Tuesday, 8 November 2011

Reorg Rebuild

The  reorg rebuild is use in the following condition :
1.       Large I/O is not being selected for queries where it is usually used, and optdiag shows a low cluster ratio for datapages, data rows, or index pages.                                                                                                                                         
2.       You used sp_chgattribute to change one or more of the exp_row_size, reservepagegap, or fillfactor space management settings and you want the changes to apply not only to future data, but also to existing rows and pages.
3.       If a table needs to be rebuilt because of a low cluster ratio, it may also need to have its space management settings changed                                                                                             

Reorg rebuild on a TABLE

Reorg rebuild uses a table's current space management settings to rewrite the rows in the table according to the table's clustered index.
All indexes on the table are dropped and re-created using   the  current space management values for reservepagegap and fillfactor.                                          
After a rebuild, a table has no forwarded rows and no unused space from deletions  or updates.
Reorg rebuild table syntax:-
reorg rebuild tablename [index_name]
Steps tobe consider before  running  reorg rebuild on a TABLE:-

1)Set the database option select into/bulkcopy/pllsort to true and run checkpoint in the database.                                                                                                
2)Make sure that additional disk space, equal to the size of the table and its indexes, is available.

Other comments
1)When reorg rebuild rebuilds a table, it rewrites all table and index rows according to the table's current settings for reservepagegap, fillfactor, and exp_row_size.           These properties all affect how quickly inserts cause a table to become fragmented, as measured by a low cluster ratio.
2)If it appears that a table quickly becomes fragmented and needs to be rebuilt too frequently, it may be a sign that you need to change the table's space management settings before you run reorg rebuild, by  using  sp_chgattribute .

Performance characteristics

Running reorg against a table can have a negative effect on performance of concurrent queries.

Reorg rebuild on a INDEX
The reorg rebuild command allows you to rebuild individual indexes while the table itself is accessible for read and update activities.
Reorg rebuild  index syntax:-
reorg rebuild table_name index_name 
Steps tobe  consider  before  running  reorg rebuild on a INDEX:-
1) You do not need to set select into to rebuild an index.
2) You must be the table owner or the Database Owner, or having SA  privileges.
3) You needs space for the pages copied on each transaction since rebuilding an index works in small transactions, and deallocates pages once they are copied
4) You can rebuild the index on a table while transaction level scans (dirty reads) are active 
5)  If  we  omit the index name, the entire table is rebuilt.
6)If we  specify an index name , only that index is rebuilt.

Other comments

The reorg command applies only to tables using datarows or datapages locking. You cannot run reorg on a table that uses allpages locking.
You cannot run reorg on a text index, the name from sysindexes associated with a text chain.
You cannot run reorg within a transaction.
You can do a dump tran on a table after rebuilding its index. However, you cannot do a dump tran if the entire table has been rebuilt.
You can rebuild the index for systabstats, but you cannot run reorg rebuild on the table itself.
Reorg rebuild may not rebuild those parts of the index that are already well clustered and have the desired space utilization.

Performance characteristics

Index scans are faster after you run reorg.
Reorg rebuild accomplishes everything that dropping and re-creating a clustered index does and takes less time.

Sql for finding the frgmentaion of the tables in databse

select o.name "Table Name",i.name "Table/Index Name",i.indid,
Case when i.indid=0 then
 derived_stat(o.name,i.indid,"dpcr")
else  derived_stat(o.name,i.indid,"ipcr")
end "CLUSTER RATIO",
derived_stat(o.name,i.indid,"lgio") "Large IO Effeciency",
derived_stat(o.name,i.indid,"sput") "Space Utilization",
s.pagecnt "Pages",
s.rowcnt "Rows",
s.forwrowcnt "forward row count",
s.delrowcnt "Deleted ROw Count"
--,ss.moddate "Last Update Stats Run",
--ss.sequence
from sysindexes i,sysobjects o, systabstats s  --,sysstatistics ss
where o.id=i.id and o.id = s.id   --and o.id=ss.id
and o.type = 'U'
order by s.rowcnt desc


Also see the new post for Fragmentation and its new Resolution steps.

Sybase Migration - Sybmigrate

Introduction

Sybmigrate lets you migrate databases and servers across Adaptive Server versions, platforms, and page sizes. It is supported by Adaptive Server versions 12.0 and later when used as the source server.

Benefits of sybmigrate

  • Aids users in changing the page sizes of their database applications.
  • Provides a manageable and smooth migration process.
  • Allows customers to take advantage of the variable page size feature for existing databases with user data, thus realizing the full benefit of Adaptive Server versions 12.5 and later.

Data Affected

During the setup portion of the migration process, the following server data is migrated to the target Adaptive Server:
  • Remote servers
  • Logins
  • Login attributes
  • Server roles
  • Login roles
  • Role attributes
  • Users
  • Alternate users
  • Roles
  • Permissions
  • Remote logins
  • External login attributes
  • Timer
  • Resource limits
  • Replication attributes
  • Display level attributes
  • User messages in the master database
  • Java classes in the master database
  • JAR files in the master database
During the migration portion of the migration process, the following database-specific data is migrated to the target database:
  • Defaults
  • User-defined datatypes
  • Rules
  • User tables
  • User table data
  • Views
  • Triggers
  • Indexes
  • Stored procedures
  • Extended stored procedures
  • Users
  • Logins
  • Roles
  • Remote servers
  • Database data
  • Users
  • Alternate users
  • Roles
  • Role attributes
  • Permissions
  • User messages
  • Java classes
  • JAR files
  • Defaults
  • Rules
  • User-defined types
  • Tables
  • Indexes
  • Referential constraints
  • Views
  • Stored procedures
  • Triggers

The following items must be migrated manually:

  • User-defined thresholds
  • Abstract plan definitions maintained in sysqueryplans
  • All system databases except the model database
  • Any required database options like cache binding, recovery order, and the associated log I/O size as specified by sp_logiosize
  • Proxy databases
  • Engine groups
  • Engine bindings
  • Execution classes
  • Cache configurations
  • Auditing tables and auditing configuration
  • Server-wide row-lock promotion settings
  • Access rules
  • Drop access rules before beginning data migration; they can prevent the Database Owner from accessing all rows in a table, which prevents complete data migration.
  • Compiled objects with hidden SQL text
  • User-defined segments
  • Constraints are migrated but when they are bound by name to user-defined message numbers, the bindings must be re-created manually
  • Settings for objects such as ascinserts, maxwritedes, indextrips, oamtrips, datatrips, and sortbufsize created using dbcc tune
  • Device definitions
  • SQLJ functions
  • Proxy tables for external files
  • Audit options and audit events
  • Server configuration
  • Database suspect threshold
  • Recovery orders

Prerequisites

  • Set $SYBASE_JRE to $JAVA_HOME.
  • Make sure that you can connect to Source server from target and vice versa.
  • Run full dbcc checks on your database(s) to be migrated, and fix reported problems.
  • Perform backups of the database(s) to be migrated.
  • sybmigrate requires allow resource limits to be set to 0.
  • Manually create segments on target database before starting migration
  • There are three types of data that are migrated: server data, database data, and user objects. To migrate metadata (the server and database data), the target Adaptive Server must be newly installed so that the migrated metadata does not conflict with any residual data from previous usage.
  • Determine the size of the named caches and buffer pools on the target Adaptive Server. sybmigrate does not migrate cache configurations. You can use the information that is generated by ddlgen and apply it to the target Adaptive Server, or you can choose to configure larger amounts of memory, in light of the larger page size being used.
  • If multibyte character sets are configured on the target Adaptive Server after initiating the migration process, you must manually run dbcc fix_text on the sysattributes and sysxtypes system catalogs to make the text columns in these catalogs consistent with the multibyte character sets.Sybase recommends that you configure the target server character set first, and then initiate the migration process.
  • Check the following Parameters in source:
    • cis packet size – should be equal to max page size of the target Adaptive Server.
    • number of user connections – should be high enough to accommodate the migration of multiple tables simultaneously according to the value of COPY_THREADS and INDEX_THREADS
    • max parallel degree – should be set to a value that is larger than the largest number of partitions in a single table.
    • number of worker processes – data migration for partitioned tables requires one worker thread per partition. Therefore, if  10 partitioned tables with 2 partitions each are migrating simultaneously, configure a total of 10*2 worker threads on the source Adaptive Server.
  • Check the following parameters on Target:
    • Number of user connections : should be set to accommodate the migration of multiple tables in parallel and partitioned tables.
    • Number of sort buffers : Default is 500 which is sufficient during migration process.
During migration process 2 databases will be created on source server, make sure that you have enough free space in devices(approx 20% of Databases to be migrated)


Migration Process

Execute the sybmigation in GUI mode / commandline.

Following steps are involved:

            Set up target database for migration
           


Select database and click Add path below



Migrate Data



Validate Data






Post Migration

  • Need to create statistics for NonIndex Columns
  • Any message requiring user attention preceded by the word “attention” and logged in the migration log.
  • Run the object migrations status report to verify that all objects have been migrated


ASE 15 Upgrade STEPS with DUMP/LOAD

1. Login to existing 12.5 server and Unpartition all the partition tables (if any)

2. Lock all logins & take backup of all user databases and master and model databases etc.

3. Take BCP out of following System Tables from master database
a. sysusers
b. sysprotects
c. sysalternates
d. syslogins
e. sysloginroles
f. sysdevices
g. sysdatabases
h. sysusages

4. Take BCP out of following system tables from each user database
a. sysusers
b. sysprotects
c. sysalternates

 
6. If possible, have SA increase allocated memory to sybase (update /etc/project file).

7. Have SA create a new file system and have him install wrap ase1503a.

8. Have SA allocate Raw device for master device, sybsystemprocs device and tempdb device

9. Have SA allocate free devices for user databases to be created.

10. Copy 12.5.4 cfg file to ASE 15.0 cfg file

11. Modify this CFG file, Increase procedure cache and default data cache 4 – 5 times.

12. Build new DataServer and Backup Server & all databases  (Command – srvbuildres –r <resource file>)

13. Modify .profile for sybase to set ASE 15 environment variables

14. Generate License file from https://sybase.subscribenet.com/and copy it to $SYBASE/SYSAM-2_0/
licenses

15. Following are the steps to generate license file

a. Login to https://sybase.subscribenet.com/
b. Choose approprite option
c. From Product list choose Adaptive Server Enterprise
d. Choose ASE Enterprise Edition
e. Click on link for the version you have installed ASE Enterprise Edition 15.0.3 for Sun Solaris
x64 - 64bit
f. Read license agreement for your region and click on ‘I Accept’.
g. Click on License Keys
h. Choose the radio button, which is left of license type, scroll down and click on select to
generate.
i. In license model choose un-served license, click next.
j. In license quantity enter 1, click next
k. Enter hostid, hostname and quantity (Number of engines), Click on Generate.

17 .  load all the databases & bcp in the table.
18. run the consistency check for any error and correct it
19 Ask application team to do checkout
20. Shutdown ASE 12.5 server.

UPGRADE FROM ASE-15.0.2 to ASE-15.0.3

Step for UPGRADE FROM ASE-15.0.2 to ASE-15.0.3


1) HAVE SA INSTALL ASE 15.0.3 Binaries in seperate partition

2) HAVE SA point link '150' in /optware/sybase to ase1503a

3)TAKE BACKUP OF USER DATABASES and MASTER

dump database userdb1 , master  model below mentioned :-

"/sybdump/ASE_PRI/BACKUP_DONT_DELETE/userdb1.dmp"

"/sybdump/ASE_PRI/BACKUP_DONT_DELETE/master.dmp"

"/sybdump/ASE_PRI/BACKUP_DONT_DELETE/model.dmp"

5)TAKE BCP OUT OF BELOW TABLES:-

COPY BELOW FILES AS MENTIONED
cp $SYBASE/interfaces  /optware/sybase/ase1503a
cp $SYBASE/$SYBASE_ASE/ASE_PRI.cfg  /optware/sybase/ase1503a/ASE-15*
cp $SYBASE/$SYBASE_OCS/config/libtcl.cfg /optware/sybase/ase1503a/$SYBASE_OCS/config/
cp $SYBASE/SYSAM-2_0/licenses/*.lic       /optware/sybase/ase1503a/SYSAM-2_0/licenses/
cp /optware/sybase/150/ASE-15_0/install/RUN_*    /optware/sybase/ase1503a/ASE-15_0/install/

5) ENSURE THAT $SYBASE IS POINTING TO NEW BINARIES

6)SHUTDOWN OLD 15.0.2 SERVER

7)START Server from NEW $SYBASE.

8)VERIFY SERVER VERSION

9)EXECUTE BELOW COMMANDS:-

isql -Usa -SASE_PRI -iinstallmaster -oinstallmaster.out
isql -Usa -SASE_PRI -iinstallcommit -oinstallcommit.out
isql -Usa -SASE_PRI -iinstallmsgsvss -oinstallmsgsvss.out

10) Generate License from SPDC and place it in appropriate dir

11) RESTART SERVER

sql : free disk space,negative logsegment space,Table Partitioning ,rename db

1) Display Available Device Space - Free Disk Space .
Determining the amount of free space on a server's set of devices
is not an easy task; the procedure below produces a nice  report ,
showing total allocations and available space on each device.

create procedure sp_freedisk
as
set nocount on
select
logical_name = substring(d.name, 1, 12),
physical_name = substring(d.phyname, 1, 17),
vdevno = d.low / power(2, 24),
size = (d.high - d.low + 1) * 2 / 1024,
reserved = isnull(sum(u.size) * 2 / 1024, 0),
left = (d.high - d.low + 1) * 2 / 1024 -
isnull(sum(u.size) * 2 / 1024, 0)
from
master.dbo.sysdevices d, master.dbo.sysusages u
where
d.status & 2 = 2
and u.vstart / power(2, 24)=*d.low / power(2, 24)
group by
substring(d.name, 1, 12),
substring(d.phyname, 1, 17),
d.low / power(2, 24),
(d.high - d.low + 1) * 2 / 1024
order by vdevno
return



DBName            SegName                Allocated    Available   PctUsed
--------------- -------------------- ---------- ---------- ---------
reports             default                    289000.0  13153.5       95.4
reports             logsegment              3010.0      3857.7       -28.1
reports             system                    289000.0  13153.5       95.4

--Need to recycle server to take effect

--Negative value for log free space

--The following is undocumented from sybase

-- Connect to your server as sa and run the following to fix it.

dbcc gam(dbname, 0, 0, "fix")
go
dbcc usedextents(dbname, 0, 1, 1)
go

3) Table Partitioning
Insert performance on partitioned tables is improved, as multiple
'entry points' (last page entries) are created. Partitioned tables
require slightly more disk space and need a chunk of memory also.

/* create 4 partitions for the invoice table */

alter table invoice
partition 4
go

bcp pubs2..publishers out "/optware/sybase/publishers.dat”
-Uuser1 –Pabc456 -c &
Bcp pub2..publishers out "/optware/sybase/ publishers.dat 
To bcp into several partitions split your file into 4 parts and bcp them into the partitions.

You have to specify the partition number like this:

bcp mydb..mytable:1 in ...
bcp mydb..mytable:2 in ...
bcp mydb..mytable:3 in ...

bcp mydb..mytable:4 in ...

***
Partitioning in ASE 15***

Semantic "smart" partitioning is new in ASE 15.
Tables can be partitioned by range, value, or round-robin. By default, all tables in ASE 15 are partitioned, round-robin.
4)Clustered vs non-clustered indexes

Typically, a clustered index will be created on the primary key
of a table, and non-clustered indexes are used where needed.



Non-clustered indexes


l>Leaves are stored in b-tree
li>Lower overhead on inserts, vs clustered
li>Best for single key queries
<li>Last page of index can become a 'hot spot'
</ul>

Clustered indexes

<ul>
<li>Records in table are sorted physically by key values
<li>Only one clustered index per table
<li>Higher overhead on inserts, if re-org on table is required
<li>Best for queries requesting a range of records
<li>Index must exist on same segment as table
</ul>

Note! With "lock datapages" or "lock datarows" ... clustered indexes
are sorted physically only upon creation. After that, the indexes
behave like non-clustered indexes.

6) Setting the thresholds
Threshold settings allow customized procedures to be run when database segments
approach a defined capacity.

The "last chance threshold" is set by default, to execute sp_thresholdaction within the
current database, when a segment reaches 95% of capacity. The procedure sp_thresholdaction
needs to be created by the DBA. Here is a sample:


create proc sp_thresholdaction (
@dbname varchar(30),
@segmentname varchar(30),
@space_left int,
@status int ) as

declare @msg varchar(80),
@date1 datetime,
@fname varchar(80),
@fdate varchar(20),

@fpath varchar(40)

select @fpath = '/usr/dumps/logs/'

select @date1 = getdate()

select @fdate =
convert(varchar(2),datepart(MM,@date1)) +
convert(varchar(2),datepart(DD,@date1)) +
convert(varchar(2),datepart(HH,@date1)) +
convert(varchar(2),datepart(MI,@date1))

select @fname = @fpath + 'log_' + @dbname + @fdate + '.dmp'

select @msg = '***!! Last Chance Threshold reached, for ' + @dbname + '(' + @segmentname + ')'

print @msg

if @segmentname = 'logsegment'
dump tran @dbname to @fname

return

Other threshold levels can be created, for specific segments. They can be set up
to print informational messages to the error log, as a forewarning to the DBA. Here's a
sample which reflects the command syntax:


1> sp_addthreshold dbname,logsegment,400,'proc_log_threshold'
2> go
Adding threshold for segment 'logsegment' at '400' pages.
DBCC execution completed. If DBCC printed error messages, contact a user with
System Administrator (SA) role.
(return status = 0)

8)Renaming a database
 Renaming a database requires that it be put in single-user mode first.


-- Renaming a database

use master
go

sp_dboption 'warehouse003','single user',true
go
use warehouse003
go
checkpoint
go
use master
go

sp_renamedb 'warehouse003','warehouse009'
go

sp_dboption 'warehouse009','single user',false
go
use warehouse009
go
checkpoint
go

use master
go

sp_helpdb warehouse009
go