Lessons learnt from creating indexes…
A project required a DBA to execute a SQL script for them to create a few indexes on their production database. The script was provided to them by their vendor. I have learnt quite a few lessons from this exercise, being such a complete Oracle n00b, I would like to write about my experiences:
CREATE INDEX IDX_PL_N1 ON PRODUCTION_LOG (TIMESTAMP)
STORAGE(INITIAL 4096M NEXT 160M)
- Ask a few questions to understand the reasons behind the index.
* How will the index used, why the initial extent has to be 4GB, is the index necessary… etc.
- Size of the table.
* The segment size (table) is 10GB.
- Free space to fit the index.
* There weren’t enough space in the PROD_INDEX tablespace, so I checked whether there’s enough disk space in the filesystem for the tablespace to expand. Some datafiles are created with AUTOTEXTENSIBLE set to NO, also beware of MAXSIZE set on the datafiles.
- Database version.
- Are there enough TEMP space for this work? because I didn’t…
* According to Metalink Note 19047.1 (OERR: ORA 1652 unable to extend temp segment…): “The index create performs a SORT in the users default TEMP tablespace and ALSO uses a TEMP segment to build the final index in the INDEX tablespace. Once the index build is complete the segment type is changed.”
* Metalink Note 100492.1 (ORA-01652: Estimate Space Needed to CREATE INDEX) describes the method to avoid ORA-01652 errors.
* Also take into account TEMP space required to keep the application running (if it cannot be shutdown).
- Get a feel how fast the database server is.
* Solaris 9, 4x 750MHz CPUs, 8GB Memory, datafiles on SAN
- Allow plenty of time.
* The index took 1 hour 20 minutes to create.
- Consider CREATE INDEX with ONLINE option.
* The ONLINE option allows DML to continue whilst CREATE INDEX is running, changes to the table are kept and applied when CREATE INDEX is completed. Without this option, CREATE INDEX will hold an exclusive DDL lock, preventing any DML (insert, update, delete) on the table.
* There’s still a few things to consider with this feature though, large amount of DMLs is still not recommended (reference).
* Furthermore, it is important to check whether there are problems with this feature for a particular database version. Bug 3566511 (ONLINE index rebuild can cause DML sessions to hang) affects 188.8.131.52 and 184.108.40.206, but does this bug affect create index?
- Discuss with the business owner about stopping the application, reports, scheduled dbms jobs that might impact the environment.
* It isn’t always possible to stop the application if uptime is critical, then that’s a whole different scenario to mine. The business owner was okay to stop the application, reports running during the maintenance given to me. Even with ONLINE option, the CREATE INDEX cannot start unless all DML transactions on the table is completed because of locks (I will demonstrate this in my next blog). For a table like production_log, this may mean *hardly*.
* Another thing to consider is system resources, the application, reports and the CREATE INDEX will compete for CPU, Memory, I/O resources. Will this have a major impact on the environment? Will this cause CREATE INDEX to take longer to complete?
* The most important thing to note, if the CREATE INDEX is not used with ONLINE option, and the application is not shutdown, there will most likely be sessions issuing DMLs against the table. These tables will be in lockwait whilst CREATE INDEX is running. In my case, I observed there were > 100 sessions in lockwait 10 minutes after CREATE INDEX started… I really had to cancel it.
- Even after the application is stopped or shutdown, check the locks.
* So after I learnt my first lesson (see my last point), we re-scheduled another maintenance window to try again. This time, I asked the application owner to stop the application before I start. However, application owner could only stop the application processes that would write to the database, a few database sessions still remained.
* I used a script supplied by Metalink Note 232815.1 (Script to Show current Users and SQL being Executed – which uses v$session and v$sqltext_with_newlines), to see whether there are any DMLs running on the table.
* I also used a script supplied by Metalink Note 1020008.6 (Script to Show Locking), to see whether there are any locks on the table. It turned out there’s an uncommitted transaction (a simple INSERT statement) by a session that hasn’t been active for a few days (last_call_et=1022617 – 284 hours). So I had to kill the session before proceeding with the CREATE INDEX.
- Beware of Internet and VPN dropouts.
* It was scheduled maintenance work at night, so I remotely login to the Production environment via VPN. The VPN dropped out after 1 hour into creating the index. Had to start all-over again – not much fun.
- Monitor the listener and alert logs for abnormal activities.
* These logs are tell-tale signs when things go wrong…
Anyone’s got good experiences to add?