RMAN Incremental Backups to Roll Forward a Physical Standby Database

* DataGuard setup between primary and standby databases - ORACLE 11gR2

* Some of the archive logs  in primary were deleted accidentally and these were noto shipped to Standby. We realized it after 3 days.

* We followed the approach mentioned in below link by just taking incremental backup in primary to rollforward standby database.

https://docs.oracle.com/cd/B19306_01/server.102/b14239/scenarios.htm#CIHIAADC

ORACLE 11gR2 - ASM Start failed with ORA-04031

This is 2 node RAC with ASM. 

ASM and DB was up and running in Node 1 where as ASM start in Node 2 (11gR2)failed with ORA-04031: unable to allocate 32 bytes of shared memory ("shared pool","unknown object","KGLH0^f6c30305","kglHeapInitialize:temp").

DB is configured to use Automatic Memory Management feature.

Please see below doc for AMM feature.

http://www.orafaq.com/wiki/Memory_target

I Double the value for memory_target, memory_max_target to get rid of error.

Logged in second node as 'grid' user and increased the value for both parameters and started ASM in Node 1.

 alter system set memory_max_target=4096m scope=spfile;

 alter system set memory_target=1024m scope=spfile;

As it was scope=spfile new value will reflect after next Instance restart in Node 2.

File Transfer failed with "File too large"

Most of you might be aware of this already.

I was trying to move files from ServerA to ServerB. After copying 10GB, file transfer failed with "File too Large".

Login Target server as root and put the following entry in /etc/security/limits for that particular user.

user1:
        fsize = -1
        nofiles = -1

This makes the size of file transfer as "unlimited".

Hope this helps.

Oracle Goldengate - ERROR: sending message to REPLICAT (Timeout waiting for message).

ORACLE 11gR2 - GoldenGate

Recently we observed  lag in one of the Replicat in Golden Gate. Tried to get 'stats','lag','stop' this replicat. However all these commands failed with 
"ERROR: sending message to REPLICAT <Replicat Name> (Timeout waiting for message)."

Identified Unix PID for this replicat and killed and restarted it. 
ps -ef| grep <replicat name>; kill -9 <pid>

This resolved the issue and we see laggap decreasing after this.

ORA-01555 on Standby

DB - 11gR2

Users were running select queries on Primary where as the same select was failing with ORA-01555 in Active standby. 

Undo_retention =   900 sec --Primary
Undo_retention = 21600 sec --Standby

Inspite of higher retention at standby users were facing ORA-01555. 

Later realized that undo_retention at stantby does not have any significance. Oracle Dataguard just replicates the undo_retention value from Primary. Increased the Undo_retention in Primary to get rid of this error.

Below note  really helped in understanding this.
http://alexeymoseyev.wordpress.com/2013/10/24/ora-01555-on-standby/

ORA-12502: TNS:listener received no CONNECT_DATA from client.

Users were able to connect from One client where as they were having issues from another client.
Jobs were failing with ORA-12502 error.

This is RAC env. 11gR2

From both clients
1. we can telnet,ping to db hosnames
2. we can telnet,ping to scan

however telnet to VIP was not working for problematic client. Informed apps team to Open firewall request for VIPs from client. This resolved the issue.

Below two links helped me in troubleshooting the issue.

http://levipereira.wordpress.com/2011/05/03/configuring-client-to-use-scan-11-2-0/
http://stelliosdba.blogspot.sg/2012/02/ora-12502-tnslistener-received-no.html

DB2 V9.7 Pending Free Pages

Pending free pages is a big headache in db2 v9.7. This is with automatic storage tablespaces.

Pending free pages does not release space back to File system.

Following query gives the top 25 tablespaces based on TBSP_PENDING_FREE_PAGES in desc order.

select char(TBSP_NAME,20) TBSP_NAME, sum(TBSP_USED_PAGES)TBSP_USED_PAGES, sum(TBSP_FREE_PAGES)TBSP_FREE_PAGES, sum(TBSP_TOTAL_PAGES)TBSP_TOTAL_PAGES, sum(TBSP_PAGE_TOP) TBSP_PAGE_TOP, sum(TBSP_PENDING_FREE_PAGES)TBSP_PENDING_FREE_PAGES FROM TABLE(MON_GET_TABLESPACE('',-2)) AS t where TBSP_CONTENT_TYPE NOT IN  ('USRTEMP','SYSTEMP') group by tbsp_name order by  TBSP_PENDING_FREE_PAGES desc fetch first 25 rows only

So far, DB2 does not give any option which application is holding the pending free pages so that it can be killed.  Raised this concern with IBM labs.

So far I was successful in the following ways

1. take the back of tablespace

2. db2_all "db2 connect to <dbname> ;db2 list tablespaces show detail;db2 terminate;" > /dev/null

3. Sometimes stopping eventmonitors also help

Below two methods I use to verify if the extent movement is in lock wait status.
1. db2 "select * from sysibmadm.SNAPLOCKWAIT"

2.db2 list applications global show detail | grep <instanceName> | grep -v db2fw | grep -v db2bp | grep -i db2ExtMov