Oracle Goldengate - ERROR: sending message to REPLICAT (Timeout waiting for message).

ORACLE 11gR2 - GoldenGate

Recently we observed  lag in one of the Replicat in Golden Gate. Tried to get 'stats','lag','stop' this replicat. However all these commands failed with 
"ERROR: sending message to REPLICAT <Replicat Name> (Timeout waiting for message)."

Identified Unix PID for this replicat and killed and restarted it. 
ps -ef| grep <replicat name>; kill -9 <pid>

This resolved the issue and we see laggap decreasing after this.

ORA-01555 on Standby

DB - 11gR2

Users were running select queries on Primary where as the same select was failing with ORA-01555 in Active standby. 

Undo_retention =   900 sec --Primary
Undo_retention = 21600 sec --Standby

Inspite of higher retention at standby users were facing ORA-01555. 

Later realized that undo_retention at stantby does not have any significance. Oracle Dataguard just replicates the undo_retention value from Primary. Increased the Undo_retention in Primary to get rid of this error.

Below note  really helped in understanding this.
http://alexeymoseyev.wordpress.com/2013/10/24/ora-01555-on-standby/

ORA-12502: TNS:listener received no CONNECT_DATA from client.

Users were able to connect from One client where as they were having issues from another client.
Jobs were failing with ORA-12502 error.

This is RAC env. 11gR2

From both clients
1. we can telnet,ping to db hosnames
2. we can telnet,ping to scan

however telnet to VIP was not working for problematic client. Informed apps team to Open firewall request for VIPs from client. This resolved the issue.

Below two links helped me in troubleshooting the issue.

http://levipereira.wordpress.com/2011/05/03/configuring-client-to-use-scan-11-2-0/
http://stelliosdba.blogspot.sg/2012/02/ora-12502-tnslistener-received-no.html

DB2 V9.7 Pending Free Pages

Pending free pages is a big headache in db2 v9.7. This is with automatic storage tablespaces.

Pending free pages does not release space back to File system.

Following query gives the top 25 tablespaces based on TBSP_PENDING_FREE_PAGES in desc order.

select char(TBSP_NAME,20) TBSP_NAME, sum(TBSP_USED_PAGES)TBSP_USED_PAGES, sum(TBSP_FREE_PAGES)TBSP_FREE_PAGES, sum(TBSP_TOTAL_PAGES)TBSP_TOTAL_PAGES, sum(TBSP_PAGE_TOP) TBSP_PAGE_TOP, sum(TBSP_PENDING_FREE_PAGES)TBSP_PENDING_FREE_PAGES FROM TABLE(MON_GET_TABLESPACE('',-2)) AS t where TBSP_CONTENT_TYPE NOT IN  ('USRTEMP','SYSTEMP') group by tbsp_name order by  TBSP_PENDING_FREE_PAGES desc fetch first 25 rows only

So far, DB2 does not give any option which application is holding the pending free pages so that it can be killed.  Raised this concern with IBM labs.

So far I was successful in the following ways

1. take the back of tablespace

2. db2_all "db2 connect to <dbname> ;db2 list tablespaces show detail;db2 terminate;" > /dev/null

3. Sometimes stopping eventmonitors also help

Below two methods I use to verify if the extent movement is in lock wait status.
1. db2 "select * from sysibmadm.SNAPLOCKWAIT"

2.db2 list applications global show detail | grep <instanceName> | grep -v db2fw | grep -v db2bp | grep -i db2ExtMov 

India-Consolidated Mutual fund Holdings Statement

https://www.karvymfs.com/platformservice/

The above website gives the consolidated statement of your (India)  mutual fund holdings. 

DB2 v9.7 - SQLCODE=-727, SQLSTATE=56098, SQLERRMC=2

xx Apr 20xx 09:52:06,363 [quartzScheduler_Worker-3] ERROR org.hibernate.util.JDBCExceptionReporter - DB2 SQL Error: SQLCODE=-727, SQLSTATE=56098, SQLERRMC=2;-206;42703;THIS_.ISO_CNTRY_CD, DRIVER=3.50.152
xx Apr 20xx 09:52:06,363 [quartzScheduler_Worker-3] ERROR org.hibernate.util.JDBCExceptionReporter - DB2 SQL Error: SQLCODE=-727, SQLSTATE=56098, SQLERRMC=2;-206;42703;THIS_.ISO_CNTRY_CD, DRIVER=3.50.152

Users were facing this issue after deployment. Tired doing rebind package but it did not work.

Later realized that ISO_CNTRY_CD column does not exist in the table.  they changed the column name in the job and issue got resolved.

Partition detach hanged in DB2 v9.7

I accidentally issued folloiwng command twice and it was showing partition detach status for a while for small table.

alter table T1 detach partition PART5 intoT1_02

db2 list utilities was showing AIC was running for 90 mins..

db2 list utilities show detail | grep -p -i detach
ID                               = 1208
Type                             = ASYNCHRONOUS PARTITION DETACH
Database Name                    = DB1
Partition Number                 = 0
Description                      = Finalize detach for partition '9' of table 'T1' and make table 'T1_02' available
Start Time                       = 01/28/2014 10:48:37.682113
State                            = Executing
Invocation Type                  = Automatic
Progress Monitoring:
      Description                = Waiting for old access to the partitioned table to complete.
      Start Time                 = 01/28/2014 13:08:40.480134

db2pd -util

Database Partition 0 -- Active -- Up 93 days 14:48:44 -- Date 2014-01-28-13.18.33.334414

Utilities:
Address            ID         Type                   State      Invoker    Priority   StartTime           DBName   NumPhases  CurPhase   Description
0x0780000001FAE4C0 1208       ASYNCHRONOUS PARTITION DETACH 1          1          0          Tue Jan 28 10:48:37 DB1  -1          -1          Finalize detach for partition '9' of table 'T1' and make table 'T1_02' available

Progress:
Address            ID         PhaseNum   CompletedWork                TotalWork                    StartTime           Description

I killed the hanging application as per the following link..

http://www-01.ibm.com/support/docview.wss?uid=swg21601085

Hope this helps..