ORACLE GG: EXTRACT ABEND with OGG-01629,OGG-01423,OGG-01449

1.   We started receiving alerts for /backup FS is full and apparently EX01 extract ABENDED with following messages in log.

2015-05-17 06:12:22  INFO    OGG-01629  Oracle GoldenGate Capture for Oracle, EX01.prm:  BOUNDED RECOVERY: PERSISTED OBJECTS RECOVERED: <<NONE TO RECOVER>>.

2015-05-17 06:12:22  WARNING OGG-01423  Oracle GoldenGate Capture for Oracle, EX01.prm:  No valid default archive log destination directory found for thread 1.

2.   Restart of EX01 extract failed with below error message

2015-05-17 06:12:26  ERROR   OGG-01449  Oracle GoldenGate Capture for Oracle, EX01.prm:  Scan failed in trail file /u01/app/oracle/product/GoldenGate11.2/dirdat/aa017499, with scan start seqno 17499, rba 185844458.

3.   At this point we decided to rollover EX01 to next trail file.

ALTER EXT EX01, ETROLLOVER

2015-05-17 07:14:52  INFO    OGG-00993  Oracle GoldenGate Capture for Oracle, EX01.prm:  EXTRACT EX01 started.

2015-05-17 07:14:52  INFO    OGG-01517  Oracle GoldenGate Capture for Oracle, EX01.prm:  Position of first record processed for Thread 3, Sequence 91429, RBA 62615056, SCN 14.602109362, May 17, 2015 5:47:18 AM.

2015-05-17 07:14:52  INFO    OGG-01052  Oracle GoldenGate Capture for Oracle, EX01.prm:  No recovery is required for target file /u01/app/oracle/product/GoldenGate11.2/dirdat/aa017500, at RBA 0 (file not opened).

4.  However data pumps DP01, DP02 were still waiting at aa017499 (as shown below) where as EX01 moved to aa017587

GGSCI 5> info extract DP01

EXTRACT    DP01      Last Started 2015-05-17 06:49   Status RUNNING

Checkpoint Lag       00:00:00 (updated 00:00:08 ago)

Log Read Checkpoint  File /u01/app/oracle/product/GoldenGate11.2/dirdat/aa017499

                     First Record  RBA 187818293

GGSCI 6> info extract DP02

EXTRACT    DP02      Last Started 2015-05-17 06:49   Status RUNNING

Checkpoint Lag       00:00:00 (updated 00:00:08 ago)

Log Read Checkpoint  File /u01/app/oracle/product/GoldenGate11.2/dirdat/aa017499

                     First Record  RBA 187818293

GGSCI 7> info extract EX01

EXTRACT    EX01      Last Started 2015-05-17 07:14   Status RUNNING

Checkpoint Lag       00:00:00 (updated 00:00:01 ago)

Log Read Checkpoint  Oracle Redo Logs

                     2015-05-18 14:14:52  Thread 1, Seqno 75662, RBA 55462912

                     SCN 14.728406038 (60857948182)

Log Read Checkpoint  Oracle Redo Logs

                     2015-05-18 14:14:50  Thread 2, Seqno 162005, RBA 382260556

                     SCN 14.728404148 (60857946292)

Log Read Checkpoint  Oracle Redo Logs

                     2015-05-18 14:14:52  Thread 3, Seqno 91632, RBA 189293056

                     SCN 14.728405751 (60857947895)

5.  Then we decided to stop DP01/2 and start from next trail file and this fixed the issue.

alter extract DP01,EXTSEQNO 17500,EXTRBA 0

alter extract DP02,EXTSEQNO 17500,EXTRBA 0

6.  Similarly we worked with other team to restart replicats at their end.

2015-05-17 06:49:54  INFO    OGG-01670  Oracle GoldenGate Collector for Oracle:  Closing /main/input/tr017538.

2015-05-17 06:49:54  INFO    OGG-01669  Oracle GoldenGate Collector for Oracle:  Opening /main/input/tr017539 (byte -1, current EOF 0).

alter REPLICAT MAINP,EXTSEQNO 17539,EXTRBA 0

2015-05-17 06:49:54  INFO    OGG-01735  Oracle GoldenGate Collector for Oracle:  Synchronizing /standby/input/tr017537 to disk.

2015-05-17 06:49:54  INFO    OGG-01735  Oracle GoldenGate Collector for Oracle:  Synchronizing /standby/input/tr017537 to disk.

2015-05-17 06:49:54  INFO    OGG-01670  Oracle GoldenGate Collector for Oracle:  Closing /standby/input/tr017537.

2015-05-17 06:49:54  INFO    OGG-01669  Oracle GoldenGate Collector for Oracle:  Opening /standby/input/tr017538 (byte -1, current EOF 0).

alter REPLICAT STANDBYP,EXTSEQNO 17538,EXTRBA 0

This is what we followed to fix the issue.