1. We
started receiving alerts for /backup FS is full and apparently EX01 extract
ABENDED with following messages in log.
2015-05-17 06:12:22 INFO
OGG-01629 Oracle GoldenGate
Capture for Oracle, EX01.prm: BOUNDED
RECOVERY: PERSISTED OBJECTS RECOVERED: <<NONE TO RECOVER>>.
2015-05-17 06:12:22 WARNING OGG-01423 Oracle GoldenGate Capture for Oracle,
EX01.prm: No valid default archive log
destination directory found for thread 1.
2. Restart
of EX01 extract failed with below error message
2015-05-17 06:12:26 ERROR
OGG-01449 Oracle GoldenGate
Capture for Oracle, EX01.prm: Scan
failed in trail file /u01/app/oracle/product/GoldenGate11.2/dirdat/aa017499, with scan start
seqno 17499, rba 185844458.
3. At
this point we decided to rollover EX01 to next trail file.
ALTER EXT EX01, ETROLLOVER
2015-05-17 07:14:52 INFO
OGG-00993 Oracle GoldenGate
Capture for Oracle, EX01.prm: EXTRACT
EX01 started.
2015-05-17 07:14:52 INFO
OGG-01517 Oracle GoldenGate
Capture for Oracle, EX01.prm: Position
of first record processed for Thread 3, Sequence 91429, RBA 62615056, SCN
14.602109362, May 17, 2015 5:47:18 AM.
2015-05-17 07:14:52 INFO
OGG-01052 Oracle GoldenGate
Capture for Oracle, EX01.prm: No
recovery is required for target file
/u01/app/oracle/product/GoldenGate11.2/dirdat/aa017500, at RBA 0 (file not opened).
4. However
data pumps DP01, DP02 were still waiting at aa017499 (as shown below) where as EX01 moved to
aa017587
GGSCI 5> info
extract DP01
EXTRACT DP01
Last Started 2015-05-17 06:49
Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:08 ago)
Log Read Checkpoint File
/u01/app/oracle/product/GoldenGate11.2/dirdat/aa017499
First Record RBA 187818293
GGSCI 6> info
extract DP02
EXTRACT DP02
Last Started 2015-05-17 06:49
Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:08 ago)
Log Read Checkpoint File
/u01/app/oracle/product/GoldenGate11.2/dirdat/aa017499
First Record RBA 187818293
GGSCI 7> info
extract EX01
EXTRACT EX01
Last Started 2015-05-17 07:14
Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:01 ago)
Log Read Checkpoint Oracle Redo Logs
2015-05-18 14:14:52 Thread 1, Seqno 75662, RBA 55462912
SCN 14.728406038
(60857948182)
Log Read Checkpoint Oracle Redo Logs
2015-05-18 14:14:50 Thread 2, Seqno 162005, RBA 382260556
SCN 14.728404148
(60857946292)
Log Read Checkpoint Oracle Redo Logs
2015-05-18 14:14:52 Thread 3, Seqno 91632, RBA 189293056
SCN 14.728405751
(60857947895)
5. Then
we decided to stop DP01/2 and start from next trail file and this fixed the
issue.
alter extract DP01,EXTSEQNO 17500,EXTRBA
0
alter extract DP02,EXTSEQNO 17500,EXTRBA
0
6. Similarly
we worked with other team to restart replicats at their end.
2015-05-17 06:49:54 INFO
OGG-01670 Oracle GoldenGate
Collector for Oracle: Closing /main/input/tr017538.
2015-05-17 06:49:54 INFO
OGG-01669 Oracle GoldenGate
Collector for Oracle: Opening /main/input/tr017539 (byte -1, current EOF 0).
alter REPLICAT MAINP,EXTSEQNO 17539,EXTRBA 0
2015-05-17 06:49:54 INFO
OGG-01735 Oracle GoldenGate Collector
for Oracle: Synchronizing /standby/input/tr017537 to disk.
2015-05-17 06:49:54 INFO
OGG-01735 Oracle GoldenGate
Collector for Oracle: Synchronizing /standby/input/tr017537 to disk.
2015-05-17 06:49:54 INFO
OGG-01670 Oracle GoldenGate
Collector for Oracle: Closing /standby/input/tr017537.
2015-05-17 06:49:54 INFO
OGG-01669 Oracle GoldenGate
Collector for Oracle: Opening /standby/input/tr017538 (byte -1, current EOF 0).
alter REPLICAT STANDBYP,EXTSEQNO 17538,EXTRBA 0
This is what we followed to fix the issue.
