ORACLE GG: EXTRACT ABEND with OGG-01629,OGG-01423,OGG-01449

1.   We started receiving alerts for /backup FS is full and apparently EX01 extract ABENDED with following messages in log.

2015-05-17 06:12:22  INFO    OGG-01629  Oracle GoldenGate Capture for Oracle, EX01.prm:  BOUNDED RECOVERY: PERSISTED OBJECTS RECOVERED: <<NONE TO RECOVER>>.

2015-05-17 06:12:22  WARNING OGG-01423  Oracle GoldenGate Capture for Oracle, EX01.prm:  No valid default archive log destination directory found for thread 1.

2.   Restart of EX01 extract failed with below error message

2015-05-17 06:12:26  ERROR   OGG-01449  Oracle GoldenGate Capture for Oracle, EX01.prm:  Scan failed in trail file /u01/app/oracle/product/GoldenGate11.2/dirdat/aa017499, with scan start seqno 17499, rba 185844458.

3.   At this point we decided to rollover EX01 to next trail file.

ALTER EXT EX01, ETROLLOVER

2015-05-17 07:14:52  INFO    OGG-00993  Oracle GoldenGate Capture for Oracle, EX01.prm:  EXTRACT EX01 started.

2015-05-17 07:14:52  INFO    OGG-01517  Oracle GoldenGate Capture for Oracle, EX01.prm:  Position of first record processed for Thread 3, Sequence 91429, RBA 62615056, SCN 14.602109362, May 17, 2015 5:47:18 AM.

2015-05-17 07:14:52  INFO    OGG-01052  Oracle GoldenGate Capture for Oracle, EX01.prm:  No recovery is required for target file /u01/app/oracle/product/GoldenGate11.2/dirdat/aa017500, at RBA 0 (file not opened).

4.  However data pumps DP01, DP02 were still waiting at aa017499 (as shown below) where as EX01 moved to aa017587

GGSCI 5> info extract DP01

EXTRACT    DP01      Last Started 2015-05-17 06:49   Status RUNNING

Checkpoint Lag       00:00:00 (updated 00:00:08 ago)

Log Read Checkpoint  File /u01/app/oracle/product/GoldenGate11.2/dirdat/aa017499

                     First Record  RBA 187818293

GGSCI 6> info extract DP02

EXTRACT    DP02      Last Started 2015-05-17 06:49   Status RUNNING

Checkpoint Lag       00:00:00 (updated 00:00:08 ago)

Log Read Checkpoint  File /u01/app/oracle/product/GoldenGate11.2/dirdat/aa017499

                     First Record  RBA 187818293

GGSCI 7> info extract EX01

EXTRACT    EX01      Last Started 2015-05-17 07:14   Status RUNNING

Checkpoint Lag       00:00:00 (updated 00:00:01 ago)

Log Read Checkpoint  Oracle Redo Logs

                     2015-05-18 14:14:52  Thread 1, Seqno 75662, RBA 55462912

                     SCN 14.728406038 (60857948182)

Log Read Checkpoint  Oracle Redo Logs

                     2015-05-18 14:14:50  Thread 2, Seqno 162005, RBA 382260556

                     SCN 14.728404148 (60857946292)

Log Read Checkpoint  Oracle Redo Logs

                     2015-05-18 14:14:52  Thread 3, Seqno 91632, RBA 189293056

                     SCN 14.728405751 (60857947895)

5.  Then we decided to stop DP01/2 and start from next trail file and this fixed the issue.

alter extract DP01,EXTSEQNO 17500,EXTRBA 0

alter extract DP02,EXTSEQNO 17500,EXTRBA 0

6.  Similarly we worked with other team to restart replicats at their end.

2015-05-17 06:49:54  INFO    OGG-01670  Oracle GoldenGate Collector for Oracle:  Closing /main/input/tr017538.

2015-05-17 06:49:54  INFO    OGG-01669  Oracle GoldenGate Collector for Oracle:  Opening /main/input/tr017539 (byte -1, current EOF 0).

alter REPLICAT MAINP,EXTSEQNO 17539,EXTRBA 0

2015-05-17 06:49:54  INFO    OGG-01735  Oracle GoldenGate Collector for Oracle:  Synchronizing /standby/input/tr017537 to disk.

2015-05-17 06:49:54  INFO    OGG-01735  Oracle GoldenGate Collector for Oracle:  Synchronizing /standby/input/tr017537 to disk.

2015-05-17 06:49:54  INFO    OGG-01670  Oracle GoldenGate Collector for Oracle:  Closing /standby/input/tr017537.

2015-05-17 06:49:54  INFO    OGG-01669  Oracle GoldenGate Collector for Oracle:  Opening /standby/input/tr017538 (byte -1, current EOF 0).

alter REPLICAT STANDBYP,EXTSEQNO 17538,EXTRBA 0

This is what we followed to fix the issue.

SQL30082N Security processing failed with reason "24" ("USERNAME AND/OR PASSWORD INVALID")

One of the user reported SQL 30082, Rc=24 in DB2 LUW V10.1 database.

He was sure that pasword supplied was correct and his password not expired and no failed login counts as well.

DB: DB2 LUW
version: 10.1, 9.7
OS: AIX

Users having special character like ! or @ # $  etc. may face RC=24.

Set the below registry variable to fix the issue.

db2set DB2AUTH=OSAUTHDB

Some interesting resources.

DB2 users receive SQL30082N, although OS authentication is possible.
http://www-01.ibm.com/support/docview.wss?uid=swg21651924

Password Validation Messages in db2diag.log
http://www-01.ibm.com/support/docview.wss?uid=swg21623221 

addNode.sh failed with PRCF-2023 while transfering files between RAC nodes

ORACLE GRID 11.2.0.4, Database 11.2.0.3

addNode.sh failed with below error.
./addNode.sh "CLUSTER_NEW_NODES={d002}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={d002-vip}"

PRCF-2023 : The following contents are not transferred as they are non-readable.
Directories:

   1) /u01/app/11.2.0.4/grid/home_2/crs/utl

This directory is owned by root, permissions are 740 on this directory where as addNode.sh is being as 'grid' user. Hence I modified the permissions as chmod -R o+r utl.

utl directory will have four files which need 755 permissions. Make sure these permissions exist. 
-rwxr-xr-x 1 root root  4555 Apr 22 07:40 rootaddnode.sh
-rwxr-xr-x 1 root root  5190 Apr 22 07:40 rootdeinstall.sh
-rwxr-xr-x 1 root root  5992 Apr 22 07:40 rootdelete.sh
-rwxr-xr-x 1 root root  1954 Apr 22 07:40 rootdeletenode.sh

After this change addNode.sh was successfully completed.

root.sh failed with "clscfg: Error in retrieving own node information" while adding node to the RAC cluster

Oracle Grid in 11.2.0.4 and DB in 11.2.0.3

Accidentally all cluster related files were deleted and i have to remove this node from cluster and add it back.

While running root.sh in this node log was showing below error message
---------------------------------------------------------------------
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.4/grid/home_2
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.4/grid/home_2/crs/install/crsconfig_params
Installing Trace File Analyzer
Unable to retrieve local node number 1.
Internal Error Information:
  Category: 0
  Operation:
  Location:
  Other:
  Dep: 0
clscfg: Error in retrieving own node information
/u01/app/11.2.0.4/grid/home_2/bin/clscfg -add failed
/u01/app/11.2.0.4/grid/home_2/perl/bin/perl -I/u01/app/11.2.0.4/grid/home_2/perl/lib -I/u01/app/11.2.0.4/grid/home_2/crs/install /u01/app/11.2.0.4/grid/home_2/crs/install/rootcrs.pl execution failed


Tried lot of options but all of them failed. At the end tried to deconfig. However this was not succesful 
-------------------------------------------------------------------------
[root@ ~]# /u01/app/11.2.0.4/grid/home_2/crs/install/rootcrs.pl -deconfig
Using configuration parameter file: /u01/app/11.2.0.4/grid/home_2/crs/install/crsconfig_params
Oracle Clusterware stack is not active on this node
Restart the clusterware stack (use /u01/app/11.2.0.4/grid/home_2/bin/crsctl start crs) and retry
Failed to verify resources

This time same command with force option did the trick
------------------------------------------------------------------------------
[root@ ~]# /u01/app/11.2.0.4/grid/home_2/crs/install/rootcrs.pl -deconfig -force
Using configuration parameter file: /u01/app/11.2.0.4/grid/home_2/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node

Executed orainstRoot.sh once again
---------------------------------------------------------------------------------
[root ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

This time root.sh was successful.
----------------------------------------------------------------------------
[root ~]# /u01/app/11.2.0.4/grid/home_2/root.sh
Check /u01/app/11.2.0.4/grid/home_2/install/root_d002_2015-04-23_14-57-51.log for the output of root script

[root ~]# tail -100f /u01/app/11.2.0.4/grid/home_2/install/root_d002_2015-04-23_14-57-51.log
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.4/grid/home_2
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.4/grid/home_2/crs/install/crsconfig_params
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node d001, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Preparing packages for installation...
cvuqdisk-1.0.9-1

Configure Oracle Grid Infrastructure for a Cluster ... succeeded

RMAN Incremental Backups to Roll Forward a Physical Standby Database

* DataGuard setup between primary and standby databases - ORACLE 11gR2

* Some of the archive logs  in primary were deleted accidentally and these were noto shipped to Standby. We realized it after 3 days.

* We followed the approach mentioned in below link by just taking incremental backup in primary to rollforward standby database.

https://docs.oracle.com/cd/B19306_01/server.102/b14239/scenarios.htm#CIHIAADC

ORACLE 11gR2 - ASM Start failed with ORA-04031

This is 2 node RAC with ASM. 

ASM and DB was up and running in Node 1 where as ASM start in Node 2 (11gR2)failed with ORA-04031: unable to allocate 32 bytes of shared memory ("shared pool","unknown object","KGLH0^f6c30305","kglHeapInitialize:temp").

DB is configured to use Automatic Memory Management feature.

Please see below doc for AMM feature.

http://www.orafaq.com/wiki/Memory_target

I Double the value for memory_target, memory_max_target to get rid of error.

Logged in second node as 'grid' user and increased the value for both parameters and started ASM in Node 1.

 alter system set memory_max_target=4096m scope=spfile;

 alter system set memory_target=1024m scope=spfile;

As it was scope=spfile new value will reflect after next Instance restart in Node 2.

File Transfer failed with "File too large"

Most of you might be aware of this already.

I was trying to move files from ServerA to ServerB. After copying 10GB, file transfer failed with "File too Large".

Login Target server as root and put the following entry in /etc/security/limits for that particular user.

user1:
        fsize = -1
        nofiles = -1

This makes the size of file transfer as "unlimited".

Hope this helps.