ORACLE startup upgrade failed with ORA-27041 ORA-15025

ORACLE DB - Upgrade from 11.2.0.3 to 11.2.0.4

Startup upgrade failed with ORA-27041 ORA-15025 in 11.2.0.4.

However we were able to start DB from old (11.2.0.3) home. 

After spending sometime we were able to fix it by changing ownership like below.

[grid@Node02 ~]$ ls -l /u01/app/oracle/product/11.2.0.4/bin/oracle

-rwsr-s--x 1 oracle oinstall 239772898 Aug 25 05:25 /u01/app/oracle/product/11.2.0.4/bin/oracle

[grid@Node02 ~]$  setasmgidwrap o=/u01/app/oracle/product/11.2.0.4/bin/oracle

[grid@Node02 ~]$ ls -l /u01/app/oracle/product/11.2.0.4/bin/oracle

-rwsr-s--x 1 oracle asmadmin 239772898 Aug 25 05:25 /u01/app/oracle/product/11.2.0.4/bin/oracle

Add/Register new RAC database to Cluster resources

Recently we created test database and below were the steps followed to add this new database to cluster resources.

[oracle@ ~]$ . oraenv
ORACLE_SID = [+ASM1] ? test1
The Oracle base has been changed from /u01/app/grid to /u01/app/oracle

-- Checking the location of PFILE or SPFILE

[oracle@ ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Jun 11 01:40:02 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1119043584 bytes
Fixed Size                  2227624 bytes
Variable Size             838861400 bytes
Database Buffers          268435456 bytes
Redo Buffers                9519104 bytes
Database mounted.
Database opened.
SQL> show parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +DATADG/test/spfiletest1.ora

-- Checking if TEST DB is configured already

[oracle@~]$ srvctl config database -d test

PRCD-1120 : The resource for database test could not be found.

PRCR-1001 : Resource ora.test.db does not exist

--Adding TEST DB with ORACLE_HOME (-o) and SPFILE (-p_ location

[oracle@~]$ srvctl add database -d test -o /u01/app/oracle/product/11.2.0.3/dbhome_1 -p +DATADG/test/spfiletest1.ora

[oracle@~]$ srvctl config database -d test

Database unique name: test

Database name:

Oracle home: /u01/app/oracle/product/11.2.0.3/dbhome_1

Oracle user: oracle

Spfile: +DATADG/test/spfiletest1.ora

Domain:

Start options: open

Stop options: immediate

Database role: PRIMARY

Management policy: AUTOMATIC

Server pools: test

Database instances:   <<< Here no instances  are configured yet >>>>

Disk Groups:

Mount point paths:

Services:

Type: RAC

Database is administrator managed

[oracle@~]$ srvctl status database -d test

Database is not running.

--Below error comes as we have not configured any instances

[oracle@~]$ srvctl start database -d test

PRKO-3119 : Database test cannot be started since it has no configured instances. 

--Configure instances now

[oracle@~]$ srvctl add instance -d test -i test1 -n node1

[oracle@~]$ srvctl add instance -d test -i test2 -n node2

[oracle@~]$ srvctl status database -d test

Instance test1 is not running on node node1

Instance test2 is not running on node node2

[oracle@~]$ srvctl config database -d test

Database unique name: test

Database name:

Oracle home: /u01/app/oracle/product/11.2.0.3/dbhome_1

Oracle user: oracle

Spfile: +DATADG/test/spfiletest1.ora

Domain:

Start options: open

Stop options: immediate

Database role: PRIMARY

Management policy: AUTOMATIC

Server pools: test

Database instances: test1,test2  <<Compare with previous values>>

Disk Groups:

Mount point paths:

Services:

Type: RAC

Database is administrator managed

[oracle@~]$ srvctl start database -d test

[oracle@~]$ srvctl status database -d test

Instance test1 is running on node node1

Instance test2 is running on node node2

ora.test.db

      1        ONLINE  ONLINE       node1                   Open

      2        ONLINE  ONLINE       node2                   Open

addNode.sh failed with PRCF-2023 while transfering files between RAC nodes

ORACLE GRID 11.2.0.4, Database 11.2.0.3

addNode.sh failed with below error.
./addNode.sh "CLUSTER_NEW_NODES={d002}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={d002-vip}"

PRCF-2023 : The following contents are not transferred as they are non-readable.
Directories:

   1) /u01/app/11.2.0.4/grid/home_2/crs/utl

This directory is owned by root, permissions are 740 on this directory where as addNode.sh is being as 'grid' user. Hence I modified the permissions as chmod -R o+r utl.

utl directory will have four files which need 755 permissions. Make sure these permissions exist. 
-rwxr-xr-x 1 root root  4555 Apr 22 07:40 rootaddnode.sh
-rwxr-xr-x 1 root root  5190 Apr 22 07:40 rootdeinstall.sh
-rwxr-xr-x 1 root root  5992 Apr 22 07:40 rootdelete.sh
-rwxr-xr-x 1 root root  1954 Apr 22 07:40 rootdeletenode.sh

After this change addNode.sh was successfully completed.

root.sh failed with "clscfg: Error in retrieving own node information" while adding node to the RAC cluster

Oracle Grid in 11.2.0.4 and DB in 11.2.0.3

Accidentally all cluster related files were deleted and i have to remove this node from cluster and add it back.

While running root.sh in this node log was showing below error message
---------------------------------------------------------------------
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.4/grid/home_2
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.4/grid/home_2/crs/install/crsconfig_params
Installing Trace File Analyzer
Unable to retrieve local node number 1.
Internal Error Information:
  Category: 0
  Operation:
  Location:
  Other:
  Dep: 0
clscfg: Error in retrieving own node information
/u01/app/11.2.0.4/grid/home_2/bin/clscfg -add failed
/u01/app/11.2.0.4/grid/home_2/perl/bin/perl -I/u01/app/11.2.0.4/grid/home_2/perl/lib -I/u01/app/11.2.0.4/grid/home_2/crs/install /u01/app/11.2.0.4/grid/home_2/crs/install/rootcrs.pl execution failed


Tried lot of options but all of them failed. At the end tried to deconfig. However this was not succesful 
-------------------------------------------------------------------------
[root@ ~]# /u01/app/11.2.0.4/grid/home_2/crs/install/rootcrs.pl -deconfig
Using configuration parameter file: /u01/app/11.2.0.4/grid/home_2/crs/install/crsconfig_params
Oracle Clusterware stack is not active on this node
Restart the clusterware stack (use /u01/app/11.2.0.4/grid/home_2/bin/crsctl start crs) and retry
Failed to verify resources

This time same command with force option did the trick
------------------------------------------------------------------------------
[root@ ~]# /u01/app/11.2.0.4/grid/home_2/crs/install/rootcrs.pl -deconfig -force
Using configuration parameter file: /u01/app/11.2.0.4/grid/home_2/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node

Executed orainstRoot.sh once again
---------------------------------------------------------------------------------
[root ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

This time root.sh was successful.
----------------------------------------------------------------------------
[root ~]# /u01/app/11.2.0.4/grid/home_2/root.sh
Check /u01/app/11.2.0.4/grid/home_2/install/root_d002_2015-04-23_14-57-51.log for the output of root script

[root ~]# tail -100f /u01/app/11.2.0.4/grid/home_2/install/root_d002_2015-04-23_14-57-51.log
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.4/grid/home_2
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.4/grid/home_2/crs/install/crsconfig_params
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node d001, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Preparing packages for installation...
cvuqdisk-1.0.9-1

Configure Oracle Grid Infrastructure for a Cluster ... succeeded

ORACLE 11gR2 - ASM Start failed with ORA-04031

This is 2 node RAC with ASM. 

ASM and DB was up and running in Node 1 where as ASM start in Node 2 (11gR2)failed with ORA-04031: unable to allocate 32 bytes of shared memory ("shared pool","unknown object","KGLH0^f6c30305","kglHeapInitialize:temp").

DB is configured to use Automatic Memory Management feature.

Please see below doc for AMM feature.

http://www.orafaq.com/wiki/Memory_target

I Double the value for memory_target, memory_max_target to get rid of error.

Logged in second node as 'grid' user and increased the value for both parameters and started ASM in Node 1.

 alter system set memory_max_target=4096m scope=spfile;

 alter system set memory_target=1024m scope=spfile;

As it was scope=spfile new value will reflect after next Instance restart in Node 2.