[root@racserver2 bin]# ./crsctl stop crs CRS-2796: The command may not proceed when Cluster Ready Services is not running CRS-4687: Shutdown command has completed with errors. CRS-4000: Command Stop failed, or completed with errors. [root@racserver2 bin]# ./crsctl start crs CRS-4640: Oracle High Availability Services is already active CRS-4000: Command Start failed, or completed with errors.
2014-11-26 13:20:50.154 [/u01/app/11.2.0/grid/bin/oraagent.bin(3115)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/racserver2/agent/ohasd/oraagent_grid/oraagent_grid.log"
2014-11-26 13:31:10.539 [/u01/app/11.2.0/grid/bin/cssdagent(3449)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/11.2.0/grid/log/racserver2/agent/ohasd/oracssdagent_root/oracssdagent_root.log. 2014-11-26 13:31:10.539 [cssd(3462)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0/grid/log/racserver2/cssd/ocssd.log
2014-11-26 13:31:10.539: [ CSSD][1563055872]################################### 2014-11-26 13:31:10.539: [ CSSD][1563055872]clssscExit: CSSD aborting from thread GMClientListener 2014-11-26 13:31:10.539: [ CSSD][1563055872]################################### 2014-11-26 13:31:10.539: [ CSSD][1563055872](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally 2014-11-26 13:31:10.540: [ CSSD][1563055872]clssgmUpdateEventValue: CmInfo State val 0, changes 1 2014-11-26 13:31:10.778: [ CSSD][1547863808]clssnmPollingThread: state(1) clusterState(0) exit 2014-11-26 13:31:10.778: [ CSSD][1547863808]clssscExit: abort already set 0 2014-11-26 13:31:11.146: [ CSSD][1554196224]clssnmvDHBValidateNCopy: node 1, racserver1, has a disk HB, but no network HB, DHB has rcfg 253810092, wrtcnt, 53881108, LATS 4294689760, lastSeqNo 53881107, uniqueness 1406180021, timestamp 1416976270/2203792632
[root@racserver1 ~]# /u01/app/11.2.0/grid/bin/crsctl stop crs
[root@racserver1 ~]# /u01/app/11.2.0/grid/bin/crsctl start crs
less /u01/app/11.2.0/grid/log/racserver2/alertracserver2.log [cssd(6902)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/racserver2/cssd/ocssd.log
2012-06-11 23:29:14.162 [/u01/app/11.2.0/grid/bin/oraagent.bin(6698)]CRS-5815:Agent '/u01/app/11.2.0/grid/bin/oraagent_grid' could not find any base type entry points for type 'ora.daemon.type'. Details at (:CRSAGF00108:) {0:2:2} in /u01/app/11.2.0/grid/log/racserver2/agent/ohasd/oraagent_grid/oraagent_grid.log. 2012-06-11 23:29:14.190 [/u01/app/11.2.0/grid/bin/oraagent.bin(6698)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/racserver2/agent/ohasd/oraagent_grid/oraagent_grid.log"
[grid@racserver2 ~]$ crsctl check has CRS-4638: Oracle High Availability Services is online [grid@racserver2 ~]$ crsctl check css CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
[oracle@racserver2 ~]$ srvctl start instance -d rac -i rac2 PRCR-1013 : Failed to start resource ora.rac.db PRCR-1064 : Failed to start resource ora.rac.db on node racserver2 CRS-5017: The resource action "ora.DATA.dg start" encountered the following error: ORA-15032: not all alterations performed ORA-15017: diskgroup "DATA" cannot be mounted ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"
/usr/sbin > ./oracleasm listdisks /usr/sbin > ./oracleasm scandisks