weblogic.nodemanager.common.ConfigException: Identity key store file not found:

---------------------------------------------------
Weblogic 12.1.2 | Node Manager Startup Issues
----------------------------------------------------
Issue: Node Manager was throwing error when we invoked the script startNodeManager.sh script from $WLS_HOME/server/bin




[fusion@wlsadmin bin]$ sh startNodeManager.sh
+ CLASSPATH=/u02/jdk1.7.0_45/lib/tools.jar:/u02/Middleware/wlserver/server/lib/weblogic_sp.jar:/u02/Middleware/wlserver/server/lib/weblogic.jar:/u02/Middleware/wlserver/server/lib/webservices.jar:/u02/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u02/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u02/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar:/u02/jdk1.7.0_45/lib/tools.jar:/u02/Middleware/wlserver/server/lib/weblogic_sp.jar:/u02/Middleware/wlserver/server/lib/weblogic.jar:/u02/Middleware/wlserver/server/lib/webservices.jar:/u02/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u02/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u02/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar:/u02/Middleware/wlserver/../oracle_common/modules/com.oracle.cie.config-wls-online_8.0.0.0.jar::/u02/Middleware/wlserver/common/derby/lib/derbynet.jar:/u02/Middleware/wlserver/common/derby/lib/derbyclient.jar:/u02/Middleware/wlserver/common/derby/lib/derby.jar:/u02/Middleware/wlserver/server/lib/xqrl.jar:/u02/jdk1.7.0_45/lib/tools.jar:/u02/Middleware/wlserver/server/lib/weblogic_sp.jar:/u02/Middleware/wlserver/server/lib/weblogic.jar:/u02/Middleware/wlserver/server/lib/webservices.jar:/u02/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u02/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u02/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar::/u02/Middleware/wlserver/..
+ '[' '!' -z '' ']'
+ '[' '!' -z /u02/Middleware/wlserver/../oracle_common/modules/com.oracle.cie.config-wls-online_8.0.0.0.jar::/u02/Middleware/wlserver/common/derby/lib/derbynet.jar:/u02/Middleware/wlserver/common/derby/lib/derbyclient.jar:/u02/Middleware/wlserver/common/derby/lib/derby.jar:/u02/Middleware/wlserver/server/lib/xqrl.jar ']'
+ CLASSPATH=/u02/jdk1.7.0_45/lib/tools.jar:/u02/Middleware/wlserver/server/lib/weblogic_sp.jar:/u02/Middleware/wlserver/server/lib/weblogic.jar:/u02/Middleware/wlserver/server/lib/webservices.jar:/u02/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u02/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u02/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar:/u02/jdk1.7.0_45/lib/tools.jar:/u02/Middleware/wlserver/server/lib/weblogic_sp.jar:/u02/Middleware/wlserver/server/lib/weblogic.jar:/u02/Middleware/wlserver/server/lib/webservices.jar:/u02/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u02/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u02/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar:/u02/Middleware/wlserver/../oracle_common/modules/com.oracle.cie.config-wls-online_8.0.0.0.jar::/u02/Middleware/wlserver/common/derby/lib/derbynet.jar:/u02/Middleware/wlserver/common/derby/lib/derbyclient.jar:/u02/Middleware/wlserver/common/derby/lib/derby.jar:/u02/Middleware/wlserver/server/lib/xqrl.jar:/u02/jdk1.7.0_45/lib/tools.jar:/u02/Middleware/wlserver/server/lib/weblogic_sp.jar:/u02/Middleware/wlserver/server/lib/weblogic.jar:/u02/Middleware/wlserver/server/lib/webservices.jar:/u02/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u02/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u02/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar::/u02/Middleware/wlserver/..:/u02/Middleware/wlserver/../oracle_common/modules/com.oracle.cie.config-wls-online_8.0.0.0.jar::/u02/Middleware/wlserver/common/derby/lib/derbynet.jar:/u02/Middleware/wlserver/common/derby/lib/derbyclient.jar:/u02/Middleware/wlserver/common/derby/lib/derby.jar:/u02/Middleware/wlserver/server/lib/xqrl.jar
+ export CLASSPATH
+ export PATH
+ cd /u02/Middleware/wlserver/../oracle_common/common/nodemanager
+ set -x
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ /u02/jdk1.7.0_45/bin/java -server -Xms32m -Xmx200m -XX:MaxPermSize=128m -Dcoherence.home=/u02/Middleware/wlserver/../coherence -Dbea.home=/u02/Middleware/wlserver/.. -Dweblogic.ProductionModeEnabled=true -Xverify:none -Djava.endorsed.dirs=/u02/jdk1.7.0_45/jre/lib/endorsed:/u02/Middleware/wlserver/../oracle_common/modules/endorsed -Xverify:none -Djava.endorsed.dirs=/u02/jdk1.7.0_45/jre/lib/endorsed:/u02/Middleware/wlserver/../oracle_common/modules/endorsed -da -Dwls.home=/u02/Middleware/wlserver/server -Dweblogic.home=/u02/Middleware/wlserver/server -Xverify:none -Djava.endorsed.dirs=/u02/jdk1.7.0_45/jre/lib/endorsed:/u02/Middleware/wlserver/../oracle_common/modules/endorsed -Djava.security.policy=/u02/Middleware/wlserver/server/lib/weblogic.policy -Dweblogic.nodemanager.JavaHome=/u02/jdk1.7.0_45 weblogic.NodeManager -v
<Feb 14, 2014 11:33:44 PM IST> <INFO> <Loading domains file: /u02/Middleware/oracle_common/common/nodemanager/nodemanager.domains>
<Feb 14, 2014 11:33:44 PM IST> <WARNING> <Domains file not found: /u02/Middleware/oracle_common/common/nodemanager/nodemanager.domains>
<Feb 14, 2014 11:33:44 PM IST> <INFO> <Loading identity key store: FileName=/u02/Middleware/oracle_common/common/nodemanager/security/DemoIdentity.jks, Type=jks, PassPhraseUsed=true>
<Feb 14, 2014 11:33:44 PM IST> <SEVERE> <Fatal error in NodeManager server>
weblogic.nodemanager.common.ConfigException: Identity key store file not found: /u02/Middleware/oracle_common/common/nodemanager/security/DemoIdentity.jks
        at weblogic.nodemanager.server.SSLConfig.loadKeyStoreConfig(SSLConfig.java:170)
        at weblogic.nodemanager.server.SSLConfig.access$000(SSLConfig.java:32)
        at weblogic.nodemanager.server.SSLConfig$1.run(SSLConfig.java:111)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.nodemanager.server.SSLConfig.<init>(SSLConfig.java:108)
        at weblogic.nodemanager.server.NMServer.<init>(NMServer.java:121)
        at weblogic.nodemanager.server.NMServer.main(NMServer.java:383)
        at weblogic.NodeManager.main(NodeManager.java:31)

+ set +x
[fusion@wlsadmin bin]$ 



Solution: In Weblogic 12.1.2(12c), we have to start the node manager from DOMAIN_HOME/bin using the startNodeManager.sh script.




[fusion@wlsadmin bin]$ ./startNodeManager.sh
NODEMGR_HOME is already set to /u02/Middleware/user_projects/domains/base_domain/nodemanager
+ CLASSPATH=/u02/jdk1.7.0_45/lib/tools.jar:/u02/Middleware/wlserver/server/lib/weblogic_sp.jar:/u02/Middleware/wlserver/server/lib/weblogic.jar:/u02/Middleware/wlserver/server/lib/webservices.jar:/u02/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u02/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u02/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar:/u02/jdk1.7.0_45/lib/tools.jar:/u02/Middleware/wlserver/server/lib/weblogic_sp.jar:/u02/Middleware/wlserver/server/lib/weblogic.jar:/u02/Middleware/wlserver/server/lib/webservices.jar:/u02/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u02/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u02/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar:/u02/Middleware/wlserver/../oracle_common/modules/com.oracle.cie.config-wls-online_8.0.0.0.jar::/u02/Middleware/wlserver/common/derby/lib/derbynet.jar:/u02/Middleware/wlserver/common/derby/lib/derbyclient.jar:/u02/Middleware/wlserver/common/derby/lib/derby.jar:/u02/Middleware/wlserver/server/lib/xqrl.jar:/u02/jdk1.7.0_45/lib/tools.jar:/u02/Middleware/wlserver/server/lib/weblogic_sp.jar:/u02/Middleware/wlserver/server/lib/weblogic.jar:/u02/Middleware/wlserver/server/lib/webservices.jar:/u02/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u02/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u02/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar::/u02/Middleware/wlserver/..
+ '[' '!' -z '' ']'
+ '[' '!' -z /u02/Middleware/wlserver/../oracle_common/modules/com.oracle.cie.config-wls-online_8.0.0.0.jar::/u02/Middleware/wlserver/common/derby/lib/derbynet.jar:/u02/Middleware/wlserver/common/derby/lib/derbyclient.jar:/u02/Middleware/wlserver/common/derby/lib/derby.jar:/u02/Middleware/wlserver/server/lib/xqrl.jar ']'
+ CLASSPATH=/u02/jdk1.7.0_45/lib/tools.jar:/u02/Middleware/wlserver/server/lib/weblogic_sp.jar:/u02/Middleware/wlserver/server/lib/weblogic.jar:/u02/Middleware/wlserver/server/lib/webservices.jar:/u02/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u02/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u02/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar:/u02/jdk1.7.0_45/lib/tools.jar:/u02/Middleware/wlserver/server/lib/weblogic_sp.jar:/u02/Middleware/wlserver/server/lib/weblogic.jar:/u02/Middleware/wlserver/server/lib/webservices.jar:/u02/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u02/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u02/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar:/u02/Middleware/wlserver/../oracle_common/modules/com.oracle.cie.config-wls-online_8.0.0.0.jar::/u02/Middleware/wlserver/common/derby/lib/derbynet.jar:/u02/Middleware/wlserver/common/derby/lib/derbyclient.jar:/u02/Middleware/wlserver/common/derby/lib/derby.jar:/u02/Middleware/wlserver/server/lib/xqrl.jar:/u02/jdk1.7.0_45/lib/tools.jar:/u02/Middleware/wlserver/server/lib/weblogic_sp.jar:/u02/Middleware/wlserver/server/lib/weblogic.jar:/u02/Middleware/wlserver/server/lib/webservices.jar:/u02/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u02/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u02/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar::/u02/Middleware/wlserver/..:/u02/Middleware/wlserver/../oracle_common/modules/com.oracle.cie.config-wls-online_8.0.0.0.jar::/u02/Middleware/wlserver/common/derby/lib/derbynet.jar:/u02/Middleware/wlserver/common/derby/lib/derbyclient.jar:/u02/Middleware/wlserver/common/derby/lib/derby.jar:/u02/Middleware/wlserver/server/lib/xqrl.jar
+ export CLASSPATH
+ export PATH
+ cd /u02/Middleware/user_projects/domains/base_domain/nodemanager
+ set -x
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ /u02/jdk1.7.0_45/bin/java -server -Xms32m -Xmx200m -XX:MaxPermSize=128m -Dcoherence.home=/u02/Middleware/wlserver/../coherence -Dbea.home=/u02/Middleware/wlserver/.. -Dweblogic.ProductionModeEnabled=true -Xverify:none -Djava.endorsed.dirs=/u02/jdk1.7.0_45/jre/lib/endorsed:/u02/Middleware/wlserver/../oracle_common/modules/endorsed -Xverify:none -Djava.endorsed.dirs=/u02/jdk1.7.0_45/jre/lib/endorsed:/u02/Middleware/wlserver/../oracle_common/modules/endorsed -da -Dwls.home=/u02/Middleware/wlserver/server -Dweblogic.home=/u02/Middleware/wlserver/server -Dweblogic.RootDirectory=/u02/Middleware/user_projects/domains/base_domain -Xverify:none -Djava.endorsed.dirs=/u02/jdk1.7.0_45/jre/lib/endorsed:/u02/Middleware/wlserver/../oracle_common/modules/endorsed -Djava.security.policy=/u02/Middleware/wlserver/server/lib/weblogic.policy -Dweblogic.nodemanager.JavaHome=/u02/jdk1.7.0_45 weblogic.NodeManager -v
<Feb 14, 2014 11:37:10 PM IST> <INFO> <Loading domains file: /u02/Middleware/user_projects/domains/base_domain/nodemanager/nodemanager.domains>
<Feb 14, 2014 11:37:10 PM IST> <INFO> <Upgrade> <Setting NodeManager properties version to 12.1.2>
<Feb 14, 2014 11:37:10 PM IST> <INFO> <Upgrade> <Saving upgraded NodeManager properties to '/u02/Middleware/user_projects/domains/base_domain/nodemanager/nodemanager.properties'>
<Feb 14, 2014 11:37:10 PM IST> <INFO> <Loading domains file: /u02/Middleware/user_projects/domains/base_domain/nodemanager/nodemanager.domains>
<Feb 14, 2014 11:37:10 PM IST> <INFO> <Loading identity key store: FileName=/u02/Middleware/user_projects/domains/base_domain/security/DemoIdentity.jks, Type=jks, PassPhraseUsed=true>
<Feb 14, 2014 11:37:10 PM IST> <INFO> <Loaded NodeManager configuration properties from '/u02/Middleware/user_projects/domains/base_domain/nodemanager/nodemanager.properties'>
Node manager v12.1.2

Configuration settings:

NodeManagerHome=/u02/Middleware/user_projects/domains/base_domain/nodemanager
ListenAddress=wlsadmin.user.com
ListenPort=5556
ListenBacklog=50
SecureListener=true
AuthenticationEnabled=true
NativeVersionEnabled=true
CrashRecoveryEnabled=false
JavaHome=/u02/jdk1.7.0_45
StartScriptEnabled=true
StopScriptEnabled=false
StartScriptName=startWebLogic.sh
StopScriptName=
CoherenceStartScriptEnabled=false
CoherenceStartScriptName=null
LogFile=/u02/Middleware/user_projects/domains/base_domain/nodemanager/nodemanager.log
LogLevel=INFO
LogLimit=0
LogCount=1
LogAppend=true
LogToStderr=true
LogFormatter=weblogic.nodemanager.server.LogFormatter
DomainsFile=/u02/Middleware/user_projects/domains/base_domain/nodemanager/nodemanager.domains
DomainsFileEnabled=true
StateCheckInterval=500
UseMACBroadcast=false
DomainRegistrationEnabled=false
DomainsDirRemoteSharingEnabled=false
RotatedFileCount=7
FileSizeKB=500
NumberOfFilesLimited=false
RotationTimeStart=00:00
RotationType=bySize
FileTimeSpan=24
FileTimeSpanFactor=3600000
ProcessDestroyTimeout=20000
Domain name mappings:
base_domain -> /u02/Middleware/user_projects/domains/base_domain
<Feb 14, 2014 11:37:10 PM IST> <INFO> <WebLogic Server 12.1.2.0.0  Fri Jun 7 15:16:15 PDT 2013 1530982 WLS_12.1.2.0.0_GENERIC_130607.1100>
<Feb 14, 2014 11:37:11 PM IST> <INFO> <Secure socket listener started on port 5556, host wlsadmin.user.com/127.0.0.1>