Problem Faced:
00000016 TimeoutManage I WTRN0124I: When the timeout occurred the thread with which the transaction is, or was most recently, associated was Thread[WebContainer : 2,5,main]. The stack trace of this thread when the timeout occurred was:
java.net.PlainSocketImpl.socketConnect(Native Method)
java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
java.net.Socket.connect(Socket.java:525)
java.net.Socket.connect(Socket.java:475)
java.net.Socket.(Socket.java:372)
java.net.Socket.(Socket.java:186)
com.sun.jndi.ldap.Connection.createSocket(Connection.java:349)
com.sun.jndi.ldap.Connection.(Connection.java:184)
com.sun.jndi.ldap.LdapClient.(LdapClient.java:118)
com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1580)
com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2652)
com.sun.jndi.ldap.LdapCtx.(LdapCtx.java:293)
com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:175)
com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:134)
com.sun.jndi.url.ldap.ldapURLContextFactory.getObjectInstance(ldapURLContextFactory.java:35)
javax.naming.spi.NamingManager.getURLObject(NamingManager.java:584)
javax.naming.spi.NamingManager.processURL(NamingManager.java:364)
javax.naming.spi.NamingManager.processURLAddrs(NamingManager.java:344)
javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:316)
com.sun.jndi.ldap.LdapReferralContext.(LdapReferralContext.java:93)
com.sun.jndi.ldap.LdapReferralException.getReferralContext(LdapReferralException.java:132)
com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreReferrals(LdapNamingEnumeration.java:339)
com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(LdapNamingEnumeration.java:208)
com.sun.jndi.ldap.LdapNamingEnumeration.hasMore(LdapNamingEnumeration.java:171)
com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreElements(LdapNamingEnumeration.java:98)
com.ibm.ws.wim.adapter.ldap.LdapConnection.search(LdapConnection.java:2723)
com.ibm.ws.wim.adapter.ldap.LdapConnection.checkSearchCache(LdapConnection.java:2575)
com.ibm.ws.wim.adapter.ldap.LdapConnection.search(LdapConnection.java:2763)
com.ibm.ws.wim.adapter.ldap.LdapConnection.searchEntities(LdapConnection.java:2961)
com.ibm.ws.wim.adapter.ldap.LdapAdapter.login(LdapAdapter.java:2605)
com.ibm.ws.wim.ProfileManager.loginImpl(ProfileManager.java:3450)
com.ibm.ws.wim.ProfileManager.genericProfileManagerMethod(ProfileManager.java:281)
com.ibm.ws.wim.ProfileManager.login(ProfileManager.java:385)
com.ibm.websphere.wim.ServiceProvider.login(ServiceProvider.java:485)
com.ibm.ws.wim.registry.util.LoginBridge.checkPassword(LoginBridge.java:169)
com.ibm.ws.wim.registry.WIMUserRegistry$1.run(WIMUserRegistry.java:181)
com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:4574)
com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:4662)
com.ibm.ws.wim.security.authz.jacc.JACCSecurityManager.runAsSuperUser(JACCSecurityManager.java:432)
com.ibm.ws.wim.security.authz.ProfileSecurityManager.runAsSuperUser(ProfileSecurityManager.java:964)
com.ibm.ws.wim.registry.WIMUserRegistry.checkPassword(WIMUserRegistry.java:170)
com.ibm.ws.security.registry.UserRegistryImpl.checkPassword(UserRegistryImpl.java:337)
com.ibm.ws.security.ltpa.LTPAServerObject.authenticate(LTPAServerObject.java:973)
com.ibm.ws.security.server.lm.ltpaLoginModule.login(ltpaLoginModule.java:643)
sun.reflect.GeneratedMethodAccessor811.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)
javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
javax.security.auth.login.LoginContext.login(LoginContext.java:579)
com.ibm.ws.security.auth.JaasLoginHelper.jaas_login(JaasLoginHelper.java:354)
com.ibm.ws.security.auth.ContextManagerImpl.login(ContextManagerImpl.java:3988)
com.ibm.ws.security.auth.ContextManagerImpl.login(ContextManagerImpl.java:3692)
com.ibm.ws.security.auth.ContextManagerImpl.login(ContextManagerImpl.java:3688)
com.ibm.ws.security.auth.ContextManagerImpl.login(ContextManagerImpl.java:3459)
com.ibm.ws.sib.security.auth.login.UserNamePasswordLoginAction.login(UserNamePasswordLoginAction.java:118)
com.ibm.ws.sib.security.auth.login.BusUserNamePasswordLoginAction.login(BusUserNamePasswordLoginAction.java:90)
com.ibm.ws.sib.security.auth.login.AbstractLoginAction$1.run(AbstractLoginAction.java:130)
com.ibm.ws.sib.security.auth.login.AbstractLoginAction$1.run(AbstractLoginAction.java:127)
com.ibm.ws.sib.security.impl.BusUtilities.doInBusDomain(BusUtilities.java:115)
com.ibm.ws.sib.security.auth.login.AbstractLoginAction.run(AbstractLoginAction.java:126)
com.ibm.ws.sib.security.auth.login.AbstractLoginAction.run(AbstractLoginAction.java:67)
java.security.AccessController.doPrivileged(Native Method)
com.ibm.ws.sib.security.auth.SibLoginImpl.login(SibLoginImpl.java:537)
com.ibm.ws.sib.security.auth.SibLoginImpl.login(SibLoginImpl.java:373)
com.ibm.ws.sib.security.auth.SibLoginImpl.login(SibLoginImpl.java:327)
com.ibm.ws.sib.processor.impl.MessageProcessor.createConnection(MessageProcessor.java:751)
com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.localAttach(TrmSICoreConnectionFactoryImpl.java:445)
com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.connectFromInsideServer(TrmSICoreConnectionFactoryImpl.java:400)
com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.localBootstrap(TrmSICoreConnectionFactoryImpl.java:317)
com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.createConnection(TrmSICoreConnectionFactoryImpl.java:298)
com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.createConnection(TrmSICoreConnectionFactoryImpl.java:216)
com.ibm.ws.sib.api.jmsra.impl.JmsJcaManagedConnectionFactoryImpl.createManagedConnection(JmsJcaManagedConnectionFactoryImpl.java:465)
com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:2032)
com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1709)
com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2465)
com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1057)
com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:696)
com.ibm.ws.sib.api.jmsra.impl.JmsJcaConnectionFactoryImpl.createConnection(JmsJcaConnectionFactoryImpl.java:287)
com.ibm.ws.sib.api.jms.impl.JmsManagedConnectionFactoryImpl.createConnection(JmsManagedConnectionFactoryImpl.java:162)
com.ibm.ws.sib.api.jms.impl.JmsManagedConnectionFactoryImpl.createConnection(JmsManagedConnectionFactoryImpl.java:135)
sun.reflect.GeneratedMethodAccessor444.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
sun.reflect.GeneratedMethodAccessor718.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.ibm.rmi.util.ProxyUtil$4.run(ProxyUtil.java:612)
java.security.AccessController.doPrivileged(Native Method)
com.ibm.rmi.util.ProxyUtil.invokeWithClassLoaders(ProxyUtil.java:608)
com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1165)
$Proxy98.processCreditTransfer(Unknown Source)
sun.reflect.GeneratedMethodAccessor715.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.ibm.ws.sca.internal.java.handler.JavaReflectionAdapter$2.run(JavaReflectionAdapter.java:152)
java.security.AccessController.doPrivileged(Native Method)
com.ibm.ws.sca.internal.java.handler.JavaReflectionAdapter.invoke(JavaReflectionAdapter.java:149)
com.ibm.ws.sca.binding.ejb.handler.EJBMessageHandler.invokeLocalCORBACall(EJBMessageHandler.java:448)
com.ibm.ws.sca.binding.ejb.handler.EJBMessageHandler.processMessage(EJBMessageHandler.java:180)
com.ibm.ws.sca.binding.ejb.handler.SLSBImportNativeBindingHandler.processMessage(SLSBImportNativeBindingHandler.java:71)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:764)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1605)
com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:955)
com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionAnyActivitySessionAny(UOWNativeWASStrategyImpl.java:297)
com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:169)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:764)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1605)
com.ibm.ws.sca.internal.proxy.impl.ProxyInvocationHandlerImpl.invoke(ProxyInvocationHandlerImpl.java:1043)
com.ibm.ws.sca.internal.proxy.impl.ProxyInvocationHandlerImpl.invoke(ProxyInvocationHandlerImpl.java:862)
sun.reflect.GeneratedMethodAccessor714.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.ibm.ws.sca.internal.java.handler.JavaReflectionAdapter$2.run(JavaReflectionAdapter.java:152)
java.security.AccessController.doPrivileged(Native Method)
com.ibm.ws.sca.internal.java.handler.JavaReflectionAdapter.invoke(JavaReflectionAdapter.java:149)
com.ibm.ws.sca.internal.java.handler.JavaImplementationHandler.invokeSync(JavaImplementationHandler.java:447)
com.ibm.ws.sca.internal.java.handler.JavaImplementationHandler.processMessage(JavaImplementationHandler.java:195)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:753)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1605)
com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:955)
com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionGlobalActivitySessionAny(UOWNativeWASStrategyImpl.java:709)
com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:180)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:764)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1605)
com.ibm.ws.sca.internal.proxy.impl.ProxyInvocationHandlerImpl.invoke(ProxyInvocationHandlerImpl.java:783)
$Proxy72.processMessage(Unknown Source)
com.ibm.bpe.invocation.sca.SCAInvocationHandler.doinvoke_Sync_ReqRsp(SCAInvocationHandler.java:600)
com.ibm.bpe.invocation.sca.SCAInvocationHandler.access$000(SCAInvocationHandler.java:124)
com.ibm.bpe.invocation.sca.SCAInvocationHandler$1.run(SCAInvocationHandler.java:436)
com.ibm.bpe.invocation.sca.SCAInvocationHandler$1.run(SCAInvocationHandler.java:406)
java.security.AccessController.doPrivileged(Native Method)
com.ibm.bpe.invocation.sca.SCAInvocationHandler.doinvoke(SCAInvocationHandler.java:406)
com.ibm.bpe.invocation.sca.SCAInvocationHandler.invoke(SCAInvocationHandler.java:212)
com.ibm.bpe.engine.BpelActivityKindInvoke.doActivate(BpelActivityKindInvoke.java:545)
com.ibm.bpe.engine.BpelActivityStateInactive.activateOrSkip(BpelActivityStateInactive.java:280)
com.ibm.bpe.engine.BpelEngineCore.continueControlLink(BpelEngineCore.java:266)
com.ibm.bpe.engine.BpelContinueLinkMessage3.execute(BpelContinueLinkMessage3.java:88)
com.ibm.bpe.engine.BpelEngine.processInternalMessages(BpelEngine.java:8658)
com.ibm.bpe.framework.EngineMessageExecutionWorkObject.processAllEngineMessages(EngineMessageExecutionWorkObject.java:309)
com.ibm.bpe.framework.EngineMessageExecutionWorkObject.run(EngineMessageExecutionWorkObject.java:210)
com.ibm.bpe.framework.EngineMessageExecutor.execute(EngineMessageExecutor.java:195)
com.ibm.bpe.engine.BpelEngine.receiveEvent(BpelEngine.java:916)
com.ibm.bpe.engine.BpelEngine.receiveEvent(BpelEngine.java:819)
com.ibm.bpe.engine.BpelEngine.receiveEvent(BpelEngine.java:769)
com.ibm.bpe.framework.GenericAPIServicesImpl.invokeEngineSync(GenericAPIServicesImpl.java:725)
com.ibm.bpe.framework.sca.NativeBFMImplementationHandler.invokeEngineSync(NativeBFMImplementationHandler.java:354)
com.ibm.bpe.framework.sca.NativeBFMImplementationHandler.access$000(NativeBFMImplementationHandler.java:97)
com.ibm.bpe.framework.sca.NativeBFMImplementationHandler$1.run(NativeBFMImplementationHandler.java:171)
com.ibm.bpe.framework.sca.NativeBFMImplementationHandler$1.run(NativeBFMImplementationHandler.java:161)
com.ibm.bpe.framework.infrastructure.InfrastructureServicesSupportBean.runInCScopeTxClassLoaderContextsWithException(InfrastructureServicesSupportBean.java:125)
com.ibm.bpe.framework.infrastructure.EJSLocalStatelessInfrastructureServicesSupportBea_70e9f9d4.runInCScopeTxClassLoaderContextsWithException(Unknown Source)
com.ibm.bpe.framework.infrastructure.InfrastructureServicesImpl.runInCScopeTxClassLoaderContextsWithException(InfrastructureServicesImpl.java:246)
com.ibm.bpe.framework.sca.NativeBFMImplementationHandler.handleInboundCalls(NativeBFMImplementationHandler.java:161)
com.ibm.bpe.framework.sca.AbstractBPCImplementationHandler.processMessage(AbstractBPCImplementationHandler.java:133)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:753)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1605)
com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:955)
com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionGlobalActivitySessionAny(UOWNativeWASStrategyImpl.java:709)
com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:180)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:764)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1605)
com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:946)
com.ibm.ws.sca.internal.jaxws.handler.JaxWsExportHandler.processManagedInvoke(JaxWsExportHandler.java:552)
com.ibm.ws.sca.internal.jaxws.handler.JaxWsExportHandler.processMessage(JaxWsExportHandler.java:304)
com.ibm.wsspi.sca.jaxws.service.WsExtensionProvider.invoke(WsExtensionProvider.java:105)
com.ibm.wsspi.sca.jaxws.service.GenericJaxWsExportProvider.invoke(GenericJaxWsExportProvider.java:66)
sun.reflect.GeneratedMethodAccessor691.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.axis2.jaxws.server.dispatcher.JavaDispatcher.invokeTargetOperation(JavaDispatcher.java:82)
org.apache.axis2.jaxws.server.dispatcher.ProviderDispatcher.invoke(ProviderDispatcher.java:136)
org.apache.axis2.jaxws.server.EndpointController.invoke(EndpointController.java:110)
org.apache.axis2.jaxws.server.JAXWSMessageReceiver.receive(JAXWSMessageReceiver.java:159)
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:188)
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:275)
com.ibm.ws.websvcs.transport.http.WASAxis2Servlet.doPost(WASAxis2Servlet.java:1347)
javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1583)
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:870)
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:175)
com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:863)
com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:182)
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:455)
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:384)
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:272)
com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1016)
com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:639)
com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1772)
com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:557)
com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:607)
com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:984)
com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1069)
com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1550)
Resolution:
Changes in LDAP/JNDI context pool settings:
With the current setting for JNDI context pool, during the scenario of firewall timeout ,
the stale connection issue is noticed
The JNDI Context pool parameters to be changed as below:
The preferred size to be changed to 0 (to avoid the stale connection)
Enable Context pool timeout and set the “context pool times out” value to 2700 (to clear the unused connections )
The cache timeout to be set to 24 Hours
The same is described in the below link
http://www-01.ibm.com/support/knowledgecenter/?lang=en#!/SS7K4U_6.1.0/com.ibm.websphere.wim.doc.en/tuningtheaccesstotheldapserver.html?cp=SS7K4U_6.1.0%2F1-7-9-9-1-1-0-4-26-2-1
There is another way to resolve this problem bytuning TCP KeepAlive Interval in the application server to optimum level.
TCP_KEEPALIVE_INTERVAL
Description: The keepAlive packet ensures that a connection stays in an active and established state.
How to view or set: Use the ndd command to determine the current value or to set the value. For example:
ndd -set /dev/tcp tcp_keepalive_interval 15000
It is also described in the below link
Change in Federated Repository Settings
In case of Federated Repository , if one of the repository is down , the federated repository will not work .
In order to avoid this , we can configure a parameter which is described as below .
The default security setting in production for the parameter allowOperationIfReposDown is set as false
If Active Directory is down, the federated repository will stop working even if the file based repository is still available and the application will stop working.
Set the allowOperationIfReposDown to true
The following command to be run to set the value:
./wsadmin.sh –port 8879
$AdminTask updateIdMgrRealm {-name DEV.COM -allowOperationIfReposDown true}
$AdminConfig save
WIMCONFIG.XML after change will show as
allowOperationIfReposDown = “True”
00000016 TimeoutManage I WTRN0124I: When the timeout occurred the thread with which the transaction is, or was most recently, associated was Thread[WebContainer : 2,5,main]. The stack trace of this thread when the timeout occurred was:
java.net.PlainSocketImpl.socketConnect(Native Method)
java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
java.net.Socket.connect(Socket.java:525)
java.net.Socket.connect(Socket.java:475)
java.net.Socket.(Socket.java:372)
java.net.Socket.(Socket.java:186)
com.sun.jndi.ldap.Connection.createSocket(Connection.java:349)
com.sun.jndi.ldap.Connection.(Connection.java:184)
com.sun.jndi.ldap.LdapClient.(LdapClient.java:118)
com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1580)
com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2652)
com.sun.jndi.ldap.LdapCtx.(LdapCtx.java:293)
com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:175)
com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:134)
com.sun.jndi.url.ldap.ldapURLContextFactory.getObjectInstance(ldapURLContextFactory.java:35)
javax.naming.spi.NamingManager.getURLObject(NamingManager.java:584)
javax.naming.spi.NamingManager.processURL(NamingManager.java:364)
javax.naming.spi.NamingManager.processURLAddrs(NamingManager.java:344)
javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:316)
com.sun.jndi.ldap.LdapReferralContext.(LdapReferralContext.java:93)
com.sun.jndi.ldap.LdapReferralException.getReferralContext(LdapReferralException.java:132)
com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreReferrals(LdapNamingEnumeration.java:339)
com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(LdapNamingEnumeration.java:208)
com.sun.jndi.ldap.LdapNamingEnumeration.hasMore(LdapNamingEnumeration.java:171)
com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreElements(LdapNamingEnumeration.java:98)
com.ibm.ws.wim.adapter.ldap.LdapConnection.search(LdapConnection.java:2723)
com.ibm.ws.wim.adapter.ldap.LdapConnection.checkSearchCache(LdapConnection.java:2575)
com.ibm.ws.wim.adapter.ldap.LdapConnection.search(LdapConnection.java:2763)
com.ibm.ws.wim.adapter.ldap.LdapConnection.searchEntities(LdapConnection.java:2961)
com.ibm.ws.wim.adapter.ldap.LdapAdapter.login(LdapAdapter.java:2605)
com.ibm.ws.wim.ProfileManager.loginImpl(ProfileManager.java:3450)
com.ibm.ws.wim.ProfileManager.genericProfileManagerMethod(ProfileManager.java:281)
com.ibm.ws.wim.ProfileManager.login(ProfileManager.java:385)
com.ibm.websphere.wim.ServiceProvider.login(ServiceProvider.java:485)
com.ibm.ws.wim.registry.util.LoginBridge.checkPassword(LoginBridge.java:169)
com.ibm.ws.wim.registry.WIMUserRegistry$1.run(WIMUserRegistry.java:181)
com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:4574)
com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:4662)
com.ibm.ws.wim.security.authz.jacc.JACCSecurityManager.runAsSuperUser(JACCSecurityManager.java:432)
com.ibm.ws.wim.security.authz.ProfileSecurityManager.runAsSuperUser(ProfileSecurityManager.java:964)
com.ibm.ws.wim.registry.WIMUserRegistry.checkPassword(WIMUserRegistry.java:170)
com.ibm.ws.security.registry.UserRegistryImpl.checkPassword(UserRegistryImpl.java:337)
com.ibm.ws.security.ltpa.LTPAServerObject.authenticate(LTPAServerObject.java:973)
com.ibm.ws.security.server.lm.ltpaLoginModule.login(ltpaLoginModule.java:643)
sun.reflect.GeneratedMethodAccessor811.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)
javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
javax.security.auth.login.LoginContext.login(LoginContext.java:579)
com.ibm.ws.security.auth.JaasLoginHelper.jaas_login(JaasLoginHelper.java:354)
com.ibm.ws.security.auth.ContextManagerImpl.login(ContextManagerImpl.java:3988)
com.ibm.ws.security.auth.ContextManagerImpl.login(ContextManagerImpl.java:3692)
com.ibm.ws.security.auth.ContextManagerImpl.login(ContextManagerImpl.java:3688)
com.ibm.ws.security.auth.ContextManagerImpl.login(ContextManagerImpl.java:3459)
com.ibm.ws.sib.security.auth.login.UserNamePasswordLoginAction.login(UserNamePasswordLoginAction.java:118)
com.ibm.ws.sib.security.auth.login.BusUserNamePasswordLoginAction.login(BusUserNamePasswordLoginAction.java:90)
com.ibm.ws.sib.security.auth.login.AbstractLoginAction$1.run(AbstractLoginAction.java:130)
com.ibm.ws.sib.security.auth.login.AbstractLoginAction$1.run(AbstractLoginAction.java:127)
com.ibm.ws.sib.security.impl.BusUtilities.doInBusDomain(BusUtilities.java:115)
com.ibm.ws.sib.security.auth.login.AbstractLoginAction.run(AbstractLoginAction.java:126)
com.ibm.ws.sib.security.auth.login.AbstractLoginAction.run(AbstractLoginAction.java:67)
java.security.AccessController.doPrivileged(Native Method)
com.ibm.ws.sib.security.auth.SibLoginImpl.login(SibLoginImpl.java:537)
com.ibm.ws.sib.security.auth.SibLoginImpl.login(SibLoginImpl.java:373)
com.ibm.ws.sib.security.auth.SibLoginImpl.login(SibLoginImpl.java:327)
com.ibm.ws.sib.processor.impl.MessageProcessor.createConnection(MessageProcessor.java:751)
com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.localAttach(TrmSICoreConnectionFactoryImpl.java:445)
com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.connectFromInsideServer(TrmSICoreConnectionFactoryImpl.java:400)
com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.localBootstrap(TrmSICoreConnectionFactoryImpl.java:317)
com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.createConnection(TrmSICoreConnectionFactoryImpl.java:298)
com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.createConnection(TrmSICoreConnectionFactoryImpl.java:216)
com.ibm.ws.sib.api.jmsra.impl.JmsJcaManagedConnectionFactoryImpl.createManagedConnection(JmsJcaManagedConnectionFactoryImpl.java:465)
com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:2032)
com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1709)
com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2465)
com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1057)
com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:696)
com.ibm.ws.sib.api.jmsra.impl.JmsJcaConnectionFactoryImpl.createConnection(JmsJcaConnectionFactoryImpl.java:287)
com.ibm.ws.sib.api.jms.impl.JmsManagedConnectionFactoryImpl.createConnection(JmsManagedConnectionFactoryImpl.java:162)
com.ibm.ws.sib.api.jms.impl.JmsManagedConnectionFactoryImpl.createConnection(JmsManagedConnectionFactoryImpl.java:135)
sun.reflect.GeneratedMethodAccessor444.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
sun.reflect.GeneratedMethodAccessor718.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.ibm.rmi.util.ProxyUtil$4.run(ProxyUtil.java:612)
java.security.AccessController.doPrivileged(Native Method)
com.ibm.rmi.util.ProxyUtil.invokeWithClassLoaders(ProxyUtil.java:608)
com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1165)
$Proxy98.processCreditTransfer(Unknown Source)
sun.reflect.GeneratedMethodAccessor715.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.ibm.ws.sca.internal.java.handler.JavaReflectionAdapter$2.run(JavaReflectionAdapter.java:152)
java.security.AccessController.doPrivileged(Native Method)
com.ibm.ws.sca.internal.java.handler.JavaReflectionAdapter.invoke(JavaReflectionAdapter.java:149)
com.ibm.ws.sca.binding.ejb.handler.EJBMessageHandler.invokeLocalCORBACall(EJBMessageHandler.java:448)
com.ibm.ws.sca.binding.ejb.handler.EJBMessageHandler.processMessage(EJBMessageHandler.java:180)
com.ibm.ws.sca.binding.ejb.handler.SLSBImportNativeBindingHandler.processMessage(SLSBImportNativeBindingHandler.java:71)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:764)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1605)
com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:955)
com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionAnyActivitySessionAny(UOWNativeWASStrategyImpl.java:297)
com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:169)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:764)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1605)
com.ibm.ws.sca.internal.proxy.impl.ProxyInvocationHandlerImpl.invoke(ProxyInvocationHandlerImpl.java:1043)
com.ibm.ws.sca.internal.proxy.impl.ProxyInvocationHandlerImpl.invoke(ProxyInvocationHandlerImpl.java:862)
sun.reflect.GeneratedMethodAccessor714.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.ibm.ws.sca.internal.java.handler.JavaReflectionAdapter$2.run(JavaReflectionAdapter.java:152)
java.security.AccessController.doPrivileged(Native Method)
com.ibm.ws.sca.internal.java.handler.JavaReflectionAdapter.invoke(JavaReflectionAdapter.java:149)
com.ibm.ws.sca.internal.java.handler.JavaImplementationHandler.invokeSync(JavaImplementationHandler.java:447)
com.ibm.ws.sca.internal.java.handler.JavaImplementationHandler.processMessage(JavaImplementationHandler.java:195)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:753)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1605)
com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:955)
com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionGlobalActivitySessionAny(UOWNativeWASStrategyImpl.java:709)
com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:180)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:764)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1605)
com.ibm.ws.sca.internal.proxy.impl.ProxyInvocationHandlerImpl.invoke(ProxyInvocationHandlerImpl.java:783)
$Proxy72.processMessage(Unknown Source)
com.ibm.bpe.invocation.sca.SCAInvocationHandler.doinvoke_Sync_ReqRsp(SCAInvocationHandler.java:600)
com.ibm.bpe.invocation.sca.SCAInvocationHandler.access$000(SCAInvocationHandler.java:124)
com.ibm.bpe.invocation.sca.SCAInvocationHandler$1.run(SCAInvocationHandler.java:436)
com.ibm.bpe.invocation.sca.SCAInvocationHandler$1.run(SCAInvocationHandler.java:406)
java.security.AccessController.doPrivileged(Native Method)
com.ibm.bpe.invocation.sca.SCAInvocationHandler.doinvoke(SCAInvocationHandler.java:406)
com.ibm.bpe.invocation.sca.SCAInvocationHandler.invoke(SCAInvocationHandler.java:212)
com.ibm.bpe.engine.BpelActivityKindInvoke.doActivate(BpelActivityKindInvoke.java:545)
com.ibm.bpe.engine.BpelActivityStateInactive.activateOrSkip(BpelActivityStateInactive.java:280)
com.ibm.bpe.engine.BpelEngineCore.continueControlLink(BpelEngineCore.java:266)
com.ibm.bpe.engine.BpelContinueLinkMessage3.execute(BpelContinueLinkMessage3.java:88)
com.ibm.bpe.engine.BpelEngine.processInternalMessages(BpelEngine.java:8658)
com.ibm.bpe.framework.EngineMessageExecutionWorkObject.processAllEngineMessages(EngineMessageExecutionWorkObject.java:309)
com.ibm.bpe.framework.EngineMessageExecutionWorkObject.run(EngineMessageExecutionWorkObject.java:210)
com.ibm.bpe.framework.EngineMessageExecutor.execute(EngineMessageExecutor.java:195)
com.ibm.bpe.engine.BpelEngine.receiveEvent(BpelEngine.java:916)
com.ibm.bpe.engine.BpelEngine.receiveEvent(BpelEngine.java:819)
com.ibm.bpe.engine.BpelEngine.receiveEvent(BpelEngine.java:769)
com.ibm.bpe.framework.GenericAPIServicesImpl.invokeEngineSync(GenericAPIServicesImpl.java:725)
com.ibm.bpe.framework.sca.NativeBFMImplementationHandler.invokeEngineSync(NativeBFMImplementationHandler.java:354)
com.ibm.bpe.framework.sca.NativeBFMImplementationHandler.access$000(NativeBFMImplementationHandler.java:97)
com.ibm.bpe.framework.sca.NativeBFMImplementationHandler$1.run(NativeBFMImplementationHandler.java:171)
com.ibm.bpe.framework.sca.NativeBFMImplementationHandler$1.run(NativeBFMImplementationHandler.java:161)
com.ibm.bpe.framework.infrastructure.InfrastructureServicesSupportBean.runInCScopeTxClassLoaderContextsWithException(InfrastructureServicesSupportBean.java:125)
com.ibm.bpe.framework.infrastructure.EJSLocalStatelessInfrastructureServicesSupportBea_70e9f9d4.runInCScopeTxClassLoaderContextsWithException(Unknown Source)
com.ibm.bpe.framework.infrastructure.InfrastructureServicesImpl.runInCScopeTxClassLoaderContextsWithException(InfrastructureServicesImpl.java:246)
com.ibm.bpe.framework.sca.NativeBFMImplementationHandler.handleInboundCalls(NativeBFMImplementationHandler.java:161)
com.ibm.bpe.framework.sca.AbstractBPCImplementationHandler.processMessage(AbstractBPCImplementationHandler.java:133)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:753)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1605)
com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:955)
com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionGlobalActivitySessionAny(UOWNativeWASStrategyImpl.java:709)
com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:180)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:764)
com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1605)
com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:946)
com.ibm.ws.sca.internal.jaxws.handler.JaxWsExportHandler.processManagedInvoke(JaxWsExportHandler.java:552)
com.ibm.ws.sca.internal.jaxws.handler.JaxWsExportHandler.processMessage(JaxWsExportHandler.java:304)
com.ibm.wsspi.sca.jaxws.service.WsExtensionProvider.invoke(WsExtensionProvider.java:105)
com.ibm.wsspi.sca.jaxws.service.GenericJaxWsExportProvider.invoke(GenericJaxWsExportProvider.java:66)
sun.reflect.GeneratedMethodAccessor691.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.axis2.jaxws.server.dispatcher.JavaDispatcher.invokeTargetOperation(JavaDispatcher.java:82)
org.apache.axis2.jaxws.server.dispatcher.ProviderDispatcher.invoke(ProviderDispatcher.java:136)
org.apache.axis2.jaxws.server.EndpointController.invoke(EndpointController.java:110)
org.apache.axis2.jaxws.server.JAXWSMessageReceiver.receive(JAXWSMessageReceiver.java:159)
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:188)
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:275)
com.ibm.ws.websvcs.transport.http.WASAxis2Servlet.doPost(WASAxis2Servlet.java:1347)
javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1583)
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:870)
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:175)
com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:863)
com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:182)
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:455)
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:384)
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:272)
com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1016)
com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:639)
com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1772)
com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:557)
com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:607)
com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:984)
com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1069)
com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1550)
Resolution:
Changes in LDAP/JNDI context pool settings:
With the current setting for JNDI context pool, during the scenario of firewall timeout ,
the stale connection issue is noticed
The JNDI Context pool parameters to be changed as below:
The preferred size to be changed to 0 (to avoid the stale connection)
Enable Context pool timeout and set the “context pool times out” value to 2700 (to clear the unused connections )
The cache timeout to be set to 24 Hours
The same is described in the below link
http://www-01.ibm.com/support/knowledgecenter/?lang=en#!/SS7K4U_6.1.0/com.ibm.websphere.wim.doc.en/tuningtheaccesstotheldapserver.html?cp=SS7K4U_6.1.0%2F1-7-9-9-1-1-0-4-26-2-1
There is another way to resolve this problem by
How to view or set: Use the ndd command to determine the current value or to set the value. For example:
ndd -set /dev/tcp tcp_keepalive_interval 15000
Change in Federated Repository Settings
In case of Federated Repository , if one of the repository is down , the federated repository will not work .
In order to avoid this , we can configure a parameter which is described as below .
The default security setting in production for the parameter allowOperationIfReposDown is set as false
If Active Directory is down, the federated repository will stop working even if the file based repository is still available and the application will stop working.
Set the allowOperationIfReposDown to true
The following command to be run to set the value:
./wsadmin.sh –port 8879
$AdminTask updateIdMgrRealm {-name DEV.COM -allowOperationIfReposDown true}
$AdminConfig save
WIMCONFIG.XML after change will show as
allowOperationIfReposDown = “True”
No comments:
Post a Comment