Server Node comes down when a thick client node comes down and an update is issued within the failuredetectiontimeout

classic Classic list List threaded Threaded
2 messages Options
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Server Node comes down when a thick client node comes down and an update is issued within the failuredetectiontimeout

This post was updated on .
We have a 3 node server cluster ( Issue observed in 2.7.6, could not test in
2.8.0 because I am unable to bring up the dbeaver in 2.8.0 with
securityplugin enabled
)(http://apache-ignite-users.70518.x6.nabble.com/2-8-0-JDBC-Thin-Client-Unable-to-load-the-tables-via-DBeaver-td31681.html)

A 4th node joins as a client with a continuous query on a Table A(
Transaction_mode = transactional ).

Now If I bring the client down and issue an update to the Table A within
failureDetectionTimeout 30000 , I get the following error and this error
brings the server down since it causes a */unhandled exception in a critical
thread/*:

======================================
/*ERROR  [] - Critical system error detected. Will be handled accordingly to
configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false,
timeout=0, super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=CRITICAL_ERROR, err=java.lang.NoClassDefFoundError:
com/companyname/projectname/modulename/helper/ContinuousQueryHelper]]
java.lang.NoClassDefFoundError:
com/companyname/projectname/modulename/helper/ContinuousQueryHelper*/
        at
com.companyname.projectname.modulename.helper.ContinuousQueryHelper$ModuleTableRemoteFilterFactory$1.evaluate(ContinuousQueryHelper.java:289)
~[?:?]
    at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.filter(CacheContinuousQueryHandler.java:833)
~[ignite-core-2.7.6.jar:2.7.6]
    at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:422)
~[ignite-core-2.7.6.jar:2.7.6]
    at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:426)
~[ignite-core-2.7.6.jar:2.7.6]
    at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1584)
~[ignite-core-2.7.6.jar:2.7.6]
=================================
"(err) Failed to notify listener: GridDhtTxPrepareFuture Error"
===================================
Basically the server , tries to update the record on the Table A, and tries
to  notify Client since it had registered a continuous query for Table A.  
But since the Client Node has been brought down, it undeploys the
remotefilterfactory lambda. Hence the server is no longer able to complete
the
transaction .

*/This also brings the server down./
*
How can I resolve this issue ?
=======================================
Please find the complete stack trace for this error :


2020-03-13 17:13:40,145 sys-stripe-19-#20 ERROR [] - Critical system error
detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=CRITICAL_ERROR, err=java.lang.NoClassDefFoundError:
com/companyname/projectname/Module/helper/ContinuousQueryHelper]]
java.lang.NoClassDefFoundError:
com/companyname/projectname/Module/helper/ContinuousQueryHelper
at
com.companyname.projectname.Module.helper.ContinuousQueryHelper$ModuleTableRemoteFilterFactory$1.evaluate(ContinuousQueryHelper.java:289)
~[?:?]
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.filter(CacheContinuousQueryHandler.java:833)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:422)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:426)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1584)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:741)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.localFinish(GridNearTxLocal.java:3646)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.doFinish(GridNearTxFinishFuture.java:475)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.finish(GridNearTxFinishFuture.java:425)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal$25.apply(GridNearTxLocal.java:3788)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal$25.apply(GridNearTxLocal.java:3782)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.GridCacheCompoundFuture.onDone(GridCacheCompoundFuture.java:56)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.onComplete(GridNearOptimisticTxPrepareFuture.java:307)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.onDone(GridNearOptimisticTxPrepareFuture.java:285)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.onDone(GridNearOptimisticTxPrepareFuture.java:79)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture$MiniFuture.onResult(GridNearOptimisticTxPrepareFuture.java:1046)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture$2.apply(GridNearOptimisticTxPrepareFuture.java:612)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture$2.apply(GridNearOptimisticTxPrepareFuture.java:609)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.GridCacheCompoundFuture.onDone(GridCacheCompoundFuture.java:56)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onComplete(GridDhtTxPrepareFuture.java:1006)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:806)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:110)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture$MiniFuture.onResult(GridDhtTxPrepareFuture.java:1948)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onResult(GridDhtTxPrepareFuture.java:572)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareResponse(IgniteTxHandler.java:798)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$500(IgniteTxHandler.java:119)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:229)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:227)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.7.6.jar:2.7.6]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
Caused by: java.lang.ClassNotFoundException: Failed to peer load class
[class=com.companyname.projectname.Module.helper.ContinuousQueryHelper,
nodeClsLdrs=\{123a731e-fc4e-46cc-a9f7-bd72fbde82fb=c8be2e4d071-123a731e-fc4e-46cc-a9f7-bd72fbde82fb},
parentClsLoader=sun.misc.Launcher$AppClassLoader@73d16e93]
at
org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.sendClassRequest(GridDeploymentClassLoader.java:661)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.findClass(GridDeploymentClassLoader.java:508)
~[ignite-core-2.7.6.jar:2.7.6]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_241]
at
org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.loadClass(GridDeploymentClassLoader.java:440)
~[ignite-core-2.7.6.jar:2.7.6]
... 80 more
Caused by: org.apache.ignite.IgniteCheckedException: Failed to send message
(node may have left the grid or TCP connection cannot be established due to
firewall issues) [node=TcpDiscoveryNode
[id=123a731e-fc4e-46cc-a9f7-bd72fbde82fb, addrs=[0:0:0:0:0:0:0:1, x.x.x.44,
127.0.0.1], sockAddrs=[machinename.companyname.LOCAL/x.x.x.44:0,
/0:0:0:0:0:0:0:1:0, /127.0.0.1:0], discPort=0, order=4, intOrder=4,
lastExchangeTime=1584119608194, loc=false, ver=2.7.6#20190911-sha1:21f7ca41,
isClient=true], topic=TOPIC_CLASSLOAD, msg=GridDeploymentRequest
[rsrcName=com/companyname/projectname/Module/helper/ContinuousQueryHelper.class,
ldrId=c8be2e4d071-123a731e-fc4e-46cc-a9f7-bd72fbde82fb, isUndeploy=false,
nodeIds=null], policy=1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1667)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1731)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.deployment.GridDeploymentCommunication.sendResourceRequest(GridDeploymentCommunication.java:454)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.sendClassRequest(GridDeploymentClassLoader.java:601)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.findClass(GridDeploymentClassLoader.java:508)
~[ignite-core-2.7.6.jar:2.7.6]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_241]
at
org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.loadClass(GridDeploymentClassLoader.java:440)
~[ignite-core-2.7.6.jar:2.7.6]
... 80 more
Caused by: org.apache.ignite.spi.IgniteSpiException: Failed to send message
to remote node: TcpDiscoveryNode [id=123a731e-fc4e-46cc-a9f7-bd72fbde82fb,
addrs=[0:0:0:0:0:0:0:1, x.x.x.44, 127.0.0.1],
sockAddrs=[machinename.companyname.LOCAL/x.x.x.44:0, /0:0:0:0:0:0:0:1:0,
/127.0.0.1:0], discPort=0, order=4, intOrder=4,
lastExchangeTime=1584119608194, loc=false, ver=2.7.6#20190911-sha1:21f7ca41,
isClient=true]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2747)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2672)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1656)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1731)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.deployment.GridDeploymentCommunication.sendResourceRequest(GridDeploymentCommunication.java:454)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.sendClassRequest(GridDeploymentClassLoader.java:601)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.findClass(GridDeploymentClassLoader.java:508)
~[ignite-core-2.7.6.jar:2.7.6]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_241]
at
org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.loadClass(GridDeploymentClassLoader.java:440)
~[ignite-core-2.7.6.jar:2.7.6]
... 80 more
Caused by: org.apache.ignite.IgniteCheckedException: Failed to connect to
node (is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=123a731e-fc4e-46cc-a9f7-bd72fbde82fb, addrs=[/127.0.0.1:47103,
/0:0:0:0:0:0:0:1:47103, machinename.companyname.LOCAL/x.x.x.44:47103]]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3459)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.access$6000(TcpCommunicationSpi.java:271)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.processDisconnect(TcpCommunicationSpi.java:4489)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4294)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
~[ignite-core-2.7.6.jar:2.7.6]
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
~[ignite-core-2.7.6.jar:2.7.6]
Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to
address [addr=/127.0.0.1:47103, err=Connection refused: no further
information]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3462)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.access$6000(TcpCommunicationSpi.java:271)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.processDisconnect(TcpCommunicationSpi.java:4489)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4294)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
~[ignite-core-2.7.6.jar:2.7.6]
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
~[ignite-core-2.7.6.jar:2.7.6]
Caused by: java.net.ConnectException: Connection refused: no further
information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_241]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
~[?:1.8.0_241]
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) ~[?:1.8.0_241]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3299)
~[ignite-core-2.7.6.jar:2.7.6]
... 8 more
Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to
address [addr=/0:0:0:0:0:0:0:1:47103, err=Connection refused: no further
information]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3462)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.access$6000(TcpCommunicationSpi.java:271)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.processDisconnect(TcpCommunicationSpi.java:4489)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4294)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
~[ignite-core-2.7.6.jar:2.7.6]
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
~[ignite-core-2.7.6.jar:2.7.6]
Caused by: java.net.ConnectException: Connection refused: no further
information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_241]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
~[?:1.8.0_241]
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) ~[?:1.8.0_241]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3299)
~[ignite-core-2.7.6.jar:2.7.6]
... 8 more
Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to
address [addr=machinename.companyname.LOCAL/x.x.x.44:47103, err=Connection
refused: no further information]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3462)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.access$6000(TcpCommunicationSpi.java:271)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.processDisconnect(TcpCommunicationSpi.java:4489)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4294)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
~[ignite-core-2.7.6.jar:2.7.6]
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
~[ignite-core-2.7.6.jar:2.7.6]
Caused by: java.net.ConnectException: Connection refused: no further
information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_241]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
~[?:1.8.0_241]
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) ~[?:1.8.0_241]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3299)
~[ignite-core-2.7.6.jar:2.7.6]
... 8 more
2


I have created this jira : https://issues.apache.org/jira/browse/IGNITE-12784

Kindly suggest a workaround , since we have huge number of thick clients and restart of these clients might bring the server down.
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Server Node comes down when a thick client node comes down and an update is issued within the failuredetectiontimeout