Failed to send message to remote node error

classic Classic list List threaded Threaded
2 messages Options
rakshita04 rakshita04
Reply | Threaded
Open this post in threaded view
|

Failed to send message to remote node error

Hi Teams,

I am using apache-ignite 2.8.0 version for our C++ applications.
We are using "TcpDiscoveryVmIpFinder" for connecting our 2 ignite nodes,
running on 2 separate machines.
But while running the ignite node are getting below error on one of the
nodes and application and node is stopping-
"Failed to send message to remote node"
Below are the full logs for node 1-
[14:36:32,572][SEVERE][exchange-worker-#46][TcpCommunicationSpi] Failed to
send message to remote node [node=TcpDiscoveryNode
[id=7f683a9a-afcc-4d20-95f4-855147d759fd, consistentId=DSU_A,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.119, 127.0.0.1,
192.168.10.201, 192.168.100.100], sockAddrs=HashSet [/192.168.100.100:47500,
/0:0:0:0:0:0:0:1%lo:47500, /192.168.10.201:47500, /127.0.0.1:47500,
/10.100.126.119:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1608212160446, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
ordered=false, timeout=0, skipOnTimeout=false,
msg=GridDhtPartitionsSingleMessage [parts=HashMap
{-2100569601=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion
[topVer=-1, minorTopVer=0], updateSeq=2, size=0],
-510489548=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion
[topVer=-1, minorTopVer=0], updateSeq=2, size=0]}, partCntrs=HashMap
{-2100569601=CachePartitionPartialCountersMap {},
-510489548=CachePartitionPartialCountersMap {}}, partsSizes=null,
partHistCntrs=null, err=null, client=false, exchangeStartTime=92638170010,
finishMsg=null, super=GridDhtPartitionsAbstractMessage
[exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=6, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212191947, loc=true, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], topVer=6, nodeId8=0c98991e, msg=null, type=NODE_JOINED,
tstamp=1608212161928], nodeId=0c98991e, evt=NODE_JOINED],
lastVer=GridCacheVersion [topVer=0, order=1608212143633, nodeOrder=0],
super=GridCacheMessage [msgId=1, depInfo=null,
lastAffChangedTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0],
err=null, skipPrepare=false]]]]]
class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
Failed to send message (node left topology): TcpDiscoveryNode
[id=7f683a9a-afcc-4d20-95f4-855147d759fd, consistentId=DSU_A,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.119, 127.0.0.1,
192.168.10.201, 192.168.100.100], sockAddrs=HashSet [/192.168.100.100:47500,
/0:0:0:0:0:0:0:1%lo:47500, /192.168.10.201:47500, /127.0.0.1:47500,
/10.100.126.119:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1608212160446, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3521)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1257)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendLocalPartitions(GridDhtPartitionsExchangeFuture.java:2014)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendPartitions(GridDhtPartitionsExchangeFuture.java:2149)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1614)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:891)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3172)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3021)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.base/java.lang.Thread.run(Thread.java:834)


Below are the logs for node 2-
[14:36:31,026][SEVERE][exchange-worker-#46][TcpCommunicationSpi] Failed to
send message to remote node [node=TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_INTERNAL_DIAGNOSTIC,
topicOrd=27, ordered=false, timeout=0, skipOnTimeout=false,
msg=IgniteDiagnosticMessage [flags=1, futId=0]]]
class org.apache.ignite.IgniteCheckedException: Failed to connect to node
(is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544)
        at
org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115)
        at
org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55)
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
                ... 20 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=10000, startNanos=518549051061,
currTimeout=600000]]
                at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
                at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564)
                ... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
        ... 20 more
[14:36:31,082][SEVERE][Thread-9][TcpCommunicationSpi] Failed to send message
to remote node [node=TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], msg=GridIoMessage [plc=3, topic=TOPIC_JOB, topicOrd=0,
ordered=false, timeout=0, skipOnTimeout=false, msg=GridJobExecuteRequest
[subjId=7f683a9a-afcc-4d20-95f4-855147d759fd,
sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
startTaskTime=1608212160962, timeout=9223372036854775807,
taskName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask,
userVer=0,
taskClsName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask,
ldrParticipants=null, cpSpi=null, createTime=1608212160982,
clsLdrId=f7c73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, depMode=SHARED,
dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true,
topPred=null, part=-1, topVer=null, execName=null]]]
class org.apache.ignite.IgniteCheckedException: Failed to connect to node
(is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544)
        at
org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115)
        at
org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55)
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
                ... 20 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=10000, startNanos=518549051061,
currTimeout=600000]]
                at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
                at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564)
                ... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
        ... 20 more
[14:36:31,084][SEVERE][exchange-worker-#46][diagnostic] Failed to send
diagnostic message: class o.a.i.IgniteCheckedException: Failed to send
message (node may have left the grid or TCP connection cannot be established
due to firewall issues) [node=TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], topic=TOPIC_INTERNAL_DIAGNOSTIC,
msg=IgniteDiagnosticMessage [flags=1, futId=0], policy=2]
[14:36:31,314][SEVERE][Thread-9][GridTaskWorker] Failed to send job request:
GridJobExecuteRequest [subjId=7f683a9a-afcc-4d20-95f4-855147d759fd,
sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
startTaskTime=1608212160962, timeout=9223372036854775807,
taskName=o.a.i.i.processors.cache.GridCacheAdapter$SizeTask, userVer=0,
taskClsName=o.a.i.i.processors.cache.GridCacheAdapter$SizeTask,
ldrParticipants=null, cpSpi=null, createTime=1608212160982,
clsLdrId=f7c73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, depMode=SHARED,
dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true,
topPred=null, part=-1, topVer=null, execName=null]
class org.apache.ignite.IgniteCheckedException: Failed to send message (node
may have left the grid or TCP connection cannot be established due to
firewall issues) [node=TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], topic=TOPIC_JOB, msg=GridJobExecuteRequest
[subjId=7f683a9a-afcc-4d20-95f4-855147d759fd,
sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
startTaskTime=1608212160962, timeout=9223372036854775807,
taskName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask,
userVer=0,
taskClsName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask,
ldrParticipants=null, cpSpi=null, createTime=1608212160982,
clsLdrId=f7c73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, depMode=SHARED,
dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true,
topPred=null, part=-1, topVer=null, execName=null], policy=3]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2042)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544)
        at
org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115)
        at
org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send
message to remote node: TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2943)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031)
        ... 14 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect
to node (is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906)
        ... 16 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
                ... 20 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=10000, startNanos=518549051061,
currTimeout=600000]]
                at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
                at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564)
                ... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
        ... 20 more
[14:36:31,387][SEVERE][Thread-9][GridTaskWorker] Failed to obtain remote job
result policy for result from ComputeTask.result(..) method (will fail the
whole task): GridJobResultImpl [job=SizeJob [], sib=GridJobSiblingImpl
[sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, isJobDone=false],
jobCtx=GridJobContextImpl
[jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, timeoutObj=null,
attrs=HashMap {}], node=TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], ex=class o.a.i.spi.IgniteSpiException: Failed to send
message to remote node: TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], hasRes=true, isCancelled=false, isOccupied=true]
class org.apache.ignite.IgniteException: Remote job threw exception.
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask.result(GridCacheAdapter.java:6811)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1062)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1055)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7037)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:1055)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:862)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.processDelayedResponses(GridTaskWorker.java:711)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:542)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544)
        at
org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115)
        at
org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send
message to remote node: TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2943)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536)
        ... 10 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect
to node (is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906)
        ... 16 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
                ... 20 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=10000, startNanos=518549051061,
currTimeout=600000]]
                at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
                at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564)
                ... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
        ... 20 more
terminate called after throwing an instance of 'ignite::IgniteError'
  what():  class org.apache.ignite.IgniteCheckedException: Remote job threw
exception.


Please suggest how to resolve this issue



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Failed to send message to remote node error

Hello!

Please provide complete log files from both nodes. These errors may be a sign of node leaving some time ago.

Regards,
--
Ilya Kasnacheev


чт, 17 дек. 2020 г. в 19:43, rakshita04 <[hidden email]>:
Hi Teams,

I am using apache-ignite 2.8.0 version for our C++ applications.
We are using "TcpDiscoveryVmIpFinder" for connecting our 2 ignite nodes,
running on 2 separate machines.
But while running the ignite node are getting below error on one of the
nodes and application and node is stopping-
"Failed to send message to remote node"
Below are the full logs for node 1-
[14:36:32,572][SEVERE][exchange-worker-#46][TcpCommunicationSpi] Failed to
send message to remote node [node=TcpDiscoveryNode
[id=7f683a9a-afcc-4d20-95f4-855147d759fd, consistentId=DSU_A,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.119, 127.0.0.1,
192.168.10.201, 192.168.100.100], sockAddrs=HashSet [/192.168.100.100:47500,
/0:0:0:0:0:0:0:1%lo:47500, /192.168.10.201:47500, /127.0.0.1:47500,
/10.100.126.119:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1608212160446, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
ordered=false, timeout=0, skipOnTimeout=false,
msg=GridDhtPartitionsSingleMessage [parts=HashMap
{-2100569601=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion
[topVer=-1, minorTopVer=0], updateSeq=2, size=0],
-510489548=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion
[topVer=-1, minorTopVer=0], updateSeq=2, size=0]}, partCntrs=HashMap
{-2100569601=CachePartitionPartialCountersMap {},
-510489548=CachePartitionPartialCountersMap {}}, partsSizes=null,
partHistCntrs=null, err=null, client=false, exchangeStartTime=92638170010,
finishMsg=null, super=GridDhtPartitionsAbstractMessage
[exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=6, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212191947, loc=true, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], topVer=6, nodeId8=0c98991e, msg=null, type=NODE_JOINED,
tstamp=1608212161928], nodeId=0c98991e, evt=NODE_JOINED],
lastVer=GridCacheVersion [topVer=0, order=1608212143633, nodeOrder=0],
super=GridCacheMessage [msgId=1, depInfo=null,
lastAffChangedTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0],
err=null, skipPrepare=false]]]]]
class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
Failed to send message (node left topology): TcpDiscoveryNode
[id=7f683a9a-afcc-4d20-95f4-855147d759fd, consistentId=DSU_A,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.119, 127.0.0.1,
192.168.10.201, 192.168.100.100], sockAddrs=HashSet [/192.168.100.100:47500,
/0:0:0:0:0:0:0:1%lo:47500, /192.168.10.201:47500, /127.0.0.1:47500,
/10.100.126.119:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1608212160446, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3521)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1257)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendLocalPartitions(GridDhtPartitionsExchangeFuture.java:2014)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendPartitions(GridDhtPartitionsExchangeFuture.java:2149)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1614)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:891)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3172)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3021)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.base/java.lang.Thread.run(Thread.java:834)


Below are the logs for node 2-
[14:36:31,026][SEVERE][exchange-worker-#46][TcpCommunicationSpi] Failed to
send message to remote node [node=TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_INTERNAL_DIAGNOSTIC,
topicOrd=27, ordered=false, timeout=0, skipOnTimeout=false,
msg=IgniteDiagnosticMessage [flags=1, futId=0]]]
class org.apache.ignite.IgniteCheckedException: Failed to connect to node
(is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544)
        at
org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115)
        at
org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55)
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
                ... 20 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=10000, startNanos=518549051061,
currTimeout=600000]]
                at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
                at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564)
                ... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
        ... 20 more
[14:36:31,082][SEVERE][Thread-9][TcpCommunicationSpi] Failed to send message
to remote node [node=TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], msg=GridIoMessage [plc=3, topic=TOPIC_JOB, topicOrd=0,
ordered=false, timeout=0, skipOnTimeout=false, msg=GridJobExecuteRequest
[subjId=7f683a9a-afcc-4d20-95f4-855147d759fd,
sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
startTaskTime=1608212160962, timeout=9223372036854775807,
taskName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask,
userVer=0,
taskClsName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask,
ldrParticipants=null, cpSpi=null, createTime=1608212160982,
clsLdrId=f7c73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, depMode=SHARED,
dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true,
topPred=null, part=-1, topVer=null, execName=null]]]
class org.apache.ignite.IgniteCheckedException: Failed to connect to node
(is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544)
        at
org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115)
        at
org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55)
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
                ... 20 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=10000, startNanos=518549051061,
currTimeout=600000]]
                at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
                at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564)
                ... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
        ... 20 more
[14:36:31,084][SEVERE][exchange-worker-#46][diagnostic] Failed to send
diagnostic message: class o.a.i.IgniteCheckedException: Failed to send
message (node may have left the grid or TCP connection cannot be established
due to firewall issues) [node=TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], topic=TOPIC_INTERNAL_DIAGNOSTIC,
msg=IgniteDiagnosticMessage [flags=1, futId=0], policy=2]
[14:36:31,314][SEVERE][Thread-9][GridTaskWorker] Failed to send job request:
GridJobExecuteRequest [subjId=7f683a9a-afcc-4d20-95f4-855147d759fd,
sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
startTaskTime=1608212160962, timeout=9223372036854775807,
taskName=o.a.i.i.processors.cache.GridCacheAdapter$SizeTask, userVer=0,
taskClsName=o.a.i.i.processors.cache.GridCacheAdapter$SizeTask,
ldrParticipants=null, cpSpi=null, createTime=1608212160982,
clsLdrId=f7c73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, depMode=SHARED,
dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true,
topPred=null, part=-1, topVer=null, execName=null]
class org.apache.ignite.IgniteCheckedException: Failed to send message (node
may have left the grid or TCP connection cannot be established due to
firewall issues) [node=TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], topic=TOPIC_JOB, msg=GridJobExecuteRequest
[subjId=7f683a9a-afcc-4d20-95f4-855147d759fd,
sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
startTaskTime=1608212160962, timeout=9223372036854775807,
taskName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask,
userVer=0,
taskClsName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask,
ldrParticipants=null, cpSpi=null, createTime=1608212160982,
clsLdrId=f7c73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, depMode=SHARED,
dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true,
topPred=null, part=-1, topVer=null, execName=null], policy=3]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2042)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544)
        at
org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115)
        at
org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send
message to remote node: TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2943)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031)
        ... 14 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect
to node (is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906)
        ... 16 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
                ... 20 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=10000, startNanos=518549051061,
currTimeout=600000]]
                at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
                at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564)
                ... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
        ... 20 more
[14:36:31,387][SEVERE][Thread-9][GridTaskWorker] Failed to obtain remote job
result policy for result from ComputeTask.result(..) method (will fail the
whole task): GridJobResultImpl [job=SizeJob [], sib=GridJobSiblingImpl
[sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd,
nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, isJobDone=false],
jobCtx=GridJobContextImpl
[jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, timeoutObj=null,
attrs=HashMap {}], node=TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], ex=class o.a.i.spi.IgniteSpiException: Failed to send
message to remote node: TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], hasRes=true, isCancelled=false, isOccupied=true]
class org.apache.ignite.IgniteException: Remote job threw exception.
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask.result(GridCacheAdapter.java:6811)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1062)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1055)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7037)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:1055)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:862)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.processDelayedResponses(GridTaskWorker.java:711)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:542)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544)
        at
org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115)
        at
org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send
message to remote node: TcpDiscoveryNode
[id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1,
192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500,
/10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df,
isClient=false]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2943)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536)
        ... 10 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect
to node (is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906)
        ... 16 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
                ... 20 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100,
/10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%lo:47100]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682)
                ... 20 more
        Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=10000, startNanos=518549051061,
currTimeout=600000]]
                at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
                at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564)
                ... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Failed to perform handshake due to timeout (consider increasing
'connectionTimeout' configuration property).
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589)
        ... 20 more
terminate called after throwing an instance of 'ignite::IgniteError'
  what():  class org.apache.ignite.IgniteCheckedException: Remote job threw
exception.


Please suggest how to resolve this issue



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/