Failed to send message to remote node error
Locked
 
|
rakshita04 |
Failed to send message to remote node error
|
Hi Teams,
I am using apache-ignite 2.8.0 version for our C++ applications. We are using "TcpDiscoveryVmIpFinder" for connecting our 2 ignite nodes, running on 2 separate machines. But while running the ignite node are getting below error on one of the nodes and application and node is stopping- "Failed to send message to remote node" Below are the full logs for node 1- [14:36:32,572][SEVERE][exchange-worker-#46][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=7f683a9a-afcc-4d20-95f4-855147d759fd, consistentId=DSU_A, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.119, 127.0.0.1, 192.168.10.201, 192.168.100.100], sockAddrs=HashSet [/192.168.100.100:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.10.201:47500, /127.0.0.1:47500, /10.100.126.119:47500], discPort=47500, order=4, intOrder=3, lastExchangeTime=1608212160446, loc=false, ver=2.8.0#20200226-sha1:341b01df, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtPartitionsSingleMessage [parts=HashMap {-2100569601=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=-1, minorTopVer=0], updateSeq=2, size=0], -510489548=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=-1, minorTopVer=0], updateSeq=2, size=0]}, partCntrs=HashMap {-2100569601=CachePartitionPartialCountersMap {}, -510489548=CachePartitionPartialCountersMap {}}, partsSizes=null, partHistCntrs=null, err=null, client=false, exchangeStartTime=92638170010, finishMsg=null, super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500, /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, lastExchangeTime=1608212191947, loc=true, ver=2.8.0#20200226-sha1:341b01df, isClient=false], topVer=6, nodeId8=0c98991e, msg=null, type=NODE_JOINED, tstamp=1608212161928], nodeId=0c98991e, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0, order=1608212143633, nodeOrder=0], super=GridCacheMessage [msgId=1, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], err=null, skipPrepare=false]]]]] class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to send message (node left topology): TcpDiscoveryNode [id=7f683a9a-afcc-4d20-95f4-855147d759fd, consistentId=DSU_A, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.119, 127.0.0.1, 192.168.10.201, 192.168.100.100], sockAddrs=HashSet [/192.168.100.100:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.10.201:47500, /127.0.0.1:47500, /10.100.126.119:47500], discPort=47500, order=4, intOrder=3, lastExchangeTime=1608212160446, loc=false, ver=2.8.0#20200226-sha1:341b01df, isClient=false] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3521) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1257) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendLocalPartitions(GridDhtPartitionsExchangeFuture.java:2014) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendPartitions(GridDhtPartitionsExchangeFuture.java:2149) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1614) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:891) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3172) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3021) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.base/java.lang.Thread.run(Thread.java:834) Below are the logs for node 2- [14:36:31,026][SEVERE][exchange-worker-#46][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500, /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_INTERNAL_DIAGNOSTIC, topicOrd=27, ordered=false, timeout=0, skipOnTimeout=false, msg=IgniteDiagnosticMessage [flags=1, futId=0]]] class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100, /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, /0:0:0:0:0:0:0:1%lo:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128) at org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445) at org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664) at org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829) at org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497) at org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544) at org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115) at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55) Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100, /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, /0:0:0:0:0:0:0:1%lo:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) ... 20 more Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Failed to perform handshake due to timeout (consider increasing 'connectionTimeout' configuration property). at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) ... 20 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100, /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, /0:0:0:0:0:0:0:1%lo:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) ... 20 more Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=10000, startNanos=518549051061, currTimeout=600000]] at org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) at org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564) ... 20 more Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Failed to perform handshake due to timeout (consider increasing 'connectionTimeout' configuration property). at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) ... 20 more [14:36:31,082][SEVERE][Thread-9][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500, /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df, isClient=false], msg=GridIoMessage [plc=3, topic=TOPIC_JOB, topicOrd=0, ordered=false, timeout=0, skipOnTimeout=false, msg=GridJobExecuteRequest [subjId=7f683a9a-afcc-4d20-95f4-855147d759fd, sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, startTaskTime=1608212160962, timeout=9223372036854775807, taskName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask, userVer=0, taskClsName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask, ldrParticipants=null, cpSpi=null, createTime=1608212160982, clsLdrId=f7c73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, depMode=SHARED, dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true, topPred=null, part=-1, topVer=null, execName=null]]] class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100, /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, /0:0:0:0:0:0:0:1%lo:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128) at org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445) at org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664) at org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829) at org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497) at org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544) at org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115) at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55) Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100, /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, /0:0:0:0:0:0:0:1%lo:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) ... 20 more Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Failed to perform handshake due to timeout (consider increasing 'connectionTimeout' configuration property). at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) ... 20 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100, /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, /0:0:0:0:0:0:0:1%lo:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) ... 20 more Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=10000, startNanos=518549051061, currTimeout=600000]] at org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) at org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564) ... 20 more Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Failed to perform handshake due to timeout (consider increasing 'connectionTimeout' configuration property). at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) ... 20 more [14:36:31,084][SEVERE][exchange-worker-#46][diagnostic] Failed to send diagnostic message: class o.a.i.IgniteCheckedException: Failed to send message (node may have left the grid or TCP connection cannot be established due to firewall issues) [node=TcpDiscoveryNode [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500, /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df, isClient=false], topic=TOPIC_INTERNAL_DIAGNOSTIC, msg=IgniteDiagnosticMessage [flags=1, futId=0], policy=2] [14:36:31,314][SEVERE][Thread-9][GridTaskWorker] Failed to send job request: GridJobExecuteRequest [subjId=7f683a9a-afcc-4d20-95f4-855147d759fd, sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, startTaskTime=1608212160962, timeout=9223372036854775807, taskName=o.a.i.i.processors.cache.GridCacheAdapter$SizeTask, userVer=0, taskClsName=o.a.i.i.processors.cache.GridCacheAdapter$SizeTask, ldrParticipants=null, cpSpi=null, createTime=1608212160982, clsLdrId=f7c73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, depMode=SHARED, dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true, topPred=null, part=-1, topVer=null, execName=null] class org.apache.ignite.IgniteCheckedException: Failed to send message (node may have left the grid or TCP connection cannot be established due to firewall issues) [node=TcpDiscoveryNode [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500, /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df, isClient=false], topic=TOPIC_JOB, msg=GridJobExecuteRequest [subjId=7f683a9a-afcc-4d20-95f4-855147d759fd, sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, startTaskTime=1608212160962, timeout=9223372036854775807, taskName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask, userVer=0, taskClsName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask, ldrParticipants=null, cpSpi=null, createTime=1608212160982, clsLdrId=f7c73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, depMode=SHARED, dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true, topPred=null, part=-1, topVer=null, execName=null], policy=3] at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2042) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128) at org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445) at org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664) at org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829) at org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497) at org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544) at org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115) at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55) Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500, /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df, isClient=false] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2943) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031) ... 14 more Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100, /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, /0:0:0:0:0:0:0:1%lo:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906) ... 16 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100, /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, /0:0:0:0:0:0:0:1%lo:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) ... 20 more Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Failed to perform handshake due to timeout (consider increasing 'connectionTimeout' configuration property). at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) ... 20 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100, /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, /0:0:0:0:0:0:0:1%lo:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) ... 20 more Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=10000, startNanos=518549051061, currTimeout=600000]] at org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) at org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564) ... 20 more Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Failed to perform handshake due to timeout (consider increasing 'connectionTimeout' configuration property). at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) ... 20 more [14:36:31,387][SEVERE][Thread-9][GridTaskWorker] Failed to obtain remote job result policy for result from ComputeTask.result(..) method (will fail the whole task): GridJobResultImpl [job=SizeJob [], sib=GridJobSiblingImpl [sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, isJobDone=false], jobCtx=GridJobContextImpl [jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, timeoutObj=null, attrs=HashMap {}], node=TcpDiscoveryNode [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500, /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df, isClient=false], ex=class o.a.i.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500, /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df, isClient=false], hasRes=true, isCancelled=false, isOccupied=true] class org.apache.ignite.IgniteException: Remote job threw exception. at org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask.result(GridCacheAdapter.java:6811) at org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1062) at org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1055) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7037) at org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:1055) at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:862) at org.apache.ignite.internal.processors.task.GridTaskWorker.processDelayedResponses(GridTaskWorker.java:711) at org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:542) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829) at org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497) at org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544) at org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115) at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55) Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/192.168.100.101:47500, /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, lastExchangeTime=1608212157863, loc=false, ver=2.8.0#20200226-sha1:341b01df, isClient=false] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2943) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128) at org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445) at org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664) at org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536) ... 10 more Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100, /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, /0:0:0:0:0:0:0:1%lo:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906) ... 16 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100, /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, /0:0:0:0:0:0:0:1%lo:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) ... 20 more Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Failed to perform handshake due to timeout (consider increasing 'connectionTimeout' configuration property). at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) ... 20 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/192.168.100.101:47100, /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, /0:0:0:0:0:0:0:1%lo:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) ... 20 more Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=10000, startNanos=518549051061, currTimeout=600000]] at org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) at org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564) ... 20 more Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Failed to perform handshake due to timeout (consider increasing 'connectionTimeout' configuration property). at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) ... 20 more terminate called after throwing an instance of 'ignite::IgniteError' what(): class org.apache.ignite.IgniteCheckedException: Remote job threw exception. Please suggest how to resolve this issue -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ |
|
|
ilya.kasnacheev |
Re: Failed to send message to remote node error
|
|
Hello! Please provide complete log files from both nodes. These errors may be a sign of node leaving some time ago. Regards, -- Ilya Kasnacheev чт, 17 дек. 2020 г. в 19:43, rakshita04 <[hidden email]>: Hi Teams, |
«
Return to Apache Ignite Users
|
1 view|%1 views
| Free forum by Nabble | Edit this page |