![]() ![]() |
VeenaMithare |
![]() |
Hi ,
Our setup : Servers - 3 node cluster Reader clients : wait for an update on an entry of a cache ( around 20 of them ) Writer Client : 1 If one of the reader client restarts while the writer is writing into the entry of the cache , the server attempts to send the update to the failed client's local listener . It keeps attempting to communicate with the failed client ( client's continous query local listener ? ) till it timesout as per connTimeoutStrategy=ExponentialBackoffTimeoutStrategy . ( Please find the snippet of the exception below. The complete log is attached as an attachment ) This delays the completion of the transaction that was started by the writer client. Is there any way the writer client could complete the transaction without getting impacted by the reader client restarts ? 2020-10-08 14:35:21,465 [sys-stripe-26-#27] WARN org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Handshake timed out (will stop attempts to perform the handshake) [node=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, connTimeoutStrategy=ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=30000, startNanos=223772180706400, currTimeout=600000], err=Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=30000, startNanos=223772180706400, currTimeout=600000]], addr=MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103, failureDetectionTimeoutEnabled=true, timeout=0] 2020-10-08 14:35:21,465 [sys-stripe-26-#27] ERROR org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Failed to send message to remote node [node=TcpDiscoveryNode [id=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, consistentId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=ArrayList [0:0:0:0:0:0:0:1, 1.2.3.4, 127.0.0.1], sockAddrs=HashSet [MACHINENAME.COMPANY.LOCAL/1.2.3.4:0, /0:0:0:0:0:0:0:1:0, /127.0.0.1:0], discPort=0, order=12, intOrder=8, lastExchangeTime=1602163619453, loc=false, ver=2.8.1#20200521-sha1:86422096, isClient=true], msg=GridIoMessage [plc=2, topic=T4 [topic=TOPIC_CACHE, id1=94370aa1-e970-37ae-9471-fd583d923522, id2=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, id3=0], topicOrd=-1, ordered=true, timeout=0, skipOnTimeout=true, msg=GridContinuousMessage [type=MSG_EVT_NOTIFICATION, routineId=7dac3ff4-3460-4dc4-8324-a1ebe4561854, data=null, futId=null]]] org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=[/127.0.0.1:47103, /0:0:0:0:0:0:0:1:47103, MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3698) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:2280) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1963) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1934) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1916) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1323) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1260) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1045) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1588) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:745) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:606) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:478) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:523) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1077) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) ~[ignite-core-2.8.1.jar:2.8.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=[/127.0.0.1:47103, /0:0:0:0:0:0:0:1:47103, MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3700) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:2280) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1963) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1934) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1916) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1323) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1260) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1045) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1588) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:745) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:606) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:478) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:523) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1077) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) ~[ignite-core-2.8.1.jar:2.8.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=30000, startNanos=223742176046300, currTimeout=600000]] at org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3582) ~[ignite-core-2.8.1.jar:2.8.1] ... 41 more Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=[/127.0.0.1:47103, /0:0:0:0:0:0:0:1:47103, MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3700) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:2280) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1963) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1934) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1916) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1323) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1260) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1045) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1588) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:745) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:606) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:478) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:523) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1077) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) ~[ignite-core-2.8.1.jar:2.8.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=30000, startNanos=223772180706400, currTimeout=600000]] at org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3582) ~[ignite-core-2.8.1.jar:2.8.1] ... 41 more Caused by: org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=30000, startNanos=223712146125000, currTimeout=600000]] at org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3582) ~[ignite-core-2.8.1.jar:2.8.1] ... 41 more 2020-10-08 14:35:21,467 [sys-stripe-26-#27] ERROR org.apache.ignite.continuous.query [] - Failed to send event notification to node: 3311a67b-bfcb-41af-8c09-b2e8f2fbde9b EXPONENTIALBACKOFFSTRATEGY.txt <http://apache-ignite-users.70518.x6.nabble.com/file/t2757/EXPONENTIALBACKOFFSTRATEGY.txt> -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ |
![]() ![]() |
VeenaMithare |
![]() |
Hi Team,
Kindly awaiting a reply on this one, regards, Veena. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ |
![]() ![]() |
ilya.kasnacheev |
![]() |
In reply to this post by VeenaMithare
Hello! Looks like a network timeout, probably caused by firewall between two nodes, imparting their communication. You can try updating to 2.9 and enabling communication via discovery. Regards, -- Ilya Kasnacheev чт, 8 окт. 2020 г. в 18:17, VeenaMithare <[hidden email]>: Hi , |
![]() ![]() |
VeenaMithare |
![]() |
This post was updated on .
Hi Ilya,
The node communication issue is because one of the node is being restarted - and not due to network failure . The original issue is as below : Our setup : Servers - 3 node cluster Reader clients : wait for an update on an entry of a cache ( around 20 of them ) Writer Client : 1 If one of the reader client restarts( i.e. someone does a client deployment and restarts the node intentionally ) while the writer is writing into the entry of the cache , the server attempts to send the update to the failed client's local listener . It keeps attempting to communicate with the failed client ( client's continous query local listener ? ) till it timesout as per connTimeoutStrategy=ExponentialBackoffTimeoutStrategy . ( Please find the snippet of the exception below. The complete log is attached as an attachment ) This delays the completion of the transaction that was started by the writer client. Is there any way the writer client could complete the transaction without getting impacted by the reader client restarts ? regards, Veena. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ |
![]() ![]() |
ilya.kasnacheev |
![]() |
In reply to this post by ilya.kasnacheev
Hello! Do you have a reproducer for this behaviour that I could run and see it failing? Regards, -- Ilya Kasnacheev вт, 27 окт. 2020 г. в 22:02, VeenaMithare <[hidden email]>: Hi Ilya, The node communication issue is because one of the node is being restarted - and not due to network failure . The original issue is as below : Our setup : Servers - 3 node cluster Reader clients : wait for an update on an entry of a cache ( around 20 of them ) Writer Client : 1 If one of the reader client restarts while the writer is writing into the entry of the cache , the server attempts to send the update to the failed client's local listener . It keeps attempting to communicate with the failed client ( client's continous query local listener ? ) till it timesout as per connTimeoutStrategy=ExponentialBackoffTimeoutStrategy . ( Please find the snippet of the exception below. The complete log is attached as an attachment ) This delays the completion of the transaction that was started by the writer client. Is there any way the writer client could complete the transaction without getting impacted by the reader client restarts ? regards, Veena. |
![]() ![]() |
aealexsandrov |
![]() |
Hi, 10/30/2020 2:20 PM, Ilya Kasnacheev
пишет:
|
![]() ![]() |
VeenaMithare |
![]() |
Thank you for the suggestions.
I will try the timeout settings. It looks like the server has connected to the client on the new client id using the discovery SPI but it is trying to send the cache update to the old client id using the communication SPI. What impact does the IgniteAsyncCallback annotation on the remote filter have ? What are the pros and cons of having this annotation on the remote filter ? regards, Veena. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ |
Free forum by Nabble | Edit this page |