Apache Ignite Users

IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

classic Classic list List threaded Threaded
♦
♦
 7 messages Options
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
♦
♦
|

IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

Hi ,

Our setup :
Servers - 3 node cluster
Reader clients : wait for an update on an entry of a cache ( around 20 of
them )
Writer Client : 1

If one of the reader client restarts while the writer is writing into the
entry of the cache , the server attempts to send the update to the failed
client's local listener . It keeps attempting to communicate with the failed
client ( client's continous query local listener ? ) till it timesout as per
connTimeoutStrategy=ExponentialBackoffTimeoutStrategy . ( Please find the
snippet of the exception below. The complete log is attached as an
attachment ) This delays the completion of the transaction that was started
by the writer client.

Is there any way the writer client could complete the transaction without
getting impacted by the reader client restarts ?





2020-10-08 14:35:21,465 [sys-stripe-26-#27] WARN
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Handshake
timed out (will stop attempts to perform the handshake)
[node=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b,
connTimeoutStrategy=ExponentialBackoffTimeoutStrategy [maxTimeout=600000,
totalTimeout=30000, startNanos=223772180706400, currTimeout=600000],
err=Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=30000, startNanos=223772180706400,
currTimeout=600000]], addr=MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103,
failureDetectionTimeoutEnabled=true, timeout=0]
2020-10-08 14:35:21,465 [sys-stripe-26-#27] ERROR
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Failed to
send message to remote node [node=TcpDiscoveryNode
[id=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b,
consistentId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=ArrayList
[0:0:0:0:0:0:0:1, 1.2.3.4, 127.0.0.1], sockAddrs=HashSet
[MACHINENAME.COMPANY.LOCAL/1.2.3.4:0, /0:0:0:0:0:0:0:1:0, /127.0.0.1:0],
discPort=0, order=12, intOrder=8, lastExchangeTime=1602163619453, loc=false,
ver=2.8.1#20200521-sha1:86422096, isClient=true], msg=GridIoMessage [plc=2,
topic=T4 [topic=TOPIC_CACHE, id1=94370aa1-e970-37ae-9471-fd583d923522,
id2=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, id3=0], topicOrd=-1, ordered=true,
timeout=0, skipOnTimeout=true, msg=GridContinuousMessage
[type=MSG_EVT_NOTIFICATION, routineId=7dac3ff4-3460-4dc4-8324-a1ebe4561854,
data=null, futId=null]]]
org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node
still alive?). Make sure that each ComputeTask and cache Transaction has a
timeout set in order to prevent parties from waiting forever in case of
network issues [nodeId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b,
addrs=[/127.0.0.1:47103, /0:0:0:0:0:0:0:1:47103,
MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103]]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3698)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:2280)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1963)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1934)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1916)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1323)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1260)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1045)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1588)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:745)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:606)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:478)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:523)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1077)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.8.1.jar:2.8.1]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
        Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to
node (is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=[/127.0.0.1:47103,
/0:0:0:0:0:0:0:1:47103, MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3700)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:2280)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1963)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1934)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1916)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1323)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1260)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1045)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1588)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:745)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:606)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:478)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:523)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1077)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.8.1.jar:2.8.1]
                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
        Caused by: org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=30000, startNanos=223742176046300,
currTimeout=600000]]
                at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3582)
~[ignite-core-2.8.1.jar:2.8.1]
                ... 41 more
        Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to
node (is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=[/127.0.0.1:47103,
/0:0:0:0:0:0:0:1:47103, MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3700)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:2280)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1963)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1934)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1916)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1323)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1260)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1045)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1588)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:745)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:606)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:478)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:523)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1077)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.8.1.jar:2.8.1]
                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
        Caused by: org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=30000, startNanos=223772180706400,
currTimeout=600000]]
                at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3582)
~[ignite-core-2.8.1.jar:2.8.1]
                ... 41 more
Caused by: org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=30000, startNanos=223712146125000,
currTimeout=600000]]
        at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3582)
~[ignite-core-2.8.1.jar:2.8.1]
        ... 41 more
2020-10-08 14:35:21,467 [sys-stripe-26-#27] ERROR
org.apache.ignite.continuous.query [] - Failed to send event notification to
node: 3311a67b-bfcb-41af-8c09-b2e8f2fbde9b

EXPONENTIALBACKOFFSTRATEGY.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t2757/EXPONENTIALBACKOFFSTRATEGY.txt>  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

Hi Team,

Kindly awaiting a reply on this one,

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

In reply to this post by VeenaMithare
Hello!

Looks like a network timeout, probably caused by firewall between two nodes, imparting their communication.

You can try updating to 2.9 and enabling communication via discovery.

Regards,
--
Ilya Kasnacheev


чт, 8 окт. 2020 г. в 18:17, VeenaMithare <[hidden email]>:
Hi ,

Our setup :
Servers - 3 node cluster
Reader clients : wait for an update on an entry of a cache ( around 20 of
them )
Writer Client : 1

If one of the reader client restarts while the writer is writing into the
entry of the cache , the server attempts to send the update to the failed
client's local listener . It keeps attempting to communicate with the failed
client ( client's continous query local listener ? ) till it timesout as per
connTimeoutStrategy=ExponentialBackoffTimeoutStrategy . ( Please find the
snippet of the exception below. The complete log is attached as an
attachment ) This delays the completion of the transaction that was started
by the writer client.

Is there any way the writer client could complete the transaction without
getting impacted by the reader client restarts ?





2020-10-08 14:35:21,465 [sys-stripe-26-#27] WARN
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Handshake
timed out (will stop attempts to perform the handshake)
[node=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b,
connTimeoutStrategy=ExponentialBackoffTimeoutStrategy [maxTimeout=600000,
totalTimeout=30000, startNanos=223772180706400, currTimeout=600000],
err=Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=30000, startNanos=223772180706400,
currTimeout=600000]], addr=MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103,
failureDetectionTimeoutEnabled=true, timeout=0]
2020-10-08 14:35:21,465 [sys-stripe-26-#27] ERROR
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Failed to
send message to remote node [node=TcpDiscoveryNode
[id=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b,
consistentId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=ArrayList
[0:0:0:0:0:0:0:1, 1.2.3.4, 127.0.0.1], sockAddrs=HashSet
[MACHINENAME.COMPANY.LOCAL/1.2.3.4:0, /0:0:0:0:0:0:0:1:0, /127.0.0.1:0],
discPort=0, order=12, intOrder=8, lastExchangeTime=1602163619453, loc=false,
ver=2.8.1#20200521-sha1:86422096, isClient=true], msg=GridIoMessage [plc=2,
topic=T4 [topic=TOPIC_CACHE, id1=94370aa1-e970-37ae-9471-fd583d923522,
id2=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, id3=0], topicOrd=-1, ordered=true,
timeout=0, skipOnTimeout=true, msg=GridContinuousMessage
[type=MSG_EVT_NOTIFICATION, routineId=7dac3ff4-3460-4dc4-8324-a1ebe4561854,
data=null, futId=null]]]
org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node
still alive?). Make sure that each ComputeTask and cache Transaction has a
timeout set in order to prevent parties from waiting forever in case of
network issues [nodeId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b,
addrs=[/127.0.0.1:47103, /0:0:0:0:0:0:0:1:47103,
MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103]]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3698)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:2280)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1963)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1934)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1916)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1323)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1260)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1045)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1588)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:745)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:606)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:478)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:523)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1077)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.8.1.jar:2.8.1]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
        Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to
node (is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=[/127.0.0.1:47103,
/0:0:0:0:0:0:0:1:47103, MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3700)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:2280)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1963)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1934)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1916)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1323)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1260)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1045)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1588)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:745)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:606)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:478)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:523)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1077)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.8.1.jar:2.8.1]
                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
        Caused by: org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=30000, startNanos=223742176046300,
currTimeout=600000]]
                at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3582)
~[ignite-core-2.8.1.jar:2.8.1]
                ... 41 more
        Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to
node (is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=[/127.0.0.1:47103,
/0:0:0:0:0:0:0:1:47103, MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103]]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3700)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:2280)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1963)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1934)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1916)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1323)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1260)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1045)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1588)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:745)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:606)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:478)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:523)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1077)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.8.1.jar:2.8.1]
                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
        Caused by: org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=30000, startNanos=223772180706400,
currTimeout=600000]]
                at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
~[ignite-core-2.8.1.jar:2.8.1]
                at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3582)
~[ignite-core-2.8.1.jar:2.8.1]
                ... 41 more
Caused by: org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=30000, startNanos=223712146125000,
currTimeout=600000]]
        at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
~[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3582)
~[ignite-core-2.8.1.jar:2.8.1]
        ... 41 more
2020-10-08 14:35:21,467 [sys-stripe-26-#27] ERROR
org.apache.ignite.continuous.query [] - Failed to send event notification to
node: 3311a67b-bfcb-41af-8c09-b2e8f2fbde9b

EXPONENTIALBACKOFFSTRATEGY.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t2757/EXPONENTIALBACKOFFSTRATEGY.txt> 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

This post was updated on .
Hi Ilya,

The node communication issue is because one of the node is being restarted -
and not due to network failure . The original issue is as below :

Our setup :
Servers - 3 node cluster
Reader clients : wait for an update on an entry of a cache ( around 20 of
them )
Writer Client : 1

If one of the reader client restarts( i.e. someone does a client deployment and restarts the node intentionally ) while the writer is writing into the
entry of the cache , the server attempts to send the update to the failed
client's local listener . It keeps attempting to communicate with the failed
client ( client's continous query local listener ? ) till it timesout as per
connTimeoutStrategy=ExponentialBackoffTimeoutStrategy . ( Please find the
snippet of the exception below. The complete log is attached as an
attachment ) This delays the completion of the transaction that was started
by the writer client.

Is there any way the writer client could complete the transaction without
getting impacted by the reader client restarts ?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

In reply to this post by ilya.kasnacheev
Hello!

Do you have a reproducer for this behaviour that I could run and see it failing?

Regards,
--
Ilya Kasnacheev


вт, 27 окт. 2020 г. в 22:02, VeenaMithare <[hidden email]>:
Hi Ilya, The node communication issue is because one of the node is being restarted - and not due to network failure . The original issue is as below : Our setup : Servers - 3 node cluster Reader clients : wait for an update on an entry of a cache ( around 20 of them ) Writer Client : 1 If one of the reader client restarts while the writer is writing into the entry of the cache , the server attempts to send the update to the failed client's local listener . It keeps attempting to communicate with the failed client ( client's continous query local listener ? ) till it timesout as per connTimeoutStrategy=ExponentialBackoffTimeoutStrategy . ( Please find the snippet of the exception below. The complete log is attached as an attachment ) This delays the completion of the transaction that was started by the writer client. Is there any way the writer client could complete the transaction without getting impacted by the reader client restarts ? regards, Veena.
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

Hi,

Often, problems with establishing a communication connection can be solved with the following configuration:

1)You may have multiple network interfaces and the wrong one could be used. Solved by changing the SPI communication timeouts.:

<property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
          ...
          <property name="connectTimeout" value="5000"/>
          <property name="maxConnectTimeout" value="10000"/>
          ...
    </bean>
</property>

Otherwise, you can wait more than 10 minutes when trying to create a connection (due to the ExponentialBackoffTimeoutStrategy strategy).

2)Some operations in the cluster require communication with clients through communication. In case you have communication problems, but you can still access through the discovery SPI, such operations may hang. To avoid it please set the following property:

https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteSystemProperties.html#IGNITE_ENABLE_FORCIBLE_NODE_KILL

If these recommendations do not help, then yes, as Ilya said, we require a loudspeaker on your part.

BR,
Andrei

10/30/2020 2:20 PM, Ilya Kasnacheev пишет:
Hello!

Do you have a reproducer for this behaviour that I could run and see it failing?

Regards,
--
Ilya Kasnacheev


вт, 27 окт. 2020 г. в 22:02, VeenaMithare <[hidden email]>:
Hi Ilya, The node communication issue is because one of the node is being restarted - and not due to network failure . The original issue is as below : Our setup : Servers - 3 node cluster Reader clients : wait for an update on an entry of a cache ( around 20 of them ) Writer Client : 1 If one of the reader client restarts while the writer is writing into the entry of the cache , the server attempts to send the update to the failed client's local listener . It keeps attempting to communicate with the failed client ( client's continous query local listener ? ) till it timesout as per connTimeoutStrategy=ExponentialBackoffTimeoutStrategy . ( Please find the snippet of the exception below. The complete log is attached as an attachment ) This delays the completion of the transaction that was started by the writer client. Is there any way the writer client could complete the transaction without getting impacted by the reader client restarts ? regards, Veena.
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

Thank you for the suggestions.

I will try the timeout settings. It looks like the server has connected to
the client on the new client id using the discovery SPI but it is trying to
send the cache update to the old client id using the communication SPI.

What impact does the IgniteAsyncCallback annotation on the remote filter
have ? What are the pros and cons of having this annotation on the remote
filter ?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Free forum by Nabble Edit this page