java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

classic Classic list List threaded Threaded
9 messages Options
swara swara
Reply | Threaded
Open this post in threaded view
|

java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

This post was updated on .
Hi

Getting this error run time.
Ignite - 2.8.1

ERROR IgniteInMemoryDAO:119 - [EVENT FAILURE Anonymous:null@unknown ->
/DefaultName/com.inmemory.IgniteInMemoryDAO] Exception :
java.lang.IllegalStateException: Getting affinity for too old topology
version that is already out of history [locNode=TcpDiscoveryNode
[id=cfb6e618-4a12-43db-b14a-68efdea05a8c,
consistentId=cfb6e618-4a12-43db-b14a-68efdea05a8c, addrs=ArrayList [],
sockAddrs=HashSet [], discPort=0, order=78, intOrder=0,
lastExchangeTime=1601918903890, loc=true, ver=2.8.1#20200521-sha1:86422096,
isClient=true], grp=SQL_PUBLIC_COLL_1602059401160,
topVer=AffinityTopologyVersion [topVer=78, minorTopVer=254],
lastAffChangeTopVer=AffinityTopologyVersion [topVer=78, minorTopVer=254],
head=AffinityTopologyVersion [topVer=78, minorTopVer=255],
history=[AffinityTopologyVersion [topVer=78, minorTopVer=255]]]
        at
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:817)
        at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.assignment(GridCacheAffinityManager.java:255)
        at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.assignment(GridCacheAffinityManager.java:237)
        at
org.apache.ignite.internal.processors.query.h2.twostep.ReducePartitionMapper.stableDataNodesMap(ReducePartitionMapper.java:268)
        at
org.apache.ignite.internal.processors.query.h2.twostep.ReducePartitionMapper.stableDataNodes(ReducePartitionMapper.java:212)
       

Thank You
Swara Yerukola.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

Hello!

Looks like you have an SQL query which executes for ages and ran out of topology history.

That, or some other weird issue (you have minorTopVer of 250+ which is unusual). Can you provide logs for your server nodes?

Do you have any short-lived caches which are created and destroyed frequently? I advise against it, caches should be long-lived for optimal performance and decreased risk of running into issues like this one.

Regards,
--
Ilya Kasnacheev


пн, 12 окт. 2020 г. в 09:15, swara <[hidden email]>:
Hi

Getting this error run time.


ERROR IgniteInMemoryDAO:119 - [EVENT FAILURE Anonymous:null@unknown ->
/DefaultName/com.inmemory.IgniteInMemoryDAO] Exception :
java.lang.IllegalStateException: Getting affinity for too old topology
version that is already out of history [locNode=TcpDiscoveryNode
[id=cfb6e618-4a12-43db-b14a-68efdea05a8c,
consistentId=cfb6e618-4a12-43db-b14a-68efdea05a8c, addrs=ArrayList [],
sockAddrs=HashSet [], discPort=0, order=78, intOrder=0,
lastExchangeTime=1601918903890, loc=true, ver=2.8.1#20200521-sha1:86422096,
isClient=true], grp=SQL_PUBLIC_COLL_1602059401160,
topVer=AffinityTopologyVersion [topVer=78, minorTopVer=254],
lastAffChangeTopVer=AffinityTopologyVersion [topVer=78, minorTopVer=254],
head=AffinityTopologyVersion [topVer=78, minorTopVer=255],
history=[AffinityTopologyVersion [topVer=78, minorTopVer=255]]]
        at
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:817)
        at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.assignment(GridCacheAffinityManager.java:255)
        at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.assignment(GridCacheAffinityManager.java:237)
        at
org.apache.ignite.internal.processors.query.h2.twostep.ReducePartitionMapper.stableDataNodesMap(ReducePartitionMapper.java:268)
        at
org.apache.ignite.internal.processors.query.h2.twostep.ReducePartitionMapper.stableDataNodes(ReducePartitionMapper.java:212)


Thank You
Swara Yerukola.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
swara swara
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

This post was updated on .
we are creating caches for each table and using it for joining with other tables.
we are using ignite mainly for joining two tables whose data and definition will change frequently.
And we are creating once cache per table.

Thank You
Swara



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

Hello!

How many caches do you have? How often are they created?

Regards,
--
Ilya Kasnacheev


пн, 12 окт. 2020 г. в 13:35, swara <[hidden email]>:
We are creating caches for each table and joining tables.

Thank You
Swara



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
swara swara
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

one per table, whenever the table definition/data changes
hourly/daily/weekly.

please suggest if there is any other way to do joins without using cache?

Thank You
Swara



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

Hello!

How often do you create or drop tables? Minor version of 255 suggests a lot of operations.

Can you share logs for your nodes prior to that error?

Regards,
--
Ilya Kasnacheev


пн, 12 окт. 2020 г. в 16:04, swara <[hidden email]>:
one per table, whenever the table definition/data changes
hourly/daily/weekly.

please suggest if there is any other way to do joins without using cache?

Thank You
Swara



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
swara swara
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

Hello

Same error we got again today.

Ignite.log-

java.lang.IllegalStateException: Getting affinity for too old topology
version that is already out of history [locNode=TcpDiscoveryNode
[id=57473290-f181-49e5-9f5e-81188abc012f,
consistentId=57473290-f181-49e5-9f5e-81188abc012f, addrs=ArrayList [
127.0.0.1,], sockAddrs=HashSet [ /127.0.0.1:0,], discPort=0, order=2,
intOrder=0, lastExchangeTime=1602615417319, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=true],
grp=SQL_PUBLIC_COLL_1602660610976, topVer=AffinityTopologyVersion [topVer=2,
minorTopVer=129], lastAffChangeTopVer=AffinityTopologyVersion [topVer=2,
minorTopVer=129], head=AffinityTopologyVersion [topVer=2, minorTopVer=130],
history=[AffinityTopologyVersion [topVer=2, minorTopVer=130]]]
        at
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:817)
        at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.assignment(GridCacheAffinityManager.java:255)
        at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.assignment(GridCacheAffinityManager.java:237)


Server log-

[07:29:49,967][INFO][db-checkpoint-thread-#60][GridCacheDatabaseSharedManager]
Skipping checkpoint (no pages were modified)
[checkpointBeforeLockTime=126ms, checkpointLockWait=0ms,
checkpointListenersExecuteTime=118ms, checkpointLockHoldTime=130ms,
reason='timeout']
[07:30:10,991][INFO][exchange-worker-#46][time] Started exchange init
[topVer=AffinityTopologyVersion [topVer=2, minorTopVer=130], crd=true,
evt=DISCOVERY_CUSTOM_EVT, evtNode=57473290-f181-49e5-9f5e-81188abc012f,
customEvt=DynamicCacheChangeBatch
[id=057f3532571-1385aa3f-7385-4236-8361-193928f265ec, reqs=ArrayList
[DynamicCacheChangeRequest [cacheName=SQL_PUBLIC_COLL_1602660610976,
hasCfg=true, nodeId=57473290-f181-49e5-9f5e-81188abc012f,
clientStartOnly=false, stop=false, destroy=false, disabledAfterStartfalse]],
exchangeActions=ExchangeActions
[startCaches=[SQL_PUBLIC_COLL_1602660610976], stopCaches=null,
startGrps=[SQL_PUBLIC_COLL_1602660610976], stopGrps=[], resetParts=null,
stateChangeRequest=null], startCaches=false], allowMerge=false,
exchangeFreeSwitch=false]
[07:30:11,019][INFO][exchange-worker-#46][GridCacheProcessor] Started cache
[name=SQL_PUBLIC_COLL_1602660610976, id=238456958, dataRegionName=default,
mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false]
[07:30:11,042][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Finished waiting for partition release future
[topVer=AffinityTopologyVersion [topVer=2, minorTopVer=130], waitTime=0ms,
futInfo=NA, mode=DISTRIBUTED]
[07:30:11,042][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Finished waiting for partitions release latch: ServerLatch [permits=0,
pendingAcks=HashSet [], super=CompletableLatch [id=CompletableLatchUid
[id=exchange, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=130]]]]
[07:30:11,042][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Finished waiting for partition release future
[topVer=AffinityTopologyVersion [topVer=2, minorTopVer=130], waitTime=0ms,
futInfo=NA, mode=LOCAL]
[07:30:11,132][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], resVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130]]
[07:30:11,240][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Finish exchange future [startVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], resVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], err=null, rebalanced=true, wasRebalanced=true]
[07:30:11,257][INFO][exchange-worker-#46][GridCacheProcessor] Finish proxy
initialization, cacheName=SQL_PUBLIC_COLL_1602660610976,
localNodeId=bea04023-2512-4900-b9f0-ea363f7ef66f
[07:30:11,258][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Completed partition exchange
[localNode=bea04023-2512-4900-b9f0-ea363f7ef66f,
exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion
[topVer=2, minorTopVer=130], evt=DISCOVERY_CUSTOM_EVT,
evtNode=TcpDiscoveryNode [id=57473290-f181-49e5-9f5e-81188abc012f,
consistentId=57473290-f181-49e5-9f5e-81188abc012f, addrs=ArrayList [],
sockAddrs=HashSet [], discPort=0, order=2, intOrder=2,
lastExchangeTime=1602615417390, loc=false, ver=2.8.1#20200521-sha1:86422096,
isClient=true], done=true, newCrdFut=null], topVer=AffinityTopologyVersion
[topVer=2, minorTopVer=130]]
[07:30:11,258][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Exchange timings [startVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], resVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], stage="Waiting in exchange queue" (0 ms), stage="Exchange
parameters initialization" (0 ms), stage="Update caches registry" (1 ms),
stage="Start caches" (27 ms), stage="Affinity initialization on cache group
start" (0 ms), stage="Determine exchange type" (0 ms), stage="Preloading
notification" (0 ms), stage="WAL history reservation" (21 ms), stage="Wait
partitions release" (0 ms), stage="Wait partitions release latch" (0 ms),
stage="Wait partitions release" (0 ms), stage="Restore partition states" (4
ms), stage="After states restored callback" (85 ms), stage="Waiting for all
single messages" (0 ms), stage="Affinity recalculation (crd)" (0 ms),
stage="Collect update counters and create affinity messages" (0 ms),
stage="Validate partitions states" (8 ms), stage="Assign partitions states"
(23 ms), stage="Apply update counters" (26 ms), stage="Full message
preparing" (49 ms), stage="Full message sending" (0 ms), stage="Exchange
done" (17 ms), stage="Total time" (261 ms)]
[07:30:11,258][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Exchange longest local stages [startVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], resVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], stage="Affinity initialization on cache group start
[grp=SQL_PUBLIC_COLL_1602660610976]" (0 ms) (parent=Affinity initialization
on cache group start), stage="Restore partition states
[grp=SQL_PUBLIC_COLL_1602660610976]" (4 ms) (parent=Restore partition
states), stage="Restore partition states [grp=ignite-sys-cache]" (0 ms)
(parent=Restore partition states), stage="Restore partition states
[grp=SQL_PUBLIC_COLL_1602660408621]" (0 ms) (parent=Restore partition
states)]
[07:30:11,258][INFO][exchange-worker-#46][time] Finished exchange init
[topVer=AffinityTopologyVersion [topVer=2, minorTopVer=130], crd=true]
[07:30:11,308][INFO][exchange-worker-#46][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=2, minorTopVer=130], force=false, evt=DISCOVERY_CUSTOM_EVT,
node=57473290-f181-49e5-9f5e-81188abc012f]
[07:30:13,301][WARNING][grid-nio-worker-tcp-comm-0-#24][TcpCommunicationSpi]
Communication SPI session write timed out (consider increasing
'socketWriteTimeout' configuration property) [remoteAddr=/,
writeTimeout=2000]
[07:30:16,561][INFO][grid-nio-worker-tcp-comm-1-#25][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/, rmtAddr=/]
[07:30:16,562][INFO][grid-nio-worker-tcp-comm-1-#25][TcpCommunicationSpi]
Received incoming connection from remote node while connecting to this node,
rejecting [locNode=bea04023-2512-4900-b9f0-ea363f7ef66f, locNodeOrder=1,
rmtNode=57473290-f181-49e5-9f5e-81188abc012f, rmtNodeOrder=2]
[07:30:16,764][INFO][grid-nio-worker-tcp-comm-2-#26][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/, rmtAddr=/]
[07:30:16,764][INFO][grid-nio-worker-tcp-comm-2-#26][TcpCommunicationSpi]
Received incoming connection from remote node while connecting to this node,
rejecting [locNode=bea04023-2512-4900-b9f0-ea363f7ef66f, locNodeOrder=1,
rmtNode=57473290-f181-49e5-9f5e-81188abc012f, rmtNodeOrder=2]


Thank You
Swara



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
swara swara
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

Hello

We are using this type of cache design from last 3 years with 2.5.6 version.
we never got any issue.
When we upgraded to 2.8.1 we are getting this type of errors.
Should we downgrade to 2.8.0 or 2.5.6. Please suggest?

Thank You
Swara



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

Hello!

Why not, if it fixes the issue for you. Then you can try 2.9.0 once it is out.

Unfortunately, a small snippet of server log is not sufficient, complete log is needed. I can already spot some communication problems, though.

Regards,
--
Ilya Kasnacheev


чт, 15 окт. 2020 г. в 09:50, swara <[hidden email]>:
Hello

We are using this type of cache design from last 3 years with 2.5.6 version.
we never got any issue.
When we upgraded to 2.8.1 we are getting this type of errors.
Should we downgrade to 2.8.0 or 2.5.6. Please suggest?

Thank You
Swara



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/