Ignite Cluster got frozen/unreponsive

classic Classic list List threaded Threaded
2 messages Options
Kamlesh Joshi Kamlesh Joshi
Reply | Threaded
Open this post in threaded view
|

Ignite Cluster got frozen/unreponsive

Hi Igniters,

 

Our PROD cluster was frozen due to  deadlock as per below logs. Couldn't find further info on this, whether is it due to putall/removeall ? for which cache and keys or is it due to some other issue ?

We are using putall and removeall but not doing any ordering while using it.

 

Could you please check logs, tell us if you find any reason for this deadlock. If it is due to putall/removeall, how to get info for which cache and keys. Please find cluster configuration below,

 

######IgniteConfiguration###################

<bean class="org.apache.ignite.configuration.IgniteConfiguration">

        <property name="igniteInstanceName" value="${IGNITE_INSTANCE_NAME}" />

        <property name="peerClassLoadingEnabled" value="true"/>

        <property name="metricsLogFrequency" value="0" />

        <property name="failureDetectionTimeout" value="60000" />

        <property name="clientFailureDetectionTimeout" value="240000" />

        <property name="networkTimeout" value="60000" />

 

 

############### Cache Configuration ################

<property name="name" value="DeviceDetailsCache" />

                    <property name="cacheMode" value="PARTITIONED" />

                    <property name="atomicityMode" value="ATOMIC" />

                    <property name="backups" value="1" />

                    <property name="statisticsEnabled" value="true"/>

 

Below are logs,

 

[2020-10-17T16:12:17,158][INFO ][db-checkpoint-thread-#385%EDIFCustomerCC%][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=53afef47-ff6e-494d-a448-e35cc45b4eff, startPtr=FileWALPointer [idx=1928027, fileOff=3575743, len=63453], checkpointLockWait=0ms, checkpointLockHoldTime=33ms, walCpRecordFsyncDuration=5ms, pages=76968, reason='timeout']

[2020-10-17T16:12:17,224][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000006.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928026.wal]

[2020-10-17T16:12:18,318][INFO ][db-checkpoint-thread-#385%EDIFCustomerCC%][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=53afef47-ff6e-494d-a448-e35cc45b4eff, pages=76968, markPos=FileWALPointer [idx=1928027, fileOff=3575743, len=63453], walSegmentsCleared=5, walSegmentsCovered=[1928022 - 1928026], markDuration=67ms, pagesWrite=706ms, fsync=454ms, total=1227ms]

[2020-10-17T16:12:48,251][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=1928027, segIdx=7, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000007.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928027.wal]

[2020-10-17T16:12:48,355][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000007.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928027.wal]

[2020-10-17T16:13:27,146][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=1928028, segIdx=8, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000008.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928028.wal]

[2020-10-17T16:13:27,265][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000008.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928028.wal]

[2020-10-17T16:14:06,135][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=1928029, segIdx=9, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000009.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928029.wal]

[2020-10-17T16:14:06,250][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000009.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928029.wal]

[2020-10-17T16:15:07,128][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=1928030, segIdx=0, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000000.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928030.wal]

[2020-10-17T16:15:07,228][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000000.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928030.wal]

[2020-10-17T16:15:24,264][WARN ][grid-timeout-worker-#263%EDIFCustomerCC%][G] >>> Possible starvation in striped pool.

    Thread name: sys-stripe-40-#41%EDIFCustomerCC%

    Queue: [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252009083, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=236ba74c-3e77-43db-b06d-ae757784e7d6, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628085, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=608dea33571-1d1d3064-dfa1-4587-9131-439530617f71, miniId=808dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251655252, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=d31dfbf3-95a4-42c8-be02-99cc24d174bd, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901227431, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6b2fb754-9165-4d61-891f-5970e263c396, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[KeyCacheObjectImpl [part=40, val=null, hasValBytes=true]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=cb782ce0571-870075cf-b885-40be-a586-b862991e3798, miniId=db782ce0571-870075cf-b885-40be-a586-b862991e3798, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=fb38f867-4fb3-448b-99db-1f20ca492598, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252831927, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e5667fd1-0d1d-466a-aafd-3d4daafccb6b, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821044, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=2, arr=[142131082,142131083]]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=398dea33571-1d1d3064-dfa1-4587-9131-439530617f71, miniId=598dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628128, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442697, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251824023, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901227447, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6b2fb754-9165-4d61-891f-5970e263c396, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=d0c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, miniId=01c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=1ac4a008-e359-487e-bd60-8bc4d101085c, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628143, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=019dea33571-1d1d3064-dfa1-4587-9131-439530617f71, miniId=119dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250927127, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=0498547b-c945-4c65-908a-6ac0ddf4dff8, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=d04e1e33571-1f7c74a2-aff6-434b-a347-d88f70581bb8, miniId=e04e1e33571-1f7c74a2-aff6-434b-a347-d88f70581bb8, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=68fac742-6ea7-4bb3-9d27-6c626cd4a009, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602889898269, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602125222995, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=9afeae4c-ae1f-47a5-b6e6-032536db894e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250632521, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=41a03e2c-88fa-4ab1-9555-4774118b82a5, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901232835, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124902656, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=8fff54b3-f00a-4e4e-8421-b81c23b7919e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602137511901, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b00afce2-12e7-41b1-9147-6874d7bdca3e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252085756, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=2a3e1fb1-37cc-4c16-ace5-34f9d75e8034, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442733, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252434062, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a9252c27-1c4d-4f64-90cb-5b59a7f03eb9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=14061b33571-81fe5115-bee4-4ff8-8dc0-3b38d25cca03, miniId=24061b33571-81fe5115-bee4-4ff8-8dc0-3b38d25cca03, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=908a1300-c06b-4349-a94a-e2e4ca836cb0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251967622, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=921cd171-9236-4257-9bf0-81ffd57d8555, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=333c0b33571-bfe09cc5-6435-41be-a319-d4e6f5999644, miniId=433c0b33571-bfe09cc5-6435-41be-a319-d4e6f5999644, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=1e50ef76-cb94-4ffd-864d-29103600d567, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251239976, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=ac6bc567-29d3-40c2-8f62-b63f3e8edba6, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901308734, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6bbfc872-5022-48dc-b4a6-f8dabec35cf0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250879685, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=7da2bd3b-fb71-4d3f-a634-8fa9c6d60d14, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442873, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628186, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251824253, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901233022, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=34c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, miniId=64c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=1ac4a008-e359-487e-bd60-8bc4d101085c, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250414898, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=2a66a450-c90c-44e5-826b-ab0be7706e88, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=cab59a33571-7fc24c46-bc64-4e80-a4fc-6d6f02d8d8cd, miniId=dab59a33571-7fc24c46-bc64-4e80-a4fc-6d6f02d8d8cd, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=bff99e14-d0a6-488b-9e5b-7ab3b2190382, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821280, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251065338, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=11395929-053b-4a13-b03f-c8fc670eb8fa, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123820250, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=df729a44-6fd8-4dd1-b9bf-98d0aef3de75, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901376631, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a1377485-85fd-43ca-96b1-7b323e124662, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251643645, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=9cf24c88-db30-43c2-a8ed-2cf14c36f558, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123751037, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=d2d8e1bf-43a9-47b0-bc91-97a9fb0fa498, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123619208, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=beddfcbd-19e9-4a74-b87b-153ec35af598, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123751050, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=d2d8e1bf-43a9-47b0-bc91-97a9fb0fa498, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821399, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901287062, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821413, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821422, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901315682, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=0845ba78-0c30-4f97-952d-4c5b57ce8913, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251655371, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=d31dfbf3-95a4-42c8-be02-99cc24d174bd, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901303189, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=7251566e-d022-4c5e-af08-60feaa3fb9ee, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252832012, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e5667fd1-0d1d-466a-aafd-3d4daafccb6b, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251010602, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=f94e3dca-bf5c-44df-bb18-87fc3a9a8ca0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251967763, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=921cd171-9236-4257-9bf0-81ffd57d8555, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=276f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, miniId=676f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=425f44e2-2600-427b-a47a-1823af73692d, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=2, arr=[142346521,142346523]]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821584, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252443040, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602900084066, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=2e8b3c45-2790-471a-9aa4-2afcdf1f78b5, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602136587142, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=c2c02269-cec2-4f96-acdd-2a131a4b27fd, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251177802, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=db5b0c2d-48b5-444f-a59b-7850a2b6073c, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602899966390, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6c04a7b6-8b10-4329-b751-92912eaea781, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251968094, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=921cd171-9236-4257-9bf0-81ffd57d8555, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123464026, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=95dc0be1-d639-4920-a635-3b28075ce7e7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602900044341, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=8787e0aa-ae20-4c7d-bb5b-c8be99c88a3d, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250880196, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=7da2bd3b-fb71-4d3f-a634-8fa9c6d60d14, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901233568, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=b53d3e33571-701b81e6-a49b-44df-81c4-a063a7d1e7a6, miniId=c53d3e33571-701b81e6-a49b-44df-81c4-a063a7d1e7a6, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=22c72b15-5e2a-47eb-9949-1d5c7565ddfc, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=7d4a6243571-6fff9fa2-9496-41bd-bf6e-db2d52c4a2e4, miniId=8d4a6243571-6fff9fa2-9496-41bd-bf6e-db2d52c4a2e4, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e2dff823-0090-40bf-8aab-0a3b5bcf3611, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602900044373, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=8787e0aa-ae20-4c7d-bb5b-c8be99c88a3d, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=27337243571-cdc9b3fd-927b-4c38-b7ac-52840c0d42cc, miniId=57337243571-cdc9b3fd-927b-4c38-b7ac-52840c0d42cc, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=69e1b4e3-5082-4317-8ce0-19c956be89c0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252009770, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=236ba74c-3e77-43db-b06d-ae757784e7d6, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250415308, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=2a66a450-c90c-44e5-826b-ab0be7706e88, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901287470, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=3cdf3e33571-8b9a9b5e-83cf-469d-b424-4feba04aa2ed, miniId=5cdf3e33571-8b9a9b5e-83cf-469d-b424-4feba04aa2ed, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=95c546c4-3c34-433e-97a5-a24911be098a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=4f7f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, miniId=5f7f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=425f44e2-2600-427b-a47a-1823af73692d, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602899937408, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=f3c978a8-b2fd-443e-b1c0-9197c50521e3, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=1, arr=[142346525]]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901287655, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=9e2f3e33571-055b3332-1a13-40cf-8857-662219beebf6, miniId=be2f3e33571-055b3332-1a13-40cf-8857-662219beebf6, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=4e7366a0-6ba8-4fad-afae-b412601c99b1, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901309460, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6bbfc872-5022-48dc-b4a6-f8dabec35cf0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=0d2f1e33571-b536da48-556d-4c6b-be8f-817b8280fb82, miniId=3d2f1e33571-b536da48-556d-4c6b-be8f-817b8280fb82, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=ce6f4bab-f303-4d87-80a6-c03343400c8e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124011789, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=0b85712f-70b0-4d18-ae92-cf7955ef4464, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252005183, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=5a368e87-d5b1-4aa5-8fdd-cf2184f0a768, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]] Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250950411, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=0498547b-c945-4c65-908a-6ac0ddf4dff8, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251034391, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=f94e3dca-bf5c-44df-bb18-87fc3a9a8ca0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252175793, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3b156c84-9eba-4de0-9861-e4974e17fe1b, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124988181, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b33c781b-e83a-495c-9e72-5f9d4507a717, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251529166, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=9a3e3502-fa33-494b-a445-b6ec254e5d59, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124901688, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=31f69255-0587-49b8-bf74-8d9941967eb4, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251847453, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602889551081, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=bff99e14-d0a6-488b-9e5b-7ab3b2190382, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251845149, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123481727, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=95dc0be1-d639-4920-a635-3b28075ce7e7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=1085ba33571-e48b4a3d-7144-4e73-a0c6-f2c452b56cd7, miniId=3085ba33571-e48b4a3d-7144-4e73-a0c6-f2c452b56cd7, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=1769deb3-fd7b-4b36-9707-1808245f6fe9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251448830, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=44b5b67c-da49-468b-96af-e302b07b6a16, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251201103, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=db5b0c2d-48b5-444f-a59b-7850a2b6073c, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250596135, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=653fbfe7-dd96-4896-9b63-9b5fd3aec77f, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602125326244, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=c55ebb6d-4539-44e2-b14f-24a9bb10f077, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]]]

    Deadlock: true

    Completed: 58842256

Thread [name="sys-stripe-120-#121%EDIFCustomerCC%", id=153, state=WAITING, blockCnt=7120, waitCnt=58416728]

    Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@793496f8, ownerName=sys-stripe-40-#41%EDIFCustomerCC%, ownerId=73]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)

        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)

        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)

        at o.a.i.i.processors.cache.GridCacheMapEntry.lockEntry(GridCacheMapEntry.java:4967)

        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.lockEntries(GridDhtAtomicCache.java:2930)

        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1746)

        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)

        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3138)

       at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:135)

        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:271)

        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:266)

        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)

        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)

        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)

        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)

        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)

        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)

        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)

        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)

        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)

        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)

        at java.lang.Thread.run(Thread.java:748)

 

[2020-10-17T16:15:51,276][ERROR][disco-event-worker-#342%EDIFCustomerCC%][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=sys-stripe-120, blockedFor=77s]

[2020-10-17T16:15:51,276][WARN ][disco-event-worker-#342%EDIFCustomerCC%][G] Thread [name="sys-stripe-120-#121%EDIFCustomerCC%", id=153, state=WAITING, blockCnt=7120, waitCnt=58416728]

    Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@793496f8, ownerName=sys-stripe-40-#41%EDIFCustomerCC%, ownerId=73]

 

[2020-10-17T16:15:51,276][ERROR][disco-event-worker-#342%EDIFCustomerCC%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=sys-stripe-120, igniteInstanceName=EDIFCustomerCC, finished=false, heartbeatTs=1602931473886]]]

org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-120, igniteInstanceName=EDIFCustomerCC, finished=false, heartbeatTs=1602931473886]

                at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) [ignite-core-2.7.6.jar:2.7.6]

                at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) [ignite-core-2.7.6.jar:2.7.6]

                at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) [ignite-core-2.7.6.jar:2.7.6]

                at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) [ignite-core-2.7.6.jar:2.7.6]

                at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2827) [ignite-core-2.7.6.jar:2.7.6]

                at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.7.6.jar:2.7.6]

                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

[2020-10-17T16:15:51,278][WARN ][disco-event-worker-#342%EDIFCustomerCC%][FailureProcessor] Deadlocked threads detected (see thread dump below) [deadlockedThreadsCnt=2]

[2020-10-17T16:15:55,716][WARN ][jvm-pause-detector-worker][IgniteKernal%EDIFCustomerCC] Possible too long JVM pause: 4429 milliseconds.

[2020-10-17T16:15:55,760][WARN ][disco-event-worker-#342%EDIFCustomerCC%][FailureProcessor] Thread dump at 2020/10/17 16:15:55 IST

Thread [name="sys-#194498%EDIFCustomerCC%", id=382762, state=TIMED_WAITING, blockCnt=0, waitCnt=1]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@435ed2a3, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

Thread [name="sys-#194497%EDIFCustomerCC%", id=382761, state=TIMED_WAITING, blockCnt=0, waitCnt=1]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@435ed2a3, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

 

Thanks and Regards,

Kamlesh Joshi

 


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Cluster got frozen/unreponsive

Hello!

This looks like a transactional deadlock. Is this operation a part of transaction? Are you locking keys in the same order?

Regards,
--
Ilya Kasnacheev


вс, 18 окт. 2020 г. в 09:35, Kamlesh Joshi <[hidden email]>:

Hi Igniters,

 

Our PROD cluster was frozen due to  deadlock as per below logs. Couldn't find further info on this, whether is it due to putall/removeall ? for which cache and keys or is it due to some other issue ?

We are using putall and removeall but not doing any ordering while using it.

 

Could you please check logs, tell us if you find any reason for this deadlock. If it is due to putall/removeall, how to get info for which cache and keys. Please find cluster configuration below,

 

######IgniteConfiguration###################

<bean class="org.apache.ignite.configuration.IgniteConfiguration">

        <property name="igniteInstanceName" value="${IGNITE_INSTANCE_NAME}" />

        <property name="peerClassLoadingEnabled" value="true"/>

        <property name="metricsLogFrequency" value="0" />

        <property name="failureDetectionTimeout" value="60000" />

        <property name="clientFailureDetectionTimeout" value="240000" />

        <property name="networkTimeout" value="60000" />

 

 

############### Cache Configuration ################

<property name="name" value="DeviceDetailsCache" />

                    <property name="cacheMode" value="PARTITIONED" />

                    <property name="atomicityMode" value="ATOMIC" />

                    <property name="backups" value="1" />

                    <property name="statisticsEnabled" value="true"/>

 

Below are logs,

 

[2020-10-17T16:12:17,158][INFO ][db-checkpoint-thread-#385%EDIFCustomerCC%][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=53afef47-ff6e-494d-a448-e35cc45b4eff, startPtr=FileWALPointer [idx=1928027, fileOff=3575743, len=63453], checkpointLockWait=0ms, checkpointLockHoldTime=33ms, walCpRecordFsyncDuration=5ms, pages=76968, reason='timeout']

[2020-10-17T16:12:17,224][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000006.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928026.wal]

[2020-10-17T16:12:18,318][INFO ][db-checkpoint-thread-#385%EDIFCustomerCC%][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=53afef47-ff6e-494d-a448-e35cc45b4eff, pages=76968, markPos=FileWALPointer [idx=1928027, fileOff=3575743, len=63453], walSegmentsCleared=5, walSegmentsCovered=[1928022 - 1928026], markDuration=67ms, pagesWrite=706ms, fsync=454ms, total=1227ms]

[2020-10-17T16:12:48,251][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=1928027, segIdx=7, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000007.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928027.wal]

[2020-10-17T16:12:48,355][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000007.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928027.wal]

[2020-10-17T16:13:27,146][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=1928028, segIdx=8, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000008.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928028.wal]

[2020-10-17T16:13:27,265][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000008.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928028.wal]

[2020-10-17T16:14:06,135][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=1928029, segIdx=9, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000009.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928029.wal]

[2020-10-17T16:14:06,250][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000009.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928029.wal]

[2020-10-17T16:15:07,128][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=1928030, segIdx=0, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000000.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928030.wal]

[2020-10-17T16:15:07,228][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000000.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928030.wal]

[2020-10-17T16:15:24,264][WARN ][grid-timeout-worker-#263%EDIFCustomerCC%][G] >>> Possible starvation in striped pool.

    Thread name: sys-stripe-40-#41%EDIFCustomerCC%

    Queue: [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252009083, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=236ba74c-3e77-43db-b06d-ae757784e7d6, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628085, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=608dea33571-1d1d3064-dfa1-4587-9131-439530617f71, miniId=808dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251655252, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=d31dfbf3-95a4-42c8-be02-99cc24d174bd, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901227431, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6b2fb754-9165-4d61-891f-5970e263c396, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[KeyCacheObjectImpl [part=40, val=null, hasValBytes=true]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=cb782ce0571-870075cf-b885-40be-a586-b862991e3798, miniId=db782ce0571-870075cf-b885-40be-a586-b862991e3798, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=fb38f867-4fb3-448b-99db-1f20ca492598, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252831927, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e5667fd1-0d1d-466a-aafd-3d4daafccb6b, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821044, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=2, arr=[142131082,142131083]]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=398dea33571-1d1d3064-dfa1-4587-9131-439530617f71, miniId=598dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628128, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442697, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251824023, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901227447, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6b2fb754-9165-4d61-891f-5970e263c396, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=d0c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, miniId=01c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=1ac4a008-e359-487e-bd60-8bc4d101085c, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628143, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=019dea33571-1d1d3064-dfa1-4587-9131-439530617f71, miniId=119dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250927127, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=0498547b-c945-4c65-908a-6ac0ddf4dff8, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=d04e1e33571-1f7c74a2-aff6-434b-a347-d88f70581bb8, miniId=e04e1e33571-1f7c74a2-aff6-434b-a347-d88f70581bb8, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=68fac742-6ea7-4bb3-9d27-6c626cd4a009, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602889898269, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602125222995, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=9afeae4c-ae1f-47a5-b6e6-032536db894e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250632521, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=41a03e2c-88fa-4ab1-9555-4774118b82a5, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901232835, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124902656, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=8fff54b3-f00a-4e4e-8421-b81c23b7919e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602137511901, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b00afce2-12e7-41b1-9147-6874d7bdca3e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252085756, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=2a3e1fb1-37cc-4c16-ace5-34f9d75e8034, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442733, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252434062, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a9252c27-1c4d-4f64-90cb-5b59a7f03eb9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=14061b33571-81fe5115-bee4-4ff8-8dc0-3b38d25cca03, miniId=24061b33571-81fe5115-bee4-4ff8-8dc0-3b38d25cca03, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=908a1300-c06b-4349-a94a-e2e4ca836cb0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251967622, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=921cd171-9236-4257-9bf0-81ffd57d8555, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=333c0b33571-bfe09cc5-6435-41be-a319-d4e6f5999644, miniId=433c0b33571-bfe09cc5-6435-41be-a319-d4e6f5999644, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=1e50ef76-cb94-4ffd-864d-29103600d567, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251239976, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=ac6bc567-29d3-40c2-8f62-b63f3e8edba6, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901308734, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6bbfc872-5022-48dc-b4a6-f8dabec35cf0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250879685, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=7da2bd3b-fb71-4d3f-a634-8fa9c6d60d14, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442873, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628186, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251824253, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901233022, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=34c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, miniId=64c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=1ac4a008-e359-487e-bd60-8bc4d101085c, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250414898, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=2a66a450-c90c-44e5-826b-ab0be7706e88, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=cab59a33571-7fc24c46-bc64-4e80-a4fc-6d6f02d8d8cd, miniId=dab59a33571-7fc24c46-bc64-4e80-a4fc-6d6f02d8d8cd, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=bff99e14-d0a6-488b-9e5b-7ab3b2190382, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821280, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251065338, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=11395929-053b-4a13-b03f-c8fc670eb8fa, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123820250, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=df729a44-6fd8-4dd1-b9bf-98d0aef3de75, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901376631, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a1377485-85fd-43ca-96b1-7b323e124662, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251643645, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=9cf24c88-db30-43c2-a8ed-2cf14c36f558, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123751037, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=d2d8e1bf-43a9-47b0-bc91-97a9fb0fa498, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123619208, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=beddfcbd-19e9-4a74-b87b-153ec35af598, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123751050, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=d2d8e1bf-43a9-47b0-bc91-97a9fb0fa498, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821399, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901287062, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821413, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821422, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901315682, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=0845ba78-0c30-4f97-952d-4c5b57ce8913, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251655371, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=d31dfbf3-95a4-42c8-be02-99cc24d174bd, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901303189, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=7251566e-d022-4c5e-af08-60feaa3fb9ee, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252832012, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e5667fd1-0d1d-466a-aafd-3d4daafccb6b, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251010602, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=f94e3dca-bf5c-44df-bb18-87fc3a9a8ca0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251967763, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=921cd171-9236-4257-9bf0-81ffd57d8555, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=276f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, miniId=676f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=425f44e2-2600-427b-a47a-1823af73692d, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=2, arr=[142346521,142346523]]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821584, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252443040, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602900084066, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=2e8b3c45-2790-471a-9aa4-2afcdf1f78b5, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602136587142, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=c2c02269-cec2-4f96-acdd-2a131a4b27fd, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251177802, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=db5b0c2d-48b5-444f-a59b-7850a2b6073c, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602899966390, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6c04a7b6-8b10-4329-b751-92912eaea781, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251968094, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=921cd171-9236-4257-9bf0-81ffd57d8555, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123464026, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=95dc0be1-d639-4920-a635-3b28075ce7e7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602900044341, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=8787e0aa-ae20-4c7d-bb5b-c8be99c88a3d, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250880196, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=7da2bd3b-fb71-4d3f-a634-8fa9c6d60d14, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901233568, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=b53d3e33571-701b81e6-a49b-44df-81c4-a063a7d1e7a6, miniId=c53d3e33571-701b81e6-a49b-44df-81c4-a063a7d1e7a6, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=22c72b15-5e2a-47eb-9949-1d5c7565ddfc, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=7d4a6243571-6fff9fa2-9496-41bd-bf6e-db2d52c4a2e4, miniId=8d4a6243571-6fff9fa2-9496-41bd-bf6e-db2d52c4a2e4, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e2dff823-0090-40bf-8aab-0a3b5bcf3611, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602900044373, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=8787e0aa-ae20-4c7d-bb5b-c8be99c88a3d, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=27337243571-cdc9b3fd-927b-4c38-b7ac-52840c0d42cc, miniId=57337243571-cdc9b3fd-927b-4c38-b7ac-52840c0d42cc, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=69e1b4e3-5082-4317-8ce0-19c956be89c0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252009770, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=236ba74c-3e77-43db-b06d-ae757784e7d6, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250415308, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=2a66a450-c90c-44e5-826b-ab0be7706e88, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901287470, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=3cdf3e33571-8b9a9b5e-83cf-469d-b424-4feba04aa2ed, miniId=5cdf3e33571-8b9a9b5e-83cf-469d-b424-4feba04aa2ed, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=95c546c4-3c34-433e-97a5-a24911be098a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=4f7f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, miniId=5f7f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=425f44e2-2600-427b-a47a-1823af73692d, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602899937408, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=f3c978a8-b2fd-443e-b1c0-9197c50521e3, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=1, arr=[142346525]]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901287655, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=9e2f3e33571-055b3332-1a13-40cf-8857-662219beebf6, miniId=be2f3e33571-055b3332-1a13-40cf-8857-662219beebf6, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=4e7366a0-6ba8-4fad-afae-b412601c99b1, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901309460, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6bbfc872-5022-48dc-b4a6-f8dabec35cf0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=0d2f1e33571-b536da48-556d-4c6b-be8f-817b8280fb82, miniId=3d2f1e33571-b536da48-556d-4c6b-be8f-817b8280fb82, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=ce6f4bab-f303-4d87-80a6-c03343400c8e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124011789, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=0b85712f-70b0-4d18-ae92-cf7955ef4464, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252005183, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=5a368e87-d5b1-4aa5-8fdd-cf2184f0a768, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]] Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250950411, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=0498547b-c945-4c65-908a-6ac0ddf4dff8, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251034391, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=f94e3dca-bf5c-44df-bb18-87fc3a9a8ca0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252175793, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3b156c84-9eba-4de0-9861-e4974e17fe1b, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124988181, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b33c781b-e83a-495c-9e72-5f9d4507a717, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251529166, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=9a3e3502-fa33-494b-a445-b6ec254e5d59, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124901688, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=31f69255-0587-49b8-bf74-8d9941967eb4, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251847453, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602889551081, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=bff99e14-d0a6-488b-9e5b-7ab3b2190382, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251845149, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123481727, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=95dc0be1-d639-4920-a635-3b28075ce7e7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=1085ba33571-e48b4a3d-7144-4e73-a0c6-f2c452b56cd7, miniId=3085ba33571-e48b4a3d-7144-4e73-a0c6-f2c452b56cd7, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=1769deb3-fd7b-4b36-9707-1808245f6fe9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251448830, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=44b5b67c-da49-468b-96af-e302b07b6a16, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251201103, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=db5b0c2d-48b5-444f-a59b-7850a2b6073c, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250596135, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=653fbfe7-dd96-4896-9b63-9b5fd3aec77f, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602125326244, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=c55ebb6d-4539-44e2-b14f-24a9bb10f077, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]]]

    Deadlock: true

    Completed: 58842256

Thread [name="sys-stripe-120-#121%EDIFCustomerCC%", id=153, state=WAITING, blockCnt=7120, waitCnt=58416728]

    Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@793496f8, ownerName=sys-stripe-40-#41%EDIFCustomerCC%, ownerId=73]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)

        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)

        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)

        at o.a.i.i.processors.cache.GridCacheMapEntry.lockEntry(GridCacheMapEntry.java:4967)

        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.lockEntries(GridDhtAtomicCache.java:2930)

        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1746)

        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)

        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3138)

       at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:135)

        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:271)

        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:266)

        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)

        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)

        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)

        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)

        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)

        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)

        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)

        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)

        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)

        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)

        at java.lang.Thread.run(Thread.java:748)

 

[2020-10-17T16:15:51,276][ERROR][disco-event-worker-#342%EDIFCustomerCC%][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=sys-stripe-120, blockedFor=77s]

[2020-10-17T16:15:51,276][WARN ][disco-event-worker-#342%EDIFCustomerCC%][G] Thread [name="sys-stripe-120-#121%EDIFCustomerCC%", id=153, state=WAITING, blockCnt=7120, waitCnt=58416728]

    Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@793496f8, ownerName=sys-stripe-40-#41%EDIFCustomerCC%, ownerId=73]

 

[2020-10-17T16:15:51,276][ERROR][disco-event-worker-#342%EDIFCustomerCC%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=sys-stripe-120, igniteInstanceName=EDIFCustomerCC, finished=false, heartbeatTs=1602931473886]]]

org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-120, igniteInstanceName=EDIFCustomerCC, finished=false, heartbeatTs=1602931473886]

                at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) [ignite-core-2.7.6.jar:2.7.6]

                at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) [ignite-core-2.7.6.jar:2.7.6]

                at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) [ignite-core-2.7.6.jar:2.7.6]

                at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) [ignite-core-2.7.6.jar:2.7.6]

                at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2827) [ignite-core-2.7.6.jar:2.7.6]

                at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.7.6.jar:2.7.6]

                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

[2020-10-17T16:15:51,278][WARN ][disco-event-worker-#342%EDIFCustomerCC%][FailureProcessor] Deadlocked threads detected (see thread dump below) [deadlockedThreadsCnt=2]

[2020-10-17T16:15:55,716][WARN ][jvm-pause-detector-worker][IgniteKernal%EDIFCustomerCC] Possible too long JVM pause: 4429 milliseconds.

[2020-10-17T16:15:55,760][WARN ][disco-event-worker-#342%EDIFCustomerCC%][FailureProcessor] Thread dump at 2020/10/17 16:15:55 IST

Thread [name="sys-#194498%EDIFCustomerCC%", id=382762, state=TIMED_WAITING, blockCnt=0, waitCnt=1]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@435ed2a3, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

Thread [name="sys-#194497%EDIFCustomerCC%", id=382761, state=TIMED_WAITING, blockCnt=0, waitCnt=1]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@435ed2a3, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

 

Thanks and Regards,

Kamlesh Joshi

 


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."