Blocked Worker Thread

classic Classic list List threaded Threaded
4 messages Options
Conrad Mukai (cmukai) Conrad Mukai (cmukai)
Reply | Threaded
Open this post in threaded view
|

Blocked Worker Thread

We are running a cache in a 4 node cluster with atomicityMode set to ATOMIC and have persistence enabled. We repeatedly get a SYSTEM_WORKER_BLOCKED error on one node which is disabling the entire cluster. We were seeing a lot of sockets in TIME_WAIT state which was blocking clients from connecting so we did the following on all the nodes:

 

# ignore TIME_WAIT state on sockets
echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse
echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle

 

This made that issue go away, but may play a part in this new issue. First question is what is the root cause of the error? The second question is why does this bring down the entire cluster?

 

Here is the error message:

 

[2019-11-07 16:36:22,037][ERROR][tcp-disco-msg-worker-#2][root] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=partition-exchanger, igniteInstanceName=null, finished=false, heartbeatTs=1573090509428]]]

class org.apache.ignite.IgniteException: GridWorker [name=partition-exchanger, igniteInstanceName=null, finished=false, heartbeatTs=1573090509428]

        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)

        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)

        at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)

        at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)

        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)

        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

 

This is followed by a warning and a thread dump:

 

[2019-11-07 16:36:22,038][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] No deadlocked threads detected.

[2019-11-07 16:36:22,328][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] Thread dump at 2019/11/07 16:36:22 GMT

 

For the particular thread in the error and warning messages here is the thread dump:

 

Thread [name="tcp-disco-msg-worker-#2", id=113, state=RUNNABLE, blockCnt=211, waitCnt=4745368]

        at sun.management.ThreadImpl.dumpThreads0(Native Method)

        at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)

        at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1368)

        at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:128)

        - locked o.a.i.i.processors.failure.FailureProcessor@7e65ceba

        at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:104)

        at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1829)

        at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)

        at o.a.i.i.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)

        at o.a.i.i.util.worker.GridWorker.onIdle(GridWorker.java:297)

        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)

        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker$$Lambda$47/1047515321.run(Unknown Source)

        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)

        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)

        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)

        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)

        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

 

In addition all the system threads are in TIMED_WAITING state:

 

Thread [name="sys-#7099", id=9252, state=TIMED_WAITING, blockCnt=0, waitCnt=1]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

and all the query threads are in WAITING state:

 

Thread [name="query-#1062", id=3027, state=WAITING, blockCnt=11, waitCnt=3365]

    Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)

        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)

        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)

        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)

        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)

        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)

        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)

        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)

        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)

        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)

        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)

        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)

       at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)

        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)

        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)

        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)

        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)

        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)

        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)

        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)

        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)

        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

    Locked synchronizers:

        java.util.concurrent.ThreadPoolExecutor$Worker@c1349cf

        [hidden email]

 

Many other threads are in WAITING state, in particular this one may be of interest:

 

Thread [name="srvc-deploy-#91", id=164, state=WAITING, blockCnt=0, waitCnt=5182]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7310e7da, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

As well as all the checkpoint-runner threads:

 

Thread [name="checkpoint-runner-#82", id=152, state=WAITING, blockCnt=2, waitCnt=3282307]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c8ae71a, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

and the db-checkpoint-thread:

 

Thread [name="db-checkpoint-thread-#78", id=148, state=WAITING, blockCnt=297, waitCnt=1337]

    Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)

       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)

        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)

        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:3690)

        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3300)

        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3199)

        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)

        at java.lang.Thread.run(Thread.java:748)

 

Thanks in advance,

Conrad Mukai

 

aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Blocked Worker Thread

Hi Conrad,

The reasons can be different. Could you please share the logs?

BR,
Andrei

11/7/2019 10:35 PM, Conrad Mukai (cmukai) пишет:

We are running a cache in a 4 node cluster with atomicityMode set to ATOMIC and have persistence enabled. We repeatedly get a SYSTEM_WORKER_BLOCKED error on one node which is disabling the entire cluster. We were seeing a lot of sockets in TIME_WAIT state which was blocking clients from connecting so we did the following on all the nodes:

 

# ignore TIME_WAIT state on sockets
echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse
echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle

 

This made that issue go away, but may play a part in this new issue. First question is what is the root cause of the error? The second question is why does this bring down the entire cluster?

 

Here is the error message:

 

[2019-11-07 16:36:22,037][ERROR][tcp-disco-msg-worker-#2][root] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=partition-exchanger, igniteInstanceName=null, finished=false, heartbeatTs=1573090509428]]]

class org.apache.ignite.IgniteException: GridWorker [name=partition-exchanger, igniteInstanceName=null, finished=false, heartbeatTs=1573090509428]

        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)

        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)

        at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)

        at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)

        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)

        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

 

This is followed by a warning and a thread dump:

 

[2019-11-07 16:36:22,038][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] No deadlocked threads detected.

[2019-11-07 16:36:22,328][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] Thread dump at 2019/11/07 16:36:22 GMT

 

For the particular thread in the error and warning messages here is the thread dump:

 

Thread [name="tcp-disco-msg-worker-#2", id=113, state=RUNNABLE, blockCnt=211, waitCnt=4745368]

        at sun.management.ThreadImpl.dumpThreads0(Native Method)

        at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)

        at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1368)

        at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:128)

        - locked o.a.i.i.processors.failure.FailureProcessor@7e65ceba

        at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:104)

        at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1829)

        at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)

        at o.a.i.i.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)

        at o.a.i.i.util.worker.GridWorker.onIdle(GridWorker.java:297)

        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)

        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker$$Lambda$47/1047515321.run(Unknown Source)

        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)

        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)

        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)

        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)

        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

 

In addition all the system threads are in TIMED_WAITING state:

 

Thread [name="sys-#7099", id=9252, state=TIMED_WAITING, blockCnt=0, waitCnt=1]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

and all the query threads are in WAITING state:

 

Thread [name="query-#1062", id=3027, state=WAITING, blockCnt=11, waitCnt=3365]

    Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)

        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)

        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)

        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)

        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)

        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)

        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)

        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)

        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)

        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)

        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)

        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)

       at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)

        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)

        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)

        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)

        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)

        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)

        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)

        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)

        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)

        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

    Locked synchronizers:

        java.util.concurrent.ThreadPoolExecutor$Worker@c1349cf

        [hidden email]

 

Many other threads are in WAITING state, in particular this one may be of interest:

 

Thread [name="srvc-deploy-#91", id=164, state=WAITING, blockCnt=0, waitCnt=5182]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7310e7da, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

As well as all the checkpoint-runner threads:

 

Thread [name="checkpoint-runner-#82", id=152, state=WAITING, blockCnt=2, waitCnt=3282307]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c8ae71a, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

and the db-checkpoint-thread:

 

Thread [name="db-checkpoint-thread-#78", id=148, state=WAITING, blockCnt=297, waitCnt=1337]

    Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)

       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)

        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)

        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:3690)

        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3300)

        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3199)

        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)

        at java.lang.Thread.run(Thread.java:748)

 

Thanks in advance,

Conrad Mukai

 

Conrad Mukai (cmukai) Conrad Mukai (cmukai)
Reply | Threaded
Open this post in threaded view
|

Re: Blocked Worker Thread

Here is the error and warnings with the thread dump:

 

[2019-11-07 16:36:22,037][ERROR][tcp-disco-msg-worker-#2][root] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=partition-exchanger, igniteInstanceName=null, finished=false, heartbeatTs=1573090509428]]]
class
org.apache.ignite.IgniteException: GridWorker [name=partition-exchanger, igniteInstanceName=null, finished=false, heartbeatTs=1573090509428]
   at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
   at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
   at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
   at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
   at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
   at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
   at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
   at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
   at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
   at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[2019-11-07 16:36:22,038][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] No deadlocked threads detected.
[
2019-11-07 16:36:22,328][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] Thread dump at 2019/11/07 16:36:22 GMT
Thread [name=
"sys-#7099", id=9252, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#7098", id=9251, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#7097", id=9250, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#7096", id=9249, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#7095", id=9248, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="query-#1078", id=3051, state=WAITING, blockCnt=7, waitCnt=3190]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@2d4a8835
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@4166bfee
Thread [name=
"query-#1077", id=3050, state=WAITING, blockCnt=15, waitCnt=3235]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@c0a3312
        java.util.concurrent.ThreadPoolExecutor
$Worker@735561c9
Thread [name=
"query-#1076", id=3049, state=WAITING, blockCnt=8, waitCnt=3309]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
       at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1141)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@19007848
        java.util.concurrent.locks.ReentrantLock
$NonfairSync@714ddb67
Thread [name=
"query-#1075", id=3048, state=WAITING, blockCnt=10, waitCnt=3246]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@20f0fee8
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@5a0836f
Thread [name="query-#1074", id=3047, state=WAITING, blockCnt=9, waitCnt=3282]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@12b78a6d
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@55608d7f
Thread [name=
"query-#1073", id=3046, state=WAITING, blockCnt=12, waitCnt=3273]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@4297fe42
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@8afa662
Thread [name="query-#1072", id=3045, state=WAITING, blockCnt=5, waitCnt=3315]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
       at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@50eeac6
        java.util.concurrent.locks.ReentrantLock
$NonfairSync@1726c2f3
Thread [name=
"query-#1071", id=3044, state=WAITING, blockCnt=11, waitCnt=3374]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@28d59ac2
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@169c9083
Thread [name=
"query-#1070", id=3043, state=WAITING, blockCnt=9, waitCnt=3259]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@31dc0609
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@4746fc5a
Thread [name=
"query-#1069", id=3042, state=WAITING, blockCnt=13, waitCnt=3312]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@3668b5e8
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@322fde66
Thread [name=
"query-#1068", id=3041, state=WAITING, blockCnt=9, waitCnt=3482]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@6d8c916e
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@13bfa38a
Thread [name=
"query-#1067", id=3040, state=WAITING, blockCnt=12, waitCnt=3238]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@2ba1cf40
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@61d531f6
Thread [name=
"query-#1066", id=3039, state=WAITING, blockCnt=7, waitCnt=3294]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@76a8189
        java.util.concurrent.locks.ReentrantLock
$NonfairSync@1101b5ba
Thread [name=
"query-#1065", id=3038, state=WAITING, blockCnt=10, waitCnt=3316]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@9ee918
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@5d118512
Thread [name=
"query-#1064", id=3037, state=WAITING, blockCnt=3, waitCnt=3267]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@3d0f7d55
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@335ccec9
Thread [name=
"query-#1062", id=3027, state=WAITING, blockCnt=11, waitCnt=3365]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@c1349cf
        java.util.concurrent.locks.ReentrantLock
$NonfairSync@7bce13d5
Thread [name=
"srvc-deploy-#91", id=164, state=WAITING, blockCnt=0, waitCnt=5182]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7310e7da, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="checkpoint-runner-#82", id=152, state=WAITING, blockCnt=2, waitCnt=3282307]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c8ae71a, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="checkpoint-runner-#81", id=151, state=WAITING, blockCnt=7, waitCnt=3283481]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c8ae71a, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="checkpoint-runner-#80", id=150, state=WAITING, blockCnt=8, waitCnt=3282180]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c8ae71a, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="checkpoint-runner-#79", id=149, state=WAITING, blockCnt=10, waitCnt=3282388]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c8ae71a, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="db-checkpoint-thread-#78", id=148, state=WAITING, blockCnt=297, waitCnt=1337]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:3690)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3300)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3199)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="ttl-cleanup-worker-#77", id=147, state=WAITING, blockCnt=11117, waitCnt=2816036]
    Lock [object=
java.util.concurrent.locks.ReentrantLock$NonfairSync@335ccec9, ownerName=query-#1064, ownerId=3037]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
        at o.a.i.i.processors.cache.GridCacheMapEntry.lockEntry(GridCacheMapEntry.java:4967)
        at o.a.i.i.processors.cache.GridCacheMapEntry.obsolete(GridCacheMapEntry.java:2874)
        at o.a.i.i.processors.cache.GridCacheConcurrentMapImpl.putEntryIfObsoleteOrAbsent(GridCacheConcurrentMapImpl.java:124)
        at o.a.i.i.processors.cache.GridCacheConcurrentMapImpl.putEntryIfObsoleteOrAbsent(GridCacheConcurrentMapImpl.java:69)
        at o.a.i.i.processors.cache.distributed.dht.GridCachePartitionedConcurrentMap.putEntryIfObsoleteOrAbsent(GridCachePartitionedConcurrentMap.java:94)
        at o.a.i.i.processors.cache.GridCacheAdapter.entryEx(GridCacheAdapter.java:1008)
        at o.a.i.i.processors.cache.distributed.dht.GridDhtCacheAdapter.entryEx(GridDhtCacheAdapter.java:544)
        at o.a.i.i.processors.cache.GridCacheAdapter.entryEx(GridCacheAdapter.java:999)
        at o.a.i.i.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpiredInternal(GridCacheOffheapManager.java:2224)
        at o.a.i.i.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpired(GridCacheOffheapManager.java:2157)
        at o.a.i.i.processors.cache.persistence.GridCacheOffheapManager.expire(GridCacheOffheapManager.java:845)
        at o.a.i.i.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:207)
        at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:139)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="wal-segment-syncer-#74", id=143, state=TIMED_WAITING, blockCnt=0, waitCnt=103412]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7778)
        at o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$WalSegmentSyncer.body(FileWriteAheadLogManager.java:3518)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="tcp-disco-sock-reader-#12", id=128, state=RUNNABLE, blockCnt=48, waitCnt=6]
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked java.io.BufferedInputStream@32ce513d
        at o.a.i.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
        at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2663)
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2679)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
        at o.a.i.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:43)
        at o.a.i.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:137)
        at o.a.i.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
        at o.a.i.i.util.IgniteUtils.unmarshal(IgniteUtils.java:10113)
        at o.a.i.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:6246)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="session-timeout-worker-#54", id=83, state=TIMED_WAITING, blockCnt=0, waitCnt=57215]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.processors.rest.GridRestProcessor$4.body(GridRestProcessor.java:495)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="exchange-worker-#63", id=121, state=WAITING, blockCnt=202, waitCnt=30351]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initTopologies(GridDhtPartitionsExchangeFuture.java:966)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:824)
        at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2681)
        at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2553)
       at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="ignite-update-notifier-timer", id=120, state=TIMED_WAITING, blockCnt=1, waitCnt=35]
    Lock [object=
java.util.TaskQueue@39f7d249, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        at java.util.TimerThread.run(Timer.java:505)

Thread [name="upd-ver-checker", id=119, state=TIMED_WAITING, blockCnt=0, waitCnt=11688]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.processors.cluster.GridUpdateNotifier$1.run(GridUpdateNotifier.java:115)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="disco-event-worker-#62", id=118, state=WAITING, blockCnt=201, waitCnt=31392]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64c4c081, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2856)
        at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2825)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="tcp-disco-srvr-#3", id=114, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5845)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at o.a.i.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5763)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-msg-worker-#2", id=113, state=RUNNABLE, blockCnt=211, waitCnt=4745368]
        at sun.management.ThreadImpl.dumpThreads0(Native Method)
        at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
        at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1368)
        at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:128)
        - locked o.a.i.i.processors.failure.FailureProcessor@7e65ceba
        at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:104)
        at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1829)
        at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
        at o.a.i.i.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
        at o.a.i.i.util.worker.GridWorker.onIdle(GridWorker.java:297)
        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker$$Lambda$47/1047515321.run(Unknown Source)
        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="disco-notifier-worker-#61", id=112, state=WAITING, blockCnt=0, waitCnt=109690]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40336649, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2660)
        at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2705)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-data-loader-flusher-#60", id=110, state=WAITING, blockCnt=0, waitCnt=1]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4751919c, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-109", id=109, state=TIMED_WAITING, blockCnt=0, waitCnt=1289]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-108", id=108, state=TIMED_WAITING, blockCnt=0, waitCnt=1290]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-107", id=107, state=TIMED_WAITING, blockCnt=0, waitCnt=1272]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-106", id=106, state=TIMED_WAITING, blockCnt=0, waitCnt=1289]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-105", id=105, state=TIMED_WAITING, blockCnt=0, waitCnt=1283]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-104", id=104, state=TIMED_WAITING, blockCnt=0, waitCnt=1261]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-103", id=103, state=TIMED_WAITING, blockCnt=0, waitCnt=1257]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-102", id=102, state=TIMED_WAITING, blockCnt=0, waitCnt=1318]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-101", id=101, state=TIMED_WAITING, blockCnt=0, waitCnt=1279]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-100", id=100, state=TIMED_WAITING, blockCnt=0, waitCnt=1299]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-99-acceptor-1@7012ca9f-ServerConnector@7ed52aed{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}", id=99, state=RUNNABLE, blockCnt=0, waitCnt=1]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        - locked java.lang.Object@2185a38e
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-98-acceptor-0@2b02a2d4-ServerConnector@7ed52aed{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}", id=98, state=BLOCKED, blockCnt=1, waitCnt=2]
    Lock [object=
java.lang.Object@2185a38e, ownerName=qtp1710783020-99-acceptor-1@7012ca9f-ServerConnector@7ed52aed{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}, ownerId=99]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-97", id=97, state=RUNNABLE, blockCnt=3, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@7a5f98d9
        - locked
java.util.Collections$UnmodifiableSet@31493b7f
        - locked
sun.nio.ch.EPollSelectorImpl@288a1e00
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-96", id=96, state=RUNNABLE, blockCnt=4, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@249f514c
        - locked
java.util.Collections$UnmodifiableSet@22808c56
        - locked
sun.nio.ch.EPollSelectorImpl@5aead9ba
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-95", id=95, state=RUNNABLE, blockCnt=3, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@22e8a6f9
        - locked
java.util.Collections$UnmodifiableSet@7ca396d9
        - locked
sun.nio.ch.EPollSelectorImpl@5a675727
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-94", id=94, state=RUNNABLE, blockCnt=3, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@56cc62d8
        - locked
java.util.Collections$UnmodifiableSet@31f4d996
        - locked
sun.nio.ch.EPollSelectorImpl@5101842f
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-93", id=93, state=RUNNABLE, blockCnt=3, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@313f3f47
        - locked
java.util.Collections$UnmodifiableSet@4d0511a
        - locked sun.nio.ch.EPollSelectorImpl@13ff107f
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-92", id=92, state=RUNNABLE, blockCnt=5, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@4e82ded3
        - locked
java.util.Collections$UnmodifiableSet@60f0543
        - locked sun.nio.ch.EPollSelectorImpl@51cc19
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-91", id=91, state=RUNNABLE, blockCnt=4, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@6b464884
        - locked
java.util.Collections$UnmodifiableSet@33ea6779
        - locked
sun.nio.ch.EPollSelectorImpl@4d31595a
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-90", id=90, state=RUNNABLE, blockCnt=1, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@430e75e1
        - locked
java.util.Collections$UnmodifiableSet@1ad5076c
        - locked
sun.nio.ch.EPollSelectorImpl@7a647ae0
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-rest-3-#58", id=87, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@3dd8077f
        - locked
java.util.Collections$UnmodifiableSet@dc9dc58
        - locked sun.nio.ch.EPollSelectorImpl@3f9544ea
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-rest-2-#57", id=86, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@667f8a82
        - locked
java.util.Collections$UnmodifiableSet@1b619acd
        - locked
sun.nio.ch.EPollSelectorImpl@5889bee5
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-rest-1-#56", id=85, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@4dae130a
        - locked
java.util.Collections$UnmodifiableSet@571fbba7
        - locked
sun.nio.ch.EPollSelectorImpl@403ff47b
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-rest-0-#55", id=84, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@5554040f
        - locked
java.util.Collections$UnmodifiableSet@68df6bec
        - locked
sun.nio.ch.EPollSelectorImpl@42b727d3
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-tcp-rest-#59", id=88, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@7d148fd2
        - locked
java.util.Collections$UnmodifiableSet@8159803
        - locked sun.nio.ch.EPollSelectorImpl@1b0602e6
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2926)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2874)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-3-#52", id=81, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@60f437ea
        - locked
java.util.Collections$UnmodifiableSet@39c74e39
        - locked
sun.nio.ch.EPollSelectorImpl@1d361c82
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-2-#51", id=80, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@360c3cf0
        - locked
java.util.Collections$UnmodifiableSet@778616bc
        - locked
sun.nio.ch.EPollSelectorImpl@743338f3
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-1-#50", id=79, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@594cf8fb
        - locked
java.util.Collections$UnmodifiableSet@ef0a49e
        - locked sun.nio.ch.EPollSelectorImpl@4904600a
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-0-#49", id=78, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@249d2d79
        - locked
java.util.Collections$UnmodifiableSet@1a289231
        - locked
sun.nio.ch.EPollSelectorImpl@117701be
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-client-listener-#53", id=82, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@7b9937aa
        - locked
java.util.Collections$UnmodifiableSet@36223b45
        - locked
sun.nio.ch.EPollSelectorImpl@68458eaa
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2926)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2874)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="tcp-comm-worker-#1", id=77, state=TIMED_WAITING, blockCnt=3, waitCnt=112]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5bcf0aa0, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4287)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at o.a.i.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="grid-nio-worker-tcp-comm-7-#47", id=72, state=RUNNABLE, blockCnt=19478, waitCnt=2]
       at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@54dfa634
        - locked
java.util.Collections$UnmodifiableSet@10ab42db
        - locked
sun.nio.ch.EPollSelectorImpl@610bf143
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-6-#46", id=71, state=RUNNABLE, blockCnt=8814, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@10e226b3
        - locked
java.util.Collections$UnmodifiableSet@39a5dc9d
        - locked
sun.nio.ch.EPollSelectorImpl@7352fce8
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-5-#45", id=70, state=RUNNABLE, blockCnt=7958, waitCnt=4]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@4ded9a28
        - locked
java.util.Collections$UnmodifiableSet@4cbbfa98
        - locked
sun.nio.ch.EPollSelectorImpl@5bed96a6
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-4-#44", id=69, state=RUNNABLE, blockCnt=16692, waitCnt=5]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@3fbc848f
        - locked
java.util.Collections$UnmodifiableSet@1ea1658a
        - locked
sun.nio.ch.EPollSelectorImpl@ddd33fc
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-3-#43", id=68, state=RUNNABLE, blockCnt=9365, waitCnt=2]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@3a32cc47
        - locked
java.util.Collections$UnmodifiableSet@6171bf72
        - locked
sun.nio.ch.EPollSelectorImpl@74e90be3
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-2-#42", id=67, state=RUNNABLE, blockCnt=16342, waitCnt=3]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@104f9451
        - locked
java.util.Collections$UnmodifiableSet@3c3322eb
        - locked
sun.nio.ch.EPollSelectorImpl@50559030
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-1-#41", id=66, state=RUNNABLE, blockCnt=11611, waitCnt=3]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@6cdf3fca
        - locked
java.util.Collections$UnmodifiableSet@dec31c
        - locked
sun.nio.ch.EPollSelectorImpl@25df8d96
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-0-#40", id=65, state=RUNNABLE, blockCnt=11225, waitCnt=3]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@59f75d0d
        - locked
java.util.Collections$UnmodifiableSet@4c431896
        - locked
sun.nio.ch.EPollSelectorImpl@3af3e421
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-tcp-comm-#48", id=76, state=RUNNABLE, blockCnt=28, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@63197dd4
        - locked
java.util.Collections$UnmodifiableSet@e54b8f
        - locked
sun.nio.ch.EPollSelectorImpl@54b664ad
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2926)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2874)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-timeout-worker-#39", id=64, state=TIMED_WAITING, blockCnt=260, waitCnt=532667]
    Lock [object=
java.lang.Object@714140f6, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:269)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="ignite-clock", id=63, state=TIMED_WAITING, blockCnt=0, waitCnt=4724639]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.util.IgniteUtils$22.run(IgniteUtils.java:3383)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="jvm-pause-detector-worker", id=56, state=TIMED_WAITING, blockCnt=0, waitCnt=967819]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:100)

Thread [name="data-streamer-stripe-15-#32", id=55, state=WAITING, blockCnt=4149, waitCnt=2121970]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-14-#31", id=54, state=WAITING, blockCnt=4140, waitCnt=2130293]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-13-#30", id=53, state=WAITING, blockCnt=4178, waitCnt=2131313]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-12-#29", id=52, state=WAITING, blockCnt=3964, waitCnt=2133028]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-11-#28", id=51, state=WAITING, blockCnt=4342, waitCnt=2144768]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-10-#27", id=50, state=WAITING, blockCnt=4860, waitCnt=2162164]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-9-#26", id=49, state=WAITING, blockCnt=4268, waitCnt=2150445]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-8-#25", id=48, state=WAITING, blockCnt=4301, waitCnt=2143068]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-7-#24", id=47, state=WAITING, blockCnt=4669, waitCnt=2146662]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-6-#23", id=46, state=WAITING, blockCnt=4598, waitCnt=2155185]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
       at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-5-#22", id=45, state=WAITING, blockCnt=4344, waitCnt=2145319]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-4-#21", id=44, state=WAITING, blockCnt=4512, waitCnt=2126296]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-3-#20", id=43, state=WAITING, blockCnt=3926, waitCnt=2152280]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
       at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-2-#19", id=42, state=WAITING, blockCnt=4512, waitCnt=2143955]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-1-#18", id=41, state=WAITING, blockCnt=4149, waitCnt=2145031]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-0-#17", id=40, state=WAITING, blockCnt=4062, waitCnt=2165334]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
       at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-15-#16", id=39, state=WAITING, blockCnt=5772, waitCnt=2653735]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-14-#15", id=38, state=WAITING, blockCnt=6154, waitCnt=2722611]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-13-#14", id=37, state=WAITING, blockCnt=6286, waitCnt=2705888]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-12-#13", id=36, state=WAITING, blockCnt=3580, waitCnt=1958196]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-11-#12", id=35, state=WAITING, blockCnt=6031, waitCnt=2732193]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-10-#11", id=34, state=WAITING, blockCnt=4802, waitCnt=2526993]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-9-#10", id=33, state=WAITING, blockCnt=5896, waitCnt=2699439]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-8-#9", id=32, state=WAITING, blockCnt=5361, waitCnt=2633071]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-7-#8", id=31, state=WAITING, blockCnt=3817, waitCnt=2077662]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-6-#7", id=30, state=WAITING, blockCnt=4702, waitCnt=2379880]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-5-#6", id=29, state=WAITING, blockCnt=3489, waitCnt=1929836]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-4-#5", id=28, state=WAITING, blockCnt=4286, waitCnt=2157413]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-3-#4", id=27, state=WAITING, blockCnt=4002, waitCnt=2066821]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-2-#3", id=26, state=WAITING, blockCnt=3989, waitCnt=2158051]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-1-#2", id=25, state=WAITING, blockCnt=7154, waitCnt=2792749]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-0-#1", id=24, state=WAITING, blockCnt=5014, waitCnt=2549470]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="FileWatchdog", id=23, state=TIMED_WAITING, blockCnt=0, waitCnt=987]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)

Thread [name="RMI TCP Accept-0", id=22, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="RMI TCP Accept-49112", id=21, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="RMI TCP Accept-0", id=20, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="Signal Dispatcher", id=5, state=RUNNABLE, blockCnt=0, waitCnt=0]

Thread [name=
"Finalizer", id=3, state=WAITING, blockCnt=1629, waitCnt=1481]
    Lock [object=
java.lang.ref.ReferenceQueue$Lock@599e74d7, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

Thread [name="Reference Handler", id=2, state=WAITING, blockCnt=1494, waitCnt=1481]
    Lock [object=
java.lang.ref.Reference$Lock@730de558, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

Thread [name="main", id=1, state=WAITING, blockCnt=2, waitCnt=6]
    Lock [object=
java.util.concurrent.CountDownLatch$Sync@14f96498, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at o.a.i.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:334)

 

 

From: Andrei Aleksandrov <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, November 8, 2019 at 3:49 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: Blocked Worker Thread

 

Hi Conrad,

The reasons can be different. Could you please share the logs?

BR,
Andrei

11/7/2019 10:35 PM, Conrad Mukai (cmukai) пишет:

We are running a cache in a 4 node cluster with atomicityMode set to ATOMIC and have persistence enabled. We repeatedly get a SYSTEM_WORKER_BLOCKED error on one node which is disabling the entire cluster. We were seeing a lot of sockets in TIME_WAIT state which was blocking clients from connecting so we did the following on all the nodes:

 

# ignore TIME_WAIT state on sockets
echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse
echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle

 

This made that issue go away, but may play a part in this new issue. First question is what is the root cause of the error? The second question is why does this bring down the entire cluster?

 

Here is the error message:

 

[2019-11-07 16:36:22,037][ERROR][tcp-disco-msg-worker-#2][root] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=partition-exchanger, igniteInstanceName=null, finished=false, heartbeatTs=1573090509428]]]

class org.apache.ignite.IgniteException: GridWorker [name=partition-exchanger, igniteInstanceName=null, finished=false, heartbeatTs=1573090509428]

        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)

        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)

        at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)

        at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)

        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)

        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

 

This is followed by a warning and a thread dump:

 

[2019-11-07 16:36:22,038][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] No deadlocked threads detected.

[2019-11-07 16:36:22,328][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] Thread dump at 2019/11/07 16:36:22 GMT

 

For the particular thread in the error and warning messages here is the thread dump:

 

Thread [name="tcp-disco-msg-worker-#2", id=113, state=RUNNABLE, blockCnt=211, waitCnt=4745368]

        at sun.management.ThreadImpl.dumpThreads0(Native Method)

        at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)

        at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1368)

        at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:128)

        - locked o.a.i.i.processors.failure.FailureProcessor@7e65ceba

        at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:104)

        at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1829)

        at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)

        at o.a.i.i.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)

        at o.a.i.i.util.worker.GridWorker.onIdle(GridWorker.java:297)

        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)

        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker$$Lambda$47/1047515321.run(Unknown Source)

        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)

        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)

        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)

        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)

        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

 

In addition all the system threads are in TIMED_WAITING state:

 

Thread [name="sys-#7099", id=9252, state=TIMED_WAITING, blockCnt=0, waitCnt=1]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

and all the query threads are in WAITING state:

 

Thread [name="query-#1062", id=3027, state=WAITING, blockCnt=11, waitCnt=3365]

    Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)

        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)

        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)

        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)

        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)

        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)

        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)

        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)

        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)

        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)

        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)

        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)

       at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)

        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)

        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)

        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)

        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)

        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)

        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)

        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)

        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)

        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

    Locked synchronizers:

        java.util.concurrent.ThreadPoolExecutor$Worker@c1349cf

        [hidden email]

 

Many other threads are in WAITING state, in particular this one may be of interest:

 

Thread [name="srvc-deploy-#91", id=164, state=WAITING, blockCnt=0, waitCnt=5182]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7310e7da, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

As well as all the checkpoint-runner threads:

 

Thread [name="checkpoint-runner-#82", id=152, state=WAITING, blockCnt=2, waitCnt=3282307]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c8ae71a, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

and the db-checkpoint-thread:

 

Thread [name="db-checkpoint-thread-#78", id=148, state=WAITING, blockCnt=297, waitCnt=1337]

    Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)

       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)

        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)

        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:3690)

        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3300)

        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3199)

        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)

        at java.lang.Thread.run(Thread.java:748)

 

Thanks in advance,

Conrad Mukai

 

aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Blocked Worker Thread

Hi,

Thread dumps aren't logs from the server. However, I guess that some of transactions work for a long time. It means that until it wasn't completed or roll backed other queries will not able to work with the data used in current transaction.

To avoid current situation please set next timeouts:

https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/TransactionConfiguration.html#setDefaultTxTimeout-long-
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/TransactionConfiguration.html#setTxTimeoutOnPartitionMapExchange-long-

Also, you can try to set

https://apacheignite.readme.io/v1.4/docs/cluster-config#section-failure-detection-timeout

BR,
Andrei

11/8/2019 8:20 PM, Conrad Mukai (cmukai) пишет:

Here is the error and warnings with the thread dump:

 

[2019-11-07 16:36:22,037][ERROR][tcp-disco-msg-worker-#2][root] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=partition-exchanger, igniteInstanceName=null, finished=false, heartbeatTs=1573090509428]]]
class
org.apache.ignite.IgniteException: GridWorker [name=partition-exchanger, igniteInstanceName=null, finished=false, heartbeatTs=1573090509428]
   at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
   at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
   at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
   at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
   at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
   at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
   at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
   at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
   at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
   at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[2019-11-07 16:36:22,038][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] No deadlocked threads detected.
[
2019-11-07 16:36:22,328][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] Thread dump at 2019/11/07 16:36:22 GMT
Thread [name=
"sys-#7099", id=9252, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#7098", id=9251, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#7097", id=9250, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#7096", id=9249, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#7095", id=9248, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="query-#1078", id=3051, state=WAITING, blockCnt=7, waitCnt=3190]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@2d4a8835
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@4166bfee
Thread [name=
"query-#1077", id=3050, state=WAITING, blockCnt=15, waitCnt=3235]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@c0a3312
        java.util.concurrent.ThreadPoolExecutor
$Worker@735561c9
Thread [name=
"query-#1076", id=3049, state=WAITING, blockCnt=8, waitCnt=3309]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
       at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1141)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@19007848
        java.util.concurrent.locks.ReentrantLock
$NonfairSync@714ddb67
Thread [name=
"query-#1075", id=3048, state=WAITING, blockCnt=10, waitCnt=3246]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@20f0fee8
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@5a0836f
Thread [name="query-#1074", id=3047, state=WAITING, blockCnt=9, waitCnt=3282]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@12b78a6d
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@55608d7f
Thread [name=
"query-#1073", id=3046, state=WAITING, blockCnt=12, waitCnt=3273]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@4297fe42
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@8afa662
Thread [name="query-#1072", id=3045, state=WAITING, blockCnt=5, waitCnt=3315]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
       at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@50eeac6
        java.util.concurrent.locks.ReentrantLock
$NonfairSync@1726c2f3
Thread [name=
"query-#1071", id=3044, state=WAITING, blockCnt=11, waitCnt=3374]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@28d59ac2
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@169c9083
Thread [name=
"query-#1070", id=3043, state=WAITING, blockCnt=9, waitCnt=3259]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
  ��     at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@31dc0609
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@4746fc5a
Thread [name=
"query-#1069", id=3042, state=WAITING, blockCnt=13, waitCnt=3312]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@3668b5e8
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@322fde66
Thread [name=
"query-#1068", id=3041, state=WAITING, blockCnt=9, waitCnt=3482]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@6d8c916e
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@13bfa38a
Thread [name=
"query-#1067", id=3040, state=WAITING, blockCnt=12, waitCnt=3238]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@2ba1cf40
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@61d531f6
Thread [name=
"query-#1066", id=3039, state=WAITING, blockCnt=7, waitCnt=3294]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@76a8189
        java.util.concurrent.locks.ReentrantLock
$NonfairSync@1101b5ba
Thread [name=
"query-#1065", id=3038, state=WAITING, blockCnt=10, waitCnt=3316]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@9ee918
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@5d118512
Thread [name=
"query-#1064", id=3037, state=WAITING, blockCnt=3, waitCnt=3267]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@3d0f7d55
       
java.util.concurrent.locks.ReentrantLock$NonfairSync@335ccec9
Thread [name=
"query-#1062", id=3027, state=WAITING, blockCnt=11, waitCnt=3365]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4049)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:522)
        at o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:472)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3010)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at o.a.i.i.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at o.a.i.i.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at o.a.i.i.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at o.a.i.i.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
       
java.util.concurrent.ThreadPoolExecutor$Worker@c1349cf
        java.util.concurrent.locks.ReentrantLock
$NonfairSync@7bce13d5
Thread [name=
"srvc-deploy-#91", id=164, state=WAITING, blockCnt=0, waitCnt=5182]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7310e7da, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="checkpoint-runner-#82", id=152, state=WAITING, blockCnt=2, waitCnt=3282307]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c8ae71a, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="checkpoint-runner-#81", id=151, state=WAITING, blockCnt=7, waitCnt=3283481]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c8ae71a, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="checkpoint-runner-#80", id=150, state=WAITING, blockCnt=8, waitCnt=3282180]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c8ae71a, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="checkpoint-runner-#79", id=149, state=WAITING, blockCnt=10, waitCnt=3282388]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c8ae71a, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="db-checkpoint-thread-#78", id=148, state=WAITING, blockCnt=297, waitCnt=1337]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:3690)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3300)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3199)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="ttl-cleanup-worker-#77", id=147, state=WAITING, blockCnt=11117, waitCnt=2816036]
    Lock [object=
java.util.concurrent.locks.ReentrantLock$NonfairSync@335ccec9, ownerName=query-#1064, ownerId=3037]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
        at o.a.i.i.processors.cache.GridCacheMapEntry.lockEntry(GridCacheMapEntry.java:4967)
        at o.a.i.i.processors.cache.GridCacheMapEntry.obsolete(GridCacheMapEntry.java:2874)
        at o.a.i.i.processors.cache.GridCacheConcurrentMapImpl.putEntryIfObsoleteOrAbsent(GridCacheConcurrentMapImpl.java:124)
        at o.a.i.i.processors.cache.GridCacheConcurrentMapImpl.putEntryIfObsoleteOrAbsent(GridCacheConcurrentMapImpl.java:69)
        at o.a.i.i.processors.cache.distributed.dht.GridCachePartitionedConcurrentMap.putEntryIfObsoleteOrAbsent(GridCachePartitionedConcurrentMap.java:94)
        at o.a.i.i.processors.cache.GridCacheAdapter.entryEx(GridCacheAdapter.java:1008)
        at o.a.i.i.processors.cache.distributed.dht.GridDhtCacheAdapter.entryEx(GridDhtCacheAdapter.java:544)
        at o.a.i.i.processors.cache.GridCacheAdapter.entryEx(GridCacheAdapter.java:999)
        at o.a.i.i.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpiredInternal(GridCacheOffheapManager.java:2224)
        at o.a.i.i.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpired(GridCacheOffheapManager.java:2157)
        at o.a.i.i.processors.cache.persistence.GridCacheOffheapManager.expire(GridCacheOffheapManager.java:845)
        at o.a.i.i.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:207)
        at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:139)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="wal-segment-syncer-#74", id=143, state=TIMED_WAITING, blockCnt=0, waitCnt=103412]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7778)
        at o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$WalSegmentSyncer.body(FileWriteAheadLogManager.java:3518)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="tcp-disco-sock-reader-#12", id=128, state=RUNNABLE, blockCnt=48, waitCnt=6]
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked java.io.BufferedInputStream@32ce513d
        at o.a.i.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
        at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2663)
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2679)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
        at o.a.i.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:43)
        at o.a.i.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:137)
        at o.a.i.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
        at o.a.i.i.util.IgniteUtils.unmarshal(IgniteUtils.java:10113)
        at o.a.i.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:6246)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="session-timeout-worker-#54", id=83, state=TIMED_WAITING, blockCnt=0, waitCnt=57215]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.processors.rest.GridRestProcessor$4.body(GridRestProcessor.java:495)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="exchange-worker-#63", id=121, state=WAITING, blockCnt=202, waitCnt=30351]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initTopologies(GridDhtPartitionsExchangeFuture.java:966)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:824)
        at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2681)
        at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2553)
       at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="ignite-update-notifier-timer", id=120, state=TIMED_WAITING, blockCnt=1, waitCnt=35]
    Lock [object=
java.util.TaskQueue@39f7d249, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        at java.util.TimerThread.run(Timer.java:505)

Thread [name="upd-ver-checker", id=119, state=TIMED_WAITING, blockCnt=0, waitCnt=11688]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.processors.cluster.GridUpdateNotifier$1.run(GridUpdateNotifier.java:115)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="disco-event-worker-#62", id=118, state=WAITING, blockCnt=201, waitCnt=31392]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64c4c081, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2856)
        at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2825)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="tcp-disco-srvr-#3", id=114, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5845)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at o.a.i.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5763)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-msg-worker-#2", id=113, state=RUNNABLE, blockCnt=211, waitCnt=4745368]
        at sun.management.ThreadImpl.dumpThreads0(Native Method)
        at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
        at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1368)
        at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:128)
        - locked o.a.i.i.processors.failure.FailureProcessor@7e65ceba
        at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:104)
        at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1829)
        at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
        at o.a.i.i.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
        at o.a.i.i.util.worker.GridWorker.onIdle(GridWorker.java:297)
        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker$$Lambda$47/1047515321.run(Unknown Source)
        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="disco-notifier-worker-#61", id=112, state=WAITING, blockCnt=0, waitCnt=109690]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40336649, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2660)
        at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2705)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-data-loader-flusher-#60", id=110, state=WAITING, blockCnt=0, waitCnt=1]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4751919c, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-109", id=109, state=TIMED_WAITING, blockCnt=0, waitCnt=1289]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-108", id=108, state=TIMED_WAITING, blockCnt=0, waitCnt=1290]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-107", id=107, state=TIMED_WAITING, blockCnt=0, waitCnt=1272]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-106", id=106, state=TIMED_WAITING, blockCnt=0, waitCnt=1289]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-105", id=105, state=TIMED_WAITING, blockCnt=0, waitCnt=1283]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-104", id=104, state=TIMED_WAITING, blockCnt=0, waitCnt=1261]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-103", id=103, state=TIMED_WAITING, blockCnt=0, waitCnt=1257]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-102", id=102, state=TIMED_WAITING, blockCnt=0, waitCnt=1318]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-101", id=101, state=TIMED_WAITING, blockCnt=0, waitCnt=1279]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-100", id=100, state=TIMED_WAITING, blockCnt=0, waitCnt=1299]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d3913f, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

Thread [name=[hidden email], id=99, state=RUNNABLE, blockCnt=0, waitCnt=1]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        - locked java.lang.Object@2185a38e
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name=[hidden email], id=98, state=BLOCKED, blockCnt=1, waitCnt=2]
    Lock [object=
java.lang.Object@2185a38e, ownerName=qtp1710783020-99-acceptor-1@7012ca9f-ServerConnector@7ed52aed{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}, ownerId=99]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-97", id=97, state=RUNNABLE, blockCnt=3, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@7a5f98d9
        - locked
java.util.Collections$UnmodifiableSet@31493b7f
        - locked
sun.nio.ch.EPollSelectorImpl@288a1e00
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-96", id=96, state=RUNNABLE, blockCnt=4, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@249f514c
        - locked
java.util.Collections$UnmodifiableSet@22808c56
        - locked
sun.nio.ch.EPollSelectorImpl@5aead9ba
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-95", id=95, state=RUNNABLE, blockCnt=3, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@22e8a6f9
        - locked
java.util.Collections$UnmodifiableSet@7ca396d9
        - locked
sun.nio.ch.EPollSelectorImpl@5a675727
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-94", id=94, state=RUNNABLE, blockCnt=3, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@56cc62d8
        - locked
java.util.Collections$UnmodifiableSet@31f4d996
        - locked
sun.nio.ch.EPollSelectorImpl@5101842f
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-93", id=93, state=RUNNABLE, blockCnt=3, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@313f3f47
        - locked
java.util.Collections$UnmodifiableSet@4d0511a
        - locked sun.nio.ch.EPollSelectorImpl@13ff107f
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-92", id=92, state=RUNNABLE, blockCnt=5, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@4e82ded3
        - locked
java.util.Collections$UnmodifiableSet@60f0543
        - locked sun.nio.ch.EPollSelectorImpl@51cc19
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-91", id=91, state=RUNNABLE, blockCnt=4, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@6b464884
        - locked
java.util.Collections$UnmodifiableSet@33ea6779
        - locked
sun.nio.ch.EPollSelectorImpl@4d31595a
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="qtp1710783020-90", id=90, state=RUNNABLE, blockCnt=1, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@430e75e1
        - locked
java.util.Collections$UnmodifiableSet@1ad5076c
        - locked
sun.nio.ch.EPollSelectorImpl@7a647ae0
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$35/1420012431.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-rest-3-#58", id=87, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@3dd8077f
        - locked
java.util.Collections$UnmodifiableSet@dc9dc58
        - locked sun.nio.ch.EPollSelectorImpl@3f9544ea
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-rest-2-#57", id=86, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@667f8a82
        - locked
java.util.Collections$UnmodifiableSet@1b619acd
        - locked
sun.nio.ch.EPollSelectorImpl@5889bee5
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-rest-1-#56", id=85, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@4dae130a
        - locked
java.util.Collections$UnmodifiableSet@571fbba7
        - locked
sun.nio.ch.EPollSelectorImpl@403ff47b
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-rest-0-#55", id=84, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@5554040f
        - locked
java.util.Collections$UnmodifiableSet@68df6bec
        - locked
sun.nio.ch.EPollSelectorImpl@42b727d3
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-tcp-rest-#59", id=88, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@7d148fd2
        - locked
java.util.Collections$UnmodifiableSet@8159803
        - locked sun.nio.ch.EPollSelectorImpl@1b0602e6
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2926)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2874)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-3-#52", id=81, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@60f437ea
        - locked
java.util.Collections$UnmodifiableSet@39c74e39
        - locked
sun.nio.ch.EPollSelectorImpl@1d361c82
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-2-#51", id=80, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@360c3cf0
        - locked
java.util.Collections$UnmodifiableSet@778616bc
        - locked
sun.nio.ch.EPollSelectorImpl@743338f3
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-1-#50", id=79, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@594cf8fb
        - locked
java.util.Collections$UnmodifiableSet@ef0a49e
        - locked sun.nio.ch.EPollSelectorImpl@4904600a
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-0-#49", id=78, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@249d2d79
        - locked
java.util.Collections$UnmodifiableSet@1a289231
        - locked
sun.nio.ch.EPollSelectorImpl@117701be
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-client-listener-#53", id=82, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@7b9937aa
        - locked
java.util.Collections$UnmodifiableSet@36223b45
        - locked
sun.nio.ch.EPollSelectorImpl@68458eaa
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2926)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2874)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="tcp-comm-worker-#1", id=77, state=TIMED_WAITING, blockCnt=3, waitCnt=112]
    Lock [object=
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5bcf0aa0, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4287)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at o.a.i.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="grid-nio-worker-tcp-comm-7-#47", id=72, state=RUNNABLE, blockCnt=19478, waitCnt=2]
       at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@54dfa634
        - locked
java.util.Collections$UnmodifiableSet@10ab42db
        - locked
sun.nio.ch.EPollSelectorImpl@610bf143
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-6-#46", id=71, state=RUNNABLE, blockCnt=8814, waitCnt=1]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@10e226b3
        - locked
java.util.Collections$UnmodifiableSet@39a5dc9d
        - locked
sun.nio.ch.EPollSelectorImpl@7352fce8
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-5-#45", id=70, state=RUNNABLE, blockCnt=7958, waitCnt=4]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@4ded9a28
        - locked
java.util.Collections$UnmodifiableSet@4cbbfa98
        - locked
sun.nio.ch.EPollSelectorImpl@5bed96a6
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-4-#44", id=69, state=RUNNABLE, blockCnt=16692, waitCnt=5]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@3fbc848f
        - locked
java.util.Collections$UnmodifiableSet@1ea1658a
        - locked
sun.nio.ch.EPollSelectorImpl@ddd33fc
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-3-#43", id=68, state=RUNNABLE, blockCnt=9365, waitCnt=2]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@3a32cc47
        - locked
java.util.Collections$UnmodifiableSet@6171bf72
        - locked
sun.nio.ch.EPollSelectorImpl@74e90be3
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-2-#42", id=67, state=RUNNABLE, blockCnt=16342, waitCnt=3]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@104f9451
        - locked
java.util.Collections$UnmodifiableSet@3c3322eb
        - locked
sun.nio.ch.EPollSelectorImpl@50559030
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-1-#41", id=66, state=RUNNABLE, blockCnt=11611, waitCnt=3]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@6cdf3fca
        - locked
java.util.Collections$UnmodifiableSet@dec31c
        - locked
sun.nio.ch.EPollSelectorImpl@25df8d96
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-0-#40", id=65, state=RUNNABLE, blockCnt=11225, waitCnt=3]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@59f75d0d
        - locked
java.util.Collections$UnmodifiableSet@4c431896
        - locked
sun.nio.ch.EPollSelectorImpl@3af3e421
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-tcp-comm-#48", id=76, state=RUNNABLE, blockCnt=28, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@63197dd4
        - locked
java.util.Collections$UnmodifiableSet@e54b8f
        - locked
sun.nio.ch.EPollSelectorImpl@54b664ad
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2926)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2874)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-timeout-worker-#39", id=64, state=TIMED_WAITING, blockCnt=260, waitCnt=532667]
    Lock [object=
java.lang.Object@714140f6, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:269)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="ignite-clock", id=63, state=TIMED_WAITING, blockCnt=0, waitCnt=4724639]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.util.IgniteUtils$22.run(IgniteUtils.java:3383)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="jvm-pause-detector-worker", id=56, state=TIMED_WAITING, blockCnt=0, waitCnt=967819]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:100)

Thread [name="data-streamer-stripe-15-#32", id=55, state=WAITING, blockCnt=4149, waitCnt=2121970]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-14-#31", id=54, state=WAITING, blockCnt=4140, waitCnt=2130293]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-13-#30", id=53, state=WAITING, blockCnt=4178, waitCnt=2131313]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-12-#29", id=52, state=WAITING, blockCnt=3964, waitCnt=2133028]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-11-#28", id=51, state=WAITING, blockCnt=4342, waitCnt=2144768]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-10-#27", id=50, state=WAITING, blockCnt=4860, waitCnt=2162164]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-9-#26", id=49, state=WAITING, blockCnt=4268, waitCnt=2150445]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-8-#25", id=48, state=WAITING, blockCnt=4301, waitCnt=2143068]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-7-#24", id=47, state=WAITING, blockCnt=4669, waitCnt=2146662]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-6-#23", id=46, state=WAITING, blockCnt=4598, waitCnt=2155185]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
       at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-5-#22", id=45, state=WAITING, blockCnt=4344, waitCnt=2145319]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-4-#21", id=44, state=WAITING, blockCnt=4512, waitCnt=2126296]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-3-#20", id=43, state=WAITING, blockCnt=3926, waitCnt=2152280]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
       at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-2-#19", id=42, state=WAITING, blockCnt=4512, waitCnt=2143955]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-1-#18", id=41, state=WAITING, blockCnt=4149, waitCnt=2145031]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-0-#17", id=40, state=WAITING, blockCnt=4062, waitCnt=2165334]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1739)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
       at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
        at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-15-#16", id=39, state=WAITING, blockCnt=5772, waitCnt=2653735]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-14-#15", id=38, state=WAITING, blockCnt=6154, waitCnt=2722611]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-13-#14", id=37, state=WAITING, blockCnt=6286, waitCnt=2705888]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-12-#13", id=36, state=WAITING, blockCnt=3580, waitCnt=1958196]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-11-#12", id=35, state=WAITING, blockCnt=6031, waitCnt=2732193]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-10-#11", id=34, state=WAITING, blockCnt=4802, waitCnt=2526993]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-9-#10", id=33, state=WAITING, blockCnt=5896, waitCnt=2699439]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-8-#9", id=32, state=WAITING, blockCnt=5361, waitCnt=2633071]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-7-#8", id=31, state=WAITING, blockCnt=3817, waitCnt=2077662]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-6-#7", id=30, state=WAITING, blockCnt=4702, waitCnt=2379880]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-5-#6", id=29, state=WAITING, blockCnt=3489, waitCnt=1929836]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-4-#5", id=28, state=WAITING, blockCnt=4286, waitCnt=2157413]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-3-#4", id=27, state=WAITING, blockCnt=4002, waitCnt=2066821]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-2-#3", id=26, state=WAITING, blockCnt=3989, waitCnt=2158051]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-1-#2", id=25, state=WAITING, blockCnt=7154, waitCnt=2792749]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-0-#1", id=24, state=WAITING, blockCnt=5014, waitCnt=2549470]
    Lock [object=
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1500)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3217)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="FileWatchdog", id=23, state=TIMED_WAITING, blockCnt=0, waitCnt=987]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)

Thread [name="RMI TCP Accept-0", id=22, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="RMI TCP Accept-49112", id=21, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="RMI TCP Accept-0", id=20, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="Signal Dispatcher", id=5, state=RUNNABLE, blockCnt=0, waitCnt=0]

Thread [name=
"Finalizer", id=3, state=WAITING, blockCnt=1629, waitCnt=1481]
    Lock [object=
java.lang.ref.ReferenceQueue$Lock@599e74d7, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

Thread [name="Reference Handler", id=2, state=WAITING, blockCnt=1494, waitCnt=1481]
    Lock [object=
java.lang.ref.Reference$Lock@730de558, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

Thread [name="main", id=1, state=WAITING, blockCnt=2, waitCnt=6]
    Lock [object=
java.util.concurrent.CountDownLatch$Sync@14f96498, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at o.a.i.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:334)

 

 

From: Andrei Aleksandrov [hidden email]
Reply-To: [hidden email] [hidden email]
Date: Friday, November 8, 2019 at 3:49 AM
To: [hidden email] [hidden email]
Subject: Re: Blocked Worker Thread

 

Hi Conrad,

The reasons can be different. Could you please share the logs?

BR,
Andrei

11/7/2019 10:35 PM, Conrad Mukai (cmukai) пишет:

We are running a cache in a 4 node cluster with atomicityMode set to ATOMIC and have persistence enabled. We repeatedly get a SYSTEM_WORKER_BLOCKED error on one node which is disabling the entire cluster. We were seeing a lot of sockets in TIME_WAIT state which was blocking clients from connecting so we did the following on all the nodes:

 

# ignore TIME_WAIT state on sockets
echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse
echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle

 

This made that issue go away, but may play a part in this new issue. First question is what is the root cause of the error? The second question is why does this bring down the entire cluster?

 

Here is the error message:

 

[2019-11-07 16:36:22,037][ERROR][tcp-disco-msg-worker-#2][root] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=partition-exchanger, igniteInstanceName=null, finished=false, heartbeatTs=1573090509428]]]

class org.apache.ignite.IgniteException: GridWorker [name=partition-exchanger, igniteInstanceName=null, finished=false, heartbeatTs=1573090509428]

        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)

        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)

        at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)

        at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)

        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)

        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)

        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

 

This is followed by a warning and a thread dump:

 

[2019-11-07 16:36:22,038][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] No deadlocked threads detected.

[2019-11-07 16:36:22,328][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] Thread dump at 2019/11/07 16:36:22 GMT

 

For the particular thread in the error and warning messages here is the thread dump:

 

Thread [name="tcp-disco-msg-worker-#2", id=113, state=RUNNABLE, blockCnt=211, waitCnt=4745368]

        at sun.management.ThreadImpl.dumpThreads0(Native Method)

        at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)

        at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1368)

        at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:128)

        - locked o.a.i.i.processors.failure.FailureProcessor@7e65ceba

        at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:104)

        at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1829)

        at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)

        at o.a.i.i.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)

        at o.a.i.i.util.worker.GridWorker.onIdle(GridWorker.java:297)

        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)

        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker$$Lambda$47/1047515321.run(Unknown Source)

        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)

        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)

        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)

        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)

        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

 

In addition all the system threads are in TIMED_WAITING state:

 

Thread [name="sys-#7099", id=9252, state=TIMED_WAITING, blockCnt=0, waitCnt=1]

    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573, ownerName=null, ownerId=-1]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

and all the query threads are in WAITING state:

 

Thread [name="query-#1062", id=3027, state=WAITING, blockCnt=11, waitCnt=3365]

    Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2061baf, o