Large durable caches

classic Classic list List threaded Threaded
27 messages Options
12
lawrencefinn lawrencefinn
Reply | Threaded
Open this post in threaded view
|

Large durable caches

I am testing a very large durable cache that mostly cannot fit into memory.
I start loading in a lot of data via a data streamer.  At some point the
data becomes too large to fit into memory so ignite starts writing a lot to
disk during checkpoints.  But after some point after that, the datastreamer
stops streaming.  It's stuck.  Doing a jstack on ignite i see the
datastreamer threads are all stuck in a parked state.  Any thoughts?

"data-streamer-stripe-4-#37" #62 prio=5 os_prio=0 tid=0x00007fe8c212d000
nid=0x1410 waiting on condition [0x00007fe73d1d6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

The checkpoint thread seems to be parked too
"checkpoint-runner-#354" #397 prio=5 os_prio=0 tid=0x00007fe49c04f000
nid=0x177c waiting on condition [0x00007fdc54a1c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000002c08a6ef8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None


My configuration is as follows:

    <bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="publicThreadPoolSize" value="128"/>
    <property name="systemThreadPoolSize" value="128"/>
    <property name="dataStreamerThreadPoolSize" value="128"/>
    <property name="deploymentMode" value="SHARED"/>
    <property name="peerClassLoadingEnabled" value="true"/>
    <property name="dataStorageConfiguration">
    <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
        <property name="storagePath" value="/per/persistence"/>
        <property name="walPath" value="/wal"/>
        <property name="walMode" value="BACKGROUND"/>
        <property name="walFlushFrequency" value="10000"/>
        <property name="checkpointThreads" value="128"/>
        <property name="writeThrottlingEnabled" value="true"/>
        <property name="walArchivePath" value="/wal/archive"/>
        <property name="defaultDataRegionConfiguration">
                <bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
                        <property name="name" value="Default_Region"/>
                        <property name="maxSize" value="#{100L * 1024 *
1024}"/>
                </bean>
        </property>
        <property name="dataRegionConfigurations">
            <list>
              <bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
                <property name="name" value="20G_Region_Eviction_P"/>
                <property name="initialSize" value="#{10L * 1024 * 1024}"/>
                <property name="maxSize" value="#{20 * 1024L * 1024 *
1024}"/>
                <property name="persistenceEnabled" value="true"/>
                <property name="pageEvictionMode" value="RANDOM_2_LRU"/>
                <property name="checkpointPageBufferSize" value="#{2 * 1024L
* 1024 * 1024}"/>
              </bean>






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ivan Rakov Ivan Rakov
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

Hello,

Seems like checkpoint-runner thread is idle. There's no active
checkpoint at the moment of dump, and thus there's no work for
checkpoint thread.
If the problem persists, please share full thread dump. It would be
better to take 2-3 consecutive thread dumps when your data streamer stops.

Best Regards,
Ivan Rakov

On 19.02.2018 0:41, lawrencefinn wrote:

> I am testing a very large durable cache that mostly cannot fit into memory.
> I start loading in a lot of data via a data streamer.  At some point the
> data becomes too large to fit into memory so ignite starts writing a lot to
> disk during checkpoints.  But after some point after that, the datastreamer
> stops streaming.  It's stuck.  Doing a jstack on ignite i see the
> datastreamer threads are all stuck in a parked state.  Any thoughts?
>
> "data-streamer-stripe-4-#37" #62 prio=5 os_prio=0 tid=0x00007fe8c212d000
> nid=0x1410 waiting on condition [0x00007fe73d1d6000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
> The checkpoint thread seems to be parked too
> "checkpoint-runner-#354" #397 prio=5 os_prio=0 tid=0x00007fe49c04f000
> nid=0x177c waiting on condition [0x00007fdc54a1c000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000002c08a6ef8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
>
> My configuration is as follows:
>
>      <bean id="ignite.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
>      <property name="publicThreadPoolSize" value="128"/>
>      <property name="systemThreadPoolSize" value="128"/>
>      <property name="dataStreamerThreadPoolSize" value="128"/>
>      <property name="deploymentMode" value="SHARED"/>
>      <property name="peerClassLoadingEnabled" value="true"/>
>      <property name="dataStorageConfiguration">
>      <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
>          <property name="storagePath" value="/per/persistence"/>
>          <property name="walPath" value="/wal"/>
>          <property name="walMode" value="BACKGROUND"/>
>          <property name="walFlushFrequency" value="10000"/>
>          <property name="checkpointThreads" value="128"/>
>          <property name="writeThrottlingEnabled" value="true"/>
>          <property name="walArchivePath" value="/wal/archive"/>
>          <property name="defaultDataRegionConfiguration">
>                  <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
>                          <property name="name" value="Default_Region"/>
>                          <property name="maxSize" value="#{100L * 1024 *
> 1024}"/>
>                  </bean>
>          </property>
>          <property name="dataRegionConfigurations">
>              <list>
>                <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
>                  <property name="name" value="20G_Region_Eviction_P"/>
>                  <property name="initialSize" value="#{10L * 1024 * 1024}"/>
>                  <property name="maxSize" value="#{20 * 1024L * 1024 *
> 1024}"/>
>                  <property name="persistenceEnabled" value="true"/>
>                  <property name="pageEvictionMode" value="RANDOM_2_LRU"/>
>                  <property name="checkpointPageBufferSize" value="#{2 * 1024L
> * 1024 * 1024}"/>
>                </bean>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

lawrencefinn lawrencefinn
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

Okay I am trying to reproduce.  It hasn't got stuck yet, but the client got
disconnected and reconnected recently.  I don't think it is related to GC
because I am recording GC times and it does not jump up that much.  Could
the system get slow on a lot of io?  i see this in the ignite log:

[19:13:01,988][WARNING][grid-timeout-worker-#71][diagnostic] Found long
running cache future [startTime=19:11:56.656, curTime=19:13:01.911,
 fut=GridNearAtomicSingleUpdateFuture [reqState=Primary
[id=62a2a255-3320-4040-aa23-ffb86dec7586, opRes=false, expCnt=-1, rcvdCnt=0,
primaryRes=false, done=false, waitFor=null, rcvd=null],
super=GridNearAtomicAbstractUpdateFuture [remapCnt=100,
topVer=AffinityTopologyVersion [topVer=3, minorTopVer=14], remapTopVer=null,
err=null, futId=313296239, super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=1229092316]]]]
[19:13:01,988][WARNING][grid-timeout-worker-#71][diagnostic] Found long
running cache future [startTime=19:11:39.917, curTime=19:13:01.911,
fut=GridNearAtomicSingleUpdateFuture [reqState=Primary
[id=62a2a255-3320-4040-aa23-ffb86dec7586, opRes=false, expCnt=-1, rcvdCnt=0,
primaryRes=false, done=false, waitFor=null, rcvd=null],
super=GridNearAtomicAbstractUpdateFuture [remapCnt=100,
topVer=AffinityTopologyVersion [topVer=3, minorTopVer=14], remapTopVer=null,
err=null, futId=312914655, super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=15435296]]]]
[19:13:51,057][INFO][db-checkpoint-thread-#110][GridCacheDatabaseSharedManager]
Checkpoint started [checkpointId=77744626-04e6-4e17-bda7-23ecb50bbe19,
startPtr=FileWALPointer [idx=9600, fileOffset=35172819, len=124303,
forceFlush=true], checkpointLockWait=57708ms, checkpointLockHoldTime=64ms,
pages=3755135, reason='too many dirty pages']
[19:14:01,919][INFO][grid-timeout-worker-#71][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=62a2a255, uptime=01:42:41.752]
    ^-- H/N/C [hosts=2, nodes=3, CPUs=64]
    ^-- CPU [cur=77.83%, avg=39.11%, GC=0.13%]
    ^-- PageMemory [pages=5111642]
    ^-- Heap [used=11669MB, free=43.02%, comm=20480MB]
    ^-- Non heap [used=67MB, free=95.56%, comm=69MB]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=6, qSize=0]
    ^-- Outbound messages queue [size=0]
[19:15:03,470][INFO][tcp-disco-srvr-#2][TcpDiscoverySpi] TCP discovery
accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=33542]
[19:15:03,470][INFO][tcp-disco-srvr-#2][TcpDiscoverySpi] TCP discovery
spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=33542]


My app log has:
2018-02-19 19:15:02,176 [WARN] from
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi in
tcp-client-disco-reconnector-#5%cabooseGrid% - Timed out waiting for message
to be read (most probably, the reason is long GC pauses on remote node)
[curTimeout=5000, rmtAddr=/127.0.0.1:47500, rmtPort=47500]
2018-02-19 19:15:02,176 [ERROR] from
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi in
tcp-client-disco-reconnector-#5%cabooseGrid% - Exception on joining: Failed
to deserialize object with given class loader:
sun.misc.Launcher$AppClassLoader@28d93b30
org.apache.ignite.IgniteCheckedException: Failed to deserialize object with
given class loader: sun.misc.Launcher$AppClassLoader@28d93b30
        at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:129)
        at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
        at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
        at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.readMessage(TcpDiscoverySpi.java:1590)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:627)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:524)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:124)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$Reconnector.body(ClientImpl.java:1377)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.net.SocketTimeoutException: Read timed out



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ivan Rakov Ivan Rakov
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

It's hard to determine the problem by these messages. I don't see
anything unhealthy regarding persistence - checkpoint start is a regular
event.

There were cases when excessive GC load on client side seriously
affected throughput/latency of data streaming. You may consider playing
with the following data streamer parameters:

> public int perNodeBufferSize(int bufSize) - defines how many items
> should be saved in buffer before send it to server node
>  public void perNodeParallelOperations(int parallelOps) - defines how
> many buffers can be sent to the node without acknowledge that it was
> processed

Best Regards,
Ivan Rakov

On 19.02.2018 22:24, lawrencefinn wrote:

> Okay I am trying to reproduce.  It hasn't got stuck yet, but the client got
> disconnected and reconnected recently.  I don't think it is related to GC
> because I am recording GC times and it does not jump up that much.  Could
> the system get slow on a lot of io?  i see this in the ignite log:
>
> [19:13:01,988][WARNING][grid-timeout-worker-#71][diagnostic] Found long
> running cache future [startTime=19:11:56.656, curTime=19:13:01.911,
>   fut=GridNearAtomicSingleUpdateFuture [reqState=Primary
> [id=62a2a255-3320-4040-aa23-ffb86dec7586, opRes=false, expCnt=-1, rcvdCnt=0,
> primaryRes=false, done=false, waitFor=null, rcvd=null],
> super=GridNearAtomicAbstractUpdateFuture [remapCnt=100,
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=14], remapTopVer=null,
> err=null, futId=313296239, super=GridFutureAdapter [ignoreInterrupts=false,
> state=INIT, res=null, hash=1229092316]]]]
> [19:13:01,988][WARNING][grid-timeout-worker-#71][diagnostic] Found long
> running cache future [startTime=19:11:39.917, curTime=19:13:01.911,
> fut=GridNearAtomicSingleUpdateFuture [reqState=Primary
> [id=62a2a255-3320-4040-aa23-ffb86dec7586, opRes=false, expCnt=-1, rcvdCnt=0,
> primaryRes=false, done=false, waitFor=null, rcvd=null],
> super=GridNearAtomicAbstractUpdateFuture [remapCnt=100,
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=14], remapTopVer=null,
> err=null, futId=312914655, super=GridFutureAdapter [ignoreInterrupts=false,
> state=INIT, res=null, hash=15435296]]]]
> [19:13:51,057][INFO][db-checkpoint-thread-#110][GridCacheDatabaseSharedManager]
> Checkpoint started [checkpointId=77744626-04e6-4e17-bda7-23ecb50bbe19,
> startPtr=FileWALPointer [idx=9600, fileOffset=35172819, len=124303,
> forceFlush=true], checkpointLockWait=57708ms, checkpointLockHoldTime=64ms,
> pages=3755135, reason='too many dirty pages']
> [19:14:01,919][INFO][grid-timeout-worker-#71][IgniteKernal]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>      ^-- Node [id=62a2a255, uptime=01:42:41.752]
>      ^-- H/N/C [hosts=2, nodes=3, CPUs=64]
>      ^-- CPU [cur=77.83%, avg=39.11%, GC=0.13%]
>      ^-- PageMemory [pages=5111642]
>      ^-- Heap [used=11669MB, free=43.02%, comm=20480MB]
>      ^-- Non heap [used=67MB, free=95.56%, comm=69MB]
>      ^-- Public thread pool [active=0, idle=0, qSize=0]
>      ^-- System thread pool [active=0, idle=6, qSize=0]
>      ^-- Outbound messages queue [size=0]
> [19:15:03,470][INFO][tcp-disco-srvr-#2][TcpDiscoverySpi] TCP discovery
> accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=33542]
> [19:15:03,470][INFO][tcp-disco-srvr-#2][TcpDiscoverySpi] TCP discovery
> spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=33542]
>
>
> My app log has:
> 2018-02-19 19:15:02,176 [WARN] from
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi in
> tcp-client-disco-reconnector-#5%cabooseGrid% - Timed out waiting for message
> to be read (most probably, the reason is long GC pauses on remote node)
> [curTimeout=5000, rmtAddr=/127.0.0.1:47500, rmtPort=47500]
> 2018-02-19 19:15:02,176 [ERROR] from
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi in
> tcp-client-disco-reconnector-#5%cabooseGrid% - Exception on joining: Failed
> to deserialize object with given class loader:
> sun.misc.Launcher$AppClassLoader@28d93b30
> org.apache.ignite.IgniteCheckedException: Failed to deserialize object with
> given class loader: sun.misc.Launcher$AppClassLoader@28d93b30
>          at
> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:129)
>          at
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>          at
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
>          at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.readMessage(TcpDiscoverySpi.java:1590)
>          at
> org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:627)
>          at
> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:524)
>          at
> org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:124)
>          at
> org.apache.ignite.spi.discovery.tcp.ClientImpl$Reconnector.body(ClientImpl.java:1377)
>          at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Caused by: java.net.SocketTimeoutException: Read timed out
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

lawrencefinn lawrencefinn
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

Should I decrease these?  One other thing to note is im monitoring GC and the
GC times do not correlate with these issues (GC times are pretty low
anyway).  I honestly think that persisting to disk somehow causes things to
freeze up.  Could it be an AWS related issue?  Im using EBS IO1 with 20,000
IOPS, one driver for persistence and one for wal.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ivan Rakov Ivan Rakov
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

Decreasing perNodeBufferSize and setting perNodeParallelOperations to
server_cpu_cores * 2 may help if GC load is high.

Yes, it can be AWS related issue, either connectivity or disk I/O. There
were reports about cluster segmentation due to network problems on AWS.
Please share full server/client logs and full thread dumps taken at the
moment of freeze. It will certainly help in analysis.

Best Regards,
Ivan Rakov

On 20.02.2018 22:24, lawrencefinn wrote:

> Should I decrease these?  One other thing to note is im monitoring GC and the
> GC times do not correlate with these issues (GC times are pretty low
> anyway).  I honestly think that persisting to disk somehow causes things to
> freeze up.  Could it be an AWS related issue?  Im using EBS IO1 with 20,000
> IOPS, one driver for persistence and one for wal.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

lawrencefinn lawrencefinn
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

I'll give that tweaking a try.  It's hard to do a thread dump just when it
freezes, do you think there is harm in doing a thread dump every 10 seconds
or something?

I tried a new setup with more nodes to test out how that affects this
problem (from 2 to 4).  I saw less datastreaming errors, but it appears that
the client disconnects pretty frequently.  The client is running on the same
machine as the server it is connecting to, so i dont see a network issue
being real).  The time seems to overlap with checkpoints starting.  Here is
the log from the client:

2018-02-21 03:14:19,720 [ERROR] from
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi in
tcp-client-disco-sock-writer-#2%cabooseGrid% - Failed to send message: null
java.io.IOException: Failed to get acknowledge for message:
TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=7a5d3c5b161-6b443ecd-f658-4782-8f8e-3ed6d6407fc1,
verifierNodeId=null, topVer=0, pendin
gIdx=0, failedNodes=null, isClient=true]]
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1276)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
2018-02-21 03:14:19,720 [ERROR] from
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi in
tcp-client-disco-sock-reader-#3%cabooseGrid% - Failed to read message
[sock=Socket[addr=/127.0.0.1,port=47500,localport=59446],
locNodeId=6b443ecd-f658-4782-8f8e-3ed6d6407fc1,
rmtNodeId=93be80c3-c2b5-498b-a897-265c8bacb648]
org.apache.ignite.IgniteCheckedException: Failed to deserialize object with
given class loader: sun.misc.Launcher$AppClassLoader@28d93b30
        at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:129)
        at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
        at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketReader.body(ClientImpl.java:1001)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.net.SocketException: Socket closed
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        at
org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
        at
java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2338)
        at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2351)
        at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)
        at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
        at
org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
        at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
        ... 4 common frames omitted
2018-02-21 03:14:19,720 [ERROR] from
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi in
tcp-client-disco-sock-reader-#3%cabooseGrid% - Connection failed
[sock=Socket[addr=/127.0.0.1,port=47500,localport=59446],
locNodeId=6b443ecd-f658-4782-8f8e-3ed6d6407fc1]
java.net.SocketException: Socket closed
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        at
org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
        at
java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2338)
        at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2351)
        at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)
        at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
        at
org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
        at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
        at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
        at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketReader.body(ClientImpl.java:1001)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)


On the server:

[03:13:21,146][INFO][db-checkpoint-thread-#77][GridCacheDatabaseSharedManager]
Checkpoint started [checkpointId=e2648942-961a-45c4-a7f9-41129e76b70f,
startPtr=FileWALPointer [idx=2131, fileOffset=1035404, len=60889,
forceFlush=true], checkpointLockWait=0ms, checkpointLockHoldTime=362ms,
pages=801041, reason='timeout']
[03:13:42,179][INFO][grid-timeout-worker-#47][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=93be80c3, uptime=02:46:00.162]
    ^-- H/N/C [hosts=4, nodes=5, CPUs=32]
    ^-- CPU [cur=0.13%, avg=16.01%, GC=0%]
    ^-- PageMemory [pages=1279170]
    ^-- Heap [used=6044MB, free=50.81%, comm=12288MB]
    ^-- Non heap [used=63MB, free=95.82%, comm=68MB]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=6, qSize=0]
    ^-- Outbound messages queue [size=0]
[03:14:36,112][INFO][tcp-disco-srvr-#2][TcpDiscoverySpi] TCP discovery
accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=44204]
[03:14:36,119][INFO][tcp-disco-srvr-#2][TcpDiscoverySpi] TCP discovery
spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=44204]
[03:14:36,119][INFO][tcp-disco-sock-reader-#18][TcpDiscoverySpi] Finished
serving remote node connection [rmtAddr=/127.0.0.1:59446, rmtPort=59446
[03:14:36,119][INFO][tcp-disco-sock-reader-#20][TcpDiscoverySpi] Started
serving remote node connection [rmtAddr=/127.0.0.1:44204, rmtPort=44204]
[03:14:36,596][INFO][db-checkpoint-thread-#77][GridCacheDatabaseSharedManager]
Checkpoint finished [cpId=e2648942-961a-45c4-a7f9-41129e76b70f,
pages=801041, markPos=FileWALPointer [idx=2131, fileOffset=1035404,
len=60889, forceFlush=true], walSegmentsCleared=50, markDuration=913ms,
pagesWrite=3798ms, fsync=71652ms, total=76363ms]



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
lawrencefinn lawrencefinn
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

Here is a thread dump from when the client disconnects.  looks like the only
thread doing something interesting is db-checkpoint-thread

018-02-21 19:31:36
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode):

"tcp-disco-client-message-worker-#36" #1128 prio=10 os_prio=0
tid=0x00007f800c001800 nid=0x2d89 waiting on condition [0x00007f7f37afb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000075d9d7790> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at
java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6643)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"tcp-disco-sock-reader-#34" #1126 prio=10 os_prio=0 tid=0x00007f7f74001800
nid=0x2d88 runnable [0x00007f7b20400000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked <0x000000075d812028> (a java.io.BufferedInputStream)
        at
org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
        at
java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2338)
        at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2351)
        at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)
        at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
        at
org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
        at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
        at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
        at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"sys-#1062" #1125 prio=5 os_prio=0 tid=0x00007f7f24031800 nid=0x2d86 waiting
on condition [0x00007f7f3477b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c0045108> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None
"sys-#1062" #1125 prio=5 os_prio=0 tid=0x00007f7f24031800 nid=0x2d86 waiting
on condition [0x00007f7f3477b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c0045108> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"sys-#1061" #1123 prio=5 os_prio=0 tid=0x00007f7f2402e800 nid=0x2d7b waiting
on condition [0x00007f7b1fcd3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c0045108> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"Attach Listener" #913 daemon prio=9 os_prio=0 tid=0x00007f7fb4027000
nid=0x19f5 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#106" #141 prio=5 os_prio=0 tid=0x00007f7fb403f800
nid=0x13b9 waiting on condition [0x00007f7b1eec7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#105" #140 prio=5 os_prio=0 tid=0x00007f7fb403e800
nid=0x13b8 waiting on condition [0x00007f7b1edc6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#104" #139 prio=5 os_prio=0 tid=0x00007f7fb403d000
nid=0x13b7 waiting on condition [0x00007f7b1ecc5000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#103" #138 prio=5 os_prio=0 tid=0x00007f7fb403c000
nid=0x13b6 waiting on condition [0x00007f7b1e976000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#102" #137 prio=5 os_prio=0 tid=0x00007f7fb403b000
nid=0x13b5 waiting on condition [0x00007f7b1ebc4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#101" #136 prio=5 os_prio=0 tid=0x00007f7fb403a000
nid=0x13b4 waiting on condition [0x00007f7b1fefb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#101" #136 prio=5 os_prio=0 tid=0x00007f7fb403a000
nid=0x13b4 waiting on condition [0x00007f7b1fefb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#100" #135 prio=5 os_prio=0 tid=0x00007f7fb4038800
nid=0x13b3 waiting on condition [0x00007f7b1f9d2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#99" #134 prio=5 os_prio=0 tid=0x00007f7fb4037800
nid=0x13b2 waiting on condition [0x00007f7b1f8d1000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None
"checkpoint-runner-#98" #133 prio=5 os_prio=0 tid=0x00007f7fb4036800
nid=0x13b1 waiting on condition [0x00007f7b1f7d0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#97" #132 prio=5 os_prio=0 tid=0x00007f7fb4035800
nid=0x13b0 waiting on condition [0x00007f7b1eac3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#96" #131 prio=5 os_prio=0 tid=0x00007f7fb4034800
nid=0x13af waiting on condition [0x00007f7b1f4cd000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#95" #130 prio=5 os_prio=0 tid=0x00007f7fb4033800
nid=0x13ae waiting on condition [0x00007f7b1f3cc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#94" #129 prio=5 os_prio=0 tid=0x00007f7fb4032800
nid=0x13ad waiting on condition [0x00007f7b1f2cb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#93" #128 prio=5 os_prio=0 tid=0x00007f7fb4032000
nid=0x13ac waiting on condition [0x00007f7b1f1ca000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#92" #127 prio=5 os_prio=0 tid=0x00007f7fb4031000
nid=0x13ab waiting on condition [0x00007f7f34a7c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"checkpoint-runner-#91" #126 prio=5 os_prio=0 tid=0x00007f7fb402f000
nid=0x13aa waiting on condition [0x00007f7f34278000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c1231e90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"ttl-cleanup-worker-#61" #96 prio=5 os_prio=0 tid=0x00007f7f8429a800
nid=0x12f2 sleeping[0x00007f7b1fdfa000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7533)
        at
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"srvc-deploy-#58" #93 prio=5 os_prio=0 tid=0x00007f7fb000b000 nid=0x12e9
waiting on condition [0x00007f7b1fffc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c080cb28> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"db-checkpoint-thread-#56" #91 prio=5 os_prio=0 tid=0x00007f7f84247800
nid=0x12e6 runnable [0x00007f7b201fe000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.FileDispatcherImpl.force0(Native Method)
        at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
        at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)
        at
org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIO.force(RandomAccessFileIO.java:87)
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.sync(FilePageStore.java:495)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:2156)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:2010)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - <0x00000004c2b000e0> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)

"wal-file-archiver%null" #86 prio=5 os_prio=0 tid=0x00007f7f84004000
nid=0x12e1 in Object.wait() [0x00007f7f34579000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.run(FileWriteAheadLogManager.java:1228)
        - locked <0x00000004c076f208> (a
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver)

   Locked ownable synchronizers:
        - None

"tcp-disco-sock-reader-#8" #82 prio=10 os_prio=0 tid=0x00007f7f74003800
nid=0x1247 runnable [0x00007f7f352db000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked <0x00000004c0829b50> (a java.io.BufferedInputStream)
        at
org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
        at
java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2338)
        at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2351)
        at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)
        at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
        at
org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
        at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
        at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
        at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"session-timeout-worker-#42" #63 daemon prio=5 os_prio=0
tid=0x00007f804dd6d000 nid=0x1240 sleeping[0x00007f7f353dc000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.processors.rest.GridRestProcessor$4.body(GridRestProcessor.java:437)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"exchange-worker-#50" #74 prio=5 os_prio=0 tid=0x00007f804dd79000 nid=0x123f
waiting on condition [0x00007f7f354dd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c0878178> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at
java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2225)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"ignite-update-notifier-timer" #73 daemon prio=5 os_prio=0
tid=0x00007f804dd77800 nid=0x123e in Object.wait() [0x00007f7f355de000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x00000004c0878348> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

   Locked ownable synchronizers:
        - None

"upd-ver-checker" #72 daemon prio=5 os_prio=0 tid=0x00007f804dd76000
nid=0x123d waiting on condition [0x00007f7f356df000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.processors.cluster.GridUpdateNotifier$1.run(GridUpdateNotifier.java:115)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"disco-event-worker-#49" #71 prio=5 os_prio=0 tid=0x00007f804dd70000
nid=0x123c waiting on condition [0x00007f7f357e0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c004ec38> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2552)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2534)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"tcp-disco-srvr-#2" #56 prio=10 os_prio=0 tid=0x00007f804dd63800 nid=0x123b
runnable [0x00007f7f358e1000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5623)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"tcp-disco-msg-worker-#3" #70 prio=10 os_prio=0 tid=0x00007f804dd60800
nid=0x123a runnable [0x00007f7f359e2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f8698> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at
java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6643)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2533)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"grid-data-loader-flusher-#48" #69 prio=5 os_prio=0 tid=0x00007f804dd49800
nid=0x1239 waiting on condition [0x00007f7f35ce3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c0878748> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:108)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None
"grid-nio-worker-tcp-rest-3-#47" #68 prio=5 os_prio=0 tid=0x00007f804dd3f000
nid=0x1238 runnable [0x00007f7f35de4000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c09728b8> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c09728a8> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c09728d8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-rest-2-#46" #67 prio=5 os_prio=0 tid=0x00007f804dd3d000
nid=0x1237 runnable [0x00007f7f35ee5000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c08f8890> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c08f8880> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c08f88b0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-rest-1-#45" #66 prio=5 os_prio=0 tid=0x00007f804dcf7800
nid=0x1236 runnable [0x00007f7f35fe6000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c08789c0> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c08789b0> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c08789e0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None
"grid-nio-worker-tcp-rest-0-#44" #65 prio=5 os_prio=0 tid=0x00007f804dcf6000
nid=0x1235 runnable [0x00007f7f360e7000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c076f9b8> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c076f9a8> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c076f9d8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"nio-acceptor-#43" #64 prio=5 os_prio=0 tid=0x00007f804dcf4800 nid=0x1234
runnable [0x00007f7f361e8000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c0878c78> (a sun.nio.ch.Util$2)
        - locked <0x00000004c0878c68> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c0878c88> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2772)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2744)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-3-#41" #62 prio=5 os_prio=0
tid=0x00007f804dcd3800 nid=0x1233 runnable [0x00007f7f362e9000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c076fc70> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c076fc60> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c076fc90> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-2-#40" #61 prio=5 os_prio=0
tid=0x00007f804dcd2000 nid=0x1232 runnable [0x00007f7f363ea000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c08f8b48> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c08f8b38> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c08f8b68> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-1-#39" #60 prio=5 os_prio=0
tid=0x00007f804dcd0800 nid=0x1231 runnable [0x00007f7f364eb000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c076ff28> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c076ff18> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c076ff48> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-0-#38" #59 prio=5 os_prio=0
tid=0x00007f804dcce800 nid=0x1230 runnable [0x00007f7f365ec000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c0878ee0> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c0878ed0> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c0878f00> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None
"nio-acceptor-#37" #58 prio=5 os_prio=0 tid=0x00007f804dc8a000 nid=0x122f
runnable [0x00007f7f366ed000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c08f8e00> (a sun.nio.ch.Util$2)
        - locked <0x00000004c08f8df0> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c08f8e10> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2772)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2744)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"tcp-comm-worker-#1" #55 prio=5 os_prio=0 tid=0x00007f804dab0800 nid=0x122e
waiting on condition [0x00007f7f36dee000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c004ee70> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4037)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-3-#36" #51 prio=5 os_prio=0 tid=0x00007f804daae800
nid=0x122d runnable [0x00007f7f36eef000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c004f0f0> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c004f0e0> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c004f110> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-2-#35" #50 prio=5 os_prio=0 tid=0x00007f804daad000
nid=0x122c runnable [0x00007f7f36ff0000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c004f3a0> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c004f390> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c004f3c0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-1-#34" #49 prio=5 os_prio=0 tid=0x00007f804daac000
nid=0x122b runnable [0x00007f7f371f2000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c004f690> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c004f680> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c004f6b0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-0-#33" #48 prio=5 os_prio=0 tid=0x00007f804da5e000
nid=0x122a runnable [0x00007f7f372f3000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c004f940> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c004f930> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c004f960> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"nio-acceptor-#32" #47 prio=5 os_prio=0 tid=0x00007f804daaa800 nid=0x1229
runnable [0x00007f7f370f1000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c0045df0> (a sun.nio.ch.Util$2)
        - locked <0x00000004c0045de0> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c0045e00> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2772)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2744)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"grid-timeout-worker-#31" #46 prio=5 os_prio=0 tid=0x00007f804da41000
nid=0x1225 in Object.wait() [0x00007f7f373f4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:195)
        - locked <0x00000004c0046000> (a java.lang.Object)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"ignite-clock" #45 daemon prio=10 os_prio=0 tid=0x00007f804da4a000
nid=0x1224 sleeping[0x00007f7f374f5000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.util.IgniteUtils$22.run(IgniteUtils.java:3263)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-15-#24" #38 prio=5 os_prio=0 tid=0x00007f804d949800
nid=0x121d waiting on condition [0x00007f7f37dfc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-14-#23" #37 prio=5 os_prio=0 tid=0x00007f804d948000
nid=0x121c waiting on condition [0x00007f7f37efd000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-13-#22" #36 prio=5 os_prio=0 tid=0x00007f804d946000
nid=0x121b waiting on condition [0x00007f7f37ffe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-12-#21" #35 prio=5 os_prio=0 tid=0x00007f804d944800
nid=0x121a waiting on condition [0x00007f7fc41ee000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-11-#20" #34 prio=5 os_prio=0 tid=0x00007f804d943000
nid=0x1219 waiting on condition [0x00007f7fc42ef000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-10-#19" #33 prio=5 os_prio=0 tid=0x00007f804d941000
nid=0x1218 waiting on condition [0x00007f7fc43f0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-9-#18" #32 prio=5 os_prio=0 tid=0x00007f804d93f800
nid=0x1217 waiting on condition [0x00007f7fc44f1000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-8-#17" #31 prio=5 os_prio=0 tid=0x00007f804d93d800
nid=0x1216 waiting on condition [0x00007f7fc45f2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-7-#16" #30 prio=5 os_prio=0 tid=0x00007f804d93b800
nid=0x1215 waiting on condition [0x00007f7fc46f3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-6-#15" #29 prio=5 os_prio=0 tid=0x00007f804d939800
nid=0x1214 waiting on condition [0x00007f7fc47f4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-5-#14" #28 prio=5 os_prio=0 tid=0x00007f804d937800
nid=0x1213 waiting on condition [0x00007f7fc48f5000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-4-#13" #27 prio=5 os_prio=0 tid=0x00007f804d936000
nid=0x1212 waiting on condition [0x00007f7fc49f6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-3-#12" #26 prio=5 os_prio=0 tid=0x00007f804d934000
nid=0x1211 waiting on condition [0x00007f7fc4af7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-2-#11" #25 prio=5 os_prio=0 tid=0x00007f804d932800
nid=0x1210 waiting on condition [0x00007f7fc4bf8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-1-#10" #24 prio=5 os_prio=0 tid=0x00007f804d930800
nid=0x120f waiting on condition [0x00007f7fc4cf9000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-0-#9" #23 prio=5 os_prio=0 tid=0x00007f804d92f000
nid=0x120e waiting on condition [0x00007f7fc4dfa000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"sys-stripe-7-#8" #22 prio=5 os_prio=0 tid=0x00007f804d92d000 nid=0x120d
waiting on condition [0x00007f7fc4efb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"sys-stripe-6-#7" #21 prio=5 os_prio=0 tid=0x00007f804d92b800 nid=0x120c
waiting on condition [0x00007f7fc4ffc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"sys-stripe-5-#6" #20 prio=5 os_prio=0 tid=0x00007f804d92a000 nid=0x120b
waiting on condition [0x00007f7fc50fd000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"sys-stripe-4-#5" #19 prio=5 os_prio=0 tid=0x00007f804d928000 nid=0x120a
waiting on condition [0x00007f7fc51fe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"sys-stripe-3-#4" #18 prio=5 os_prio=0 tid=0x00007f804d926800 nid=0x1209
waiting on condition [0x00007f7fc52ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"sys-stripe-2-#3" #17 prio=5 os_prio=0 tid=0x00007f804d924800 nid=0x1208
waiting on condition [0x00007f7fc5400000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"sys-stripe-1-#2" #16 prio=5 os_prio=0 tid=0x00007f804d91e800 nid=0x1207
waiting on condition [0x00007f7fc5501000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

"sys-stripe-0-#1" #15 prio=5 os_prio=0 tid=0x00007f804d91a800 nid=0x1206
waiting on condition [0x00007f7fc5602000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None


"Service Thread" #10 daemon prio=9 os_prio=0 tid=0x00007f804d3e6800
nid=0x1201 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C1 CompilerThread3" #9 daemon prio=9 os_prio=0 tid=0x00007f804d3e3800
nid=0x1200 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f804d3e1800
nid=0x11ff waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f804d3df800
nid=0x11fe waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f804d3dd800
nid=0x11fd waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f804d3db000
nid=0x11fc runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
tid=0x00007f804d3da000 nid=0x11fb waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f804d3a7000 nid=0x11fa in
Object.wait() [0x00007f7fc73fc000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000004c0045858> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

   Locked ownable synchronizers:
        - None

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f804d3a2800
nid=0x11f9 in Object.wait() [0x00007f7fc74fd000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x00000004c0049648> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

   Locked ownable synchronizers:
        - None

"main" #1 prio=5 os_prio=0 tid=0x00007f804c00b800 nid=0x11e3 waiting on
condition [0x00007f8052502000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f8ea8> (a
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:335)

   Locked ownable synchronizers:
        - None

"VM Thread" os_prio=0 tid=0x00007f804d39b000 nid=0x11f8 runnable

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c023000
nid=0x11e4 runnable

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c024800
nid=0x11e5 runnable

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c026800
nid=0x11e6 runnable

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c028000
nid=0x11e7 runnable

"Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c02a000
nid=0x11e8 runnable

"Gang worker#5 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c02b800
nid=0x11e9 runnable

"Gang worker#6 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c02d800
nid=0x11ea runnable

"Gang worker#7 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c02f000
nid=0x11eb runnable

"G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007f804c050800
nid=0x11f5 runnable

"Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0
tid=0x00007f804c052800 nid=0x11f6 runnable
"Gang worker#1 (G1 Parallel Marking Threads)" os_prio=0
tid=0x00007f804c054000 nid=0x11f7 runnable

"G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007f804c040000
nid=0x11f4 runnable

"G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007f804c03e800
nid=0x11f3 runnable

"G1 Concurrent Refinement Thread#2" os_prio=0 tid=0x00007f804c03c800
nid=0x11f2 runnable

"G1 Concurrent Refinement Thread#3" os_prio=0 tid=0x00007f804c03b000
nid=0x11f1 runnable

"G1 Concurrent Refinement Thread#4" os_prio=0 tid=0x00007f804c039000
nid=0x11f0 runnable

"G1 Concurrent Refinement Thread#5" os_prio=0 tid=0x00007f804c037000
nid=0x11ef runnable

"G1 Concurrent Refinement Thread#6" os_prio=0 tid=0x00007f804c035800
nid=0x11ee runnable

"G1 Concurrent Refinement Thread#7" os_prio=0 tid=0x00007f804c033800
nid=0x11ed runnable

"G1 Concurrent Refinement Thread#8" os_prio=0 tid=0x00007f804c032000
nid=0x11ec runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f804d5ee000 nid=0x1205
waiting on condition

JNI global references: 267



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ivan Rakov Ivan Rakov
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

> [03:14:36,596][INFO][db-checkpoint-thread-#77][GridCacheDatabaseSharedManager]
> Checkpoint finished [cpId=e2648942-961a-45c4-a7f9-41129e76b70f,
> pages=801041, markPos=FileWALPointer [idx=2131, fileOffset=1035404,
> len=60889, forceFlush=true], walSegmentsCleared=50, markDuration=913ms,
> pagesWrite=3798ms, fsync=71652ms, total=76363ms]

 From this message I can conclude that you have quite slow disks.
Checkpointing of 800K pages (3.2G of data) took 76 seconds, and FSYNC
took 71 seconds of them.
Freeze is expected when your disk bandwidth is way lower than you data
load bandwidth. Node can accept limited amount of updates during
checkpoint. If that limit is exceeded, all update operations will block
until the end of current checkpoint.
Ignite has write throttling mechanism (see
https://apacheignite.readme.io/docs/durable-memory-tuning) to avoid such
situations, but in 2.3 it's not tuned for the case when FSYNC takes most
of checkpoint time. Improved version of throttling that adapts to long
FSYNC will be released in 2.5:
https://issues.apache.org/jira/browse/IGNITE-7533

However, I still can't say why your client disconnects from server. The
only explanation I see that it's artifact of AWS virtual environment:
excessive use of disk I/O during checkpoint somehow affects quality of
network connection. Try to decrease throughput of data loading (your
hardware can't handle more than 40MB/sec anyway) and see how system will
react.
You can achieve lower throughput with tuning your data streaming, or
don't using data streamer at all.

Best Regards,
Ivan Rakov

On 21.02.2018 22:56, lawrencefinn wrote:

> Here is a thread dump from when the client disconnects.  looks like the only
> thread doing something interesting is db-checkpoint-thread
>
> 018-02-21 19:31:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode):
>
> "tcp-disco-client-message-worker-#36" #1128 prio=10 os_prio=0
> tid=0x00007f800c001800 nid=0x2d89 waiting on condition [0x00007f7f37afb000]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x000000075d9d7790> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at
> java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
>          at
> java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
>          at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6643)
>          at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
>     Locked ownable synchronizers:
>          - None
>
> "tcp-disco-sock-reader-#34" #1126 prio=10 os_prio=0 tid=0x00007f7f74001800
> nid=0x2d88 runnable [0x00007f7b20400000]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>          at java.net.SocketInputStream.read(SocketInputStream.java:170)
>          at java.net.SocketInputStream.read(SocketInputStream.java:141)
>          at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>          at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>          at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>          - locked <0x000000075d812028> (a java.io.BufferedInputStream)
>          at
> org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
>          at
> java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2338)
>          at
> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2351)
>          at
> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)
>          at
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
>          at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
>          at
> org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
>          at
> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
>          at
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>          at
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
>          at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
>          at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
>     Locked ownable synchronizers:
>          - None
>
> "sys-#1062" #1125 prio=5 os_prio=0 tid=0x00007f7f24031800 nid=0x2d86 waiting
> on condition [0x00007f7f3477b000]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c0045108> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
> "sys-#1062" #1125 prio=5 os_prio=0 tid=0x00007f7f24031800 nid=0x2d86 waiting
> on condition [0x00007f7f3477b000]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c0045108> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "sys-#1061" #1123 prio=5 os_prio=0 tid=0x00007f7f2402e800 nid=0x2d7b waiting
> on condition [0x00007f7b1fcd3000]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c0045108> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "Attach Listener" #913 daemon prio=9 os_prio=0 tid=0x00007f7fb4027000
> nid=0x19f5 waiting on condition [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#106" #141 prio=5 os_prio=0 tid=0x00007f7fb403f800
> nid=0x13b9 waiting on condition [0x00007f7b1eec7000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#105" #140 prio=5 os_prio=0 tid=0x00007f7fb403e800
> nid=0x13b8 waiting on condition [0x00007f7b1edc6000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#104" #139 prio=5 os_prio=0 tid=0x00007f7fb403d000
> nid=0x13b7 waiting on condition [0x00007f7b1ecc5000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#103" #138 prio=5 os_prio=0 tid=0x00007f7fb403c000
> nid=0x13b6 waiting on condition [0x00007f7b1e976000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#102" #137 prio=5 os_prio=0 tid=0x00007f7fb403b000
> nid=0x13b5 waiting on condition [0x00007f7b1ebc4000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#101" #136 prio=5 os_prio=0 tid=0x00007f7fb403a000
> nid=0x13b4 waiting on condition [0x00007f7b1fefb000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#101" #136 prio=5 os_prio=0 tid=0x00007f7fb403a000
> nid=0x13b4 waiting on condition [0x00007f7b1fefb000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#100" #135 prio=5 os_prio=0 tid=0x00007f7fb4038800
> nid=0x13b3 waiting on condition [0x00007f7b1f9d2000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#99" #134 prio=5 os_prio=0 tid=0x00007f7fb4037800
> nid=0x13b2 waiting on condition [0x00007f7b1f8d1000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
> "checkpoint-runner-#98" #133 prio=5 os_prio=0 tid=0x00007f7fb4036800
> nid=0x13b1 waiting on condition [0x00007f7b1f7d0000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#97" #132 prio=5 os_prio=0 tid=0x00007f7fb4035800
> nid=0x13b0 waiting on condition [0x00007f7b1eac3000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#96" #131 prio=5 os_prio=0 tid=0x00007f7fb4034800
> nid=0x13af waiting on condition [0x00007f7b1f4cd000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#95" #130 prio=5 os_prio=0 tid=0x00007f7fb4033800
> nid=0x13ae waiting on condition [0x00007f7b1f3cc000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#94" #129 prio=5 os_prio=0 tid=0x00007f7fb4032800
> nid=0x13ad waiting on condition [0x00007f7b1f2cb000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#93" #128 prio=5 os_prio=0 tid=0x00007f7fb4032000
> nid=0x13ac waiting on condition [0x00007f7b1f1ca000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#92" #127 prio=5 os_prio=0 tid=0x00007f7fb4031000
> nid=0x13ab waiting on condition [0x00007f7f34a7c000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "checkpoint-runner-#91" #126 prio=5 os_prio=0 tid=0x00007f7fb402f000
> nid=0x13aa waiting on condition [0x00007f7f34278000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c1231e90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "ttl-cleanup-worker-#61" #96 prio=5 os_prio=0 tid=0x00007f7f8429a800
> nid=0x12f2 sleeping[0x00007f7b1fdfa000]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>          at java.lang.Thread.sleep(Native Method)
>          at
> org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7533)
>          at
> org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "srvc-deploy-#58" #93 prio=5 os_prio=0 tid=0x00007f7fb000b000 nid=0x12e9
> waiting on condition [0x00007f7b1fffc000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c080cb28> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "db-checkpoint-thread-#56" #91 prio=5 os_prio=0 tid=0x00007f7f84247800
> nid=0x12e6 runnable [0x00007f7b201fe000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.FileDispatcherImpl.force0(Native Method)
>          at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
>          at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)
>          at
> org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIO.force(RandomAccessFileIO.java:87)
>          at
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.sync(FilePageStore.java:495)
>          at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:2156)
>          at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:2010)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - <0x00000004c2b000e0> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
>
> "wal-file-archiver%null" #86 prio=5 os_prio=0 tid=0x00007f7f84004000
> nid=0x12e1 in Object.wait() [0x00007f7f34579000]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:502)
>          at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.run(FileWriteAheadLogManager.java:1228)
>          - locked <0x00000004c076f208> (a
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver)
>
>     Locked ownable synchronizers:
>          - None
>
> "tcp-disco-sock-reader-#8" #82 prio=10 os_prio=0 tid=0x00007f7f74003800
> nid=0x1247 runnable [0x00007f7f352db000]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>          at java.net.SocketInputStream.read(SocketInputStream.java:170)
>          at java.net.SocketInputStream.read(SocketInputStream.java:141)
>          at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>          at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>          at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>          - locked <0x00000004c0829b50> (a java.io.BufferedInputStream)
>          at
> org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
>          at
> java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2338)
>          at
> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2351)
>          at
> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)
>          at
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
>          at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
>          at
> org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
>          at
> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
>          at
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>          at
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
>          at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
>          at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
>     Locked ownable synchronizers:
>          - None
>
> "session-timeout-worker-#42" #63 daemon prio=5 os_prio=0
> tid=0x00007f804dd6d000 nid=0x1240 sleeping[0x00007f7f353dc000]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>          at java.lang.Thread.sleep(Native Method)
>          at
> org.apache.ignite.internal.processors.rest.GridRestProcessor$4.body(GridRestProcessor.java:437)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "exchange-worker-#50" #74 prio=5 os_prio=0 tid=0x00007f804dd79000 nid=0x123f
> waiting on condition [0x00007f7f354dd000]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c0878178> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at
> java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
>          at
> java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
>          at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2225)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "ignite-update-notifier-timer" #73 daemon prio=5 os_prio=0
> tid=0x00007f804dd77800 nid=0x123e in Object.wait() [0x00007f7f355de000]
>     java.lang.Thread.State: TIMED_WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.util.TimerThread.mainLoop(Timer.java:552)
>          - locked <0x00000004c0878348> (a java.util.TaskQueue)
>          at java.util.TimerThread.run(Timer.java:505)
>
>     Locked ownable synchronizers:
>          - None
>
> "upd-ver-checker" #72 daemon prio=5 os_prio=0 tid=0x00007f804dd76000
> nid=0x123d waiting on condition [0x00007f7f356df000]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>          at java.lang.Thread.sleep(Native Method)
>          at
> org.apache.ignite.internal.processors.cluster.GridUpdateNotifier$1.run(GridUpdateNotifier.java:115)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "disco-event-worker-#49" #71 prio=5 os_prio=0 tid=0x00007f804dd70000
> nid=0x123c waiting on condition [0x00007f7f357e0000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c004ec38> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2552)
>          at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2534)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "tcp-disco-srvr-#2" #56 prio=10 os_prio=0 tid=0x00007f804dd63800 nid=0x123b
> runnable [0x00007f7f358e1000]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.PlainSocketImpl.socketAccept(Native Method)
>          at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
>          at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>          at java.net.ServerSocket.accept(ServerSocket.java:513)
>          at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5623)
>          at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
>     Locked ownable synchronizers:
>          - None
>
> "tcp-disco-msg-worker-#3" #70 prio=10 os_prio=0 tid=0x00007f804dd60800
> nid=0x123a runnable [0x00007f7f359e2000]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c08f8698> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at
> java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
>          at
> java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
>          at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6643)
>          at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2533)
>          at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
>     Locked ownable synchronizers:
>          - None
>
> "grid-data-loader-flusher-#48" #69 prio=5 os_prio=0 tid=0x00007f804dd49800
> nid=0x1239 waiting on condition [0x00007f7f35ce3000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c0878748> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
>          at
> org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:108)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
> "grid-nio-worker-tcp-rest-3-#47" #68 prio=5 os_prio=0 tid=0x00007f804dd3f000
> nid=0x1238 runnable [0x00007f7f35de4000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c09728b8> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>          - locked <0x00000004c09728a8> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c09728d8> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "grid-nio-worker-tcp-rest-2-#46" #67 prio=5 os_prio=0 tid=0x00007f804dd3d000
> nid=0x1237 runnable [0x00007f7f35ee5000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c08f8890> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>          - locked <0x00000004c08f8880> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c08f88b0> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "grid-nio-worker-tcp-rest-1-#45" #66 prio=5 os_prio=0 tid=0x00007f804dcf7800
> nid=0x1236 runnable [0x00007f7f35fe6000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c08789c0> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>          - locked <0x00000004c08789b0> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c08789e0> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
> "grid-nio-worker-tcp-rest-0-#44" #65 prio=5 os_prio=0 tid=0x00007f804dcf6000
> nid=0x1235 runnable [0x00007f7f360e7000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c076f9b8> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>          - locked <0x00000004c076f9a8> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c076f9d8> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "nio-acceptor-#43" #64 prio=5 os_prio=0 tid=0x00007f804dcf4800 nid=0x1234
> runnable [0x00007f7f361e8000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c0878c78> (a sun.nio.ch.Util$2)
>          - locked <0x00000004c0878c68> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c0878c88> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2772)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2744)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "grid-nio-worker-client-listener-3-#41" #62 prio=5 os_prio=0
> tid=0x00007f804dcd3800 nid=0x1233 runnable [0x00007f7f362e9000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c076fc70> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>          - locked <0x00000004c076fc60> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c076fc90> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "grid-nio-worker-client-listener-2-#40" #61 prio=5 os_prio=0
> tid=0x00007f804dcd2000 nid=0x1232 runnable [0x00007f7f363ea000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c08f8b48> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>          - locked <0x00000004c08f8b38> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c08f8b68> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "grid-nio-worker-client-listener-1-#39" #60 prio=5 os_prio=0
> tid=0x00007f804dcd0800 nid=0x1231 runnable [0x00007f7f364eb000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c076ff28> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>          - locked <0x00000004c076ff18> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c076ff48> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "grid-nio-worker-client-listener-0-#38" #59 prio=5 os_prio=0
> tid=0x00007f804dcce800 nid=0x1230 runnable [0x00007f7f365ec000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c0878ee0> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>          - locked <0x00000004c0878ed0> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c0878f00> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
> "nio-acceptor-#37" #58 prio=5 os_prio=0 tid=0x00007f804dc8a000 nid=0x122f
> runnable [0x00007f7f366ed000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c08f8e00> (a sun.nio.ch.Util$2)
>          - locked <0x00000004c08f8df0> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c08f8e10> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2772)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2744)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "tcp-comm-worker-#1" #55 prio=5 os_prio=0 tid=0x00007f804dab0800 nid=0x122e
> waiting on condition [0x00007f7f36dee000]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c004ee70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>          at
> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4037)
>          at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
>     Locked ownable synchronizers:
>          - None
>
> "grid-nio-worker-tcp-comm-3-#36" #51 prio=5 os_prio=0 tid=0x00007f804daae800
> nid=0x122d runnable [0x00007f7f36eef000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c004f0f0> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>          - locked <0x00000004c004f0e0> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c004f110> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "grid-nio-worker-tcp-comm-2-#35" #50 prio=5 os_prio=0 tid=0x00007f804daad000
> nid=0x122c runnable [0x00007f7f36ff0000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c004f3a0> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>          - locked <0x00000004c004f390> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c004f3c0> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "grid-nio-worker-tcp-comm-1-#34" #49 prio=5 os_prio=0 tid=0x00007f804daac000
> nid=0x122b runnable [0x00007f7f371f2000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c004f690> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>          - locked <0x00000004c004f680> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c004f6b0> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "grid-nio-worker-tcp-comm-0-#33" #48 prio=5 os_prio=0 tid=0x00007f804da5e000
> nid=0x122a runnable [0x00007f7f372f3000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c004f940> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>          - locked <0x00000004c004f930> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c004f960> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "nio-acceptor-#32" #47 prio=5 os_prio=0 tid=0x00007f804daaa800 nid=0x1229
> runnable [0x00007f7f370f1000]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>          - locked <0x00000004c0045df0> (a sun.nio.ch.Util$2)
>          - locked <0x00000004c0045de0> (a
> java.util.Collections$UnmodifiableSet)
>          - locked <0x00000004c0045e00> (a sun.nio.ch.EPollSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2772)
>          at
> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2744)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "grid-timeout-worker-#31" #46 prio=5 os_prio=0 tid=0x00007f804da41000
> nid=0x1225 in Object.wait() [0x00007f7f373f4000]
>     java.lang.Thread.State: TIMED_WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:195)
>          - locked <0x00000004c0046000> (a java.lang.Object)
>          at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "ignite-clock" #45 daemon prio=10 os_prio=0 tid=0x00007f804da4a000
> nid=0x1224 sleeping[0x00007f7f374f5000]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>          at java.lang.Thread.sleep(Native Method)
>          at
> org.apache.ignite.internal.util.IgniteUtils$22.run(IgniteUtils.java:3263)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-15-#24" #38 prio=5 os_prio=0 tid=0x00007f804d949800
> nid=0x121d waiting on condition [0x00007f7f37dfc000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-14-#23" #37 prio=5 os_prio=0 tid=0x00007f804d948000
> nid=0x121c waiting on condition [0x00007f7f37efd000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-13-#22" #36 prio=5 os_prio=0 tid=0x00007f804d946000
> nid=0x121b waiting on condition [0x00007f7f37ffe000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-12-#21" #35 prio=5 os_prio=0 tid=0x00007f804d944800
> nid=0x121a waiting on condition [0x00007f7fc41ee000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-11-#20" #34 prio=5 os_prio=0 tid=0x00007f804d943000
> nid=0x1219 waiting on condition [0x00007f7fc42ef000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-10-#19" #33 prio=5 os_prio=0 tid=0x00007f804d941000
> nid=0x1218 waiting on condition [0x00007f7fc43f0000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-9-#18" #32 prio=5 os_prio=0 tid=0x00007f804d93f800
> nid=0x1217 waiting on condition [0x00007f7fc44f1000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-8-#17" #31 prio=5 os_prio=0 tid=0x00007f804d93d800
> nid=0x1216 waiting on condition [0x00007f7fc45f2000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-7-#16" #30 prio=5 os_prio=0 tid=0x00007f804d93b800
> nid=0x1215 waiting on condition [0x00007f7fc46f3000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-6-#15" #29 prio=5 os_prio=0 tid=0x00007f804d939800
> nid=0x1214 waiting on condition [0x00007f7fc47f4000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-5-#14" #28 prio=5 os_prio=0 tid=0x00007f804d937800
> nid=0x1213 waiting on condition [0x00007f7fc48f5000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-4-#13" #27 prio=5 os_prio=0 tid=0x00007f804d936000
> nid=0x1212 waiting on condition [0x00007f7fc49f6000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-3-#12" #26 prio=5 os_prio=0 tid=0x00007f804d934000
> nid=0x1211 waiting on condition [0x00007f7fc4af7000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-2-#11" #25 prio=5 os_prio=0 tid=0x00007f804d932800
> nid=0x1210 waiting on condition [0x00007f7fc4bf8000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-1-#10" #24 prio=5 os_prio=0 tid=0x00007f804d930800
> nid=0x120f waiting on condition [0x00007f7fc4cf9000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "data-streamer-stripe-0-#9" #23 prio=5 os_prio=0 tid=0x00007f804d92f000
> nid=0x120e waiting on condition [0x00007f7fc4dfa000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "sys-stripe-7-#8" #22 prio=5 os_prio=0 tid=0x00007f804d92d000 nid=0x120d
> waiting on condition [0x00007f7fc4efb000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "sys-stripe-6-#7" #21 prio=5 os_prio=0 tid=0x00007f804d92b800 nid=0x120c
> waiting on condition [0x00007f7fc4ffc000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "sys-stripe-5-#6" #20 prio=5 os_prio=0 tid=0x00007f804d92a000 nid=0x120b
> waiting on condition [0x00007f7fc50fd000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "sys-stripe-4-#5" #19 prio=5 os_prio=0 tid=0x00007f804d928000 nid=0x120a
> waiting on condition [0x00007f7fc51fe000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "sys-stripe-3-#4" #18 prio=5 os_prio=0 tid=0x00007f804d926800 nid=0x1209
> waiting on condition [0x00007f7fc52ff000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "sys-stripe-2-#3" #17 prio=5 os_prio=0 tid=0x00007f804d924800 nid=0x1208
> waiting on condition [0x00007f7fc5400000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "sys-stripe-1-#2" #16 prio=5 os_prio=0 tid=0x00007f804d91e800 nid=0x1207
> waiting on condition [0x00007f7fc5501000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
> "sys-stripe-0-#1" #15 prio=5 os_prio=0 tid=0x00007f804d91a800 nid=0x1206
> waiting on condition [0x00007f7fc5602000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>          at
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>          at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>          at java.lang.Thread.run(Thread.java:745)
>
>     Locked ownable synchronizers:
>          - None
>
>
> "Service Thread" #10 daemon prio=9 os_prio=0 tid=0x00007f804d3e6800
> nid=0x1201 runnable [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
>     Locked ownable synchronizers:
>          - None
>
> "C1 CompilerThread3" #9 daemon prio=9 os_prio=0 tid=0x00007f804d3e3800
> nid=0x1200 waiting on condition [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
>     Locked ownable synchronizers:
>          - None
>
> "C2 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f804d3e1800
> nid=0x11ff waiting on condition [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
>     Locked ownable synchronizers:
>          - None
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f804d3df800
> nid=0x11fe waiting on condition [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
>     Locked ownable synchronizers:
>          - None
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f804d3dd800
> nid=0x11fd waiting on condition [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
>     Locked ownable synchronizers:
>          - None
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f804d3db000
> nid=0x11fc runnable [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
>     Locked ownable synchronizers:
>          - None
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007f804d3da000 nid=0x11fb waiting on condition [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
>     Locked ownable synchronizers:
>          - None
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f804d3a7000 nid=0x11fa in
> Object.wait() [0x00007f7fc73fc000]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>          - locked <0x00000004c0045858> (a java.lang.ref.ReferenceQueue$Lock)
>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>          at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
>     Locked ownable synchronizers:
>          - None
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f804d3a2800
> nid=0x11f9 in Object.wait() [0x00007f7fc74fd000]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:502)
>          at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
>          - locked <0x00000004c0049648> (a java.lang.ref.Reference$Lock)
>          at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
>
>     Locked ownable synchronizers:
>          - None
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f804c00b800 nid=0x11e3 waiting on
> condition [0x00007f8052502000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000004c08f8ea8> (a
> java.util.concurrent.CountDownLatch$Sync)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>          at
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>          at
> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:335)
>
>     Locked ownable synchronizers:
>          - None
>
> "VM Thread" os_prio=0 tid=0x00007f804d39b000 nid=0x11f8 runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c023000
> nid=0x11e4 runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c024800
> nid=0x11e5 runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c026800
> nid=0x11e6 runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c028000
> nid=0x11e7 runnable
>
> "Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c02a000
> nid=0x11e8 runnable
>
> "Gang worker#5 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c02b800
> nid=0x11e9 runnable
>
> "Gang worker#6 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c02d800
> nid=0x11ea runnable
>
> "Gang worker#7 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c02f000
> nid=0x11eb runnable
>
> "G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007f804c050800
> nid=0x11f5 runnable
>
> "Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0
> tid=0x00007f804c052800 nid=0x11f6 runnable
> "Gang worker#1 (G1 Parallel Marking Threads)" os_prio=0
> tid=0x00007f804c054000 nid=0x11f7 runnable
>
> "G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007f804c040000
> nid=0x11f4 runnable
>
> "G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007f804c03e800
> nid=0x11f3 runnable
>
> "G1 Concurrent Refinement Thread#2" os_prio=0 tid=0x00007f804c03c800
> nid=0x11f2 runnable
>
> "G1 Concurrent Refinement Thread#3" os_prio=0 tid=0x00007f804c03b000
> nid=0x11f1 runnable
>
> "G1 Concurrent Refinement Thread#4" os_prio=0 tid=0x00007f804c039000
> nid=0x11f0 runnable
>
> "G1 Concurrent Refinement Thread#5" os_prio=0 tid=0x00007f804c037000
> nid=0x11ef runnable
>
> "G1 Concurrent Refinement Thread#6" os_prio=0 tid=0x00007f804c035800
> nid=0x11ee runnable
>
> "G1 Concurrent Refinement Thread#7" os_prio=0 tid=0x00007f804c033800
> nid=0x11ed runnable
>
> "G1 Concurrent Refinement Thread#8" os_prio=0 tid=0x00007f804c032000
> nid=0x11ec runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f804d5ee000 nid=0x1205
> waiting on condition
>
> JNI global references: 267
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ivan Rakov Ivan Rakov
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

Of course, there is another option that may help - trying better AWS
instances.

Best Regards,
Ivan Rakov

On 22.02.2018 0:58, Ivan Rakov wrote:

>> [03:14:36,596][INFO][db-checkpoint-thread-#77][GridCacheDatabaseSharedManager]
>>
>> Checkpoint finished [cpId=e2648942-961a-45c4-a7f9-41129e76b70f,
>> pages=801041, markPos=FileWALPointer [idx=2131, fileOffset=1035404,
>> len=60889, forceFlush=true], walSegmentsCleared=50, markDuration=913ms,
>> pagesWrite=3798ms, fsync=71652ms, total=76363ms]
>
> From this message I can conclude that you have quite slow disks.
> Checkpointing of 800K pages (3.2G of data) took 76 seconds, and FSYNC
> took 71 seconds of them.
> Freeze is expected when your disk bandwidth is way lower than you data
> load bandwidth. Node can accept limited amount of updates during
> checkpoint. If that limit is exceeded, all update operations will
> block until the end of current checkpoint.
> Ignite has write throttling mechanism (see
> https://apacheignite.readme.io/docs/durable-memory-tuning) to avoid
> such situations, but in 2.3 it's not tuned for the case when FSYNC
> takes most of checkpoint time. Improved version of throttling that
> adapts to long FSYNC will be released in 2.5:
> https://issues.apache.org/jira/browse/IGNITE-7533
>
> However, I still can't say why your client disconnects from server.
> The only explanation I see that it's artifact of AWS virtual
> environment: excessive use of disk I/O during checkpoint somehow
> affects quality of network connection. Try to decrease throughput of
> data loading (your hardware can't handle more than 40MB/sec anyway)
> and see how system will react.
> You can achieve lower throughput with tuning your data streaming, or
> don't using data streamer at all.
>
> Best Regards,
> Ivan Rakov
>
> On 21.02.2018 22:56, lawrencefinn wrote:
>> Here is a thread dump from when the client disconnects.  looks like
>> the only
>> thread doing something interesting is db-checkpoint-thread
>>
>> 018-02-21 19:31:36
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed
>> mode):
>>
>> "tcp-disco-client-message-worker-#36" #1128 prio=10 os_prio=0
>> tid=0x00007f800c001800 nid=0x2d89 waiting on condition
>> [0x00007f7f37afb000]
>>     java.lang.Thread.State: TIMED_WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x000000075d9d7790> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>
>>          at
>> java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
>>
>>          at
>> java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
>>
>>          at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6643)
>>
>>          at
>> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "tcp-disco-sock-reader-#34" #1126 prio=10 os_prio=0
>> tid=0x00007f7f74001800
>> nid=0x2d88 runnable [0x00007f7b20400000]
>>     java.lang.Thread.State: RUNNABLE
>>          at java.net.SocketInputStream.socketRead0(Native Method)
>>          at
>> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>>          at java.net.SocketInputStream.read(SocketInputStream.java:170)
>>          at java.net.SocketInputStream.read(SocketInputStream.java:141)
>>          at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>>          at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>>          at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>>          - locked <0x000000075d812028> (a java.io.BufferedInputStream)
>>          at
>> org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
>>
>>          at
>> java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2338)
>>
>>          at
>> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2351)
>>
>>          at
>> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)
>>
>>          at
>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
>>          at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
>>          at
>> org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
>>
>>          at
>> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
>>
>>          at
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>
>>          at
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
>>
>>          at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
>>
>>          at
>> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "sys-#1062" #1125 prio=5 os_prio=0 tid=0x00007f7f24031800 nid=0x2d86
>> waiting
>> on condition [0x00007f7f3477b000]
>>     java.lang.Thread.State: TIMED_WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c0045108> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>> "sys-#1062" #1125 prio=5 os_prio=0 tid=0x00007f7f24031800 nid=0x2d86
>> waiting
>> on condition [0x00007f7f3477b000]
>>     java.lang.Thread.State: TIMED_WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c0045108> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "sys-#1061" #1123 prio=5 os_prio=0 tid=0x00007f7f2402e800 nid=0x2d7b
>> waiting
>> on condition [0x00007f7b1fcd3000]
>>     java.lang.Thread.State: TIMED_WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c0045108> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "Attach Listener" #913 daemon prio=9 os_prio=0 tid=0x00007f7fb4027000
>> nid=0x19f5 waiting on condition [0x0000000000000000]
>>     java.lang.Thread.State: RUNNABLE
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#106" #141 prio=5 os_prio=0 tid=0x00007f7fb403f800
>> nid=0x13b9 waiting on condition [0x00007f7b1eec7000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#105" #140 prio=5 os_prio=0 tid=0x00007f7fb403e800
>> nid=0x13b8 waiting on condition [0x00007f7b1edc6000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#104" #139 prio=5 os_prio=0 tid=0x00007f7fb403d000
>> nid=0x13b7 waiting on condition [0x00007f7b1ecc5000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#103" #138 prio=5 os_prio=0 tid=0x00007f7fb403c000
>> nid=0x13b6 waiting on condition [0x00007f7b1e976000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#102" #137 prio=5 os_prio=0 tid=0x00007f7fb403b000
>> nid=0x13b5 waiting on condition [0x00007f7b1ebc4000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#101" #136 prio=5 os_prio=0 tid=0x00007f7fb403a000
>> nid=0x13b4 waiting on condition [0x00007f7b1fefb000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#101" #136 prio=5 os_prio=0 tid=0x00007f7fb403a000
>> nid=0x13b4 waiting on condition [0x00007f7b1fefb000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#100" #135 prio=5 os_prio=0 tid=0x00007f7fb4038800
>> nid=0x13b3 waiting on condition [0x00007f7b1f9d2000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#99" #134 prio=5 os_prio=0 tid=0x00007f7fb4037800
>> nid=0x13b2 waiting on condition [0x00007f7b1f8d1000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>> "checkpoint-runner-#98" #133 prio=5 os_prio=0 tid=0x00007f7fb4036800
>> nid=0x13b1 waiting on condition [0x00007f7b1f7d0000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#97" #132 prio=5 os_prio=0 tid=0x00007f7fb4035800
>> nid=0x13b0 waiting on condition [0x00007f7b1eac3000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#96" #131 prio=5 os_prio=0 tid=0x00007f7fb4034800
>> nid=0x13af waiting on condition [0x00007f7b1f4cd000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#95" #130 prio=5 os_prio=0 tid=0x00007f7fb4033800
>> nid=0x13ae waiting on condition [0x00007f7b1f3cc000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#94" #129 prio=5 os_prio=0 tid=0x00007f7fb4032800
>> nid=0x13ad waiting on condition [0x00007f7b1f2cb000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#93" #128 prio=5 os_prio=0 tid=0x00007f7fb4032000
>> nid=0x13ac waiting on condition [0x00007f7b1f1ca000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#92" #127 prio=5 os_prio=0 tid=0x00007f7fb4031000
>> nid=0x13ab waiting on condition [0x00007f7f34a7c000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "checkpoint-runner-#91" #126 prio=5 os_prio=0 tid=0x00007f7fb402f000
>> nid=0x13aa waiting on condition [0x00007f7f34278000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c1231e90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "ttl-cleanup-worker-#61" #96 prio=5 os_prio=0 tid=0x00007f7f8429a800
>> nid=0x12f2 sleeping[0x00007f7b1fdfa000]
>>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>>          at java.lang.Thread.sleep(Native Method)
>>          at
>> org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7533)
>>          at
>> org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "srvc-deploy-#58" #93 prio=5 os_prio=0 tid=0x00007f7fb000b000 nid=0x12e9
>> waiting on condition [0x00007f7b1fffc000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c080cb28> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>          at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "db-checkpoint-thread-#56" #91 prio=5 os_prio=0 tid=0x00007f7f84247800
>> nid=0x12e6 runnable [0x00007f7b201fe000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.FileDispatcherImpl.force0(Native Method)
>>          at
>> sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
>>          at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)
>>          at
>> org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIO.force(RandomAccessFileIO.java:87)
>>
>>          at
>> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.sync(FilePageStore.java:495)
>>
>>          at
>> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:2156)
>>
>>          at
>> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:2010)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - <0x00000004c2b000e0> (a
>> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
>>
>> "wal-file-archiver%null" #86 prio=5 os_prio=0 tid=0x00007f7f84004000
>> nid=0x12e1 in Object.wait() [0x00007f7f34579000]
>>     java.lang.Thread.State: WAITING (on object monitor)
>>          at java.lang.Object.wait(Native Method)
>>          at java.lang.Object.wait(Object.java:502)
>>          at
>> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.run(FileWriteAheadLogManager.java:1228)
>>
>>          - locked <0x00000004c076f208> (a
>> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver)
>>
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "tcp-disco-sock-reader-#8" #82 prio=10 os_prio=0 tid=0x00007f7f74003800
>> nid=0x1247 runnable [0x00007f7f352db000]
>>     java.lang.Thread.State: RUNNABLE
>>          at java.net.SocketInputStream.socketRead0(Native Method)
>>          at
>> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>>          at java.net.SocketInputStream.read(SocketInputStream.java:170)
>>          at java.net.SocketInputStream.read(SocketInputStream.java:141)
>>          at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>>          at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>>          at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>>          - locked <0x00000004c0829b50> (a java.io.BufferedInputStream)
>>          at
>> org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
>>
>>          at
>> java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2338)
>>
>>          at
>> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2351)
>>
>>          at
>> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)
>>
>>          at
>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
>>          at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
>>          at
>> org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
>>
>>          at
>> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
>>
>>          at
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>
>>          at
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
>>
>>          at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
>>
>>          at
>> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "session-timeout-worker-#42" #63 daemon prio=5 os_prio=0
>> tid=0x00007f804dd6d000 nid=0x1240 sleeping[0x00007f7f353dc000]
>>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>>          at java.lang.Thread.sleep(Native Method)
>>          at
>> org.apache.ignite.internal.processors.rest.GridRestProcessor$4.body(GridRestProcessor.java:437)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "exchange-worker-#50" #74 prio=5 os_prio=0 tid=0x00007f804dd79000
>> nid=0x123f
>> waiting on condition [0x00007f7f354dd000]
>>     java.lang.Thread.State: TIMED_WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c0878178> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>
>>          at
>> java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
>>
>>          at
>> java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
>>
>>          at
>> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2225)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "ignite-update-notifier-timer" #73 daemon prio=5 os_prio=0
>> tid=0x00007f804dd77800 nid=0x123e in Object.wait() [0x00007f7f355de000]
>>     java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>          at java.lang.Object.wait(Native Method)
>>          at java.util.TimerThread.mainLoop(Timer.java:552)
>>          - locked <0x00000004c0878348> (a java.util.TaskQueue)
>>          at java.util.TimerThread.run(Timer.java:505)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "upd-ver-checker" #72 daemon prio=5 os_prio=0 tid=0x00007f804dd76000
>> nid=0x123d waiting on condition [0x00007f7f356df000]
>>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>>          at java.lang.Thread.sleep(Native Method)
>>          at
>> org.apache.ignite.internal.processors.cluster.GridUpdateNotifier$1.run(GridUpdateNotifier.java:115)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "disco-event-worker-#49" #71 prio=5 os_prio=0 tid=0x00007f804dd70000
>> nid=0x123c waiting on condition [0x00007f7f357e0000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c004ec38> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>          at
>> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2552)
>>
>>          at
>> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2534)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "tcp-disco-srvr-#2" #56 prio=10 os_prio=0 tid=0x00007f804dd63800
>> nid=0x123b
>> runnable [0x00007f7f358e1000]
>>     java.lang.Thread.State: RUNNABLE
>>          at java.net.PlainSocketImpl.socketAccept(Native Method)
>>          at
>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
>>
>>          at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>          at java.net.ServerSocket.accept(ServerSocket.java:513)
>>          at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5623)
>>
>>          at
>> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "tcp-disco-msg-worker-#3" #70 prio=10 os_prio=0 tid=0x00007f804dd60800
>> nid=0x123a runnable [0x00007f7f359e2000]
>>     java.lang.Thread.State: TIMED_WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c08f8698> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>
>>          at
>> java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
>>
>>          at
>> java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
>>
>>          at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6643)
>>
>>          at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2533)
>>
>>          at
>> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "grid-data-loader-flusher-#48" #69 prio=5 os_prio=0
>> tid=0x00007f804dd49800
>> nid=0x1239 waiting on condition [0x00007f7f35ce3000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c0878748> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>          at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
>>          at
>> org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:108)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>> "grid-nio-worker-tcp-rest-3-#47" #68 prio=5 os_prio=0
>> tid=0x00007f804dd3f000
>> nid=0x1238 runnable [0x00007f7f35de4000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c09728b8> (a
>> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>>          - locked <0x00000004c09728a8> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c09728d8> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "grid-nio-worker-tcp-rest-2-#46" #67 prio=5 os_prio=0
>> tid=0x00007f804dd3d000
>> nid=0x1237 runnable [0x00007f7f35ee5000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c08f8890> (a
>> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>>          - locked <0x00000004c08f8880> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c08f88b0> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "grid-nio-worker-tcp-rest-1-#45" #66 prio=5 os_prio=0
>> tid=0x00007f804dcf7800
>> nid=0x1236 runnable [0x00007f7f35fe6000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c08789c0> (a
>> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>>          - locked <0x00000004c08789b0> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c08789e0> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>> "grid-nio-worker-tcp-rest-0-#44" #65 prio=5 os_prio=0
>> tid=0x00007f804dcf6000
>> nid=0x1235 runnable [0x00007f7f360e7000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c076f9b8> (a
>> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>>          - locked <0x00000004c076f9a8> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c076f9d8> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "nio-acceptor-#43" #64 prio=5 os_prio=0 tid=0x00007f804dcf4800
>> nid=0x1234
>> runnable [0x00007f7f361e8000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c0878c78> (a sun.nio.ch.Util$2)
>>          - locked <0x00000004c0878c68> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c0878c88> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2772)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2744)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "grid-nio-worker-client-listener-3-#41" #62 prio=5 os_prio=0
>> tid=0x00007f804dcd3800 nid=0x1233 runnable [0x00007f7f362e9000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c076fc70> (a
>> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>>          - locked <0x00000004c076fc60> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c076fc90> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "grid-nio-worker-client-listener-2-#40" #61 prio=5 os_prio=0
>> tid=0x00007f804dcd2000 nid=0x1232 runnable [0x00007f7f363ea000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c08f8b48> (a
>> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>>          - locked <0x00000004c08f8b38> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c08f8b68> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "grid-nio-worker-client-listener-1-#39" #60 prio=5 os_prio=0
>> tid=0x00007f804dcd0800 nid=0x1231 runnable [0x00007f7f364eb000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c076ff28> (a
>> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>>          - locked <0x00000004c076ff18> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c076ff48> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "grid-nio-worker-client-listener-0-#38" #59 prio=5 os_prio=0
>> tid=0x00007f804dcce800 nid=0x1230 runnable [0x00007f7f365ec000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c0878ee0> (a
>> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>>          - locked <0x00000004c0878ed0> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c0878f00> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>> "nio-acceptor-#37" #58 prio=5 os_prio=0 tid=0x00007f804dc8a000
>> nid=0x122f
>> runnable [0x00007f7f366ed000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c08f8e00> (a sun.nio.ch.Util$2)
>>          - locked <0x00000004c08f8df0> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c08f8e10> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2772)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2744)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "tcp-comm-worker-#1" #55 prio=5 os_prio=0 tid=0x00007f804dab0800
>> nid=0x122e
>> waiting on condition [0x00007f7f36dee000]
>>     java.lang.Thread.State: TIMED_WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c004ee70> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>          at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>
>>          at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>>
>>          at
>> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4037)
>>
>>          at
>> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "grid-nio-worker-tcp-comm-3-#36" #51 prio=5 os_prio=0
>> tid=0x00007f804daae800
>> nid=0x122d runnable [0x00007f7f36eef000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c004f0f0> (a
>> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>>          - locked <0x00000004c004f0e0> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c004f110> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "grid-nio-worker-tcp-comm-2-#35" #50 prio=5 os_prio=0
>> tid=0x00007f804daad000
>> nid=0x122c runnable [0x00007f7f36ff0000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c004f3a0> (a
>> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>>          - locked <0x00000004c004f390> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c004f3c0> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "grid-nio-worker-tcp-comm-1-#34" #49 prio=5 os_prio=0
>> tid=0x00007f804daac000
>> nid=0x122b runnable [0x00007f7f371f2000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c004f690> (a
>> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>>          - locked <0x00000004c004f680> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c004f6b0> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "grid-nio-worker-tcp-comm-0-#33" #48 prio=5 os_prio=0
>> tid=0x00007f804da5e000
>> nid=0x122a runnable [0x00007f7f372f3000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c004f940> (a
>> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>>          - locked <0x00000004c004f930> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c004f960> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "nio-acceptor-#32" #47 prio=5 os_prio=0 tid=0x00007f804daaa800
>> nid=0x1229
>> runnable [0x00007f7f370f1000]
>>     java.lang.Thread.State: RUNNABLE
>>          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>          at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>          at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>          at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>          - locked <0x00000004c0045df0> (a sun.nio.ch.Util$2)
>>          - locked <0x00000004c0045de0> (a
>> java.util.Collections$UnmodifiableSet)
>>          - locked <0x00000004c0045e00> (a sun.nio.ch.EPollSelectorImpl)
>>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2772)
>>
>>          at
>> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2744)
>>
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "grid-timeout-worker-#31" #46 prio=5 os_prio=0 tid=0x00007f804da41000
>> nid=0x1225 in Object.wait() [0x00007f7f373f4000]
>>     java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>          at java.lang.Object.wait(Native Method)
>>          at
>> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:195)
>>
>>          - locked <0x00000004c0046000> (a java.lang.Object)
>>          at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "ignite-clock" #45 daemon prio=10 os_prio=0 tid=0x00007f804da4a000
>> nid=0x1224 sleeping[0x00007f7f374f5000]
>>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>>          at java.lang.Thread.sleep(Native Method)
>>          at
>> org.apache.ignite.internal.util.IgniteUtils$22.run(IgniteUtils.java:3263)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-15-#24" #38 prio=5 os_prio=0
>> tid=0x00007f804d949800
>> nid=0x121d waiting on condition [0x00007f7f37dfc000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-14-#23" #37 prio=5 os_prio=0
>> tid=0x00007f804d948000
>> nid=0x121c waiting on condition [0x00007f7f37efd000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-13-#22" #36 prio=5 os_prio=0
>> tid=0x00007f804d946000
>> nid=0x121b waiting on condition [0x00007f7f37ffe000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-12-#21" #35 prio=5 os_prio=0
>> tid=0x00007f804d944800
>> nid=0x121a waiting on condition [0x00007f7fc41ee000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-11-#20" #34 prio=5 os_prio=0
>> tid=0x00007f804d943000
>> nid=0x1219 waiting on condition [0x00007f7fc42ef000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-10-#19" #33 prio=5 os_prio=0
>> tid=0x00007f804d941000
>> nid=0x1218 waiting on condition [0x00007f7fc43f0000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-9-#18" #32 prio=5 os_prio=0 tid=0x00007f804d93f800
>> nid=0x1217 waiting on condition [0x00007f7fc44f1000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-8-#17" #31 prio=5 os_prio=0 tid=0x00007f804d93d800
>> nid=0x1216 waiting on condition [0x00007f7fc45f2000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-7-#16" #30 prio=5 os_prio=0 tid=0x00007f804d93b800
>> nid=0x1215 waiting on condition [0x00007f7fc46f3000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-6-#15" #29 prio=5 os_prio=0 tid=0x00007f804d939800
>> nid=0x1214 waiting on condition [0x00007f7fc47f4000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-5-#14" #28 prio=5 os_prio=0 tid=0x00007f804d937800
>> nid=0x1213 waiting on condition [0x00007f7fc48f5000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-4-#13" #27 prio=5 os_prio=0 tid=0x00007f804d936000
>> nid=0x1212 waiting on condition [0x00007f7fc49f6000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-3-#12" #26 prio=5 os_prio=0 tid=0x00007f804d934000
>> nid=0x1211 waiting on condition [0x00007f7fc4af7000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-2-#11" #25 prio=5 os_prio=0 tid=0x00007f804d932800
>> nid=0x1210 waiting on condition [0x00007f7fc4bf8000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-1-#10" #24 prio=5 os_prio=0 tid=0x00007f804d930800
>> nid=0x120f waiting on condition [0x00007f7fc4cf9000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "data-streamer-stripe-0-#9" #23 prio=5 os_prio=0 tid=0x00007f804d92f000
>> nid=0x120e waiting on condition [0x00007f7fc4dfa000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "sys-stripe-7-#8" #22 prio=5 os_prio=0 tid=0x00007f804d92d000 nid=0x120d
>> waiting on condition [0x00007f7fc4efb000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "sys-stripe-6-#7" #21 prio=5 os_prio=0 tid=0x00007f804d92b800 nid=0x120c
>> waiting on condition [0x00007f7fc4ffc000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "sys-stripe-5-#6" #20 prio=5 os_prio=0 tid=0x00007f804d92a000 nid=0x120b
>> waiting on condition [0x00007f7fc50fd000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "sys-stripe-4-#5" #19 prio=5 os_prio=0 tid=0x00007f804d928000 nid=0x120a
>> waiting on condition [0x00007f7fc51fe000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "sys-stripe-3-#4" #18 prio=5 os_prio=0 tid=0x00007f804d926800 nid=0x1209
>> waiting on condition [0x00007f7fc52ff000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "sys-stripe-2-#3" #17 prio=5 os_prio=0 tid=0x00007f804d924800 nid=0x1208
>> waiting on condition [0x00007f7fc5400000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "sys-stripe-1-#2" #16 prio=5 os_prio=0 tid=0x00007f804d91e800 nid=0x1207
>> waiting on condition [0x00007f7fc5501000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "sys-stripe-0-#1" #15 prio=5 os_prio=0 tid=0x00007f804d91a800 nid=0x1206
>> waiting on condition [0x00007f7fc5602000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
>>
>>          at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
>>
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>>
>> "Service Thread" #10 daemon prio=9 os_prio=0 tid=0x00007f804d3e6800
>> nid=0x1201 runnable [0x0000000000000000]
>>     java.lang.Thread.State: RUNNABLE
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "C1 CompilerThread3" #9 daemon prio=9 os_prio=0 tid=0x00007f804d3e3800
>> nid=0x1200 waiting on condition [0x0000000000000000]
>>     java.lang.Thread.State: RUNNABLE
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "C2 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f804d3e1800
>> nid=0x11ff waiting on condition [0x0000000000000000]
>>     java.lang.Thread.State: RUNNABLE
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f804d3df800
>> nid=0x11fe waiting on condition [0x0000000000000000]
>>     java.lang.Thread.State: RUNNABLE
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f804d3dd800
>> nid=0x11fd waiting on condition [0x0000000000000000]
>>     java.lang.Thread.State: RUNNABLE
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f804d3db000
>> nid=0x11fc runnable [0x0000000000000000]
>>     java.lang.Thread.State: RUNNABLE
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>> tid=0x00007f804d3da000 nid=0x11fb waiting on condition
>> [0x0000000000000000]
>>     java.lang.Thread.State: RUNNABLE
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f804d3a7000
>> nid=0x11fa in
>> Object.wait() [0x00007f7fc73fc000]
>>     java.lang.Thread.State: WAITING (on object monitor)
>>          at java.lang.Object.wait(Native Method)
>>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>          - locked <0x00000004c0045858> (a
>> java.lang.ref.ReferenceQueue$Lock)
>>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>          at
>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f804d3a2800
>> nid=0x11f9 in Object.wait() [0x00007f7fc74fd000]
>>     java.lang.Thread.State: WAITING (on object monitor)
>>          at java.lang.Object.wait(Native Method)
>>          at java.lang.Object.wait(Object.java:502)
>>          at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
>>          - locked <0x00000004c0049648> (a java.lang.ref.Reference$Lock)
>>          at
>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "main" #1 prio=5 os_prio=0 tid=0x00007f804c00b800 nid=0x11e3 waiting on
>> condition [0x00007f8052502000]
>>     java.lang.Thread.State: WAITING (parking)
>>          at sun.misc.Unsafe.park(Native Method)
>>          - parking to wait for  <0x00000004c08f8ea8> (a
>> java.util.concurrent.CountDownLatch$Sync)
>>          at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>
>>          at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>
>>          at
>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>          at
>> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:335)
>>
>>
>>     Locked ownable synchronizers:
>>          - None
>>
>> "VM Thread" os_prio=0 tid=0x00007f804d39b000 nid=0x11f8 runnable
>>
>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c023000
>> nid=0x11e4 runnable
>>
>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c024800
>> nid=0x11e5 runnable
>>
>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c026800
>> nid=0x11e6 runnable
>>
>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c028000
>> nid=0x11e7 runnable
>>
>> "Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c02a000
>> nid=0x11e8 runnable
>>
>> "Gang worker#5 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c02b800
>> nid=0x11e9 runnable
>>
>> "Gang worker#6 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c02d800
>> nid=0x11ea runnable
>>
>> "Gang worker#7 (Parallel GC Threads)" os_prio=0 tid=0x00007f804c02f000
>> nid=0x11eb runnable
>>
>> "G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007f804c050800
>> nid=0x11f5 runnable
>>
>> "Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0
>> tid=0x00007f804c052800 nid=0x11f6 runnable
>> "Gang worker#1 (G1 Parallel Marking Threads)" os_prio=0
>> tid=0x00007f804c054000 nid=0x11f7 runnable
>>
>> "G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007f804c040000
>> nid=0x11f4 runnable
>>
>> "G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007f804c03e800
>> nid=0x11f3 runnable
>>
>> "G1 Concurrent Refinement Thread#2" os_prio=0 tid=0x00007f804c03c800
>> nid=0x11f2 runnable
>>
>> "G1 Concurrent Refinement Thread#3" os_prio=0 tid=0x00007f804c03b000
>> nid=0x11f1 runnable
>>
>> "G1 Concurrent Refinement Thread#4" os_prio=0 tid=0x00007f804c039000
>> nid=0x11f0 runnable
>>
>> "G1 Concurrent Refinement Thread#5" os_prio=0 tid=0x00007f804c037000
>> nid=0x11ef runnable
>>
>> "G1 Concurrent Refinement Thread#6" os_prio=0 tid=0x00007f804c035800
>> nid=0x11ee runnable
>>
>> "G1 Concurrent Refinement Thread#7" os_prio=0 tid=0x00007f804c033800
>> nid=0x11ed runnable
>>
>> "G1 Concurrent Refinement Thread#8" os_prio=0 tid=0x00007f804c032000
>> nid=0x11ec runnable
>>
>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f804d5ee000 nid=0x1205
>> waiting on condition
>>
>> JNI global references: 267
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>


Dave Harvey Dave Harvey
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

In reply to this post by Ivan Rakov
I fought with trying to get Ignite Persistence to work well on AWS GP2
volumes, and finally gave up, and moved to i3 instances, where the $ per
write IOP are much lower, and a i3.8xlarge gets 720,000 4K write IOPS vs on
the order of 10,000 for about the same cost.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
lawrencefinn lawrencefinn
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

I could try a different AWS instance.  Im running these tests on r4.8xlarge
boxes, which are pretty beefy and "ebs optimized".  I tried the same tests
using IO1 disks at 20,000 IOPS but still had issues.  

Dave with the i3 instances, were you using the local ssds?  Or still using
EBS?  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Dave Harvey Dave Harvey
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

Using the local SSDs, which are radically faster than EBS.

On Thu, Feb 22, 2018 at 10:34 AM, lawrencefinn <[hidden email]> wrote:
I could try a different AWS instance.  Im running these tests on r4.8xlarge
boxes, which are pretty beefy and "ebs optimized".  I tried the same tests
using IO1 disks at 20,000 IOPS but still had issues.

Dave with the i3 instances, were you using the local ssds?  Or still using
EBS?



Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more Click Here.

Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

RE: Large durable caches

If you’re using local SSDs, these are not durable by definition. How are you ensuring your data survives termination of the instance, or a failure of the underlying physical host?

 

From: David Harvey [mailto:[hidden email]]
Sent: Friday, February 23, 2018 10:03 AM
To: [hidden email]
Subject: Re: Large durable caches

 

Using the local SSDs, which are radically faster than EBS.

 

On Thu, Feb 22, 2018 at 10:34 AM, lawrencefinn <[hidden email]> wrote:

I could try a different AWS instance.  Im running these tests on r4.8xlarge
boxes, which are pretty beefy and "ebs optimized".  I tried the same tests
using IO1 disks at 20,000 IOPS but still had issues.

Dave with the i3 instances, were you using the local ssds?  Or still using
EBS?

 

 

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more Click Here.

Dave Harvey Dave Harvey
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

In reply to this post by lawrencefinn
In our case, we use 1+ backups, so Ignite replicates the data to other nodes.   My  understanding:  Ignite replicates changes at the K/V or record level and sends them to the backup node's WAL.  The backup node may differ per partition.   The backup node applies the changes to its storage, and also updates the local SQL indices (which only reference records on the local node).   Ignite persistence writes data to the SSDs in pages, so if you change 80 bytes, it must write multiple 4K pages on the SSDs, but only send around those 80 bytes over the network.  So you need a lot more SSD bandwidth than network bandwidth when using Ignite persistence with backups.

-DH 

On Thu, Feb 22, 2018 at 6:52 PM, Raymond Wilson <[hidden email]> wrote:

If you’re using local SSDs, these are not durable by definition. How are you ensuring your data survives termination of the instance, or a failure of the underlying physical host?

 

From: David Harvey [mailto:[hidden email]]
Sent: Friday, February 23, 2018 10:03 AM
To: [hidden email]
Subject: Re: Large durable caches

 

Using the local SSDs, which are radically faster than EBS.

 

On Thu, Feb 22, 2018 at 10:34 AM, lawrencefinn <[hidden email]> wrote:

I could try a different AWS instance.  Im running these tests on r4.8xlarge
boxes, which are pretty beefy and "ebs optimized".  I tried the same tests
using IO1 disks at 20,000 IOPS but still had issues.

Dave with the i3 instances, were you using the local ssds?  Or still using
EBS?

 

 

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more Click Here.




Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more Click Here.

lawrencefinn lawrencefinn
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

Switching instance types helped a lot.  But the writes slow down in some
relationship to the amount of data loaded. I already had 430,300,000 rows in
my cache.  I started adding more rows.  It slowed down greatly:

2018-03-01 14:19:07,709 [INFO] from application in
application-context.ignite-6 - Loaded 100,000
2018-03-01 14:20:53,842 [INFO] from application in
application-context.ignite-6 - Loaded 100,000
2018-03-01 14:22:37,162 [INFO] from application in
application-context.ignite-6 - Loaded 100,000

That is around 2 minutes to load 100,000 entries.  When I had less data it
was around 20s.  Any thoughts?

I have 3 indexes on the cache, could that be affecting write performance by
this much?  The index.bin file is pretty huge, ~ 14G per node on 4 nodes.
The checkpoints take ~ 8s

[14:34:23,992][INFO][db-checkpoint-thread-#74][GridCacheDatabaseSharedManager]
Checkpoint started [checkpointId=32303723-4c66-475e-a9de-6b6d7181561c,
startPtr=FileWALPointer [idx=25029, fileOffset=51625907, len=65335,
forceFlush=true], checkpointLockWait=12ms, checkpointLockHoldTime=30ms,
pages=219117, reason='timeout']
[14:34:31,951][INFO][db-checkpoint-thread-#74][GridCacheDatabaseSharedManager]
Checkpoint finished [cpId=32303723-4c66-475e-a9de-6b6d7181561c,
pages=219117, markPos=FileWALPointer [idx=25029, fileOffset=51625907,
len=65335, forceFlush=true], walSegmentsCleared=14, markDuration=282ms,
pagesWrite=3845ms, fsync=4113ms, total=8252ms]

The thread dump looks as follows:

2018-03-01 14:34:09
Full thread dump OpenJDK 64-Bit Server VM (25.151-b12 mixed mode):

"sys-#12845" #12950 prio=5 os_prio=0 tid=0x00007fe8980cc800 nid=0x3d99
waiting on condition [0x00007fe4a1fe5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c001fd28> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"sys-#12844" #12949 prio=5 os_prio=0 tid=0x00007fe898053800 nid=0x3d96
waiting on condition [0x00007fe49ecb6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c001fd28> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"sys-#12843" #12948 prio=5 os_prio=0 tid=0x00007fe89804a000 nid=0x3d95
waiting on condition [0x00007fe4a23e7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c001fd28> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"sys-#12842" #12947 prio=5 os_prio=0 tid=0x00007fe898048000 nid=0x3d94
waiting on condition [0x00007fe4a1be1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c001fd28> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"sys-#12841" #12946 prio=5 os_prio=0 tid=0x00007fe8980de800 nid=0x3d93
waiting on condition [0x00007fe49f2bc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c001fd28> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"sys-#12840" #12945 prio=5 os_prio=0 tid=0x00007fe8980de000 nid=0x3d92
waiting on condition [0x00007fe49e6b0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c001fd28> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"tcp-disco-client-message-worker-#10" #8811 prio=10 os_prio=0
tid=0x00007fe8cc1ea800 nid=0x24d1 waiting on condition [0x00007fe87c4ad000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006165d3e88> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at
java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6643)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

"tcp-disco-sock-reader-#9" #8810 prio=10 os_prio=0 tid=0x00007fe92d9a5800
nid=0x24d0 runnable [0x00007fe4a22e6000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked <0x00000006165d9568> (a java.io.BufferedInputStream)
        at
org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
        at
java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2657)
        at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2673)
        at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3150)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:859)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:355)
        at
org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
        at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
        at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
        at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

"Attach Listener" #3808 daemon prio=9 os_prio=0 tid=0x00007fe8bc032000
nid=0x697 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"checkpoint-runner-#156" #188 prio=5 os_prio=0 tid=0x00007fe8cc171800
nid=0x4a1e waiting on condition [0x00007fe4a05cd000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#155" #187 prio=5 os_prio=0 tid=0x00007fe8cc170800
nid=0x4a1d waiting on condition [0x00007fe49fcc4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#154" #186 prio=5 os_prio=0 tid=0x00007fe8cc16f800
nid=0x4a1c waiting on condition [0x00007fe4a24e8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#153" #185 prio=5 os_prio=0 tid=0x00007fe8cc16e000
nid=0x4a1b waiting on condition [0x00007fe49fac2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#152" #184 prio=5 os_prio=0 tid=0x00007fe8cc16c800
nid=0x4a1a waiting on condition [0x00007fe4a14dc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#151" #183 prio=5 os_prio=0 tid=0x00007fe8cc16b800
nid=0x4a19 waiting on condition [0x00007fe4a18de000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#150" #182 prio=5 os_prio=0 tid=0x00007fe8cc16a000
nid=0x4a18 waiting on condition [0x00007fe4a13db000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#149" #181 prio=5 os_prio=0 tid=0x00007fe8cc168800
nid=0x4a17 waiting on condition [0x00007fe4a12da000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#148" #180 prio=5 os_prio=0 tid=0x00007fe8cc167800
nid=0x4a16 waiting on condition [0x00007fe49f9c1000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#147" #179 prio=5 os_prio=0 tid=0x00007fe8cc166000
nid=0x4a15 waiting on condition [0x00007fe4a11d9000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#146" #178 prio=5 os_prio=0 tid=0x00007fe8cc164800
nid=0x4a14 waiting on condition [0x00007fe4a0fd7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#145" #177 prio=5 os_prio=0 tid=0x00007fe8cc163800
nid=0x4a13 waiting on condition [0x00007fe4a0ed6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#144" #176 prio=5 os_prio=0 tid=0x00007fe8cc162000
nid=0x4a12 waiting on condition [0x00007fe4a0dd5000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#143" #175 prio=5 os_prio=0 tid=0x00007fe8cc161000
nid=0x4a11 waiting on condition [0x00007fe4a0bd3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#142" #174 prio=5 os_prio=0 tid=0x00007fe8cc15f800
nid=0x4a10 waiting on condition [0x00007fe4a0ad2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#141" #173 prio=5 os_prio=0 tid=0x00007fe8cc15e000
nid=0x4a0f waiting on condition [0x00007fe4a09d1000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#140" #172 prio=5 os_prio=0 tid=0x00007fe8cc15c800
nid=0x4a0e waiting on condition [0x00007fe49f8c0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#139" #171 prio=5 os_prio=0 tid=0x00007fe8cc15a800
nid=0x4a0d waiting on condition [0x00007fe4a08d0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#138" #170 prio=5 os_prio=0 tid=0x00007fe8cc159800
nid=0x4a0c waiting on condition [0x00007fe4a07cf000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#137" #169 prio=5 os_prio=0 tid=0x00007fe8cc158000
nid=0x4a0b waiting on condition [0x00007fe4a06ce000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#136" #168 prio=5 os_prio=0 tid=0x00007fe8cc156800
nid=0x4a0a waiting on condition [0x00007fe4a04cc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#135" #167 prio=5 os_prio=0 tid=0x00007fe8cc155000
nid=0x4a09 waiting on condition [0x00007fe49f7bf000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#134" #166 prio=5 os_prio=0 tid=0x00007fe8cc154000
nid=0x4a08 waiting on condition [0x00007fe4a03cb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#133" #165 prio=5 os_prio=0 tid=0x00007fe8cc152800
nid=0x4a07 waiting on condition [0x00007fe4a02ca000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#132" #164 prio=5 os_prio=0 tid=0x00007fe8cc151800
nid=0x4a06 waiting on condition [0x00007fe4a01c9000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#131" #163 prio=5 os_prio=0 tid=0x00007fe8cc150800
nid=0x4a05 waiting on condition [0x00007fe4a00c8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#130" #162 prio=5 os_prio=0 tid=0x00007fe8cc14f000
nid=0x4a04 waiting on condition [0x00007fe49f6be000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#129" #161 prio=5 os_prio=0 tid=0x00007fe8cc14e000
nid=0x4a03 waiting on condition [0x00007fe49fec6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#128" #160 prio=5 os_prio=0 tid=0x00007fe8cc14c800
nid=0x4a02 waiting on condition [0x00007fe49fdc5000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#127" #159 prio=5 os_prio=0 tid=0x00007fe8cc14b800
nid=0x4a01 waiting on condition [0x00007fe49fbc3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#126" #158 prio=5 os_prio=0 tid=0x00007fe8cc14a800
nid=0x4a00 waiting on condition [0x00007fe87c17f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"checkpoint-runner-#125" #157 prio=5 os_prio=0 tid=0x00007fe8cc14a000
nid=0x49ff waiting on condition [0x00007fe877afd000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f63c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"ttl-cleanup-worker-#79" #111 prio=5 os_prio=0 tid=0x00007fe88017d800
nid=0x498a waiting on condition [0x00007fe4a19df000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7533)
        at
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"srvc-deploy-#76" #108 prio=5 os_prio=0 tid=0x00007fe8d400d000 nid=0x4981
waiting on condition [0x00007fe4a1ce2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c0800178> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"db-checkpoint-thread-#74" #106 prio=5 os_prio=0 tid=0x00007fe880155000
nid=0x497e in Object.wait() [0x00007fe4a1ee4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.waitCheckpointEvent(GridCacheDatabaseSharedManager.java:2235)
        - locked <0x00000004c08f6608> (a
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:1999)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"wal-file-archiver%null" #101 prio=5 os_prio=0 tid=0x00007fe880007000
nid=0x4979 in Object.wait() [0x00007fe8779fc000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.run(FileWriteAheadLogManager.java:1228)
        - locked <0x00000004c09f0730> (a
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver)

"tcp-disco-sock-reader-#8" #95 prio=10 os_prio=0 tid=0x00007fe92de1a000
nid=0x48e9 runnable [0x00007fe87c9d0000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked <0x00000004c081d178> (a java.io.BufferedInputStream)
        at
org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
        at
java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2657)
        at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2673)
        at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3150)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:859)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:355)
        at
org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
        at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
        at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
        at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

"session-timeout-worker-#58" #78 daemon prio=5 os_prio=0
tid=0x00007fe92de04000 nid=0x48e4 waiting on condition [0x00007fe87cad1000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.processors.rest.GridRestProcessor$4.body(GridRestProcessor.java:437)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"exchange-worker-#66" #89 prio=5 os_prio=0 tid=0x00007fe92de0f800 nid=0x48e3
waiting on condition [0x00007fe87cbd2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f6988> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at
java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2225)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"ignite-update-notifier-timer" #88 daemon prio=5 os_prio=0
tid=0x00007fe92de0e800 nid=0x48e2 in Object.wait() [0x00007fe87ccd3000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x00000004c08f6b90> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"upd-ver-checker" #87 daemon prio=5 os_prio=0 tid=0x00007fe92de0a800
nid=0x48e1 waiting on condition [0x00007fe87cdd4000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.processors.cluster.GridUpdateNotifier$1.run(GridUpdateNotifier.java:115)
        at java.lang.Thread.run(Thread.java:748)

"disco-event-worker-#65" #86 prio=5 os_prio=0 tid=0x00007fe92de06800
nid=0x48e0 waiting on condition [0x00007fe87ced5000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c001fdf8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2552)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2534)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"tcp-disco-srvr-#2" #71 prio=10 os_prio=0 tid=0x00007fe92ddfa800 nid=0x48df
runnable [0x00007fe87cfd6000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5623)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

"tcp-disco-msg-worker-#3" #85 prio=10 os_prio=0 tid=0x00007fe92ddf7800
nid=0x48de waiting on condition [0x00007fe87d0d7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f7188> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at
java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6643)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2533)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

"grid-data-loader-flusher-#64" #84 prio=5 os_prio=0 tid=0x00007fe92dddb800
nid=0x48dd waiting on condition [0x00007fe87d3d8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f7360> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:108)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"grid-nio-worker-tcp-rest-3-#63" #83 prio=5 os_prio=0 tid=0x00007fe92ddd5800
nid=0x48dc runnable [0x00007fe87d4d9000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c08f75d8> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c08f75c8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c08f75f8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"grid-nio-worker-tcp-rest-2-#62" #82 prio=5 os_prio=0 tid=0x00007fe92ddd4000
nid=0x48db runnable [0x00007fe87d5da000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c081d3e8> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c081d3d8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c081d408> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"grid-nio-worker-tcp-rest-1-#61" #81 prio=5 os_prio=0 tid=0x00007fe92dd52800
nid=0x48da runnable [0x00007fe87d6db000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c081d6a0> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c081d690> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c081d6c0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"grid-nio-worker-tcp-rest-0-#60" #80 prio=5 os_prio=0 tid=0x00007fe92dd50800
nid=0x48d9 runnable [0x00007fe87d7dc000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c081d958> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c081d948> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c081d978> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"nio-acceptor-#59" #79 prio=5 os_prio=0 tid=0x00007fe92dd4f000 nid=0x48d8
runnable [0x00007fe87d8dd000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c081dc10> (a sun.nio.ch.Util$3)
        - locked <0x00000004c081dc00> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c081dc20> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2772)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2744)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"grid-nio-worker-client-listener-3-#57" #77 prio=5 os_prio=0
tid=0x00007fe92dd2e800 nid=0x48d7 runnable [0x00007fe87d9de000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c081de78> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c081de68> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c081de98> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"grid-nio-worker-client-listener-2-#56" #76 prio=5 os_prio=0
tid=0x00007fe92dd2c800 nid=0x48d6 runnable [0x00007fe87dadf000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c081e130> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c081e120> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c081e150> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"grid-nio-worker-client-listener-1-#55" #75 prio=5 os_prio=0
tid=0x00007fe92dd2b000 nid=0x48d5 runnable [0x00007fe87dbe0000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c081e3e8> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c081e3d8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c081e408> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"grid-nio-worker-client-listener-0-#54" #74 prio=5 os_prio=0
tid=0x00007fe92dd29800 nid=0x48d4 runnable [0x00007fe87dce1000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c081e6a0> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c081e690> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c081e6c0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"nio-acceptor-#53" #73 prio=5 os_prio=0 tid=0x00007fe92dca9000 nid=0x48d3
runnable [0x00007fe87dde2000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c081e958> (a sun.nio.ch.Util$3)
        - locked <0x00000004c081e948> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c081e968> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2772)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2744)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"tcp-comm-worker-#1" #70 prio=5 os_prio=0 tid=0x00007fe92daf1800 nid=0x48d2
waiting on condition [0x00007fe87e2e3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c01bbea8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4037)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

"grid-nio-worker-tcp-comm-3-#52" #66 prio=5 os_prio=0 tid=0x00007fe92daf0000
nid=0x48d1 runnable [0x00007fe87e3e4000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c00202b8> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c00202a8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c00202d8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"grid-nio-worker-tcp-comm-2-#51" #65 prio=5 os_prio=0 tid=0x00007fe92daee000
nid=0x48d0 runnable [0x00007fe87e4e5000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c0020dc8> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c0020db8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c0020de8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"grid-nio-worker-tcp-comm-1-#50" #64 prio=5 os_prio=0 tid=0x00007fe92daed800
nid=0x48cf runnable [0x00007fe87e7e8000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c017b740> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c0000220> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c016bf48> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"grid-nio-worker-tcp-comm-0-#49" #63 prio=5 os_prio=0 tid=0x00007fe92dae8800
nid=0x48ce runnable [0x00007fe87e5e6000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c0021078> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000004c0021068> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c0021098> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2043)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"nio-acceptor-#48" #62 prio=5 os_prio=0 tid=0x00007fe92daea000 nid=0x48cd
runnable [0x00007fe87e6e7000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000004c0000410> (a sun.nio.ch.Util$3)
        - locked <0x00000004c0000400> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000004c0000420> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2772)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2744)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"grid-timeout-worker-#47" #61 prio=5 os_prio=0 tid=0x00007fe92da49000
nid=0x48c9 in Object.wait() [0x00007fe87e8e9000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:195)
        - locked <0x00000004c00c5170> (a java.lang.Object)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

"ignite-clock" #60 daemon prio=10 os_prio=0 tid=0x00007fe92da53000
nid=0x48c8 sleeping[0x00007fe87ebea000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.util.IgniteUtils$22.run(IgniteUtils.java:3263)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-31-#40" #53 prio=5 os_prio=0 tid=0x00007fe92d947800
nid=0x48c1 waiting on condition [0x00007fe87f2f1000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-30-#39" #52 prio=5 os_prio=0 tid=0x00007fe92d945800
nid=0x48c0 waiting on condition [0x00007fe87f3f2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-29-#38" #51 prio=5 os_prio=0 tid=0x00007fe92d943000
nid=0x48bf waiting on condition [0x00007fe87f4f3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-28-#37" #50 prio=5 os_prio=0 tid=0x00007fe92d941800
nid=0x48be waiting on condition [0x00007fe87f5f4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-27-#36" #49 prio=5 os_prio=0 tid=0x00007fe92d93f800
nid=0x48bd waiting on condition [0x00007fe87f6f5000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-26-#35" #48 prio=5 os_prio=0 tid=0x00007fe92d93d800
nid=0x48bc waiting on condition [0x00007fe87f7f6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-25-#34" #47 prio=5 os_prio=0 tid=0x00007fe92d93b800
nid=0x48bb waiting on condition [0x00007fe87f8f7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-24-#33" #46 prio=5 os_prio=0 tid=0x00007fe92d939800
nid=0x48ba waiting on condition [0x00007fe87f9f8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-23-#32" #45 prio=5 os_prio=0 tid=0x00007fe92d938000
nid=0x48b9 waiting on condition [0x00007fe87faf9000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-22-#31" #44 prio=5 os_prio=0 tid=0x00007fe92d936800
nid=0x48b8 waiting on condition [0x00007fe87fbfa000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-21-#30" #43 prio=5 os_prio=0 tid=0x00007fe92d934800
nid=0x48b7 waiting on condition [0x00007fe87fcfb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-20-#29" #42 prio=5 os_prio=0 tid=0x00007fe92d933000
nid=0x48b6 waiting on condition [0x00007fe87fdfc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-19-#28" #41 prio=5 os_prio=0 tid=0x00007fe92d931000
nid=0x48b5 waiting on condition [0x00007fe87fefd000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-18-#27" #40 prio=5 os_prio=0 tid=0x00007fe92d92f800
nid=0x48b4 waiting on condition [0x00007fe87fffe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-17-#26" #39 prio=5 os_prio=0 tid=0x00007fe92d92d800
nid=0x48b3 waiting on condition [0x00007fe8a41ef000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-16-#25" #38 prio=5 os_prio=0 tid=0x00007fe92d92c000
nid=0x48b2 waiting on condition [0x00007fe8a42f0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-15-#24" #37 prio=5 os_prio=0 tid=0x00007fe92d92a800
nid=0x48b1 waiting on condition [0x00007fe8a43f1000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-14-#23" #36 prio=5 os_prio=0 tid=0x00007fe92d928800
nid=0x48b0 waiting on condition [0x00007fe8a44f2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-13-#22" #35 prio=5 os_prio=0 tid=0x00007fe92d926800
nid=0x48af waiting on condition [0x00007fe8a45f3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-12-#21" #34 prio=5 os_prio=0 tid=0x00007fe92d925000
nid=0x48ae waiting on condition [0x00007fe8a46f4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-11-#20" #33 prio=5 os_prio=0 tid=0x00007fe92d923000
nid=0x48ad waiting on condition [0x00007fe8a47f5000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-10-#19" #32 prio=5 os_prio=0 tid=0x00007fe92d921800
nid=0x48ac waiting on condition [0x00007fe8a48f6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-9-#18" #31 prio=5 os_prio=0 tid=0x00007fe92d920000
nid=0x48ab waiting on condition [0x00007fe8a49f7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-8-#17" #30 prio=5 os_prio=0 tid=0x00007fe92d91e000
nid=0x48aa waiting on condition [0x00007fe8a4af8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-7-#16" #29 prio=5 os_prio=0 tid=0x00007fe92d91c800
nid=0x48a9 waiting on condition [0x00007fe8a4bf9000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-6-#15" #28 prio=5 os_prio=0 tid=0x00007fe92d91a800
nid=0x48a8 waiting on condition [0x00007fe8a4cfa000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-5-#14" #27 prio=5 os_prio=0 tid=0x00007fe92d919000
nid=0x48a7 waiting on condition [0x00007fe8a4dfb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-4-#13" #26 prio=5 os_prio=0 tid=0x00007fe92d917000
nid=0x48a6 waiting on condition [0x00007fe8a4efc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-3-#12" #25 prio=5 os_prio=0 tid=0x00007fe92d915800
nid=0x48a5 runnable [0x00007fe8a4ffc000]
   java.lang.Thread.State: RUNNABLE
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.FullPageIdTable.putKey(FullPageIdTable.java:290)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.FullPageIdTable.put(FullPageIdTable.java:173)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:598)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:531)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
        at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:62)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:123)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4372)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:165)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4359)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4279)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:81)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:261)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158)
        at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:319)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2254)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2006)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.put(BPlusTree.java:1977)
        at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.put(H2TreeIndex.java:197)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:472)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423)
        at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:559)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1747)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:425)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1354)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1209)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1263)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:343)
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1693)
        - locked <0x00000006e69b3db0> (a
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2419)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1882)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1735)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1116)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-2-#11" #24 prio=5 os_prio=0 tid=0x00007fe92d913800
nid=0x48a4 waiting on condition [0x00007fe8d01b2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-1-#10" #23 prio=5 os_prio=0 tid=0x00007fe92d912000
nid=0x48a3 waiting on condition [0x00007fe8d02b3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"data-streamer-stripe-0-#9" #22 prio=5 os_prio=0 tid=0x00007fe92d910800
nid=0x48a2 waiting on condition [0x00007fe8d03b4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:605)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2321)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2298)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:121)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:397)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"sys-stripe-7-#8" #21 prio=5 os_prio=0 tid=0x00007fe92d90e800 nid=0x48a1
waiting on condition [0x00007fe8d04b4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c0ccefe8> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:542)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:531)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
        at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:62)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2InnerIO.getLookupRow(H2InnerIO.java:60)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2InnerIO.getLookupRow(H2InnerIO.java:33)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:123)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4372)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:165)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4359)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4279)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:81)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:261)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158)
        at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:319)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2254)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2006)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.put(BPlusTree.java:1977)
        at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.put(H2TreeIndex.java:197)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:472)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423)
        at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:559)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1747)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:425)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1354)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1209)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1263)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:343)
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1693)
        - locked <0x00000006e5223148> (a
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2419)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1882)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1735)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"sys-stripe-6-#7" #20 prio=5 os_prio=0 tid=0x00007fe92d90d000 nid=0x48a0
runnable [0x00007fe8d05b5000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c08f77a8> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:566)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:531)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
        at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:62)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:123)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4372)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:165)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4359)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4279)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:81)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:261)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158)
        at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:319)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2254)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2006)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.put(BPlusTree.java:1977)
        at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.put(H2TreeIndex.java:197)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:472)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423)
        at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:559)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1747)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:425)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1354)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1209)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1263)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:343)
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1693)
        - locked <0x00000006e502b620> (a
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2419)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1882)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1735)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"sys-stripe-5-#6" #19 prio=5 os_prio=0 tid=0x00007fe92d90b000 nid=0x489f
runnable [0x00007fe8d06b6000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
        at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:741)
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
        at
org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIO.read(RandomAccessFileIO.java:62)
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:322)
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:274)
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:259)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:613)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:531)
        at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:129)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2246)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2006)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.put(BPlusTree.java:1977)
        at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.put(H2TreeIndex.java:197)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.addToIndex(GridH2Table.java:537)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:488)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423)
        at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:559)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1747)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:425)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1354)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1209)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1263)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:343)
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1693)
        - locked <0x00000006e585eea8> (a
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2419)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1882)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1735)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"sys-stripe-4-#5" #18 prio=5 os_prio=0 tid=0x00007fe92d909800 nid=0x489e
runnable [0x00007fe8d07b7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c12a4f50> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:542)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:531)
        at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:129)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2246)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2006)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.put(BPlusTree.java:1977)
        at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.put(H2TreeIndex.java:197)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.addToIndex(GridH2Table.java:537)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:488)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423)
        at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:559)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1747)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:425)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1354)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1209)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1263)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:343)
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1693)
        - locked <0x00000006e6684968> (a
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2419)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1882)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1735)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"sys-stripe-3-#4" #17 prio=5 os_prio=0 tid=0x00007fe92d908000 nid=0x489d
waiting on condition [0x00007fe8d08b8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c12a4f50> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:542)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:531)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
        at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:62)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2InnerIO.getLookupRow(H2InnerIO.java:60)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2InnerIO.getLookupRow(H2InnerIO.java:33)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:123)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4372)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:165)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4359)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4279)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:81)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:261)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158)
        at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:319)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2254)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2006)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.put(BPlusTree.java:1977)
        at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.put(H2TreeIndex.java:197)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:472)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423)
        at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:559)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1747)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:425)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1354)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1209)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1263)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:343)
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1693)
        - locked <0x00000006e5e332b0> (a
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2419)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1882)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1735)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"sys-stripe-2-#3" #16 prio=5 os_prio=0 tid=0x00007fe92d901800 nid=0x489c
waiting on condition [0x00007fe8d09b9000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c12a4f50> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:542)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:531)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
        at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:62)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:123)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4372)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:165)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4359)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4279)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:81)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:261)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158)
        at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:319)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2254)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2006)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.put(BPlusTree.java:1977)
        at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.put(H2TreeIndex.java:197)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:472)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423)
        at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:559)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1747)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:425)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1354)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1209)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1263)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:343)
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1693)
        - locked <0x00000006e63aa288> (a
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2419)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1882)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1735)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"sys-stripe-1-#2" #15 prio=5 os_prio=0 tid=0x00007fe92d900000 nid=0x489b
waiting on condition [0x00007fe8d0aba000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c12a4f50> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:542)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:531)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
        at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:62)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:123)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4372)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:165)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4359)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4279)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:81)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:261)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158)
        at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:319)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2254)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2006)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.put(BPlusTree.java:1977)
        at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.put(H2TreeIndex.java:197)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:472)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423)
        at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:559)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1747)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:425)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1354)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1209)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1263)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:343)
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1693)
        - locked <0x00000006e56ef190> (a
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2419)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1882)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1735)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"sys-stripe-0-#1" #14 prio=5 os_prio=0 tid=0x00007fe92d8fe000 nid=0x489a
runnable [0x00007fe8d0bbb000]
   java.lang.Thread.State: RUNNABLE
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.FullPageIdTable.getKey(FullPageIdTable.java:332)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.FullPageIdTable.get(FullPageIdTable.java:129)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:570)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:531)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:140)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
        at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:62)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
        at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:123)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4372)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:165)
        at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:40)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4359)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4279)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:81)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:261)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158)
        at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:319)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2254)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2006)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.put(BPlusTree.java:1977)
        at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.put(H2TreeIndex.java:197)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:472)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423)
        at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:559)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1747)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:425)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1354)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1209)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1263)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:343)
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1693)
        - locked <0x00000006e5a6b168> (a
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2419)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1882)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1735)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

"RMI TCP Accept-0" #13 daemon prio=5 os_prio=0 tid=0x00007fe92d61f000
nid=0x4898 runnable [0x00007fe8d15d1000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
        at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
        at java.lang.Thread.run(Thread.java:748)

"RMI TCP Accept-49127" #12 daemon prio=5 os_prio=0 tid=0x00007fe92d60e000
nid=0x4897 runnable [0x00007fe8d16d2000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
        at java.lang.Thread.run(Thread.java:748)

"RMI TCP Accept-0" #11 daemon prio=5 os_prio=0 tid=0x00007fe92d5f9800
nid=0x4896 runnable [0x00007fe8d17d3000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
        at java.lang.Thread.run(Thread.java:748)

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fe92d3da800
nid=0x4895 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fe92d3cd800
nid=0x4894 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fe92d3cb000
nid=0x4893 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fe92d3c9000
nid=0x4892 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fe92d3c7000
nid=0x4891 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
tid=0x00007fe92d3c3000 nid=0x4890 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fe92d393800 nid=0x488f in
Object.wait() [0x00007fe8d2827000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000004c00c3de0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fe92d38f000
nid=0x488e in Object.wait() [0x00007fe8d2928000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x00000004c00c3e20> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007fe92c00c000 nid=0x4881 waiting on
condition [0x00007fe93416e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c09057f8> (a
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:335)

"VM Thread" os_prio=0 tid=0x00007fe92d387800 nid=0x488d runnable

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fe92c020800
nid=0x4882 runnable

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fe92c022800
nid=0x4883 runnable

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fe92c024000
nid=0x4884 runnable

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fe92c026000
nid=0x4885 runnable

"G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007fe92c040000
nid=0x488b runnable

"Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0
tid=0x00007fe92c041800 nid=0x488c runnable

"G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007fe92c02f800
nid=0x488a runnable

"G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007fe92c02d800
nid=0x4889 runnable

"G1 Concurrent Refinement Thread#2" os_prio=0 tid=0x00007fe92c02c000
nid=0x4888 runnable

"G1 Concurrent Refinement Thread#3" os_prio=0 tid=0x00007fe92c02a000
nid=0x4887 runnable

"G1 Concurrent Refinement Thread#4" os_prio=0 tid=0x00007fe92c028800
nid=0x4886 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fe92d621000 nid=0x4899
waiting on condition

JNI global references: 280





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
lawrencefinn lawrencefinn
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

BUMP.  Can anyone verify this?  If ignite cannot scale in this manner that is
fine, i'd just want to know if what i am seeing makes sense.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Fwd: Large durable caches

Hi Lawernce,

I believe Alexey Goncharuk was working on improving this scenario. Alexey, can you provide some of your findings here? 

D.

---------- Forwarded message ----------
From: lawrencefinn <[hidden email]>
Date: Mon, Mar 5, 2018 at 1:54 PM
Subject: Re: Large durable caches
To: [hidden email]


BUMP.  Can anyone verify this?  If ignite cannot scale in this manner that is
fine, i'd just want to know if what i am seeing makes sense.

lawrencefinn lawrencefinn
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

Hi Alexey.

Were there any findings?  Any updates would be helpful.

Thanks,
-Larry

On Thu, Mar 8, 2018 at 3:48 PM, Dmitriy Setrakyan <[hidden email]> wrote:
Hi Lawernce,

I believe Alexey Goncharuk was working on improving this scenario. Alexey, can you provide some of your findings here? 

D.

---------- Forwarded message ----------
From: lawrencefinn <[hidden email]>
Date: Mon, Mar 5, 2018 at 1:54 PM
Subject: Re: Large durable caches
To: [hidden email]


BUMP.  Can anyone verify this?  If ignite cannot scale in this manner that is
fine, i'd just want to know if what i am seeing makes sense.


alexey.goncharuk alexey.goncharuk
Reply | Threaded
Open this post in threaded view
|

Re: Large durable caches

Hi Larry,

Recently we've fixed the ticket [1] which may cause a major performance degradation when page replacement is present. Can you please try building Ignite from the master branch and check if the issue is still present?


2018-03-15 19:25 GMT+03:00 Larry <[hidden email]>:
Hi Alexey.

Were there any findings?  Any updates would be helpful.

Thanks,
-Larry

On Thu, Mar 8, 2018 at 3:48 PM, Dmitriy Setrakyan <[hidden email]> wrote:
Hi Lawernce,

I believe Alexey Goncharuk was working on improving this scenario. Alexey, can you provide some of your findings here? 

D.

---------- Forwarded message ----------
From: lawrencefinn <[hidden email]>
Date: Mon, Mar 5, 2018 at 1:54 PM
Subject: Re: Large durable caches
To: [hidden email]


BUMP.  Can anyone verify this?  If ignite cannot scale in this manner that is
fine, i'd just want to know if what i am seeing makes sense.



12