Ouch! Argument is invalid: Cache name must not be null or empty.

classic Classic list List threaded Threaded
10 messages Options
sv sv
Reply | Threaded
Open this post in threaded view
|

Ouch! Argument is invalid: Cache name must not be null or empty.

This post was updated on .
Getting "Ouch! Argument is invalid: Cache name must not be null or empty."
error when trying to execute SQL from within Zeppelin, using the Node Client
type of connection (jdbc:ignite:cfg:[URL]).

Zeppelin 0.8.0, Ignite 2.6 libs.  Cluster is 2.6.  

Almost seems like it wants a cache name, but I thought this bug was fixed in
2.6?  Perhaps something else is going on.  

[[EDIT:  looks like the forum software ate the attached log, so here it is again: ]]

INFO [2018-10-29 07:42:29,294] ({main} RemoteInterpreterServer.java[main]:260) - URL:jar:file:/zeppelin/lib/interpreter/zeppelin-interpreter-0.8.0.jar!/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.class
 INFO [2018-10-29 07:42:29,354] ({main} RemoteInterpreterServer.java[<init>]:161) - Launching ThriftServer at 100.96.13.14:36339
 INFO [2018-10-29 07:42:29,360] ({main} RemoteInterpreterServer.java[<init>]:165) - Starting remote interpreter server on port 36339
 INFO [2018-10-29 07:42:29,361] ({Thread-0} RemoteInterpreterServer.java[run]:202) - Starting remote interpreter server on port 36339
 INFO [2018-10-29 07:42:30,369] ({Thread-1} RemoteInterpreterUtils.java[registerInterpreter]:165) - callbackHost: 100.96.13.14, callbackPort: 37387, callbackInfo: CallbackInfo(host:100.96.13.14, port:36339)
 INFO [2018-10-29 07:42:30,557] ({pool-1-thread-1} RemoteInterpreterServer.java[createInterpreter]:310) - Instantiate interpreter org.apache.zeppelin.ignite.IgniteInterpreter
 INFO [2018-10-29 07:42:30,560] ({pool-1-thread-1} RemoteInterpreterServer.java[createInterpreter]:310) - Instantiate interpreter org.apache.zeppelin.ignite.IgniteSqlInterpreter
 INFO [2018-10-29 07:42:30,612] ({pool-2-thread-3} SchedulerFactory.java[jobStarted]:109) - Job 20181024-193853_1151909167 started by scheduler org.apache.zeppelin.ignite.IgniteSqlInterpreter1267025233
 INFO [2018-10-29 07:42:30,667] ({pool-2-thread-3} IgniteSqlInterpreter.java[open]:83) - connect to jdbc:ignite:cfg://[[REDACTED]]
 INFO [2018-10-29 07:42:30,747] ({pool-2-thread-3} XmlBeanDefinitionReader.java[loadBeanDefinitions]:317) - Loading XML bean definitions from URL [[[REDACTED]]]
 INFO [2018-10-29 07:42:31,457] ({pool-2-thread-3} AbstractApplicationContext.java[prepareRefresh]:583) - Refreshing org.springframework.context.support.GenericApplicationContext@58b6e929: startup date [Mon Oct 29 07:42:31 UTC 2018]; root of context hierarchy
 INFO [2018-10-29 07:42:34,538] ({pool-2-thread-3} IgniteSqlInterpreter.java[open]:88) - Successfully created JDBC connection
ERROR [2018-10-29 07:42:34,543] ({pool-2-thread-3} IgniteSqlInterpreter.java[interpret]:147) - Exception in IgniteSqlInterpreter while InterpreterResult interpret:
java.sql.SQLException: Failed to query Ignite.
        at org.apache.ignite.internal.jdbc2.JdbcUtils.convertToSqlException(JdbcUtils.java:184)
        at org.apache.ignite.internal.jdbc2.JdbcUtils.convertToSqlException(JdbcUtils.java:158)
        at org.apache.ignite.internal.jdbc2.JdbcStatement.executeSingle(JdbcStatement.java:181)
        at org.apache.ignite.internal.jdbc2.JdbcStatement.execute0(JdbcStatement.java:195)
        at org.apache.ignite.internal.jdbc2.JdbcStatement.executeQuery(JdbcStatement.java:96)
        at org.apache.zeppelin.ignite.IgniteSqlInterpreter.interpret(IgniteSqlInterpreter.java:121)
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:103)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:633)
        at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
        at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Ouch! Argument is invalid: Cache name must not be null or empty.
        at org.apache.ignite.internal.util.GridArgumentCheck.ensure(GridArgumentCheck.java:109)
        at org.apache.ignite.internal.processors.cache.GridCacheUtils.validateCacheName(GridCacheUtils.java:1590)
        at org.apache.ignite.internal.IgniteKernal.cache(IgniteKernal.java:2697)
        at org.apache.ignite.internal.jdbc2.JdbcQueryTask.call(JdbcQueryTask.java:147)
        at org.apache.ignite.internal.jdbc2.JdbcStatement.executeSingle(JdbcStatement.java:170)
        ... 14 more
 INFO [2018-10-29 07:42:34,548] ({pool-2-thread-3} SchedulerFactory.java[jobFinished]:115) - Job 20181024-193853_1151909167 finished by scheduler org.apache.zeppelin.ignite.IgniteSqlInterpreter1267025233





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

Hi,

Looks like you use JDBC client driver, which needs cache name to be configured(https://apacheignite-sql.readme.io/docs/jdbc-client-driver). I'd recommend using a thin JDBC driver instead - it doesn't need.(https://apacheignite-sql.readme.io/docs/jdbc-driver)

Evgenii

пн, 29 окт. 2018 г. в 10:48, sv <[hidden email]>:
Getting "Ouch! Argument is invalid: Cache name must not be null or empty."
error when trying to execute SQL from within Zeppelin, using the Node Client
type of connection (jdbc:ignite:cfg:[URL]).

Zeppelin 0.8.0, Ignite 2.6 libs.  Cluster is 2.6. 

Almost seems like it wants a cache name, but I thought this bug was fixed in
2.6?  Perhaps something else is going on.   







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
sv sv
Reply | Threaded
Open this post in threaded view
|

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

Hi, to clarify:
1) Thin client works fine
2) Thick client using port 11211 works fine
3) The "Client Node" connection is the one that was not working.

I got the Client Node connection (#3) to work by specifying the cache name
in the connection string.  Now it connects, and all SQL works well.  

I thought that this was fixed already in 2.6, so that you didn't have to
specify the cache name anymore?  In any case, it works.  

The next problem that I ran into:  executing scala code (using %ignite)
makes the entire Ignite interpreter in Zeppelin hang, without any errors in
logs.  How do I debug this?  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

> executing scala code (using %ignite) makes the entire Ignite interpreter in Zeppelin hang, without any errors in logs.  How do I debug this?  
When something hangs, you have to check thread dumps to understand where and why it hangs. Also, sometimes, it worth to enable another level of the logging - INFO or even DEBUG.

Evgenii

чт, 1 нояб. 2018 г. в 11:05, sv <[hidden email]>:
Hi, to clarify:
1) Thin client works fine
2) Thick client using port 11211 works fine
3) The "Client Node" connection is the one that was not working.

I got the Client Node connection (#3) to work by specifying the cache name
in the connection string.  Now it connects, and all SQL works well. 

I thought that this was fixed already in 2.6, so that you didn't have to
specify the cache name anymore?  In any case, it works. 

The next problem that I ran into:  executing scala code (using %ignite)
makes the entire Ignite interpreter in Zeppelin hang, without any errors in
logs.  How do I debug this? 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
sv sv
Reply | Threaded
Open this post in threaded view
|

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

Evgenii,
I am not sure if it even gets to the Ignite cluster.

Here is what I see in Zeppelin interpreter log, when I try to execute scala
code in a notebook:  

 INFO [2018-11-02 07:29:38,005] ({pool-2-thread-4}
SchedulerFactory.java[jobStarted]:109) - Job 20181102-071721_1550658378
started by scheduler org.apache.zeppelin.ignite.IgniteInterpreter1313679265

After that, nothing.

There is absolutely nothing in Ignite logs.

I have not increased the log level for Ignite, as that would mean I would
have to reload the entire cluster.  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
sv sv
Reply | Threaded
Open this post in threaded view
|

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

Actually, I already am running each Ignite node with IGNITE_QUIET=false, so
that should already have INFO and DEBUG logging enabled.  But still,
nothing.  I suspect that the Zeppelin Ignite interpreter is not reaching the
Ignite cluster.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

IGNITE_QUIET=false enables only INFO logging. Well, then something stuck on the Zeppelin side, still it worth to collect the thread dumps from it.

Evgenii

пт, 2 нояб. 2018 г. в 10:39, sv <[hidden email]>:
Actually, I already am running each Ignite node with IGNITE_QUIET=false, so
that should already have INFO and DEBUG logging enabled.  But still,
nothing.  I suspect that the Zeppelin Ignite interpreter is not reaching the
Ignite cluster.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
sv sv
Reply | Threaded
Open this post in threaded view
|

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

Evgenii,

1) How do I enable DEBUG logging on the Ignite cluster?

2) I used jstack to produce a thread dump of the Zeppelin Ignite interpreter
(which runs in a separate process).  I pasted it below.  Does anything look
out of place to you?

2018-11-03 09:32:37
Full thread dump OpenJDK 64-Bit Server VM (25.171-b11 mixed mode):

"Attach Listener" #82 daemon prio=9 os_prio=0 tid=0x00007f202c001000
nid=0x37b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"tcp-client-disco-msg-worker-#4" #81 prio=5 os_prio=0 tid=0x00007f1fd1f2a000
nid=0x2fb waiting on condition [0x00007f1fce1e6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:522)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1846)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1561)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"tcp-client-disco-sock-reader-#3" #80 prio=5 os_prio=0
tid=0x00007f1fd1f25000 nid=0x2fa in Object.wait() [0x00007f1fce2e7000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketReader.body(ClientImpl.java:1006)
        - locked <0x00000000ee2bc268> (a java.lang.Object)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"tcp-client-disco-sock-writer-#2" #79 prio=5 os_prio=0
tid=0x00007f1fd1f24800 nid=0x2f9 in Object.wait() [0x00007f1fce3e8000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1209)
        - locked <0x00000000ee2bbd88> (a java.lang.Object)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"TcpDiscoverySpi.timer" #78 prio=5 os_prio=0 tid=0x00007f1fd1f20800
nid=0x2f8 in Object.wait() [0x00007f1fce4e9000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00000000ee228468> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

   Locked ownable synchronizers:
        - None

"grid-data-loader-flusher-#39" #77 prio=5 os_prio=0 tid=0x00007f1fd1efb000
nid=0x2f7 waiting on condition [0x00007f1fce5ea000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000edc52cc0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-3-#37" #75 prio=5 os_prio=0
tid=0x00007f1fd1eea800 nid=0x2f6 runnable [0x00007f1fce8eb000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000edb0fdf0> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000ed992ff0> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed992ed8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-2-#36" #74 prio=5 os_prio=0
tid=0x00007f1fd1ee8800 nid=0x2f5 runnable [0x00007f1fce9ec000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ed98e750> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000ed97e4f8> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed97e3e0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-1-#35" #73 prio=5 os_prio=0
tid=0x00007f1fd1ee7000 nid=0x2f4 runnable [0x00007f1fceaed000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ed979c58> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000ed969a00> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed9698e8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-0-#34" #72 prio=5 os_prio=0
tid=0x00007f1fd1e67800 nid=0x2f3 runnable [0x00007f1fcebee000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ed965160> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000ed954f08> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed954df0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"nio-acceptor-#33" #71 prio=5 os_prio=0 tid=0x00007f1fd1e66800 nid=0x2f2
runnable [0x00007f1fcecef000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ed944218> (a sun.nio.ch.Util$3)
        - locked <0x00000000ed944208> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed9440f0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2870)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2822)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"tcp-comm-worker-#1" #70 prio=5 os_prio=0 tid=0x00007f1fd1cdf800 nid=0x2f1
waiting on condition [0x00007f1fcf1f0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000fd641060> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4207)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-3-#32" #66 prio=5 os_prio=0 tid=0x00007f1fd1cdd800
nid=0x2f0 runnable [0x00007f1fcf2f1000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda143b0> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000fda16430> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda14308> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-2-#31" #65 prio=5 os_prio=0 tid=0x00007f1fd1cd7000
nid=0x2ef runnable [0x00007f1fcf6f5000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda16b38> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000fda49580> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda16a90> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-1-#30" #64 prio=5 os_prio=0 tid=0x00007f1fd1cd8800
nid=0x2ee runnable [0x00007f1fcf4f3000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda49c88> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000fda4bd08> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda49be0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-0-#29" #63 prio=5 os_prio=0 tid=0x00007f1fd1cd5800
nid=0x2ed runnable [0x00007f1fcf5f4000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda4c410> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000fda4e490> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda4c368> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"nio-acceptor-#28" #62 prio=5 os_prio=0 tid=0x00007f1fd1cdb000 nid=0x2ec
runnable [0x00007f1fcf3f2000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda13c60> (a sun.nio.ch.Util$3)
        - locked <0x00000000fda13c50> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda13838> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2870)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2822)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-timeout-worker-#27" #61 prio=5 os_prio=0 tid=0x00007f1fd1c4b800
nid=0x2e8 in Object.wait() [0x00007f1fcf7f6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:203)
        - locked <0x00000000fd9add58> (a java.lang.Object)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-clock" #60 daemon prio=10 os_prio=0 tid=0x00007f1fd1c3f000 nid=0x2e7
sleeping[0x00007f1fcf8f7000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.util.IgniteUtils$22.run(IgniteUtils.java:3308)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"jvm-pause-detector-worker" #53 daemon prio=5 os_prio=0
tid=0x00007f1fd184e800 nid=0x2e0 sleeping[0x00007f1fcfffe000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:90)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-7-#20" #52 prio=5 os_prio=0 tid=0x00007f1fd1805800
nid=0x2df waiting on condition [0x00007f1ff836b000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-6-#19" #51 prio=5 os_prio=0 tid=0x00007f1fd1803800
nid=0x2de waiting on condition [0x00007f1ff846c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-5-#18" #50 prio=5 os_prio=0 tid=0x00007f1fd1802000
nid=0x2dd waiting on condition [0x00007f1ff856d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-4-#17" #49 prio=5 os_prio=0 tid=0x00007f1fd1800000
nid=0x2dc waiting on condition [0x00007f1ff866e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-3-#16" #48 prio=5 os_prio=0 tid=0x00007f1fd17fe000
nid=0x2db waiting on condition [0x00007f1ff876f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-2-#15" #47 prio=5 os_prio=0 tid=0x00007f1fd17fc000
nid=0x2da waiting on condition [0x00007f1ff8870000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-1-#14" #46 prio=5 os_prio=0 tid=0x00007f1fd17fa800
nid=0x2d9 waiting on condition [0x00007f1ff8971000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-0-#13" #45 prio=5 os_prio=0 tid=0x00007f1fd17f8800
nid=0x2d8 waiting on condition [0x00007f1ff8a72000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-7-#12" #44 prio=5 os_prio=0 tid=0x00007f1fd17f6800 nid=0x2d7
waiting on condition [0x00007f1ff8b73000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-6-#11" #43 prio=5 os_prio=0 tid=0x00007f1fd17f4800 nid=0x2d6
waiting on condition [0x00007f1ff8c74000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-5-#10" #42 prio=5 os_prio=0 tid=0x00007f1fd17f2800 nid=0x2d5
waiting on condition [0x00007f1ff8d75000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-4-#9" #41 prio=5 os_prio=0 tid=0x00007f1fd17f0800 nid=0x2d4
waiting on condition [0x00007f1ff8e76000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-3-#8" #40 prio=5 os_prio=0 tid=0x00007f1fd17ee800 nid=0x2d3
waiting on condition [0x00007f1ff8f77000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-2-#7" #39 prio=5 os_prio=0 tid=0x00007f1fd17d3000 nid=0x2d2
waiting on condition [0x00007f1ff9078000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-1-#6" #38 prio=5 os_prio=0 tid=0x00007f1fd17d1800 nid=0x2d1
waiting on condition [0x00007f1ff9179000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-0-#5" #37 prio=5 os_prio=0 tid=0x00007f1fd17cf000 nid=0x2d0
waiting on condition [0x00007f1ffa0f1000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"JobProgressPoller, jobId=20181102-071721_1550658378" #36 prio=5 os_prio=0
tid=0x00007f1fd0003000 nid=0x2cf waiting on condition [0x00007f1ffadf2000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgressPoller.java:61)

   Locked ownable synchronizers:
        - None

"pool-2-thread-7" #35 prio=5 os_prio=0 tid=0x00007f2008001000 nid=0x2ce
waiting on condition [0x00007f1ffaef3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0006268> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pool-2-thread-6" #33 prio=5 os_prio=0 tid=0x00007f1ffc005800 nid=0x2c9
waiting on condition [0x00007f1ffb1f4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0006268> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pool-2-thread-5" #31 prio=5 os_prio=0 tid=0x00007f1ffc004800 nid=0x2c4
waiting on condition [0x00007f1ffb2f4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ee2280f0> (a
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl.spiStart(ClientImpl.java:271)
        at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
        at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
        at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721)
        at
org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
        - locked <0x00000000fd9eaa10> (a
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
        at
org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:671)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:596)
        at org.apache.ignite.Ignition.start(Ignition.java:327)
        at
org.apache.zeppelin.ignite.IgniteInterpreter.getIgnite(IgniteInterpreter.java:190)
        at
org.apache.zeppelin.ignite.IgniteInterpreter.initIgnite(IgniteInterpreter.java:208)
        at
org.apache.zeppelin.ignite.IgniteInterpreter.open(IgniteInterpreter.java:127)
        at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
        - locked <0x00000000c000f738> (a
org.apache.zeppelin.ignite.IgniteInterpreter)
        - locked <0x00000000c000f810> (a
org.apache.zeppelin.interpreter.LazyOpenInterpreter)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:617)
        at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
        at
org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000fda11fc8> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000000fda12080> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000000fda12138> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000000fda121f0> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000000c000f900> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-2-thread-4" #29 prio=5 os_prio=0 tid=0x00007f1ffc003000 nid=0x2bf
waiting on condition [0x00007f1ffb3f6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0006268> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-client-exec-3-5" #28 daemon prio=5 os_prio=0 tid=0x00007f1fd0001000
nid=0x2bb waiting on condition [0x00007f1ffb4f7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001bdb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-client-exec-3-4" #27 daemon prio=5 os_prio=0 tid=0x00007f1fe4001000
nid=0x2ba waiting on condition [0x00007f2018230000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001bdb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-client-exec-3-3" #26 daemon prio=5 os_prio=0 tid=0x00007f20001c9000
nid=0x2b9 waiting on condition [0x00007f1ffb5f8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001bdb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"a65ff0b2-c2ba-4d1c-8d67-25570203a476-topology-update" #25 daemon prio=5
os_prio=0 tid=0x00007f20001c5800 nid=0x2b8 waiting on condition
[0x00007f1ffb6f9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.client.impl.GridClientImpl$TopologyUpdaterThread.run(GridClientImpl.java:521)

   Locked ownable synchronizers:
        - None

"ignite-client-top-lsnr-1-2" #24 daemon prio=5 os_prio=0
tid=0x00007f1ff000d000 nid=0x2b7 waiting on condition [0x00007f1ffb7fa000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001c160> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-client-exec-3-2" #23 daemon prio=5 os_prio=0 tid=0x00007f20001ae000
nid=0x2b6 waiting on condition [0x00007f1ffb8fb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001bdb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-client-3-#4%gridClient%" #22 prio=5 os_prio=0
tid=0x00007f200017c800 nid=0x2b5 runnable [0x00007f1ffbbfc000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000c0007f00> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000c0007ef0> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000c0007f20> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-client-2-#3%gridClient%" #21 prio=5 os_prio=0
tid=0x00007f200017a800 nid=0x2b4 runnable [0x00007f1ffbcfd000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000c001c610> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000c001c600> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000c001c630> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-client-1-#2%gridClient%" #20 prio=5 os_prio=0
tid=0x00007f2000179000 nid=0x2b3 runnable [0x00007f1ffbdfe000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000c00237f8> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000c003a000> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000c00236e0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-client-0-#1%gridClient%" #19 prio=5 os_prio=0
tid=0x00007f2000177000 nid=0x2b2 runnable [0x00007f201812a000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000c003aa90> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000c003d658> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000c003a9e8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pool-2-thread-3" #16 prio=5 os_prio=0 tid=0x00007f1ffc001000 nid=0x2b0
waiting on condition [0x00007f2018331000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0006268> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pool-2-thread-2" #15 prio=5 os_prio=0 tid=0x00007f2004007000 nid=0x2af in
Object.wait() [0x00007f2018432000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:113)
        - locked <0x00000000c000fb00> (a java.util.LinkedList)
        - locked <0x00000000c000fb00> (a java.util.LinkedList)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c82c8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-2-thread-1" #14 prio=5 os_prio=0 tid=0x00007f2004005000 nid=0x2ae in
Object.wait() [0x00007f2018533000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:113)
        - locked <0x00000000c000f958> (a java.util.LinkedList)
        - locked <0x00000000c000f958> (a java.util.LinkedList)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c85d8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-1-thread-3" #13 prio=5 os_prio=0 tid=0x00007f2014005000 nid=0x2ad
runnable [0x00007f2018634000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked <0x00000000c00c8cf8> (a java.io.BufferedInputStream)
        at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
        at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c8f30> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-1-thread-2" #12 prio=5 os_prio=0 tid=0x00007f2014003800 nid=0x2ac in
Object.wait() [0x00007f2018948000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventClient.pollEvent(RemoteInterpreterEventClient.java:357)
        - locked <0x00000000c00804d8> (a java.util.LinkedList)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getEvent(RemoteInterpreterServer.java:970)
        at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:2000)
        at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:1985)
        at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c9808> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-1-thread-1" #11 prio=5 os_prio=0 tid=0x00007f2014002000 nid=0x2ab in
Object.wait() [0x00007f2018a49000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.interpret(RemoteInterpreterServer.java:447)
        - locked <0x00000000c00060b0> (a
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJobListener)
        at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$interpret.getResult(RemoteInterpreterService.java:1859)
        at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$interpret.getResult(RemoteInterpreterService.java:1844)
        at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c9080> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"Thread-0" #9 prio=5 os_prio=0 tid=0x00007f20645ef000 nid=0x2a6 runnable
[0x00007f2018b4a000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:129)
        at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
        at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:60)
        at
org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:161)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.run(RemoteInterpreterServer.java:203)

   Locked ownable synchronizers:
        - None

"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f20640c2000 nid=0x2a4
runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f20640bf000
nid=0x2a3 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f20640bd000
nid=0x2a2 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f20640ba000
nid=0x2a1 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f20640b8800
nid=0x2a0 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f2064086800 nid=0x29f in
Object.wait() [0x00007f2040ad0000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000000c00ceaf8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:212)

   Locked ownable synchronizers:
        - None

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f2064082000
nid=0x29e in Object.wait() [0x00007f2040bd1000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x00000000c00ced28> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

   Locked ownable synchronizers:
        - None

"main" #1 prio=5 os_prio=0 tid=0x00007f206400c800 nid=0x298 in Object.wait()
[0x00007f206bab0000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c007e0f0> (a
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer)
        at java.lang.Thread.join(Thread.java:1252)
        - locked <0x00000000c007e0f0> (a
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer)
        at java.lang.Thread.join(Thread.java:1326)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.main(RemoteInterpreterServer.java:274)

   Locked ownable synchronizers:
        - None

"VM Thread" os_prio=0 tid=0x00007f206407a000 nid=0x29d runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f2064022000 nid=0x299
runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f2064023800 nid=0x29a
runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f2064025800 nid=0x29b
runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f2064027000 nid=0x29c
runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f20640c4800 nid=0x2a5 waiting
on condition

JNI global references: 206




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

>1) How do I enable DEBUG logging on the Ignite cluster?
It depends on the logger that you use. You have to change the config file and give to the ignite package DEBUG level. It could be JUL, log4j or log4j2 config files.

> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology
Client node didn't even join the topology. Most possible you have incorrect discovery configuration. Also, I'd recommend to check client logs for more information why node can't connect to the cluster.


Evgenii

вс, 4 нояб. 2018 г. в 8:09, sv <[hidden email]>:
Evgenii,

1) How do I enable DEBUG logging on the Ignite cluster?

2) I used jstack to produce a thread dump of the Zeppelin Ignite interpreter
(which runs in a separate process).  I pasted it below.  Does anything look
out of place to you?

2018-11-03 09:32:37
Full thread dump OpenJDK 64-Bit Server VM (25.171-b11 mixed mode):

"Attach Listener" #82 daemon prio=9 os_prio=0 tid=0x00007f202c001000
nid=0x37b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"tcp-client-disco-msg-worker-#4" #81 prio=5 os_prio=0 tid=0x00007f1fd1f2a000
nid=0x2fb waiting on condition [0x00007f1fce1e6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:522)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1846)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1561)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"tcp-client-disco-sock-reader-#3" #80 prio=5 os_prio=0
tid=0x00007f1fd1f25000 nid=0x2fa in Object.wait() [0x00007f1fce2e7000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketReader.body(ClientImpl.java:1006)
        - locked <0x00000000ee2bc268> (a java.lang.Object)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"tcp-client-disco-sock-writer-#2" #79 prio=5 os_prio=0
tid=0x00007f1fd1f24800 nid=0x2f9 in Object.wait() [0x00007f1fce3e8000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1209)
        - locked <0x00000000ee2bbd88> (a java.lang.Object)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"TcpDiscoverySpi.timer" #78 prio=5 os_prio=0 tid=0x00007f1fd1f20800
nid=0x2f8 in Object.wait() [0x00007f1fce4e9000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00000000ee228468> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

   Locked ownable synchronizers:
        - None

"grid-data-loader-flusher-#39" #77 prio=5 os_prio=0 tid=0x00007f1fd1efb000
nid=0x2f7 waiting on condition [0x00007f1fce5ea000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000edc52cc0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-3-#37" #75 prio=5 os_prio=0
tid=0x00007f1fd1eea800 nid=0x2f6 runnable [0x00007f1fce8eb000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000edb0fdf0> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000ed992ff0> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed992ed8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-2-#36" #74 prio=5 os_prio=0
tid=0x00007f1fd1ee8800 nid=0x2f5 runnable [0x00007f1fce9ec000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ed98e750> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000ed97e4f8> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed97e3e0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-1-#35" #73 prio=5 os_prio=0
tid=0x00007f1fd1ee7000 nid=0x2f4 runnable [0x00007f1fceaed000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ed979c58> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000ed969a00> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed9698e8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-0-#34" #72 prio=5 os_prio=0
tid=0x00007f1fd1e67800 nid=0x2f3 runnable [0x00007f1fcebee000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ed965160> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000ed954f08> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed954df0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"nio-acceptor-#33" #71 prio=5 os_prio=0 tid=0x00007f1fd1e66800 nid=0x2f2
runnable [0x00007f1fcecef000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ed944218> (a sun.nio.ch.Util$3)
        - locked <0x00000000ed944208> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed9440f0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2870)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2822)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"tcp-comm-worker-#1" #70 prio=5 os_prio=0 tid=0x00007f1fd1cdf800 nid=0x2f1
waiting on condition [0x00007f1fcf1f0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000fd641060> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4207)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-3-#32" #66 prio=5 os_prio=0 tid=0x00007f1fd1cdd800
nid=0x2f0 runnable [0x00007f1fcf2f1000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda143b0> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000fda16430> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda14308> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-2-#31" #65 prio=5 os_prio=0 tid=0x00007f1fd1cd7000
nid=0x2ef runnable [0x00007f1fcf6f5000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda16b38> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000fda49580> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda16a90> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-1-#30" #64 prio=5 os_prio=0 tid=0x00007f1fd1cd8800
nid=0x2ee runnable [0x00007f1fcf4f3000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda49c88> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000fda4bd08> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda49be0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-0-#29" #63 prio=5 os_prio=0 tid=0x00007f1fd1cd5800
nid=0x2ed runnable [0x00007f1fcf5f4000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda4c410> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000fda4e490> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda4c368> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"nio-acceptor-#28" #62 prio=5 os_prio=0 tid=0x00007f1fd1cdb000 nid=0x2ec
runnable [0x00007f1fcf3f2000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda13c60> (a sun.nio.ch.Util$3)
        - locked <0x00000000fda13c50> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda13838> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2870)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2822)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-timeout-worker-#27" #61 prio=5 os_prio=0 tid=0x00007f1fd1c4b800
nid=0x2e8 in Object.wait() [0x00007f1fcf7f6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:203)
        - locked <0x00000000fd9add58> (a java.lang.Object)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-clock" #60 daemon prio=10 os_prio=0 tid=0x00007f1fd1c3f000 nid=0x2e7
sleeping[0x00007f1fcf8f7000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.util.IgniteUtils$22.run(IgniteUtils.java:3308)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"jvm-pause-detector-worker" #53 daemon prio=5 os_prio=0
tid=0x00007f1fd184e800 nid=0x2e0 sleeping[0x00007f1fcfffe000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:90)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-7-#20" #52 prio=5 os_prio=0 tid=0x00007f1fd1805800
nid=0x2df waiting on condition [0x00007f1ff836b000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-6-#19" #51 prio=5 os_prio=0 tid=0x00007f1fd1803800
nid=0x2de waiting on condition [0x00007f1ff846c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-5-#18" #50 prio=5 os_prio=0 tid=0x00007f1fd1802000
nid=0x2dd waiting on condition [0x00007f1ff856d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-4-#17" #49 prio=5 os_prio=0 tid=0x00007f1fd1800000
nid=0x2dc waiting on condition [0x00007f1ff866e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-3-#16" #48 prio=5 os_prio=0 tid=0x00007f1fd17fe000
nid=0x2db waiting on condition [0x00007f1ff876f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-2-#15" #47 prio=5 os_prio=0 tid=0x00007f1fd17fc000
nid=0x2da waiting on condition [0x00007f1ff8870000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-1-#14" #46 prio=5 os_prio=0 tid=0x00007f1fd17fa800
nid=0x2d9 waiting on condition [0x00007f1ff8971000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-0-#13" #45 prio=5 os_prio=0 tid=0x00007f1fd17f8800
nid=0x2d8 waiting on condition [0x00007f1ff8a72000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-7-#12" #44 prio=5 os_prio=0 tid=0x00007f1fd17f6800 nid=0x2d7
waiting on condition [0x00007f1ff8b73000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-6-#11" #43 prio=5 os_prio=0 tid=0x00007f1fd17f4800 nid=0x2d6
waiting on condition [0x00007f1ff8c74000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-5-#10" #42 prio=5 os_prio=0 tid=0x00007f1fd17f2800 nid=0x2d5
waiting on condition [0x00007f1ff8d75000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-4-#9" #41 prio=5 os_prio=0 tid=0x00007f1fd17f0800 nid=0x2d4
waiting on condition [0x00007f1ff8e76000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-3-#8" #40 prio=5 os_prio=0 tid=0x00007f1fd17ee800 nid=0x2d3
waiting on condition [0x00007f1ff8f77000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-2-#7" #39 prio=5 os_prio=0 tid=0x00007f1fd17d3000 nid=0x2d2
waiting on condition [0x00007f1ff9078000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-1-#6" #38 prio=5 os_prio=0 tid=0x00007f1fd17d1800 nid=0x2d1
waiting on condition [0x00007f1ff9179000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-0-#5" #37 prio=5 os_prio=0 tid=0x00007f1fd17cf000 nid=0x2d0
waiting on condition [0x00007f1ffa0f1000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"JobProgressPoller, jobId=20181102-071721_1550658378" #36 prio=5 os_prio=0
tid=0x00007f1fd0003000 nid=0x2cf waiting on condition [0x00007f1ffadf2000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgressPoller.java:61)

   Locked ownable synchronizers:
        - None

"pool-2-thread-7" #35 prio=5 os_prio=0 tid=0x00007f2008001000 nid=0x2ce
waiting on condition [0x00007f1ffaef3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0006268> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pool-2-thread-6" #33 prio=5 os_prio=0 tid=0x00007f1ffc005800 nid=0x2c9
waiting on condition [0x00007f1ffb1f4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0006268> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pool-2-thread-5" #31 prio=5 os_prio=0 tid=0x00007f1ffc004800 nid=0x2c4
waiting on condition [0x00007f1ffb2f4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ee2280f0> (a
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl.spiStart(ClientImpl.java:271)
        at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
        at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
        at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721)
        at
org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
        - locked <0x00000000fd9eaa10> (a
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
        at
org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:671)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:596)
        at org.apache.ignite.Ignition.start(Ignition.java:327)
        at
org.apache.zeppelin.ignite.IgniteInterpreter.getIgnite(IgniteInterpreter.java:190)
        at
org.apache.zeppelin.ignite.IgniteInterpreter.initIgnite(IgniteInterpreter.java:208)
        at
org.apache.zeppelin.ignite.IgniteInterpreter.open(IgniteInterpreter.java:127)
        at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
        - locked <0x00000000c000f738> (a
org.apache.zeppelin.ignite.IgniteInterpreter)
        - locked <0x00000000c000f810> (a
org.apache.zeppelin.interpreter.LazyOpenInterpreter)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:617)
        at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
        at
org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000fda11fc8> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000000fda12080> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000000fda12138> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000000fda121f0> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000000c000f900> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-2-thread-4" #29 prio=5 os_prio=0 tid=0x00007f1ffc003000 nid=0x2bf
waiting on condition [0x00007f1ffb3f6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0006268> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-client-exec-3-5" #28 daemon prio=5 os_prio=0 tid=0x00007f1fd0001000
nid=0x2bb waiting on condition [0x00007f1ffb4f7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001bdb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-client-exec-3-4" #27 daemon prio=5 os_prio=0 tid=0x00007f1fe4001000
nid=0x2ba waiting on condition [0x00007f2018230000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001bdb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-client-exec-3-3" #26 daemon prio=5 os_prio=0 tid=0x00007f20001c9000
nid=0x2b9 waiting on condition [0x00007f1ffb5f8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001bdb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"a65ff0b2-c2ba-4d1c-8d67-25570203a476-topology-update" #25 daemon prio=5
os_prio=0 tid=0x00007f20001c5800 nid=0x2b8 waiting on condition
[0x00007f1ffb6f9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.client.impl.GridClientImpl$TopologyUpdaterThread.run(GridClientImpl.java:521)

   Locked ownable synchronizers:
        - None

"ignite-client-top-lsnr-1-2" #24 daemon prio=5 os_prio=0
tid=0x00007f1ff000d000 nid=0x2b7 waiting on condition [0x00007f1ffb7fa000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001c160> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-client-exec-3-2" #23 daemon prio=5 os_prio=0 tid=0x00007f20001ae000
nid=0x2b6 waiting on condition [0x00007f1ffb8fb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001bdb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-client-3-#4%gridClient%" #22 prio=5 os_prio=0
tid=0x00007f200017c800 nid=0x2b5 runnable [0x00007f1ffbbfc000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000c0007f00> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000c0007ef0> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000c0007f20> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-client-2-#3%gridClient%" #21 prio=5 os_prio=0
tid=0x00007f200017a800 nid=0x2b4 runnable [0x00007f1ffbcfd000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000c001c610> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000c001c600> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000c001c630> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-client-1-#2%gridClient%" #20 prio=5 os_prio=0
tid=0x00007f2000179000 nid=0x2b3 runnable [0x00007f1ffbdfe000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000c00237f8> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000c003a000> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000c00236e0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-client-0-#1%gridClient%" #19 prio=5 os_prio=0
tid=0x00007f2000177000 nid=0x2b2 runnable [0x00007f201812a000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000c003aa90> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000c003d658> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000c003a9e8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pool-2-thread-3" #16 prio=5 os_prio=0 tid=0x00007f1ffc001000 nid=0x2b0
waiting on condition [0x00007f2018331000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0006268> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pool-2-thread-2" #15 prio=5 os_prio=0 tid=0x00007f2004007000 nid=0x2af in
Object.wait() [0x00007f2018432000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:113)
        - locked <0x00000000c000fb00> (a java.util.LinkedList)
        - locked <0x00000000c000fb00> (a java.util.LinkedList)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c82c8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-2-thread-1" #14 prio=5 os_prio=0 tid=0x00007f2004005000 nid=0x2ae in
Object.wait() [0x00007f2018533000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:113)
        - locked <0x00000000c000f958> (a java.util.LinkedList)
        - locked <0x00000000c000f958> (a java.util.LinkedList)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c85d8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-1-thread-3" #13 prio=5 os_prio=0 tid=0x00007f2014005000 nid=0x2ad
runnable [0x00007f2018634000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked <0x00000000c00c8cf8> (a java.io.BufferedInputStream)
        at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
        at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c8f30> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-1-thread-2" #12 prio=5 os_prio=0 tid=0x00007f2014003800 nid=0x2ac in
Object.wait() [0x00007f2018948000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventClient.pollEvent(RemoteInterpreterEventClient.java:357)
        - locked <0x00000000c00804d8> (a java.util.LinkedList)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getEvent(RemoteInterpreterServer.java:970)
        at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:2000)
        at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:1985)
        at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c9808> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-1-thread-1" #11 prio=5 os_prio=0 tid=0x00007f2014002000 nid=0x2ab in
Object.wait() [0x00007f2018a49000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.interpret(RemoteInterpreterServer.java:447)
        - locked <0x00000000c00060b0> (a
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJobListener)
        at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$interpret.getResult(RemoteInterpreterService.java:1859)
        at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$interpret.getResult(RemoteInterpreterService.java:1844)
        at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c9080> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"Thread-0" #9 prio=5 os_prio=0 tid=0x00007f20645ef000 nid=0x2a6 runnable
[0x00007f2018b4a000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:129)
        at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
        at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:60)
        at
org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:161)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.run(RemoteInterpreterServer.java:203)

   Locked ownable synchronizers:
        - None

"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f20640c2000 nid=0x2a4
runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f20640bf000
nid=0x2a3 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f20640bd000
nid=0x2a2 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f20640ba000
nid=0x2a1 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f20640b8800
nid=0x2a0 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f2064086800 nid=0x29f in
Object.wait() [0x00007f2040ad0000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000000c00ceaf8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:212)

   Locked ownable synchronizers:
        - None

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f2064082000
nid=0x29e in Object.wait() [0x00007f2040bd1000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x00000000c00ced28> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

   Locked ownable synchronizers:
        - None

"main" #1 prio=5 os_prio=0 tid=0x00007f206400c800 nid=0x298 in Object.wait()
[0x00007f206bab0000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c007e0f0> (a
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer)
        at java.lang.Thread.join(Thread.java:1252)
        - locked <0x00000000c007e0f0> (a
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer)
        at java.lang.Thread.join(Thread.java:1326)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.main(RemoteInterpreterServer.java:274)

   Locked ownable synchronizers:
        - None

"VM Thread" os_prio=0 tid=0x00007f206407a000 nid=0x29d runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f2064022000 nid=0x299
runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f2064023800 nid=0x29a
runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f2064025800 nid=0x29b
runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f2064027000 nid=0x29c
runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f20640c4800 nid=0x2a5 waiting
on condition

JNI global references: 206




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
sv sv
Reply | Threaded
Open this post in threaded view
|

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

Evgenii,

"Client node didn't even join the topology. Most possible you have incorrect
discovery configuration. Also, I'd recommend to check client logs for more
information why node can't connect to the cluster."

In Zeppelin setup for the Ignite interpreter, it uses the same configuration
for Scala connection as it does for Ignite SQL connections.  I have tested
the SQL connections, and they work perfectly (both the 11211 port and the
Node Client using XML).  It is odd that the Scala connection doesn't work.

As far as client logs are concerned, Zeppelin only generates the interpreter
logs, such as, zeppelin-interpreter-ignite--zeppelin-db54cc4b4-kppw2.log,  
and that log only states that the job was started:

 INFO [2018-11-03 09:15:41,893] ({pool-2-thread-5}
SchedulerFactory.java[jobStarted]:109) - Job 20181102-071721_1550658378
started by scheduler org.apache.zeppelin.ignite.IgniteInterpreter113609227

And then nothing after that, as the interpreter process hangs.  

Do you have any idea how to turn on DEBUG level logging for the Zeppelin
Ignite interpreter?  





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/