Ignite 2.7.0 : server node: null pointer exception

classic Classic list List threaded Threaded
8 messages Options
Mahesh Renduchintala Mahesh Renduchintala
Reply | Threaded
Open this post in threaded view
|

Ignite 2.7.0 : server node: null pointer exception

Hi, 


We are seeing a null pointer exception in Ignite server node. 


java.lang.NullPointerException
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.latch.ExchangeLatchManager.canSkipJoiningNodes(ExchangeLatchManager.java:327)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1401)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:806)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2667)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2539)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)

Please see attached. 




is there a known bug around this?


regards

Mahesh


ignitenullpointer.log (231K) Download Attachment
ibelyakov ibelyakov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.7.0 : server node: null pointer exception

Hi,

According to the logs I see that you've an issue with JVM pauses and nodes disconnection, seems like it is a cause of the problem. Are you able to reproduce this issue? 
Also log contains data only for the last 15 seconds and for 1 node, could you please provide full logs from all nodes in the cluster for futher investigation?

Regards,
Igor

On Tue, Jul 23, 2019 at 6:02 PM Mahesh Renduchintala <[hidden email]> wrote:

Hi, 


We are seeing a null pointer exception in Ignite server node. 


java.lang.NullPointerException
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.latch.ExchangeLatchManager.canSkipJoiningNodes(ExchangeLatchManager.java:327)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1401)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:806)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2667)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2539)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)

Please see attached. 




is there a known bug around this?


regards

Mahesh

dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.7.0 : server node: null pointer exception

Igor, thanks for looking into this. 

Even if the root cause is on an application end, our code mustn't generate an NPE.

Ignite dev community and Ivan Rakov in particular, have you seen this before? Should we file a ticket or the issue has already been resolved?

-
Denis


On Wed, Jul 24, 2019 at 7:19 AM Игорь Беляков <[hidden email]> wrote:
Hi,

According to the logs I see that you've an issue with JVM pauses and nodes disconnection, seems like it is a cause of the problem. Are you able to reproduce this issue? 
Also log contains data only for the last 15 seconds and for 1 node, could you please provide full logs from all nodes in the cluster for futher investigation?

Regards,
Igor

On Tue, Jul 23, 2019 at 6:02 PM Mahesh Renduchintala <[hidden email]> wrote:

Hi, 


We are seeing a null pointer exception in Ignite server node. 


java.lang.NullPointerException
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.latch.ExchangeLatchManager.canSkipJoiningNodes(ExchangeLatchManager.java:327)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1401)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:806)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2667)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2539)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)

Please see attached. 




is there a known bug around this?


regards

Mahesh

Mahesh Renduchintala Mahesh Renduchintala
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.7.0 : server node: null pointer exception

The clients come in and get disconnected from the cluster for many reasons - some intentionally and some due to poor network. 
Cant have Ignite nodes crashing with null pointer exception. 


slava.koptilin slava.koptilin
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.7.0 : server node: null pointer exception

Hi Mahesh,

It definitely looks like a bug. I have created this ticket in order to track the issue  https://issues.apache.org/jira/browse/IGNITE-12013  
As a temporary workaround, I would propose increasing discovery history size
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteSystemProperties.html#IGNITE_DISCOVERY_HISTORY_SIZE
The default value is 500, so let's try to use 700, for instance.
(You need to pass it to your java process -DIGNITE_DISCOVERY_HISTORY_SIZE=700)

Thanks,
S.

чт, 25 июл. 2019 г. в 08:12, Mahesh Renduchintala <[hidden email]>:
The clients come in and get disconnected from the cluster for many reasons - some intentionally and some due to poor network. 
Cant have Ignite nodes crashing with null pointer exception. 


Mahesh Renduchintala Mahesh Renduchintala
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.7.0 : server node: null pointer exception

IGNITE_DISCOVERY_HISTORY_SIZE=700

Does this go on the server side or the thick client side ?
slava.koptilin slava.koptilin
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.7.0 : server node: null pointer exception

Hi Mahesh,

Yes, it should be done on server nodes.

Thanks,
S.

чт, 25 июл. 2019 г. в 14:28, Mahesh Renduchintala <[hidden email]>:

IGNITE_DISCOVERY_HISTORY_SIZE=700

Does this go on the server side or the thick client side ?
Mahesh Renduchintala Mahesh Renduchintala
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.7.0 : server node: null pointer exception

Will try the configuration at server and report back. 



Slava, Denis, 

Can you also take a look at the below thread. I certainly believe this again a problem with ignite discovery spi or some such. I have attached all logs and configuration.


http://apache-ignite-users.70518.x6.nabble.com/Ignite-2-7-0-thick-client-has-all-system-threads-busy-indefinitely-td28880.html