Runtime error at IgniteSpiThread

classic Classic list List threaded Threaded
3 messages Options
Ambha Ambha
Reply | Threaded
Open this post in threaded view
|

Runtime error at IgniteSpiThread

I am using Ignite 1.4.0, Xmx is set to 5G and other JVM settings like -XX:+UseParNewGC -XX:+UseConcMarkSweepGC etc are also enabled. Apache Storm cluster creates 5 client connections (client mode=true) to Ignite server (single node). We also store the incoming data to Cassandra (it is running on different system) by implementing CacheStore interface. The setting works fine with 50,000 load. But when the input size increases to nearly 1 lakh (.1 million) system repeatedly fails because of runtime error. The entire stack trace is shown below.  

1. Is there any restriction on the number of clients which can be created per ignite server node? five connections in my case are created by different storm worker node
2.  The runtime error does not happen so often if I disable the cache store, any reason for this
3. The client connections are opened and kept alive till the end, do you see any side effect of this?
4. Is it better to create multimode ignite cluster (more than one server node) for an input of 5-10 lakhs (1 million)
5. Please suggest any other way of connecting different components (storm, ignite, Cassandra) which can produce the better result

************stack trace**********************
[20:45:18,050][SEVERE][tcp-disco-msg-worker-#2%testGrid][TcpDiscoverySpi] Runtime error caught during grid runnable execution: IgniteSpiThread [name=tcp-disco-msg-worker-#2%testGrid]
java.lang.NullPointerException
        at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onDiscoveryDataReceived(GridCacheProcessor.java:1899)
        at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$6.onExchange(GridDiscoveryManager.java:659)
        at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.onExchange(TcpDiscoverySpi.java:1706)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeAddedMessage(ServerImpl.java:3310)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processJoinRequestMessage(ServerImpl.java:3002)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2095)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:5382)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Runtime error at IgniteSpiThread

Hi,

As far as I remember Ignite community fixed a lot of issues related to the stability of TCP discovery SPI in version 1.5.0. The one you face should be fixed as well. So please upgrade to version 1.5.0 and check you app with it.


1. Is there any restriction on the number of clients which can be created per ignite server node? five connections in my case are created by different storm worker node

There is no any specific limitation. However the golden rule that apply to any technology or platform is to use as less resources as possible :)

2.  The runtime error does not happen so often if I disable the cache store, any reason for this

Just give a try to 1.5.0. Probably cache stored trigger that exception on your side.

3. The client connections are opened and kept alive till the end, do you see any side effect of this?

This shouldn't be the case. Close clients only if you're not going to use them for a long time.

4. Is it better to create multimode ignite cluster (more than one server node) for an input of 5-10 lakhs (1 million)

Yes, if you have several boxes/machines that you're going to use then you can start several servers nodes on them and will only benefit from this.

5. Please suggest any other way of connecting different components (storm, ignite, Cassandra) which can produce the better result

Again, I would suggest switching to 1.5.0 before. As per Cassandra there is a work on the go related to the integration of Ignite and Cassandra. [1] Hope it will be finished soon.

[1]
https://issues.apache.org/jira/browse/IGNITE-1371

Regards,
Denis
pragmaticbigdata pragmaticbigdata
Reply | Threaded
Open this post in threaded view
|

Re: Runtime error at IgniteSpiThread

I am facing a similar exception while starting apache ignite. I am using apache ignite 1.6. The exception is a bit different this time

[09:27:59] Topology snapshot [ver=269, servers=1, clients=0, CPUs=2, heap=2.0GB]
[09:27:59,957][SEVERE][exchange-worker-#46%null%][GridCachePartitionExchangeManager] Runtime error caught during grid runnable execution: GridWorker [name=partition-exchanger, gridName=null, finished=false, isCancelled=false, hashCode=1194503871, interrupted=false, runner=exchange-worker-#46%null%]
java.lang.NullPointerException
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.updateLocal(GridDhtPartitionTopologyImpl.java:1347)
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.onEvicted(GridDhtPartitionTopologyImpl.java:1444)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.onPartitionEvicted(GridDhtPreloader.java:639)
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.tryEvictAsync(GridDhtLocalPartition.java:510)
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.rent(GridDhtLocalPartition.java:478)
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.afterExchange(GridDhtPartitionTopologyImpl.java:601)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1353)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

The complete logs are shared here.