ignite problem when loading large amount of data into cache

classic Classic list List threaded Threaded
6 messages Options
kevin.zheng kevin.zheng
Reply | Threaded
Open this post in threaded view
|

ignite problem when loading large amount of data into cache

Hi there,
I am trying to load a large table content into cache (about 10g).
I have initiate 8 nodes in two servers (each has 4 nodes).
My server has 48g and 32g memory respectively.
During the loading process, I have the following exception message, and all the nodes stopped:

[14:19:15,322][SEVERE][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] TcpDiscov                                             erSpi's message worker thread failed abnormally. Stopping the node in order to p                                             revent cluster wide instability.
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject                                             .reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject                                             .awaitNanos(AbstractQueuedSynchronizer.java:2088)
        at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDequ                                             e.java:522)
        at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.jav                                             a:684)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.b                                             ody(ServerImpl.java:5779)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body                                             (ServerImpl.java:2161)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[14:19:15] Topology snapshot [ver=35, servers=1, clients=0, CPUs=8, heap=1.0GB]
[14:19:15] Ignite node stopped OK [uptime=00:36:09:525]
[14:19:19] Topology snapshot [ver=30, servers=6, clients=0, CPUs=16, heap=6.0GB]
[14:23:08,016][SEVERE][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] TcpDiscov                                             erSpi's message worker thread failed abnormally. Stopping the node in order to p                                             revent cluster wide instability.
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject                                             .reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject                                             .awaitNanos(AbstractQueuedSynchronizer.java:2088)
        at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDequ                                             e.java:522)
        at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.jav                                             a:684)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.b                                             ody(ServerImpl.java:5779)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body                                             (ServerImpl.java:2161)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)



any idea how to fix it?

best regards,
Kevin
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: ignite problem when loading large amount of data into cache

Hi Kevin,

With such large heap sizes it's very likely that you had long GC pauses that caused the topology to fall apart. To avoid this I would recommend to allocate not more that 10-12G of heap and store the data in off-heap caches [1].

[1] https://apacheignite.readme.io/docs/off-heap-memory

-Val
kevin.zheng kevin.zheng
Reply | Threaded
Open this post in threaded view
|

re: ignite problem when loading large amount of data into cache

Hi Val,
Thank you for your suggestion.
I have one more question: is it possible to change the server configuration on running time? let's say I follow the off-heap-memory suggestions and set the off-heap size,  how to make the server configuration takes effect immediately?


Cheers,
Kevin

-----邮件原件-----
发件人: vkulichenko [mailto:[hidden email]]
发送时间: 2016年4月20日 4:48
收件人: [hidden email]
主题: Re: ignite problem when loading large amount of data into cache

Hi Kevin,

With such large heap sizes it's very likely that you had long GC pauses that caused the topology to fall apart. To avoid this I would recommend to allocate not more that 10-12G of heap and store the data in off-heap caches [1].

[1] https://apacheignite.readme.io/docs/off-heap-memory

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignite-problem-when-loading-large-amount-of-data-into-cache-tp4324p4340.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: re: ignite problem when loading large amount of data into cache

Kevin,

Configuration of the existing cache can't be changed in runtime. The only option is to destroy the cache and create it again with new parameters (you will lose all in-memory data, of course). What's the use case when you can need this?

-Val
kevin.zheng kevin.zheng
Reply | Threaded
Open this post in threaded view
|

re: re: ignite problem when loading large amount of data into cache

Hi Val,
When the exception occurred, I checked the forum and reset the java vm size to 30g, and also I split my table into 10 smaller tables, each contains 1g data.
In this case, I saw your suggestion on off-heap settings. I don't want to reload all the data again, So I asked if it is possible to make the configuration take effect immediately at running time.

Btw, I see there are backup settings in partition mode,    like this  <property name="backups" value="1"/>
But I did not see where those backups are stored. Is there any settings like redis which will automatically load dump files when redis recovered from an accident?

Regards,
Kevin  

-----邮件原件-----
发件人: vkulichenko [mailto:[hidden email]]
发送时间: 2016年4月20日 12:42
收件人: [hidden email]
主题: Re: re: ignite problem when loading large amount of data into cache

Kevin,

Configuration of the existing cache can't be changed in runtime. The only option is to destroy the cache and create it again with new parameters (you will lose all in-memory data, of course). What's the use case when you can need this?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignite-problem-when-loading-large-amount-of-data-into-cache-tp4324p4347.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Vladimir Ozerov Vladimir Ozerov
Reply | Threaded
Open this post in threaded view
|

Re: re: ignite problem when loading large amount of data into cache

Hi Kevin,

Normally restarted Ignite node will pull data from other nodes. We call this process "rebalance".

Also Ignite comes with a special store type - local store (see @CacheLocalStore annotation). Local store could be used to re-load node's data after restart. However, Ignite doesn't have any local store implementations. You can find one in some Ignite-based commercial products, such as GridGain.

Vladimir.

On Wed, Apr 20, 2016 at 9:39 AM, Zhengqingzheng <[hidden email]> wrote:
Hi Val,
When the exception occurred, I checked the forum and reset the java vm size to 30g, and also I split my table into 10 smaller tables, each contains 1g data.
In this case, I saw your suggestion on off-heap settings. I don't want to reload all the data again, So I asked if it is possible to make the configuration take effect immediately at running time.

Btw, I see there are backup settings in partition mode,    like this  <property name="backups" value="1"/>
But I did not see where those backups are stored. Is there any settings like redis which will automatically load dump files when redis recovered from an accident?

Regards,
Kevin

-----邮件原件-----
发件人: vkulichenko [mailto:[hidden email]]
发送时间: 2016年4月20日 12:42
收件人: [hidden email]
主题: Re: re: ignite problem when loading large amount of data into cache

Kevin,

Configuration of the existing cache can't be changed in runtime. The only option is to destroy the cache and create it again with new parameters (you will lose all in-memory data, of course). What's the use case when you can need this?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignite-problem-when-loading-large-amount-of-data-into-cache-tp4324p4347.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.