stdout - Message queue limit is set to 0, potential OOMEs

classic Classic list List threaded Threaded
10 messages Options
javastuff.sam@gmail.com javastuff.sam@gmail.com
Reply | Threaded
Open this post in threaded view
|

stdout - Message queue limit is set to 0, potential OOMEs

Recently moved to Ignite 1.9 and noticed below line on stdout -

Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.

Not sure if this existed in older version.

Can anybody explain what does it mean and how to avoid potential OOME?  

Thanks,
-Sam
Andrew Mashenkov Andrew Mashenkov
Reply | Threaded
Open this post in threaded view
|

Re: stdout - Message queue limit is set to 0, potential OOMEs

Hi Sam,

Slow node can cause OOME, as messages for slow node can make queue grows infinitly.
Also, you can receive messages with higher rate than node able to process, that also can results in OOME.


On Tue, Apr 18, 2017 at 10:13 PM, [hidden email] <[hidden email]> wrote:
Recently moved to Ignite 1.9 and noticed below line on stdout -

/Message queue limit is set to 0 which may lead to potential OOMEs when
running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message
queues growth on sender and receiver sides./

Not sure if this existed in older version.

Can anybody explain what does it mean and how to avoid potential OOME?

Thanks,
-Sam



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-0-potential-OOMEs-tp12048.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--
Best regards,
Andrey V. Mashenkov
Regards,
Andrew.
javastuff.sam@gmail.com javastuff.sam@gmail.com
Reply | Threaded
Open this post in threaded view
|

Re: stdout - Message queue limit is set to 0, potential OOMEs

What do you mean by messages? I am not using Ignite messaging. Are these messages of rebalancing during topology change?
How do I configure it to avoid potential OOME?

Thanks,
-Sam
Andrew Mashenkov Andrew Mashenkov
Reply | Threaded
Open this post in threaded view
|

Re: stdout - Message queue limit is set to 0, potential OOMEs

Hi Sam,

Ignite use messages for inter-node communication.
You have to configure TcpCommunicationSPI in IgniteConfiguretion. Method TcpCommunicationSPI.setMessageQueueLimit() is what you need.

On Wed, Apr 19, 2017 at 11:13 AM, [hidden email] <[hidden email]> wrote:
What do you mean by messages? I am not using Ignite messaging. Are these
messages of rebalancing during topology change?
How do I configure it to avoid potential OOME?

Thanks,
-Sam




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-0-potential-OOMEs-tp12048p12066.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--
Best regards,
Andrey V. Mashenkov
Regards,
Andrew.
javastuff.sam@gmail.com javastuff.sam@gmail.com
Reply | Threaded
Open this post in threaded view
|

Re: stdout - Message queue limit is set to 0, potential OOMEs

This post was updated on .
Thank you. So high traffic scenarios are potential OOMs. Questions -
- How to identify hat value is suitable for TcpCommunicationSPI.setMessageQueueLimit()
- Why it showing up with 1.9 and not earlier?

Thanks,
-Sam
Andrew Mashenkov Andrew Mashenkov
Reply | Threaded
Open this post in threaded view
|

Re: stdout - Message queue limit is set to 0, potential OOMEs

Hi Sam,

Before 1.9, default queue limit value was 1024. 
This was changed in IGNITE-3220 [1]


On Wed, Apr 19, 2017 at 8:54 PM, [hidden email] <[hidden email]> wrote:
Thank you. Last question -
Why it showing up with 1.9 and not earlier?

Thanks,
-Sam



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-0-potential-OOMEs-tp12048p12091.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--
Best regards,
Andrey V. Mashenkov
Regards,
Andrew.
Andrew Mashenkov Andrew Mashenkov
Reply | Threaded
Open this post in threaded view
|

Re: stdout - Message queue limit is set to 0, potential OOMEs

In reply to this post by javastuff.sam@gmail.com
Actual 'outbound queue size' are printed in node mertics messages of accessible via mbean ClusterLocalNodeMetricsMXBean.


On Wed, Apr 19, 2017 at 11:13 AM, [hidden email] <[hidden email]> wrote:
What do you mean by messages? I am not using Ignite messaging. Are these
messages of rebalancing during topology change?
How do I configure it to avoid potential OOME?

Thanks,
-Sam




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-0-potential-OOMEs-tp12048p12066.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--
Best regards,
Andrey V. Mashenkov
Regards,
Andrew.
javastuff.sam@gmail.com javastuff.sam@gmail.com
Reply | Threaded
Open this post in threaded view
|

Re: stdout - Message queue limit is set to 0, potential OOMEs

This post was updated on .
Added below to TcpCommunicationSpi and message went away, able to see the new value on JMX as well.
<property name="messageQueueLimit" value="1024"/>

However, how do I know what is the good or suitable value one need to set? I have more than 50 different PARTITIONED OFF-HEAP caches defined, most likely to have 2 to 4 nodes with 10 or 20GB or more for each node.
 
in my opinion, potential OOME is big risk with default configuration. Everybody needs performance, but no one really need OOME.

Thanks,
-Sam

Andrew Mashenkov Andrew Mashenkov
Reply | Threaded
Open this post in threaded view
|

Re: stdout - Message queue limit is set to 0, potential OOMEs

Hi Sam,

If you see messageQueueSize is often close to limit and low CPU pressure on nodes,
then limit value increasing can be helpful.

On Fri, Apr 21, 2017 at 11:36 PM, [hidden email] <[hidden email]> wrote:
Added below to TcpCommunicationSpi and message went away, able to see the new
value on JMX as well.
/<property name="messageQueueLimit" value="1024"/>/

However, how do I know what is the good or suitable value one need to set? I
have more than 50 different PARTITIONED OFF-HEAP caches defined, most likely
to have 2 to 4 nodes with 20GB each or more.

in my opinion, potential OOME is big risk with default configuration.
Everybody needs performance, but no one really need OOME.

Thanks,
-Sam





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-0-potential-OOMEs-tp12048p12165.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--
Best regards,
Andrey V. Mashenkov
Regards,
Andrew.
javastuff.sam@gmail.com javastuff.sam@gmail.com
Reply | Threaded
Open this post in threaded view
|

Re: stdout - Message queue limit is set to 0, potential OOMEs

In that case I will change my configuration to have <property name="messageQueueLimit" value="1024"/> and change it if needed.

Out-of-box unlimited seems too aggressive with potential OOME risk.

Thanks,
-Sam