How to stop a node from dying when memory is full?

classic Classic list List threaded Threaded
3 messages Options
colinc colinc
Reply | Threaded
Open this post in threaded view
|

How to stop a node from dying when memory is full?

In a system that is not using native persistence, what is the recommended way
of stopping a cluster from running out of memory - or stopping it from
crashing when it does?

As per the below jira, memory monitoring appears to be unreliable in the
latest version of Ignite:
https://issues.apache.org/jira/browse/IGNITE-12096

Even when working, this is an estimate that is updated periodically, which
makes it hard to reliably avoid a critical OOM in a system that is rapidly
filling caches.

It is technically possible to create a custom failure handler - but I
understand that trapping the failure in this way is considered to be bad
practice, since it can leave Ignite in an inconsistent state.

How are people addressing this challenge?

Regards,
Colin.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ilya Kasnacheev Ilya Kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: How to stop a node from dying when memory is full?

Hello!

You can try enabling Page Eviction, in this case pages with K-V pairs contained in them will be dropped.

Regards,

On 2019/08/29 11:54:11, colinc <[hidden email]> wrote:

> In a system that is not using native persistence, what is the recommended way
> of stopping a cluster from running out of memory - or stopping it from
> crashing when it does?
>
> As per the below jira, memory monitoring appears to be unreliable in the
> latest version of Ignite:
> https://issues.apache.org/jira/browse/IGNITE-12096
>
> Even when working, this is an estimate that is updated periodically, which
> makes it hard to reliably avoid a critical OOM in a system that is rapidly
> filling caches.
>
> It is technically possible to create a custom failure handler - but I
> understand that trapping the failure in this way is considered to be bad
> practice, since it can leave Ignite in an inconsistent state.
>
> How are people addressing this challenge?
>
> Regards,
> Colin.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
colinc colinc
Reply | Threaded
Open this post in threaded view
|

Re: How to stop a node from dying when memory is full?

Thanks - that does seem to be effective at stopping the OOM condition at
least.

Is there any way to determine which cache entries were affected by the page
expiry, do you know? The EVT_CACHE_ENTRY_EVICTED doesn't seem to get fired
in this case as far as I can tell. Is that your expectation?

This is important for performing clean-up of related cache entries to ensure
referential integrity.

Regards,
Colin.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/