CacheContinuousQuery memory use and best practices

classic Classic list List threaded Threaded
3 messages Options
john_simmons john_simmons
Reply | Threaded
Open this post in threaded view
|

CacheContinuousQuery memory use and best practices

Hi, I'm looking into an issue where we have an Ignite (2.6) client node doing
a CacheContinuousQuery on a cache full of binary objects, and eventually the
server node gets Out of Memory. Usually the server node is happy to run with
2-3 gig of Heap size, however when this client is running with a
CacheContinuousQuery on a cache, it can go >20gig, until the client is
stopped, and then 20mins later objects are garbage collected on the server
node and it goes back down to 2gig. In heap dumps I see its full of
CacheContinuousQueryEvents.

Some questions:

1) In my client Continuous Query handler code, what happens when an error is
thrown:
/private void handleCacheEvent(CacheEntryEvent<? extends String, ? extends
MyBinaryObject> event) {
     // try to deserialize event but fails with error.
}/

I don't see any exception thrown, will this event stay in the cache as an
unconsumed event? Potentially causing a leak for events that have not been
handled correctly?

2) Why does CacheContinuousQueryEvent keep a reference to 'oldVal'.. i.e the
old value in the cache? This could be causing a problem, as we don't care
about old values in the cache.. can we switch that off? Why isn't that the
default

3) In the method that handles cache events, is it best practice to put the
cache event straight onto a blocking queue to make sure there is no slow
consumer problem? makes sense to me but I don't see it recommended anywhere.
If we don't I can imagine the outbound queue of the server node growing...

thanks for any pointers!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
john_simmons john_simmons
Reply | Threaded
Open this post in threaded view
|

Re: CacheContinuousQuery memory use and best practices

Also some details on the cache config:

atomicityMode=TRANSACTIONAL
cacheMode = PARTITIONED
backups=1

.. the rest are default. Cache value types are binary objects.

So here's a simpler question, if my client node does a cache.put(key=String,
value=BinaryObject) 100 times on the same key, when are the previous 99
entries put up for garbage collection?

Is it when the backup node queue has all of the events that have taken place
?
 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: CacheContinuousQuery memory use and best practices

In reply to this post by john_simmons
Hello!

Since nobody is chiming in, my opinion is that:

1) Please don't try exceptions out of this handler!
2) I don't think you can switch it off, this is implemented as in other places of JCache.
3) I think it still makes sense. Avoid blocking in continuous query handler.

As for your client-server scenario, I'm not sure what to do. If you have a lot of small updates, please try to increase pageSize perhaps? It's 1024 by default.

Regards,
--
Ilya Kasnacheev


вт, 16 апр. 2019 г. в 14:23, johnny_rotten <[hidden email]>:
Hi, I'm looking into an issue where we have an Ignite (2.6) client node doing
a CacheContinuousQuery on a cache full of binary objects, and eventually the
server node gets Out of Memory. Usually the server node is happy to run with
2-3 gig of Heap size, however when this client is running with a
CacheContinuousQuery on a cache, it can go >20gig, until the client is
stopped, and then 20mins later objects are garbage collected on the server
node and it goes back down to 2gig. In heap dumps I see its full of
CacheContinuousQueryEvents.

Some questions:

1) In my client Continuous Query handler code, what happens when an error is
thrown:
/private void handleCacheEvent(CacheEntryEvent<? extends String, ? extends
MyBinaryObject> event) {
     // try to deserialize event but fails with error.
}/

I don't see any exception thrown, will this event stay in the cache as an
unconsumed event? Potentially causing a leak for events that have not been
handled correctly?

2) Why does CacheContinuousQueryEvent keep a reference to 'oldVal'.. i.e the
old value in the cache? This could be causing a problem, as we don't care
about old values in the cache.. can we switch that off? Why isn't that the
default

3) In the method that handles cache events, is it best practice to put the
cache event straight onto a blocking queue to make sure there is no slow
consumer problem? makes sense to me but I don't see it recommended anywhere.
If we don't I can imagine the outbound queue of the server node growing...

thanks for any pointers!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/