I wrote a test for what happens in the case that a DataRegion runs out of
memory. I filled up a cache with records until I received the expected
IgniteOutOfMemoryException. Then I tried to remove entries from the cache -
expecting that memory would be freed up again.
What I found is that any cache operation such as remove() or clearAll() thew
a further IOOM exception. Although cache.size() decreased, it does not
appear that memory is freed up - it is not possible to add new entries to
the cache even after a clearAll().
Is this expected behaviour? What is the recommended approach for dealing
with an OOM condition - other than to avoid it in the first place?
Ignite DataRegionMetrics reports that memory *is* freed up when removing
items from the cache. However, Ignite continues to throw an OOM exception on
each subsequent cache removal. Cache puts are unsuccessful.
So although Ignite reports that the memory is free, it doesn't seem possible
to actually use it again following the OOM condition.
We were using Ignite 2.4 (update pending). Ignite 2.5 and later seens to
treat OOM as a critical error by default and stops the node. The reproducer
below uses a failure handler to stop this from happening. It allocates a
100MB (configurable - 100MB is quite small) region and fills it up with
data. Afterwards, it attempts to clear data from the cache. Every cache
operation (even data removal) results in OOM errors.