Odd behavior/bug with DataRegion

classic Classic list List threaded Threaded
4 messages Options
Victor Victor
Reply | Threaded
Open this post in threaded view
|

Odd behavior/bug with DataRegion

Hi,

I am running an app with a custom dataregion for one of the cache's with
following configuration,
max size = 20mb (just for example)
persistence=false
expiry policy = Random_2_LRU

I am using a 3rd party persistence (RDBMS).

When i use this configuration and add data all works fine. E.g. say i put
50k records, all 50k make it to the DB, while only 30k stay in the cache
(based on the above expiration policy). All's good so far.

However, when i start the app all over again, i preload the cache with data
from the DB. The DB has 50k records, so i expect the cache to continue to
apply the expiration policy as it nears its 90% data region capacity.

However, as i am pre loading, i am monitoring the mbeans for data region and
the overall entry count. I see the data region allocation go above 20mb and
the entry count go above 40k. And then it just crashes with OOM with the
below error,

[2020-07-23T18:41:43,286][ERROR][Test.Thread.4][ignite] Critical system
error detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.mem.IgniteOutOfMemoryException: Out of memory in data region
[name=Account, initSize=19.0 MiB, maxSize=20.0 MiB,
persistenceEnabled=false] Try the following:
  ^-- Increase maximum off-heap memory size
(DataRegionConfiguration.maxSize)
  ^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled)
  ^-- Enable eviction or expiration policies]]
org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Out of memory in
data region [name=be.gen.Concepts.Account, initSize=19.0 MiB, maxSize=20.0
MiB, persistenceEnabled=false] Try the following:
  ^-- Increase maximum off-heap memory size
(DataRegionConfiguration.maxSize)
  ^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled)
  ^-- Enable eviction or expiration policies
        at
org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:322)
[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.allocateDataPage(AbstractFreeList.java:570)
[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.writeSinglePage(AbstractFreeList.java:688)
[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.insertDataRow(AbstractFreeList.java:582)
[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.persistence.freelist.CacheFreeList.insertDataRow(CacheFreeList.java:74)
[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.persistence.freelist.CacheFreeList.insertDataRow(CacheFreeList.java:35)
[ignite-core-2.8.1.jar:2.8.1]
        at
org.apache.ignite.internal.processors.cache.persistence.RowStore.addRow(RowStore.java:108)
[ignite-core-2.8.1.jar:2.8.1]
...

Not sure why is the expiration policy not kicking in. Is this a bug?

Any inputs appreciated.

Victor



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Victor Victor
Reply | Threaded
Open this post in threaded view
|

Re: Odd behavior/bug with DataRegion

On further reading around DataRegionConfiguration. I see
'setEmptyPagesPoolSize', it says,

"Increase this parameter if cache can contain very big entries (total size
of pages in this pool should be enough to contain largest cache entry).
Increase this parameter if IgniteOutOfMemoryException occurred with enabled
page eviction."

The default value as i see is 100. So initially bumped it up to 1000. But it
failed saying the value can't go beyond 512. So i see it to 500.

I still am getting the same OOM exception, when i am preloading the cache.

Not sure if this a bug or some config is missing.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Victor Victor
Reply | Threaded
Open this post in threaded view
|

Re: Odd behavior/bug with DataRegion

Update,

Interestingly, i took the value to the wire, setting 'emptyPagesPoolSize' to
510 (< 512) and that seems to have done the trick. Atleast in my past 5 test
run's the preload has gone through fine.

Right now i have set it to pretty much the max value, since there is no good
way to identify what would be the runtime max value of an instance while
creating a cache.

Any other ideas around how to go about setting a safe value for this
property.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Odd behavior/bug with DataRegion

Hi,

There is a couple of known issues related to the small data region. I saw this behavior before and you shouldn't see this issue for regions with bigger sizes. Also, if you have really big objects, you might need to change the emptyPagesPoolSize, to make sure that there are no objects bigger than this value * 4k(default page size).

Best Regards,
Evgenii

чт, 23 июл. 2020 г. в 23:40, Victor <[hidden email]>:
Update,

Interestingly, i took the value to the wire, setting 'emptyPagesPoolSize' to
510 (< 512) and that seems to have done the trick. Atleast in my past 5 test
run's the preload has gone through fine.

Right now i have set it to pretty much the max value, since there is no good
way to identify what would be the runtime max value of an instance while
creating a cache.

Any other ideas around how to go about setting a safe value for this
property.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/