Apache Ignite DataRegion Swap

classic Classic list List threaded Threaded
4 messages Options
martinlundaskoe martinlundaskoe
Reply | Threaded
Open this post in threaded view
|

Apache Ignite DataRegion Swap

Hi

If I have an ignite node running on a machine with 16GB of physical memory and I want to create a data-region that may hold 5GB of data but only consume 1GB of memory on the off-heap, then how do I configure my cluster node to use swap when exceeding 1GB off-heap memory?

I'm trying to get my head around this example from GridGain <a href="https://www.gridgain.com/docs/latest/developers-guide/persistence/swap">https://www.gridgain.com/docs/latest/developers-guide/persistence/swap</a>

Where is the 500MB specified (besides in the name)?
It seems to me that the size of the data-region is set to 5GB and not 500MB?
ibelyakov ibelyakov
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite DataRegion Swap

Hi,

Yes, you're correct, there is a mistake in the documentation, according to the example a data region max size will be 5Gb not 500Mb.

Also you can use Native Persistence for your case:
You can specify data region with maxSize=1Gb and enabled persistence then your data will be stored on disk and GridGain will keep up to 1Gb of the "hot" data in the RAM.

Igor

On Fri, Mar 12, 2021 at 3:05 PM Martin Lund Askøe <[hidden email]> wrote:
Hi

If I have an ignite node running on a machine with 16GB of physical memory and I want to create a data-region that may hold 5GB of data but only consume 1GB of memory on the off-heap, then how do I configure my cluster node to use swap when exceeding 1GB off-heap memory?

I'm trying to get my head around this example from GridGain <a href="https://www.gridgain.com/docs/latest/developers-guide/persistence/swap">https://www.gridgain.com/docs/latest/developers-guide/persistence/swap</a>

Where is the 500MB specified (besides in the name)?
It seems to me that the size of the data-region is set to 5GB and not 500MB?
martinlundaskoe martinlundaskoe
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite DataRegion Swap

Ok, so the documentation lacks and the examples are incorrect.

I my situation Native Persistence doesn't suit me, I have a read-through
cache to a 3rd party persistence (CacheLoader API) - no write-through. I
don't expect data to be recoverable if partitions are lost, so swap would
suit me just fine, all I need is to not unnecessarily have to fetch data
from persistence.

This is my situation:

On a physical server with 128GB RAM I am running multiple ignite server
nodes each with 3.5GB heap, 1GB DirectMemory and 256MB MetaDataSpace. With
swapPath defined on the data region, when is swap supposed to kick in?

Ideally my cluster nodes should consume no more than ~5GB of ram each, but
it looks like it is consuming as much memory as it wants.

Metrics for local node
    ...
    ^-- Heap [used=2246MB, free=35.82%, comm=3500MB]
    ^-- Off-heap memory [used=1406MB, free=80.5%, allocated=1772MB]

Limited DirectMemory using -XX:MaxDirectMemorySize=2g is not adhered to,
data region keep trying to expand in memory.

I would expect a node started with -Xmx2g and -XX:MaxDirectMemorySize=2g not
to exceed 4GB of memory regardless of data region size, ie. 4GB. I would
expect swap (if configured) to be used before exceeding 4GB, and probably
expect it to crash with an OOM if swap was not configured.

What am I missing?


ibelyakov wrote

> Yes, you're correct, there is a mistake in the documentation, according to
> the example a data region max size will be 5Gb not 500Mb.
>
> Also you can use Native Persistence for your case:
> https://www.gridgain.com/docs/latest/developers-guide/persistence/native-persistence
>
> You can specify data region with maxSize=1Gb and enabled persistence then
> your data will be stored on disk and GridGain will keep up to 1Gb of the
> "hot" data in the RAM.
>
> Igor





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ibelyakov ibelyakov
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite DataRegion Swap

It happens since Ignite allocates data regions in the off-heap memory (outside of Java heap). As a result you wouldn't get a Java OOM exception due to the large amount of data stored in the region. More details regarding memory architecture can be found here:

Since you use 3rd-party persistence it's not necessary to use swapping, instead you can configure eviction policy for the data region and Ignite will evict the less used data from it in case the memory limit is exceeded for the region. 
More information regarding eviction policies can be found here:

Igor

On Fri, Mar 19, 2021 at 9:20 AM martinlundaskoe <[hidden email]> wrote:
Ok, so the documentation lacks and the examples are incorrect.

I my situation Native Persistence doesn't suit me, I have a read-through
cache to a 3rd party persistence (CacheLoader API) - no write-through. I
don't expect data to be recoverable if partitions are lost, so swap would
suit me just fine, all I need is to not unnecessarily have to fetch data
from persistence.

This is my situation:

On a physical server with 128GB RAM I am running multiple ignite server
nodes each with 3.5GB heap, 1GB DirectMemory and 256MB MetaDataSpace. With
swapPath defined on the data region, when is swap supposed to kick in?

Ideally my cluster nodes should consume no more than ~5GB of ram each, but
it looks like it is consuming as much memory as it wants.

Metrics for local node
    ...
    ^-- Heap [used=2246MB, free=35.82%, comm=3500MB]
    ^-- Off-heap memory [used=1406MB, free=80.5%, allocated=1772MB]

Limited DirectMemory using -XX:MaxDirectMemorySize=2g is not adhered to,
data region keep trying to expand in memory.

I would expect a node started with -Xmx2g and -XX:MaxDirectMemorySize=2g not
to exceed 4GB of memory regardless of data region size, ie. 4GB. I would
expect swap (if configured) to be used before exceeding 4GB, and probably
expect it to crash with an OOM if swap was not configured.

What am I missing?


ibelyakov wrote
> Yes, you're correct, there is a mistake in the documentation, according to
> the example a data region max size will be 5Gb not 500Mb.
>
> Also you can use Native Persistence for your case:
> https://www.gridgain.com/docs/latest/developers-guide/persistence/native-persistence
>
> You can specify data region with maxSize=1Gb and enabled persistence then
> your data will be stored on disk and GridGain will keep up to 1Gb of the
> "hot" data in the RAM.
>
> Igor





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/