Do we require to set MaxDirectMemorySize JVM parameter?

classic Classic list List threaded Threaded
16 messages Options
Ankit Singhai Ankit Singhai
Reply | Threaded
Open this post in threaded view
|

Do we require to set MaxDirectMemorySize JVM parameter?

Hi All,
Do we require to set MaxDirectMemorySize JVM parameter on IgniteServer to
make use of OFFHEAP?

For an example to make use of 8 GB off heap should I add MaxDirectMemorySize
if yes then how much?
                      <property name="memoryPolicies">
                          <list>
                                <bean
class="org.apache.ignite.configuration.MemoryPolicyConfiguration">
                                  <property name="name"
value="myServiceOffHeap"/>
                                  <property name="initialSize" value="#{8L *
1024 * 1024 * 1024}"/>
                                  <property name="maxSize" value="#{8L *
1024 * 1024 * 1024}"/>
                                  <property name="pageEvictionMode"
value="RANDOM_LRU"/>
                                  <property name="metricsEnabled"
value="true"/>
                                </bean>
                          </list>
                        </property>

Regards,
Ankit




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dkarachentsev dkarachentsev
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Hi Ankit,

No, Ignite uses sun.misc.Unsafe for offheap memory. Direct memory may be
used in DirectBuffers used for intercommunication. Usually defaults quite
enough.

Thanks!
-Dmitry



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
colinc colinc
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Is there any alternative way to constrain max physical RAM that Ignite uses?

My use case is to constrain physical RAM usage in a shared environment,
while allowing a relatively generous allocation of swap storage. I'm aware
of Ignite persistence, but believe that swap storage might meet our needs
better in this case.

So ideally, I'd configure the memory region max memory to determine the size
of the swap file - and use some other parameter to configure the maximum
amount of off-heap RAM that the process can consume.

What is the best way to achieve this?

Thanks,
Colin.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
summasumma summasumma
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

This post was updated on .
In reply to this post by Ankit Singhai
In the above example,

is setting  "<property name="maxSize" value="#{8L * 1024 * 1024 * 1024}"/> "
in xml config file same as adding a jvmoption "-XX:MaxDirectMemorySize=8g" ?
or its different?
If different what is the default MaxDirectMemorySize jvm option value set, if not given explicitly in apache ignite?

Can somone please clarify?

Thanks
...summa



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Павлухин Иван Павлухин Иван
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Hi collnc,

Perhaps, documentation can answer you question [1].

[1] https://apacheignite.readme.io/docs/durable-memory-tuning

2018-12-21 20:39 GMT+03:00, summasumma <[hidden email]>:

> In the above example,
>
> is setting  "<property name="maxSize" value="#{8L * 1024 * 1024 * 1024}"/>
> "
> in xml config file same as adding a jvmoption "-XX:MaxDirectMemorySize=8g"
> ?
> or its different?
>
> Can somone please clarify?
>
> Thanks
> ...summa
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>


--
Best regards,
Ivan Pavlukhin
Павлухин Иван Павлухин Иван
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Hi summasumma,

DataRegion maxSize and jvm MaxDirectMemorySize are completely different.
Ignite DataRegion uses offheap memory allocated with help of Unsafe.
And that memory is not related to "direct memory" which jvm allocates
when direct buffers are used (e.g. ByteBuffer.allocateDirect). To
constraint max amount of memory for direct buffers one can use
MaxDirectMemorySize jvm option. As far as I know, by default
MaxDirectMemorySize is equal to Xmx. Consult [1] for more details.

[1] https://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html

2018-12-23 11:17 GMT+03:00, Павлухин Иван <[hidden email]>:

> Hi collnc,
>
> Perhaps, documentation can answer you question [1].
>
> [1] https://apacheignite.readme.io/docs/durable-memory-tuning
>
> 2018-12-21 20:39 GMT+03:00, summasumma <[hidden email]>:
>> In the above example,
>>
>> is setting  "<property name="maxSize" value="#{8L * 1024 * 1024 *
>> 1024}"/>
>> "
>> in xml config file same as adding a jvmoption
>> "-XX:MaxDirectMemorySize=8g"
>> ?
>> or its different?
>>
>> Can somone please clarify?
>>
>> Thanks
>> ...summa
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>


--
Best regards,
Ivan Pavlukhin
summasumma summasumma
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Thanks Ivan.

This means having both configuration as follows:
xml        --> <property name="maxSize" value="#{8L * 1024 * 1024 * 1024}"/>
jvmption --> "-XX:MaxDirectMemorySize=8g"

Means, i should have minimum 16 Gb of RAM (8 dataregion+ 8 directmem) for
Ignite to run properly i guess.

In my current setup i have 16GB of total ram and given 12GB=maxSize
Dataregion and jvmoption as '-XX:MaxDirectMemorySize=8g'. And this is
crashing Ignite with OOME after a while (though not immediatly) when i try
to do a performance testing of Update operation. so this means either i
should increase the RAM or decrease the XX:MaxDirectMemorySize to 4g ?

Please clarify

Thanks
...summa



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Павлухин Иван Павлухин Иван
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Hi summasumma,

> Means, i should have minimum 16 Gb of RAM (8 dataregion+ 8 directmem) for
> Ignite to run properly i guess.

Not quite. Actually, I am not aware that Ignite requires some special
tuning of MaxDirectMemorySize. If direct memory causes OOME then the
exception message usually points it out (e.g. "Direct buffer memory").
Check if it is your case.

Indeed you should be careful when configure DataRegion.maxSize because
OOME is a real problem and you will get it if put more data into
Ignite than specified limit. One should carefully plan how many data
is going to be stored into Ignite when in-memory mode is used. Also,
it is possible to use Ignite native persistence or configure swap to
overcome OOME [1]. Also page about capacity planning might be useful
here [2].

[1] https://apacheignite.readme.io/docs/durable-memory
[2] https://apacheignite.readme.io/docs/capacity-planning

пн, 24 дек. 2018 г. в 09:53, summasumma <[hidden email]>:

>
> Thanks Ivan.
>
> This means having both configuration as follows:
> xml        --> <property name="maxSize" value="#{8L * 1024 * 1024 * 1024}"/>
> jvmption --> "-XX:MaxDirectMemorySize=8g"
>
> Means, i should have minimum 16 Gb of RAM (8 dataregion+ 8 directmem) for
> Ignite to run properly i guess.
>
> In my current setup i have 16GB of total ram and given 12GB=maxSize
> Dataregion and jvmoption as '-XX:MaxDirectMemorySize=8g'. And this is
> crashing Ignite with OOME after a while (though not immediatly) when i try
> to do a performance testing of Update operation. so this means either i
> should increase the RAM or decrease the XX:MaxDirectMemorySize to 4g ?
>
> Please clarify
>
> Thanks
> ...summa
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/



--
Best regards,
Ivan Pavlukhin
Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

We’re actually looking into off-heap memory usage of ignite now, specifically into the MaxDirectMemorySize needed.

So far it seems that MaxDirectMemorySize=256m should work. It may need to be larger if you use non-default walSegmentSize.

Not limiting the direct memory size may result in excessive allocation, more work for garbage collector and even OOM.

My early recommendation (note that the investigation is still in progress) would be to set the parameter to 256mb and increase it if you hit “OOME: direct memory”. We’ll have it published on readme.io when the testing is done.

Stan

> On 24 Dec 2018, at 11:42, Павлухин Иван <[hidden email]> wrote:
>
> Hi summasumma,
>
>> Means, i should have minimum 16 Gb of RAM (8 dataregion+ 8 directmem) for
>> Ignite to run properly i guess.
>
> Not quite. Actually, I am not aware that Ignite requires some special
> tuning of MaxDirectMemorySize. If direct memory causes OOME then the
> exception message usually points it out (e.g. "Direct buffer memory").
> Check if it is your case.
>
> Indeed you should be careful when configure DataRegion.maxSize because
> OOME is a real problem and you will get it if put more data into
> Ignite than specified limit. One should carefully plan how many data
> is going to be stored into Ignite when in-memory mode is used. Also,
> it is possible to use Ignite native persistence or configure swap to
> overcome OOME [1]. Also page about capacity planning might be useful
> here [2].
>
> [1] https://apacheignite.readme.io/docs/durable-memory
> [2] https://apacheignite.readme.io/docs/capacity-planning
>
> пн, 24 дек. 2018 г. в 09:53, summasumma <[hidden email]>:
>>
>> Thanks Ivan.
>>
>> This means having both configuration as follows:
>> xml        --> <property name="maxSize" value="#{8L * 1024 * 1024 * 1024}"/>
>> jvmption --> "-XX:MaxDirectMemorySize=8g"
>>
>> Means, i should have minimum 16 Gb of RAM (8 dataregion+ 8 directmem) for
>> Ignite to run properly i guess.
>>
>> In my current setup i have 16GB of total ram and given 12GB=maxSize
>> Dataregion and jvmoption as '-XX:MaxDirectMemorySize=8g'. And this is
>> crashing Ignite with OOME after a while (though not immediatly) when i try
>> to do a performance testing of Update operation. so this means either i
>> should increase the RAM or decrease the XX:MaxDirectMemorySize to 4g ?
>>
>> Please clarify
>>
>> Thanks
>> ...summa
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
colinc colinc
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Thanks for the responses. To summarise:

* JVM Heap (Xmx) - Not normally used by Ignite for caching data.
* MaxDirectMemorySize - Used by Ignite for some file operations but not for
caching data. As per above, 256m is usually sufficient.
* DataRegion maxSize - Used by Ignite to determine how much memory to
allocate using some combination of durable, swap and off-heap (Java unsafe)
RAM.

In my case, I have configured swap storage
(https://apacheignite.readme.io/docs/swap-space) but *not* Ignite durable
memory. If DataRegion maxSize is say 100GB and my physical RAM is 50GB then
the swap file will be 100GB but Ignite will also use some portion (<50GB) of
the available physical RAM for off-heap cache data storage.

My question is about how to limit the size of this portion while still
allowing the DataRegion to specify a large swap file for use as overflow of
less regularly accessed data.

For example, say I wanted my node to use 8GB off-heap physical RAM and 100GB
swap file on a machine that has a total of 50GB of physical RAM (shared with
other processes).

What parameter would I need to configure 8GB? Is it possible to control
this?

Thanks,
Colin.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
colinc colinc
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Or is it necessary to use the OS to enforce this? For example using ulimit -d



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

> In my case, I have configured swap storage
> (https://apacheignite.readme.io/docs/swap-space) but *not* Ignite durable
> memory. If DataRegion maxSize is say 100GB and my physical RAM is 50GB
> then
> the swap file will be 100GB but Ignite will also use some portion (<50GB)
> of
> the available physical RAM for off-heap cache data storage.

I assume by Durable Memory you mean Native Persistence. They're
(confusingly) different - Durable Memory is just the name of the Ignite's
memory architecture, not necessarily with Persistence enabled.

> My question is about how to limit the size of this portion while still
> allowing the DataRegion to specify a large swap file for use as overflow
> of
> less regularly accessed data.

I don't think it's possible. Just use Native Persistence instead - you'll
get the memory distribution that you want
(dataRegionConfiguration.maxSize=8gb will do the trick), and actual
persistence as a bonus.

Stan



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
colinc colinc
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Thanks. We'll give Native Persistence another try.

Our reluctance to use it stems from the fact that if something goes wrong
with the storage then there are additional production processes required to
recover - bad persistent store can cause the cluster to fail to start or
else propagate problems.

If your cache data is considered transient then the recovery or upgrade
process is a straightforward restart.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
rick_tem rick_tem
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Yes, we have a similar reluctance to use persistent store.  Ouruse case is
that Gigs of data will be running through it with several caches that we
don't necessarily want to keep around in a docker environment.  Some caches,
however, we would like persistent.  Is there a plan to have persistence
configurable at the cache level, or will it be all or nothing in the
forseeable future?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Hello!

You can have some caches persistent and some not persistent, by having several DataRegions some of whose have persistenceEnabled=true and some are not, and specifying DataRegion by name in cache configuration.


Regards,
--
Ilya Kasnacheev


ср, 16 янв. 2019 г. в 13:15, rick_tem <[hidden email]>:
Yes, we have a similar reluctance to use persistent store.  Ouruse case is
that Gigs of data will be running through it with several caches that we
don't necessarily want to keep around in a docker environment.  Some caches,
however, we would like persistent.  Is there a plan to have persistence
configurable at the cache level, or will it be all or nothing in the
forseeable future?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
rick_tem rick_tem
Reply | Threaded
Open this post in threaded view
|

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Oh, that is great.  I wasn't aware of that.  Thanks for the link!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/