off heap memory usage

classic Classic list List threaded Threaded
10 messages Options
shawn.du shawn.du
Reply | Threaded
Open this post in threaded view
|

off heap memory usage

Hi community,

I want monitor ignite cluster off heap memory usage. if off heap available memory is less than 1G then alert.
I went through ignite MBean,  find ClusterLocalNodeMetricsMXBeanImpl's NonHeapMemoryUsed. 
The value is too small. I don't think it is.  How can I get it?

Thanks
Shawn

Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: off heap memory usage

Hi!

Looks like this metric shows only non-heap memory, consumed by JVM itself. Looks like a bug.
If you want to get a picture of how much memory Ignite pages consumed, you can get DataRegionMetrics#PhysicalMemoryPages and multiply it by a size of a data page.
To do this you should enable memory metrics first. Here you can find how: https://apacheignite.readme.io/docs/memory-metrics
Note, that pages may be not full, will still be included into this metric.

Denis

чт, 4 янв. 2018 г. в 11:05, shawn.du <[hidden email]>:
Hi community,

I want monitor ignite cluster off heap memory usage. if off heap available memory is less than 1G then alert.
I went through ignite MBean,  find ClusterLocalNodeMetricsMXBeanImpl's NonHeapMemoryUsed. 
The value is too small. I don't think it is.  How can I get it?

Thanks
Shawn
colinc colinc
Reply | Threaded
Open this post in threaded view
|

Re: off heap memory usage

Hi Denis,

As per your comment, I can see pages*pageSize rising as entries are put into
the cache - but this metric doesn't come down e.g. when new nodes are added
to the cluster. I assume that the pages remain allocated but with a lower
fill factor.

So pages*pageSize gives a misleadingly pessimistic view of available memory.
It would be very useful to see the effect of adding extra capacity or
purging entries.

I note that the pagesFillFactor metric sounds like it should be able to
supply the missing information - but Ignite 2.3 seems to always return a
value of zero for me. Does this indicate a 100% fill rate, or is the metric
not in use?

Regards,
Colin.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: off heap memory usage

Collin,

To be able to see data region metrics, you should enable them either in configuration, or via MXBean.

Note, that memory metrics collection should be enabled per data region.

Denis

пт, 5 янв. 2018 г. в 12:13, colinc <[hidden email]>:
Hi Denis,

As per your comment, I can see pages*pageSize rising as entries are put into
the cache - but this metric doesn't come down e.g. when new nodes are added
to the cluster. I assume that the pages remain allocated but with a lower
fill factor.

So pages*pageSize gives a misleadingly pessimistic view of available memory.
It would be very useful to see the effect of adding extra capacity or
purging entries.

I note that the pagesFillFactor metric sounds like it should be able to
supply the missing information - but Ignite 2.3 seems to always return a
value of zero for me. Does this indicate a 100% fill rate, or is the metric
not in use?

Regards,
Colin.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
colinc colinc
Reply | Threaded
Open this post in threaded view
|

Re: off heap memory usage

I can confirm that I have metrics enabled for my region - I am able to read
allocatedPages, it's just the fillFactor that always seems to return zero.

Colin.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
colinc colinc
Reply | Threaded
Open this post in threaded view
|

Re: off heap memory usage

Please ignite my above comment, I am now able to retrieve the factor.

As such, is the following correct?

memoryUsed = pages * pageSize * pagesFillFactor



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
colinc colinc
Reply | Threaded
Open this post in threaded view
|

Re: off heap memory usage

I have given this (pages * pageSize * pagesFillFactor) a go now, but it
doesn't seem to be returning the values I'm expecting. In particular, the
value can drop significantly even when data is being inserted into the
cache.

Am I using pagesFillFactor incorrectly?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Alexey Popov Alexey Popov
Reply | Threaded
Open this post in threaded view
|

Re: off heap memory usage

Hi Colin,

Unfortunately, you can't get the exact off-heap size.

There are several tickets here
https://issues.apache.org/jira/browse/IGNITE-6814
https://issues.apache.org/jira/browse/IGNITE-5583
http://apache-ignite-users.70518.x6.nabble.com/Cache-size-in-Memory-td17226.html

You are using (pages * pageSize * pagesFillFactor) correctly (if you have
only one memory region).

BTW, pagesFillFactor is calculated using FreeLists [1] buckets. FreeLists
has a limited amount of buckets (for instance, approx 25% free, ~50% free,
~75% free).
In this case (pages * pageSize * pagesFillFactor) will give you an approx
value of memory usage (based on buckets) and you can see drops in the usage
when a page is moved from one bucket to another.

[1]
https://apacheignite.readme.io/docs/memory-architecture#section-free-lists

Thanks,
Alexey




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
colinc colinc
Reply | Threaded
Open this post in threaded view
|

Re: off heap memory usage

Thanks for this. The referenced post mentions AllocatedPages rather than
PhysicalPages. Which would you advise is the most appropriate for this
application?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Alexey Popov Alexey Popov
Reply | Threaded
Open this post in threaded view
|

Re: off heap memory usage

Hi Colin,

You should use TotalAllocatedPages if you have persistence disabled.

There is an know issue with PhysicalMemoryPages:
https://issues.apache.org/jira/browse/IGNITE-6963

Thank you,
Alexey



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/