How to know memory used by a cache or a set

classic Classic list List threaded Threaded
11 messages Options
yblazart yblazart
Reply | Threaded
Open this post in threaded view
|

How to know memory used by a cache or a set

Hello ! 

Because I have multiple process that compute files in ignite H2 database,
I really need to know how memory have been used by caches and sets, 
but the persistence is not activated (performance) and when I try to use the MxBean for CacheGroup, it send me back a 0.

Is there any solution ??

Thanks.
ibelyakov ibelyakov
Reply | Threaded
Open this post in threaded view
|

Re: How to know memory used by a cache or a set

Hi,

Did you turn on cache metrics for your data region?

To turn the metrics on, use one of the following approaches:
1. Set DataRegionConfiguration.setMetricsEnabled(true) for every region you
want to collect the metrics for.
2. Use the DataRegionMetricsMXBean.enableMetrics() method exposed by a
special JMX bean.

More information regarding cache metrics available here:
https://apacheignite.readme.io/docs/cache-metrics

Regards,
Igor




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
yblazart yblazart
Reply | Threaded
Open this post in threaded view
|

Re: How to know memory used by a cache or a set

Hello.  I'm almost sure I already did that. I will check tomorrow.

Thks

Le lun. 27 mai 2019 à 11:09, ibelyakov <[hidden email]> a écrit :
Hi,

Did you turn on cache metrics for your data region?

To turn the metrics on, use one of the following approaches:
1. Set DataRegionConfiguration.setMetricsEnabled(true) for every region you
want to collect the metrics for.
2. Use the DataRegionMetricsMXBean.enableMetrics() method exposed by a
special JMX bean.

More information regarding cache metrics available here:
https://apacheignite.readme.io/docs/cache-metrics

Regards,
Igor




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
yann.blazart@externe.bnpparibas.com yann.blazart@externe.bnpparibas.com
Reply | Threaded
Open this post in threaded view
|

Re: How to know memory used by a cache or a set

In reply to this post by ibelyakov
Hello, I'm back.

Well, I need to get memory used by each execution of my process, so I put all involved caches into the same cacheGroup.

If I use the CacheGroupMetricsBean, the size gave to me is 0 !
If I enable persistence on DataRegion, I get size, but I don't want to use the persistence enabled.

Is it a bug ?

regards.

> Le 27 mai 2019 à 11:09, ibelyakov <[hidden email]> a écrit :
>
> Hi,
>
> Did you turn on cache metrics for your data region?
>
> To turn the metrics on, use one of the following approaches:
> 1. Set DataRegionConfiguration.setMetricsEnabled(true) for every region you
> want to collect the metrics for.
> 2. Use the DataRegionMetricsMXBean.enableMetrics() method exposed by a
> special JMX bean.
>
> More information regarding cache metrics available here:
> https://apacheignite.readme.io/docs/cache-metrics
>
> Regards,
> Igor
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary, consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.
Alex Plehanov Alex Plehanov
Reply | Threaded
Open this post in threaded view
|

Re: How to know memory used by a cache or a set

Hello,

It's a known issue [1]. Now you can get cache group size via JMX only if persistence is used. 
If persistence is not used you can get allocated size only for data region (but you can have more then one data region and assign cache groups to data regions any way you want)


ср, 12 июн. 2019 г. в 16:00, [hidden email] <[hidden email]>:
Hello, I'm back.

Well, I need to get memory used by each execution of my process, so I put all involved caches into the same cacheGroup.

If I use the CacheGroupMetricsBean, the size gave to me is 0 !
If I enable persistence on DataRegion, I get size, but I don't want to use the persistence enabled.

Is it a bug ?

regards.

> Le 27 mai 2019 à 11:09, ibelyakov <[hidden email]> a écrit :
>
> Hi,
>
> Did you turn on cache metrics for your data region?
>
> To turn the metrics on, use one of the following approaches:
> 1. Set DataRegionConfiguration.setMetricsEnabled(true) for every region you
> want to collect the metrics for.
> 2. Use the DataRegionMetricsMXBean.enableMetrics() method exposed by a
> special JMX bean.
>
> More information regarding cache metrics available here:
> https://apacheignite.readme.io/docs/cache-metrics
>
> Regards,
> Igor
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary, consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.
yblazart yblazart
Reply | Threaded
Open this post in threaded view
|

Re: How to know memory used by a cache or a set

Ok, but I can't create dynamically a data region ? Because each time I  receive a new file to process, I create a cachegroup to handle it, then I clean everything.

Le jeu. 13 juin 2019 à 13:28, Alex Plehanov <[hidden email]> a écrit :
Hello,

It's a known issue [1]. Now you can get cache group size via JMX only if persistence is used. 
If persistence is not used you can get allocated size only for data region (but you can have more then one data region and assign cache groups to data regions any way you want)


ср, 12 июн. 2019 г. в 16:00, [hidden email] <[hidden email]>:
Hello, I'm back.

Well, I need to get memory used by each execution of my process, so I put all involved caches into the same cacheGroup.

If I use the CacheGroupMetricsBean, the size gave to me is 0 !
If I enable persistence on DataRegion, I get size, but I don't want to use the persistence enabled.

Is it a bug ?

regards.

> Le 27 mai 2019 à 11:09, ibelyakov <[hidden email]> a écrit :
>
> Hi,
>
> Did you turn on cache metrics for your data region?
>
> To turn the metrics on, use one of the following approaches:
> 1. Set DataRegionConfiguration.setMetricsEnabled(true) for every region you
> want to collect the metrics for.
> 2. Use the DataRegionMetricsMXBean.enableMetrics() method exposed by a
> special JMX bean.
>
> More information regarding cache metrics available here:
> https://apacheignite.readme.io/docs/cache-metrics
>
> Regards,
> Igor
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary, consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.
Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: How to know memory used by a cache or a set

+ dev list

Ignite developers,

Seems that the present solution for memory calculation doesn't work (check the thread):

Was it really broken?

--
Denis Magda


On Thu, Jun 13, 2019 at 9:45 AM yann Blazart <[hidden email]> wrote:
Ok, but I can't create dynamically a data region ? Because each time I  receive a new file to process, I create a cachegroup to handle it, then I clean everything.

Le jeu. 13 juin 2019 à 13:28, Alex Plehanov <[hidden email]> a écrit :
Hello,

It's a known issue [1]. Now you can get cache group size via JMX only if persistence is used. 
If persistence is not used you can get allocated size only for data region (but you can have more then one data region and assign cache groups to data regions any way you want)


ср, 12 июн. 2019 г. в 16:00, [hidden email] <[hidden email]>:
Hello, I'm back.

Well, I need to get memory used by each execution of my process, so I put all involved caches into the same cacheGroup.

If I use the CacheGroupMetricsBean, the size gave to me is 0 !
If I enable persistence on DataRegion, I get size, but I don't want to use the persistence enabled.

Is it a bug ?

regards.

> Le 27 mai 2019 à 11:09, ibelyakov <[hidden email]> a écrit :
>
> Hi,
>
> Did you turn on cache metrics for your data region?
>
> To turn the metrics on, use one of the following approaches:
> 1. Set DataRegionConfiguration.setMetricsEnabled(true) for every region you
> want to collect the metrics for.
> 2. Use the DataRegionMetricsMXBean.enableMetrics() method exposed by a
> special JMX bean.
>
> More information regarding cache metrics available here:
> https://apacheignite.readme.io/docs/cache-metrics
>
> Regards,
> Igor
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary, consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.
Alex Plehanov Alex Plehanov
Reply | Threaded
Open this post in threaded view
|

Re: How to know memory used by a cache or a set

Denis,

Documentation for memory usage calculation covers another case (memory usage by the node).
There is no ability (AFAIK) in released Ignite versions to calculate memory used by a cache or cache group when persistence is disabled. 
Dedicated data region can be used for some of the caches in some cases and metrics can be collected for this data region, but when the cache is destroyed (or data is deleted) memory is not deallocated, it's going to reuse list.
There is a new metric implemented DataRegionMetrics#getTotalUsedPages (count of allocated pages minus count of pages in the reuse list) which will partially help to solve Yann's problem, but this metric will be available only in the next Ignite release.
Also, as a temporary workaround, some internal API can be used to get a count of pages in the reuse list and calculate total used pages by data region manually.

ср, 19 июн. 2019 г. в 07:59, Denis Magda <[hidden email]>:
+ dev list

Ignite developers,

Seems that the present solution for memory calculation doesn't work (check
the thread):
https://apacheignite.readme.io/v2.5/docs/memory-metrics#section-memory-usage-calculation
<https://apacheignite.readme.io/docs/cache-metrics>

Was it really broken?

--
Denis Magda


On Thu, Jun 13, 2019 at 9:45 AM yann Blazart <[hidden email]> wrote:

> Ok, but I can't create dynamically a data region ? Because each time I
> receive a new file to process, I create a cachegroup to handle it, then I
> clean everything.
>
> Le jeu. 13 juin 2019 à 13:28, Alex Plehanov <[hidden email]> a
> écrit :
>
>> Hello,
>>
>> It's a known issue [1]. Now you can get cache group size via JMX only if
>> persistence is used.
>> If persistence is not used you can get allocated size only for data
>> region (but you can have more then one data region and assign cache groups
>> to data regions any way you want)
>>
>> [1] : https://issues.apache.org/jira/browse/IGNITE-8517
>>
>> ср, 12 июн. 2019 г. в 16:00, [hidden email] <
>> [hidden email]>:
>>
>>> Hello, I'm back.
>>>
>>> Well, I need to get memory used by each execution of my process, so I
>>> put all involved caches into the same cacheGroup.
>>>
>>> If I use the CacheGroupMetricsBean, the size gave to me is 0 !
>>> If I enable persistence on DataRegion, I get size, but I don't want to
>>> use the persistence enabled.
>>>
>>> Is it a bug ?
>>>
>>> regards.
>>>
>>> > Le 27 mai 2019 à 11:09, ibelyakov <[hidden email]> a écrit
>>> :
>>> >
>>> > Hi,
>>> >
>>> > Did you turn on cache metrics for your data region?
>>> >
>>> > To turn the metrics on, use one of the following approaches:
>>> > 1. Set DataRegionConfiguration.setMetricsEnabled(true) for every
>>> region you
>>> > want to collect the metrics for.
>>> > 2. Use the DataRegionMetricsMXBean.enableMetrics() method exposed by a
>>> > special JMX bean.
>>> >
>>> > More information regarding cache metrics available here:
>>> > https://apacheignite.readme.io/docs/cache-metrics
>>> >
>>> > Regards,
>>> > Igor
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>> This message and any attachments (the "message") is
>>> intended solely for the intended addressees and is confidential.
>>> If you receive this message in error,or are not the intended
>>> recipient(s),
>>> please delete it and any copies from your systems and immediately notify
>>> the sender. Any unauthorized view, use that does not comply with its
>>> purpose,
>>> dissemination or disclosure, either whole or partial, is prohibited.
>>> Since the internet
>>> cannot guarantee the integrity of this message which may not be
>>> reliable, BNP PARIBAS
>>> (and its subsidiaries) shall not be liable for the message if modified,
>>> changed or falsified.
>>> Do not print this message unless it is necessary, consider the
>>> environment.
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>
>>> Ce message et toutes les pieces jointes (ci-apres le "message")
>>> sont etablis a l'intention exclusive de ses destinataires et sont
>>> confidentiels.
>>> Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
>>> merci de le detruire ainsi que toute copie de votre systeme et d'en
>>> avertir
>>> immediatement l'expediteur. Toute lecture non autorisee, toute
>>> utilisation de
>>> ce message qui n'est pas conforme a sa destination, toute diffusion ou
>>> toute
>>> publication, totale ou partielle, est interdite. L'Internet ne
>>> permettant pas d'assurer
>>> l'integrite de ce message electronique susceptible d'alteration, BNP
>>> Paribas
>>> (et ses filiales) decline(nt) toute responsabilite au titre de ce
>>> message dans l'hypothese
>>> ou il aurait ete modifie, deforme ou falsifie.
>>> N'imprimez ce message que si necessaire, pensez a l'environnement.
>>>
>>
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: How to know memory used by a cache or a set

Looks like a mess. 

Alex Goncharuk, Nikolay Izhikov, considering the latest changes and new metrics & tracing framework, how would they one get cache/table memory size via a simple metric? There should be a way to make it workable without hacks like cachegroups, etc.


-
Denis


On Wed, Jun 19, 2019 at 3:29 AM Alex Plehanov <[hidden email]> wrote:
Denis,

Documentation for memory usage calculation covers another case (memory usage by the node).
There is no ability (AFAIK) in released Ignite versions to calculate memory used by a cache or cache group when persistence is disabled. 
Dedicated data region can be used for some of the caches in some cases and metrics can be collected for this data region, but when the cache is destroyed (or data is deleted) memory is not deallocated, it's going to reuse list.
There is a new metric implemented DataRegionMetrics#getTotalUsedPages (count of allocated pages minus count of pages in the reuse list) which will partially help to solve Yann's problem, but this metric will be available only in the next Ignite release.
Also, as a temporary workaround, some internal API can be used to get a count of pages in the reuse list and calculate total used pages by data region manually.

ср, 19 июн. 2019 г. в 07:59, Denis Magda <[hidden email]>:
+ dev list

Ignite developers,

Seems that the present solution for memory calculation doesn't work (check
the thread):
https://apacheignite.readme.io/v2.5/docs/memory-metrics#section-memory-usage-calculation
<https://apacheignite.readme.io/docs/cache-metrics>

Was it really broken?

--
Denis Magda


On Thu, Jun 13, 2019 at 9:45 AM yann Blazart <[hidden email]> wrote:

> Ok, but I can't create dynamically a data region ? Because each time I
> receive a new file to process, I create a cachegroup to handle it, then I
> clean everything.
>
> Le jeu. 13 juin 2019 à 13:28, Alex Plehanov <[hidden email]> a
> écrit :
>
>> Hello,
>>
>> It's a known issue [1]. Now you can get cache group size via JMX only if
>> persistence is used.
>> If persistence is not used you can get allocated size only for data
>> region (but you can have more then one data region and assign cache groups
>> to data regions any way you want)
>>
>> [1] : https://issues.apache.org/jira/browse/IGNITE-8517
>>
>> ср, 12 июн. 2019 г. в 16:00, [hidden email] <
>> [hidden email]>:
>>
>>> Hello, I'm back.
>>>
>>> Well, I need to get memory used by each execution of my process, so I
>>> put all involved caches into the same cacheGroup.
>>>
>>> If I use the CacheGroupMetricsBean, the size gave to me is 0 !
>>> If I enable persistence on DataRegion, I get size, but I don't want to
>>> use the persistence enabled.
>>>
>>> Is it a bug ?
>>>
>>> regards.
>>>
>>> > Le 27 mai 2019 à 11:09, ibelyakov <[hidden email]> a écrit
>>> :
>>> >
>>> > Hi,
>>> >
>>> > Did you turn on cache metrics for your data region?
>>> >
>>> > To turn the metrics on, use one of the following approaches:
>>> > 1. Set DataRegionConfiguration.setMetricsEnabled(true) for every
>>> region you
>>> > want to collect the metrics for.
>>> > 2. Use the DataRegionMetricsMXBean.enableMetrics() method exposed by a
>>> > special JMX bean.
>>> >
>>> > More information regarding cache metrics available here:
>>> > https://apacheignite.readme.io/docs/cache-metrics
>>> >
>>> > Regards,
>>> > Igor
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>> This message and any attachments (the "message") is
>>> intended solely for the intended addressees and is confidential.
>>> If you receive this message in error,or are not the intended
>>> recipient(s),
>>> please delete it and any copies from your systems and immediately notify
>>> the sender. Any unauthorized view, use that does not comply with its
>>> purpose,
>>> dissemination or disclosure, either whole or partial, is prohibited.
>>> Since the internet
>>> cannot guarantee the integrity of this message which may not be
>>> reliable, BNP PARIBAS
>>> (and its subsidiaries) shall not be liable for the message if modified,
>>> changed or falsified.
>>> Do not print this message unless it is necessary, consider the
>>> environment.
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>
>>> Ce message et toutes les pieces jointes (ci-apres le "message")
>>> sont etablis a l'intention exclusive de ses destinataires et sont
>>> confidentiels.
>>> Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
>>> merci de le detruire ainsi que toute copie de votre systeme et d'en
>>> avertir
>>> immediatement l'expediteur. Toute lecture non autorisee, toute
>>> utilisation de
>>> ce message qui n'est pas conforme a sa destination, toute diffusion ou
>>> toute
>>> publication, totale ou partielle, est interdite. L'Internet ne
>>> permettant pas d'assurer
>>> l'integrite de ce message electronique susceptible d'alteration, BNP
>>> Paribas
>>> (et ses filiales) decline(nt) toute responsabilite au titre de ce
>>> message dans l'hypothese
>>> ou il aurait ete modifie, deforme ou falsifie.
>>> N'imprimez ce message que si necessaire, pensez a l'environnement.
>>>
>>
yann.blazart@externe.bnpparibas.com yann.blazart@externe.bnpparibas.com
Reply | Threaded
Open this post in threaded view
|

Re: How to know memory used by a cache or a set

Hello, can you help me o understand the real mean of this trace ? 

Off-heap [used=79492MB, free=56.92%, comm=86642MB]
    ^--   sysMemPlc region [used=0MB, free=99.13%, comm=40MB]
    ^--   Gondor region [used=79491MB, free=56.87%, comm=86562MB]


My region is called Gondor, is "used" is the amount of memory really used in this node ? In this case what's the meaning of "free" and "comm" ?

Or do we have 79Gb allocated AND my data only take 44% of space in it ?

Thanks, and regards.


Le 26 juin 2019 à 01:28, Denis Magda <[hidden email]> a écrit :

Looks like a mess. 

Alex Goncharuk, Nikolay Izhikov, considering the latest changes and new metrics & tracing framework, how would they one get cache/table memory size via a simple metric? There should be a way to make it workable without hacks like cachegroups, etc.


-
Denis


On Wed, Jun 19, 2019 at 3:29 AM Alex Plehanov <[hidden email]> wrote:
Denis,

Documentation for memory usage calculation covers another case (memory usage by the node).
There is no ability (AFAIK) in released Ignite versions to calculate memory used by a cache or cache group when persistence is disabled. 
Dedicated data region can be used for some of the caches in some cases and metrics can be collected for this data region, but when the cache is destroyed (or data is deleted) memory is not deallocated, it's going to reuse list.
There is a new metric implemented DataRegionMetrics#getTotalUsedPages (count of allocated pages minus count of pages in the reuse list) which will partially help to solve Yann's problem, but this metric will be available only in the next Ignite release.
Also, as a temporary workaround, some internal API can be used to get a count of pages in the reuse list and calculate total used pages by data region manually.

ср, 19 июн. 2019 г. в 07:59, Denis Magda <[hidden email]>:
+ dev list

Ignite developers,

Seems that the present solution for memory calculation doesn't work (check
the thread):
https://apacheignite.readme.io/v2.5/docs/memory-metrics#section-memory-usage-calculation
<https://apacheignite.readme.io/docs/cache-metrics>

Was it really broken?

--
Denis Magda


On Thu, Jun 13, 2019 at 9:45 AM yann Blazart <[hidden email]> wrote:

> Ok, but I can't create dynamically a data region ? Because each time I
> receive a new file to process, I create a cachegroup to handle it, then I
> clean everything.
>
> Le jeu. 13 juin 2019 à 13:28, Alex Plehanov <[hidden email]> a
> écrit :
>
>> Hello,
>>
>> It's a known issue [1]. Now you can get cache group size via JMX only if
>> persistence is used.
>> If persistence is not used you can get allocated size only for data
>> region (but you can have more then one data region and assign cache groups
>> to data regions any way you want)
>>
>> [1] : https://issues.apache.org/jira/browse/IGNITE-8517
>>
>> ср, 12 июн. 2019 г. в 16:00, [hidden email] <
>> [hidden email]>:
>>
>>> Hello, I'm back.
>>>
>>> Well, I need to get memory used by each execution of my process, so I
>>> put all involved caches into the same cacheGroup.
>>>
>>> If I use the CacheGroupMetricsBean, the size gave to me is 0 !
>>> If I enable persistence on DataRegion, I get size, but I don't want to
>>> use the persistence enabled.
>>>
>>> Is it a bug ?
>>>
>>> regards.
>>>
>>> > Le 27 mai 2019 à 11:09, ibelyakov <[hidden email]> a écrit
>>> :
>>> >
>>> > Hi,
>>> >
>>> > Did you turn on cache metrics for your data region?
>>> >
>>> > To turn the metrics on, use one of the following approaches:
>>> > 1. Set DataRegionConfiguration.setMetricsEnabled(true) for every
>>> region you
>>> > want to collect the metrics for.
>>> > 2. Use the DataRegionMetricsMXBean.enableMetrics() method exposed by a
>>> > special JMX bean.
>>> >
>>> > More information regarding cache metrics available here:
>>> > https://apacheignite.readme.io/docs/cache-metrics
>>> >
>>> > Regards,
>>> > Igor
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>> This message and any attachments (the "message") is
>>> intended solely for the intended addressees and is confidential.
>>> If you receive this message in error,or are not the intended
>>> recipient(s),
>>> please delete it and any copies from your systems and immediately notify
>>> the sender. Any unauthorized view, use that does not comply with its
>>> purpose,
>>> dissemination or disclosure, either whole or partial, is prohibited.
>>> Since the internet
>>> cannot guarantee the integrity of this message which may not be
>>> reliable, BNP PARIBAS
>>> (and its subsidiaries) shall not be liable for the message if modified,
>>> changed or falsified.
>>> Do not print this message unless it is necessary, consider the
>>> environment.
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>
>>> Ce message et toutes les pieces jointes (ci-apres le "message")
>>> sont etablis a l'intention exclusive de ses destinataires et sont
>>> confidentiels.
>>> Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
>>> merci de le detruire ainsi que toute copie de votre systeme et d'en
>>> avertir
>>> immediatement l'expediteur. Toute lecture non autorisee, toute
>>> utilisation de
>>> ce message qui n'est pas conforme a sa destination, toute diffusion ou
>>> toute
>>> publication, totale ou partielle, est interdite. L'Internet ne
>>> permettant pas d'assurer
>>> l'integrite de ce message electronique susceptible d'alteration, BNP
>>> Paribas
>>> (et ses filiales) decline(nt) toute responsabilite au titre de ce
>>> message dans l'hypothese
>>> ou il aurait ete modifie, deforme ou falsifie.
>>> N'imprimez ce message que si necessaire, pensez a l'environnement.
>>>
>>

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: How to know memory used by a cache or a set

Hello!

> Or do we have 79Gb allocated AND my data only take 44% of space in it ?

This is correct.

"comm" is amount of memory allocated by underlying operating system, as far as my understanding goes.

Regards,
--
Ilya Kasnacheev


ср, 26 июн. 2019 г. в 12:14, [hidden email] <[hidden email]>:
Hello, can you help me o understand the real mean of this trace ? 

Off-heap [used=79492MB, free=56.92%, comm=86642MB]
    ^--   sysMemPlc region [used=0MB, free=99.13%, comm=40MB]
    ^--   Gondor region [used=79491MB, free=56.87%, comm=86562MB]


My region is called Gondor, is "used" is the amount of memory really used in this node ? In this case what's the meaning of "free" and "comm" ?

Or do we have 79Gb allocated AND my data only take 44% of space in it ?

Thanks, and regards.


Le 26 juin 2019 à 01:28, Denis Magda <[hidden email]> a écrit :

Looks like a mess. 

Alex Goncharuk, Nikolay Izhikov, considering the latest changes and new metrics & tracing framework, how would they one get cache/table memory size via a simple metric? There should be a way to make it workable without hacks like cachegroups, etc.


-
Denis


On Wed, Jun 19, 2019 at 3:29 AM Alex Plehanov <[hidden email]> wrote:
Denis,

Documentation for memory usage calculation covers another case (memory usage by the node).
There is no ability (AFAIK) in released Ignite versions to calculate memory used by a cache or cache group when persistence is disabled. 
Dedicated data region can be used for some of the caches in some cases and metrics can be collected for this data region, but when the cache is destroyed (or data is deleted) memory is not deallocated, it's going to reuse list.
There is a new metric implemented DataRegionMetrics#getTotalUsedPages (count of allocated pages minus count of pages in the reuse list) which will partially help to solve Yann's problem, but this metric will be available only in the next Ignite release.
Also, as a temporary workaround, some internal API can be used to get a count of pages in the reuse list and calculate total used pages by data region manually.

ср, 19 июн. 2019 г. в 07:59, Denis Magda <[hidden email]>:
+ dev list

Ignite developers,

Seems that the present solution for memory calculation doesn't work (check
the thread):
https://apacheignite.readme.io/v2.5/docs/memory-metrics#section-memory-usage-calculation
<https://apacheignite.readme.io/docs/cache-metrics>

Was it really broken?

--
Denis Magda


On Thu, Jun 13, 2019 at 9:45 AM yann Blazart <[hidden email]> wrote:

> Ok, but I can't create dynamically a data region ? Because each time I
> receive a new file to process, I create a cachegroup to handle it, then I
> clean everything.
>
> Le jeu. 13 juin 2019 à 13:28, Alex Plehanov <[hidden email]> a
> écrit :
>
>> Hello,
>>
>> It's a known issue [1]. Now you can get cache group size via JMX only if
>> persistence is used.
>> If persistence is not used you can get allocated size only for data
>> region (but you can have more then one data region and assign cache groups
>> to data regions any way you want)
>>
>> [1] : https://issues.apache.org/jira/browse/IGNITE-8517
>>
>> ср, 12 июн. 2019 г. в 16:00, [hidden email] <
>> [hidden email]>:
>>
>>> Hello, I'm back.
>>>
>>> Well, I need to get memory used by each execution of my process, so I
>>> put all involved caches into the same cacheGroup.
>>>
>>> If I use the CacheGroupMetricsBean, the size gave to me is 0 !
>>> If I enable persistence on DataRegion, I get size, but I don't want to
>>> use the persistence enabled.
>>>
>>> Is it a bug ?
>>>
>>> regards.
>>>
>>> > Le 27 mai 2019 à 11:09, ibelyakov <[hidden email]> a écrit
>>> :
>>> >
>>> > Hi,
>>> >
>>> > Did you turn on cache metrics for your data region?
>>> >
>>> > To turn the metrics on, use one of the following approaches:
>>> > 1. Set DataRegionConfiguration.setMetricsEnabled(true) for every
>>> region you
>>> > want to collect the metrics for.
>>> > 2. Use the DataRegionMetricsMXBean.enableMetrics() method exposed by a
>>> > special JMX bean.
>>> >
>>> > More information regarding cache metrics available here:
>>> > https://apacheignite.readme.io/docs/cache-metrics
>>> >
>>> > Regards,
>>> > Igor
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>> This message and any attachments (the "message") is
>>> intended solely for the intended addressees and is confidential.
>>> If you receive this message in error,or are not the intended
>>> recipient(s),
>>> please delete it and any copies from your systems and immediately notify
>>> the sender. Any unauthorized view, use that does not comply with its
>>> purpose,
>>> dissemination or disclosure, either whole or partial, is prohibited.
>>> Since the internet
>>> cannot guarantee the integrity of this message which may not be
>>> reliable, BNP PARIBAS
>>> (and its subsidiaries) shall not be liable for the message if modified,
>>> changed or falsified.
>>> Do not print this message unless it is necessary, consider the
>>> environment.
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>
>>> Ce message et toutes les pieces jointes (ci-apres le "message")
>>> sont etablis a l'intention exclusive de ses destinataires et sont
>>> confidentiels.
>>> Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
>>> merci de le detruire ainsi que toute copie de votre systeme et d'en
>>> avertir
>>> immediatement l'expediteur. Toute lecture non autorisee, toute
>>> utilisation de
>>> ce message qui n'est pas conforme a sa destination, toute diffusion ou
>>> toute
>>> publication, totale ou partielle, est interdite. L'Internet ne
>>> permettant pas d'assurer
>>> l'integrite de ce message electronique susceptible d'alteration, BNP
>>> Paribas
>>> (et ses filiales) decline(nt) toute responsabilite au titre de ce
>>> message dans l'hypothese
>>> ou il aurait ete modifie, deforme ou falsifie.
>>> N'imprimez ce message que si necessaire, pensez a l'environnement.
>>>
>>