Drop index do not release memory used ?

classic Classic list List threaded Threaded
4 messages Options
yann.blazart@externe.bnpparibas.com yann.blazart@externe.bnpparibas.com
Reply | Threaded
Open this post in threaded view
|

Drop index do not release memory used ?

Hello all.

If you can remember, I found a way to compute the real size of memory used in offheap, using the reuselist size.

As I'm facing some limits on my hardware, I'm trying to optimize my memory consumption, in pure memory, no persistence on hdd or ssd.

For that as I have to execute plenty of request on my stored data, I saw that indexes consumes a lot of memory.

To improve that, in my algorithm I tried to create tables with only pk, no indexes at first.

Then before each request, I tried to create indexes , execute request, then drop indexes.

What I see, is that drop index do not release memory...

Everything is release only when we drop table.

Is this normal  ?


Thnaks and regards.
This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary, consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

a.dolmatov a.dolmatov
Reply | Threaded
Open this post in threaded view
|

Re: Drop index do not release memory used ?

Ignite believe that Ignite will reuse that memory. But it is a question, does Ignite reuse index data blocks for data blocks.

On Fri, Oct 25, 2019, 15:28 [hidden email] <[hidden email]> wrote:
Hello all.

If you can remember, I found a way to compute the real size of memory used in offheap, using the reuselist size.

As I'm facing some limits on my hardware, I'm trying to optimize my memory consumption, in pure memory, no persistence on hdd or ssd.

For that as I have to execute plenty of request on my stored data, I saw that indexes consumes a lot of memory.

To improve that, in my algorithm I tried to create tables with only pk, no indexes at first.

Then before each request, I tried to create indexes , execute request, then drop indexes.

What I see, is that drop index do not release memory...

Everything is release only when we drop table.

Is this normal  ?


Thnaks and regards.
This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary, consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

yblazart yblazart
Reply | Threaded
Open this post in threaded view
|

Re: Drop index do not release memory used ?

Yes the data pages in reuse list are reused for new objects.

If you drop table, you get back all pages used in reuse list.

But it seems that it's not true for drop indexes

Le sam. 26 oct. 2019 à 16:18, Andrey Dolmatov <[hidden email]> a écrit :
Ignite believe that Ignite will reuse that memory. But it is a question, does Ignite reuse index data blocks for data blocks.

On Fri, Oct 25, 2019, 15:28 [hidden email] <[hidden email]> wrote:
Hello all.

If you can remember, I found a way to compute the real size of memory used in offheap, using the reuselist size.

As I'm facing some limits on my hardware, I'm trying to optimize my memory consumption, in pure memory, no persistence on hdd or ssd.

For that as I have to execute plenty of request on my stored data, I saw that indexes consumes a lot of memory.

To improve that, in my algorithm I tried to create tables with only pk, no indexes at first.

Then before each request, I tried to create indexes , execute request, then drop indexes.

What I see, is that drop index do not release memory...

Everything is release only when we drop table.

Is this normal  ?


Thnaks and regards.
This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary, consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

Re: Drop index do not release memory used ?

Hi,

What version do you use?
There was an issue with recycling pages between data and indexes which has been fixed in 2.7 https://issues.apache.org/jira/browse/IGNITE-4958.
In AI 2.7 and later this should be working fine.

Stan

On Sat, Oct 26, 2019 at 5:22 PM yann Blazart <[hidden email]> wrote:
Yes the data pages in reuse list are reused for new objects.

If you drop table, you get back all pages used in reuse list.

But it seems that it's not true for drop indexes

Le sam. 26 oct. 2019 à 16:18, Andrey Dolmatov <[hidden email]> a écrit :
Ignite believe that Ignite will reuse that memory. But it is a question, does Ignite reuse index data blocks for data blocks.

On Fri, Oct 25, 2019, 15:28 [hidden email] <[hidden email]> wrote:
Hello all.

If you can remember, I found a way to compute the real size of memory used in offheap, using the reuselist size.

As I'm facing some limits on my hardware, I'm trying to optimize my memory consumption, in pure memory, no persistence on hdd or ssd.

For that as I have to execute plenty of request on my stored data, I saw that indexes consumes a lot of memory.

To improve that, in my algorithm I tried to create tables with only pk, no indexes at first.

Then before each request, I tried to create indexes , execute request, then drop indexes.

What I see, is that drop index do not release memory...

Everything is release only when we drop table.

Is this normal  ?


Thnaks and regards.
This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary, consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.