Limiting the size of Persistent Store and clearing data on restart

classic Classic list List threaded Threaded
8 messages Options
userx userx
Reply | Threaded
Open this post in threaded view
|

Limiting the size of Persistent Store and clearing data on restart

Hi All,

1) Like the conventional configuration of https://apacheignite.readme.io/docs/memory-configuration#section-memory-policies where in we could limit the size of the memory, how do we define a configuration which can limit the size of Persistent Store so that say out of a 500GB HDisk, I don't want this Ignite related data to go beyond 300GB ?

2) Also, is there a configuration such that when Ignite data grid servers are restarted (say some code change either on client or server side), whatever was persisted before gets wiped out completely on all the participating nodes and only fresh persisted data is there ?
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Limiting the size of Persistent Store and clearing data on restart

Hi,

1) There is no possibility to limit Persistent Store size right now, but you always can use separate disk volume with size 300GB for your case to use it for Persistent Store.

2) Well it's not usual to use case, but after stopping node you can clear data from work folder or just invoke removeAll on cache

Evgenii
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Limiting the size of Persistent Store and clearing data on restart

Regarding your second question: looks like you don't actually need persistence. Its purpose is the opposite: to save cache data between restarts. 
If you use persistence to store more data than RAM available, then you can enable swap space: https://apacheignite.readme.io/v1.9/docs/off-heap-memory#section-swap-space

вт, 29 авг. 2017 г. в 20:06, ezhuravlev <[hidden email]>:
Hi,

1) There is no possibility to limit Persistent Store size right now, but you
always can use separate disk volume with size 300GB for your case to use it
for Persistent Store.

2) Well it's not usual to use case, but after stopping node you can clear
data from work folder or just invoke removeAll on cache

Evgenii



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Limiting-the-size-of-Persistent-Store-and-clearing-data-on-restart-tp16507p16515.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Limiting the size of Persistent Store and clearing data on restart



On Fri, Sep 1, 2017 at 1:41 PM, Denis Mekhanikov <[hidden email]> wrote:
Regarding your second question: looks like you don't actually need persistence. Its purpose is the opposite: to save cache data between restarts. 
If you use persistence to store more data than RAM available, then you can enable swap space: https://apacheignite.readme.io/v1.9/docs/off-heap-memory#section-swap-space

Denis, starting with 2.1, Ignite does not have the swap space anymore, since it never worked well and kept hurting the performance. We recommend now that users do use persistence whenever the data size is bigger than the available memory size.

dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Limiting the size of Persistent Store and clearing data on restart

In reply to this post by userx


On Tue, Aug 29, 2017 at 7:00 PM, userx <[hidden email]> wrote:
Hi All,

1) Like the conventional configuration of
https://apacheignite.readme.io/docs/memory-configuration#section-memory-policies
where in we could limit the size of the memory, how do we define a
configuration which can limit the size of Persistent Store so that say out
of a 500GB HDisk, I don't want this Ignite related data to go beyond 300GB ?

You would have to control your data on disk by yourself. Evicting from disk is very sensitive and use case specific, so it would be almost impossible to automate it.
 

2) Also, is there a configuration such that when Ignite data grid servers
are restarted (say some code change either on client or server side),
whatever was persisted before gets wiped out completely on all the
participating nodes and only fresh persisted data is there ?

How about calling destroyCache(...) on startup?

Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Limiting the size of Persistent Store and clearing data on restart

In reply to this post by dsetrakyan
Sorry, I didn't know that. I found a bunch of matches by "swapEnabled" keyword on master branch in the project, mostly in benchmarks configurations. These properties should probably be removed not to confuse readers.

пт, 1 сент. 2017 г. в 19:23, Dmitriy Setrakyan <[hidden email]>:
On Fri, Sep 1, 2017 at 1:41 PM, Denis Mekhanikov <[hidden email]> wrote:
Regarding your second question: looks like you don't actually need persistence. Its purpose is the opposite: to save cache data between restarts. 
If you use persistence to store more data than RAM available, then you can enable swap space: https://apacheignite.readme.io/v1.9/docs/off-heap-memory#section-swap-space

Denis, starting with 2.1, Ignite does not have the swap space anymore, since it never worked well and kept hurting the performance. We recommend now that users do use persistence whenever the data size is bigger than the available memory size.
userx userx
Reply | Threaded
Open this post in threaded view
|

Re: Limiting the size of Persistent Store and clearing data on restart

Hi all,

Regarding question 2, let me put it this way.

Like ignite has eviction policy for RAM described at
https://apacheignite.readme.io/docs/evictions
Is there an eviction policy for a persistent store ? Say for a use case
where in I cannot allocate a dedicated 1 or 2 tb space (like we do for
relational dbms), I would want to define an upper limit of say 300 GB and if
$IGNITE_HOME/PersistentStore goes beyond the same same, evict the lru cache.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Limiting the size of Persistent Store and clearing data on restart



On Sat, Sep 2, 2017 at 5:11 AM, userx <[hidden email]> wrote:
Hi all,

Regarding question 2, let me put it this way.

Like ignite has eviction policy for RAM described at
https://apacheignite.readme.io/docs/evictions
Is there an eviction policy for a persistent store ? Say for a use case
where in I cannot allocate a dedicated 1 or 2 tb space (like we do for
relational dbms), I would want to define an upper limit of say 300 GB and if
$IGNITE_HOME/PersistentStore goes beyond the same same, evict the lru cache.


Evictions for RAM make sense because the data is still on disk and is not lost. Evictions from disk do not make sense for me, nor am I aware of other databases that have this feature. I believe it should be handled by users based on the application requirements.

I think the Ignite community may agree to consider this feature as a possible enhancement. Can you provide an example of a similar feature you may have seen in another persistent store or a database?
 


--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/