Persistent store and eviction policy.

classic Classic list List threaded Threaded
5 messages Options
userx userx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Persistent store and eviction policy.

Hi team,

I was going through the documentation of durable memory at https://apacheignite.readme.io/docs/durable-memory

As per the documentation, durable memory comes into picture when PersistentStore configuration is enabled. Now durable memory uses both RAM(hot data) and disk (superset). When the RAM part reaches a threshold (80% by default as per the documentation), the durable data retains only hot data in RAM and rest on the disk.

QUESTION 1
So does that mean that there is a default eviction policy which comes into existence ? Or does the user explicitly has to mention the same in configuration ? What happens if he does not mention the eviction policy in configuration ?

Suppose there are 2 nodes N1 (different physical box) and N2 (different physical box) and the data is distributed in PARTITIONED mode and persistent store is enabled.

Here is the example of entries

N1-> "1","X"
N2-> "2","Y"

QUESTION 2
Suppose N1 crashes and goes down and does not come up at all for say 5 hours. Is "1" retrievable at all during that time if N1 went down after an entry to its WAL file or we loose N1 ? If say the entry was not able to be made in WAL file and should we have configured a back up as N3 (different physical box), would it have saved "1" ?



Alexey Kukushkin Alexey Kukushkin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Persistent store and eviction policy.

Hi,

1. Persistence and Data Eviction are alternative options to handle out-of-memory scenarios.
Persistence makes Ignite fill all available RAM and move the oldest page to the “disk” part of the cache when there is not enough memory.
Data eviction policy makes Ignite to completely remove some entries or memory pages from the cache depending on the policy.
Persistence comes at cost of performance but gives us virtually unlimited reliable cache.
Use the one or the other.

2. No, if you have no backups configured and a node goes down the data becomes unavailable. Persistence will not help - only partitions that a nodes owns are persisted on the node. Configure backups to address the issue.

Best regards, 
Alexey



On Thursday, August 10, 2017, 5:38:50 AM GMT+3, userx <[hidden email]> wrote:


Hi team,

I was going through the documentation of durable memory at

As per the documentation, durable memory comes into picture when
PersistentStore configuration is enabled. Now durable memory uses both
RAM(hot data) and disk (superset). When the RAM part reaches a threshold
(80% by default as per the documentation), the durable data retains only hot
data in RAM and rest on the disk.

QUESTION 1
So does that mean that there is a default eviction policy which comes into
existence ? Or does the user explicitly has to mention the same in
configuration ? What happens if he does not mention the eviction policy in
configuration ?

Suppose there are 2 nodes N1 (different physical box) and N2 (different
physical box) and the data is distributed in PARTITIONED mode and persistent
store is enabled.

Here is the example of entries

N1-> "1","X"
N2-> "2","Y"

QUESTION 2
Suppose N1 crashes and goes down and does not come up at all for say 5
hours. Is "1" retrievable at all during that time if N1 went down after an
entry to its WAL file or we loose N1 ? If say the entry was not able to be
made in WAL file and should we have configured a back up as N3 (different
physical box), would it have saved "1" ?







--
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Ivan Rakov Ivan Rakov
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Persistent store and eviction policy.

In reply to this post by userx

Hi!


I'm afraid, description of page-based eviction in documentation is not quite correct.
Page-based eviction (RANDOM_LRU or RANDOM_2_LRU) can be activated only if persistent store is disabled. It defines algorithm for choosing page in RAM to remove all contents completely.

On the other hand, when persistent store is enabled, eviction from RAM to disk is enabled by default and is not customizable by user. So answer to first question 1 is no - you shouldn't specify anything in configuration to make disk eviction work.

About question 2 - You will lose partition after crash of N1. Losing of partition is always undesirable scenario. By default, partition will be reset, and you can't say for sure which version of partition (old from disk or new from another node) will be resolved as actual after join of N1.
Safe solution - configure backup nodes. Your data will be safe at N3.
Another safe solution - set safe partition loss policy (e.g. PartitionLossPolicy.READ_WRITE_SAFE). All reads/writes from lost partition will throw an exception until crashed node N1 returns to the topology. After that, "1" entry will be recovered.


Thanks for raising this topic, we'll fix the documentation soon.

Best Regards,
Ivan Rakov

On 10.08.2017 5:38, userx wrote:
Hi team,

I was going through the documentation of durable memory at
https://apacheignite.readme.io/docs/durable-memory

As per the documentation, durable memory comes into picture when
PersistentStore configuration is enabled. Now durable memory uses both
RAM(hot data) and disk (superset). When the RAM part reaches a threshold
(80% by default as per the documentation), the durable data retains only hot
data in RAM and rest on the disk.

QUESTION 1
So does that mean that there is a default eviction policy which comes into
existence ? Or does the user explicitly has to mention the same in
configuration ? What happens if he does not mention the eviction policy in
configuration ?

Suppose there are 2 nodes N1 (different physical box) and N2 (different
physical box) and the data is distributed in PARTITIONED mode and persistent
store is enabled. 

Here is the example of entries

N1-> "1","X"
N2-> "2","Y"

QUESTION 2
Suppose N1 crashes and goes down and does not come up at all for say 5
hours. Is "1" retrievable at all during that time if N1 went down after an
entry to its WAL file or we loose N1 ? If say the entry was not able to be
made in WAL file and should we have configured a back up as N3 (different
physical box), would it have saved "1" ?







--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Persistent-store-and-eviction-policy-tp16092.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

userx userx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Persistent store and eviction policy.

Thank you.
Denis Magda-2 Denis Magda-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Persistent store and eviction policy.

In reply to this post by Ivan Rakov
Folks,

I’ve updated the documentation avoiding any misunderstanding - " If Ignite Persistence is enabled then the page-based evictions have no effect because the oldest pages will be evicted from RAM automatically if there is not enough space available.


BTW, why can’t we make the eviction customizable when the store is used? Don’t see any issue with that from the user perspective.

Denis

On Aug 10, 2017, at 2:18 AM, Ivan Rakov <[hidden email]> wrote:

Hi!


I'm afraid, description of page-based eviction in documentation is not quite correct.
Page-based eviction (RANDOM_LRU or RANDOM_2_LRU) can be activated only if persistent store is disabled. It defines algorithm for choosing page in RAM to remove all contents completely.

On the other hand, when persistent store is enabled, eviction from RAM to disk is enabled by default and is not customizable by user. So answer to first question 1 is no - you shouldn't specify anything in configuration to make disk eviction work.

About question 2 - You will lose partition after crash of N1. Losing of partition is always undesirable scenario. By default, partition will be reset, and you can't say for sure which version of partition (old from disk or new from another node) will be resolved as actual after join of N1.
Safe solution - configure backup nodes. Your data will be safe at N3.
Another safe solution - set safe partition loss policy (e.g. PartitionLossPolicy.READ_WRITE_SAFE). All reads/writes from lost partition will throw an exception until crashed node N1 returns to the topology. After that, "1" entry will be recovered.


Thanks for raising this topic, we'll fix the documentation soon.

Best Regards,
Ivan Rakov

On 10.08.2017 5:38, userx wrote:
Hi team,

I was going through the documentation of durable memory at
https://apacheignite.readme.io/docs/durable-memory

As per the documentation, durable memory comes into picture when
PersistentStore configuration is enabled. Now durable memory uses both
RAM(hot data) and disk (superset). When the RAM part reaches a threshold
(80% by default as per the documentation), the durable data retains only hot
data in RAM and rest on the disk.

QUESTION 1
So does that mean that there is a default eviction policy which comes into
existence ? Or does the user explicitly has to mention the same in
configuration ? What happens if he does not mention the eviction policy in
configuration ?

Suppose there are 2 nodes N1 (different physical box) and N2 (different
physical box) and the data is distributed in PARTITIONED mode and persistent
store is enabled. 

Here is the example of entries

N1-> "1","X"
N2-> "2","Y"

QUESTION 2
Suppose N1 crashes and goes down and does not come up at all for say 5
hours. Is "1" retrievable at all during that time if N1 went down after an
entry to its WAL file or we loose N1 ? If say the entry was not able to be
made in WAL file and should we have configured a back up as N3 (different
physical box), would it have saved "1" ?







--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Persistent-store-and-eviction-policy-tp16092.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


Loading...