Ignite persistence: Data of node is lost after being excluded from baseline topology

classic Classic list List threaded Threaded
3 messages Options
VincentCE VincentCE
Reply | Threaded
Open this post in threaded view
|

Ignite persistence: Data of node is lost after being excluded from baseline topology

This post was updated on .
Hi!

in our project we are currently using ignite 2.8.1 without ignite native
persistence enabled. No we would like to enable this feature to prevent data
loss during node restart.

Important: We use AttributeNodeFilters to separate our data, e.g data of
type1 only lives in the type1-clustergroup and so on.

I have two question regarding *native persistence*:

1. After some time we would like to shut down nodes of type1 to save
resources (but possibly would like to use it in the future again if business
requires it -> we would like the data to remain there safely/persisted).
However in order to start up the cluster again the type1 nodes need to be
excluded from baseline topology. But then if later on we would like to reuse
type1 nodes again their data is getting deleted as soon as they are
rejoining the baseline topology. Is there a way to prevent this?

2. When using AttributeNodeFilter it seems that we need to use fixed
consistentIds (e.g. consistentId = "type1-0") since otherwise a node of
type2 would potentially use the directory of a type1 node in the persistence
storage directory which finally would break the data separation when new
data is loaded into the caches. Is that correct?

Thanks in advance!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VincentCE VincentCE
Reply | Threaded
Open this post in threaded view
|

Re: Ignite persistence: Data of node is lost after being excluded from baseline topology

Hi igniters!

could I have short answers to my questions?

Thanks a lot!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite persistence: Data of node lost that got excluded from baseline topology

In reply to this post by VincentCE
Hello!

1. I guess it's possible to activate a cluster manually when not all baseline nodes are in. Later you can join your node back to baseline, hopefully. Please note that there's still chance that old data will be deleted from node upon join: We can only re-use data from old partition when we're confident that it's identical to new data of same partition (inactive cluster) or when we can apply historical rebalancing (apply changes from WAL to the same partition).Historical rebalance is usually only feasible when there's few differences between old and new partition. If the difference is significant, it's easier to rebalance the whole partition.

It may seem strange that we throw a lot of data out, but we don't have any obvious way to tell which of that data is current and which is not. So it's useless in the end.

2. Yes, it is recommended to always have consistentId in persistent cluster. Otherwise the node will just assume first available data dir as its own.

Regards,
--
Ilya Kasnacheev


пн, 23 нояб. 2020 г. в 11:17, VincentCE <[hidden email]>:
Hi!

in our project we are currently using ignite 2.8.1 without ignite native
persistence enabled. No we would like to enable this feature to prevent data
loss during node restart.

Important: We use AttributeNodeFilters to separate our data, e.g data of
type1 only lives in the type1-clustergroup and so on.

I have two question regarding *native persistence*:

1. After some time we would like to shut down nodes of type1 to save
resources (but possibly would like to use it in the future again if business
requires it -> we would like the data to remain there safely/persisted).
However in order to start up the cluster again the type1 nodes need to be
excluded from baseline topology. But then if later on we would like to reuse
type1 nodes again their data is getting deleted as soon as they are
rejoining the baseline topology. Is there a way to prevent this?

2. When using AttributeNodeFilter it seems that we need to use fixed
consistentIds (e.g. consistentId = "type1-0") since otherwise a node of
type2 would potentially use the directory of a type1 node in the persistence
storage directory which finally would break the data separation when new
data is loaded into the caches.

Thanks in advance!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/