Baseline topology - Data inconsistency

classic Classic list List threaded Threaded
10 messages Options
userx userx
Reply | Threaded
Open this post in threaded view
|

Baseline topology - Data inconsistency

Hi team,

I went through https://apacheignite.readme.io/docs/baseline-topology , one
thing remained a question for me is what happens to the consistency of data
in the following case

1) Say 5 nodes are started as part of the cluster with BackUps = 1. M1, M2,
M3, M4 and M5 with persistence enabled.
2) The cluster is activated with all the nodes and all of the nodes form a
baseline topology. The cache is partitioned.
3) Say a cache c1 is created and we say c1.put ("1",1); say the entry with
key "1" is stored on M1 and the back up is stored on M5.
4) M5 goes down. But since it was a baseline node, re-balancing did not
happen since it is still a part of baseline topology.
5) C1.put("1",2) happens.
6) M5 comes back and loads the data from disk, still no re-balancing because
it was a baseline node.
7) If a client says c1.get("1"), whats the value it gets ?

What in short is the importance of baseline topology when it comes to this
kind of a situation ?







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
userx userx
Reply | Threaded
Open this post in threaded view
|

Re: Baseline topology - Data inconsistency

Ivan Pavlukhin Ivan Pavlukhin
Reply | Threaded
Open this post in threaded view
|

Re: Baseline topology - Data inconsistency

In described case node M5 should rebalance entry <1, 2> from node M1
as M5 has a stale data. The decision to perform rebalance or not is
based on internal mechanism called "partition update counter". In
described case partition for key "1" on M5 will have updater lesser
than corresponding one on M1. So, rebalance should take place. If
counters are equal, there will be no rebalance for the partition.

пн, 18 нояб. 2019 г. в 11:03, userx <[hidden email]>:
>
> Any pointers ?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/



--
Best regards,
Ivan Pavlukhin
userx userx
Reply | Threaded
Open this post in threaded view
|

Re: Baseline topology - Data inconsistency

userx userx
Reply | Threaded
Open this post in threaded view
|

Re: Baseline topology - Data inconsistency

In reply to this post by Ivan Pavlukhin
So one quick question Ivan,

When M5 comes up, do we have

a) two clusters -  (M1 to M4) and (M5 alone)
b) one merged cluster (M1 to M5)

What I am trying to ask is that is it possible without manual intervention
to have different clusters if M5 was the part of the original cluster (M1 to
M5) during the activation.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ivan Pavlukhin Ivan Pavlukhin
Reply | Threaded
Open this post in threaded view
|

Re: Baseline topology - Data inconsistency

Expected behavior is that there will be one merged cluster. As my
understanding goes if you have a net split (M5 cannot reach neither
node from M1 to M4 by network) than M5 can start in separate cluster
alone. I hope that such cluster will be inactive, but you definitely
should check it manually. You can try it as follows:
1. Shutdown M5.
2. Shutdown cluster (M1-M4).
3. Start M5.

вт, 19 нояб. 2019 г. в 07:24, userx <[hidden email]>:

>
> So one quick question Ivan,
>
> When M5 comes up, do we have
>
> a) two clusters -  (M1 to M4) and (M5 alone)
> b) one merged cluster (M1 to M5)
>
> What I am trying to ask is that is it possible without manual intervention
> to have different clusters if M5 was the part of the original cluster (M1 to
> M5) during the activation.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/



--
Best regards,
Ivan Pavlukhin
userx userx
Reply | Threaded
Open this post in threaded view
|

Re: Baseline topology - Data inconsistency

So in the case which you highlighted where M5 starts in a different cluster
because of net issues. Now in such a case there is a possibility that if M5
tries to go in the first cluster (M1 to M4), there is an exception related
to "Mismatch of hashid of M5".

So what happens when M1 receives cache.put("1",1) and M5 receieves
caches.put("1",2) ?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ivan Pavlukhin Ivan Pavlukhin
Reply | Threaded
Open this post in threaded view
|

Re: Baseline topology - Data inconsistency

Sorry did not get the questions from the last message. Could you
please rephrase?

вт, 19 нояб. 2019 г. в 14:43, userx <[hidden email]>:

>
> So in the case which you highlighted where M5 starts in a different cluster
> because of net issues. Now in such a case there is a possibility that if M5
> tries to go in the first cluster (M1 to M4), there is an exception related
> to "Mismatch of hashid of M5".
>
> So what happens when M1 receives cache.put("1",1) and M5 receieves
> caches.put("1",2) ?
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/



--
Best regards,
Ivan Pavlukhin
userx userx
Reply | Threaded
Open this post in threaded view
|

Re: Baseline topology - Data inconsistency

Sorry if the questions wasn't clear.

What I wanted to ask is that say, if M5 comes up in a different cluster, so
eventually there will be two clusters

Cluster 1 :- M1 to M4
Cluster 2:- M5

Then if they share a common cache (before M5 went down) and there are two
client requests

Cluster 1:- commonCache.put("1", 1)
Cluster 2:- commonCache.put("1", 2)

isn't that a conflicting situation, there  are two different values of "1"



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ivan Pavlukhin Ivan Pavlukhin
Reply | Threaded
Open this post in threaded view
|

Re: Baseline topology - Data inconsistency

As I wrote before. I hope that split M5 cluster will start as
inactive. You can think that it is an administrator responsibility to
check that there is no split clusters upon activation. If both
clusters are active, yes they will act separately and data will
diverge.

вт, 19 нояб. 2019 г. в 16:46, userx <[hidden email]>:

>
> Sorry if the questions wasn't clear.
>
> What I wanted to ask is that say, if M5 comes up in a different cluster, so
> eventually there will be two clusters
>
> Cluster 1 :- M1 to M4
> Cluster 2:- M5
>
> Then if they share a common cache (before M5 went down) and there are two
> client requests
>
> Cluster 1:- commonCache.put("1", 1)
> Cluster 2:- commonCache.put("1", 2)
>
> isn't that a conflicting situation, there  are two different values of "1"
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/



--
Best regards,
Ivan Pavlukhin