Is ID generator split brain compliant?

classic Classic list List threaded Threaded
19 messages Options
abatra abatra
Reply | Threaded
Open this post in threaded view
|

Is ID generator split brain compliant?

Hi,

I have a requirement to create a distributed cluster-unique ID generator
microservice. I have done a PoC on it using Apache Ignite ID Generator.

I created a 2 node cluster with two instances of microservices running on
each node. Nodes are in the same datacenter (in fact in the same network and
will always be deployed in the same network) and I use TCP/IP discovery to
discover cluster nodes.

So far, it looks pretty good except that it does not provide persistence out
of the box. But I can work around it by backing latest generated ID in a
persistent cache and initializing ID generator with the latest value on a
cluster restart.

However, one thing I could not find an answer for is if the out of the box
ID generator is split brain compliant. I cannot afford to have a duplicate
ID and want to understand if duplicate ID(s) could occur in a split-brain
scenario. If yes, what is the recommended approach to handling that
scenario?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Jörn Franke Jörn Franke
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

I think you need to also look at the processes that are using the id in case of a split brain scenario.
A unique identifier is always some centralistic approach either it is done by one central service or a central rule that is enforced in a distributed fashion.

For instance, in your case you can have x nodes and each of them generates a unique id starting with nodename_localuniqueid of the node. In this case you can be also sure in case of a splitbrain scenario that the ids are unique if and only if the node names are unique.

> On 19. Sep 2018, at 21:36, abatra <[hidden email]> wrote:
>
> Hi,
>
> I have a requirement to create a distributed cluster-unique ID generator
> microservice. I have done a PoC on it using Apache Ignite ID Generator.
>
> I created a 2 node cluster with two instances of microservices running on
> each node. Nodes are in the same datacenter (in fact in the same network and
> will always be deployed in the same network) and I use TCP/IP discovery to
> discover cluster nodes.
>
> So far, it looks pretty good except that it does not provide persistence out
> of the box. But I can work around it by backing latest generated ID in a
> persistent cache and initializing ID generator with the latest value on a
> cluster restart.
>
> However, one thing I could not find an answer for is if the out of the box
> ID generator is split brain compliant. I cannot afford to have a duplicate
> ID and want to understand if duplicate ID(s) could occur in a split-brain
> scenario. If yes, what is the recommended approach to handling that
> scenario?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Magda-2 Denis Magda-2
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

In reply to this post by abatra
So far, it looks pretty good except that it does not provide persistence out
of the box. But I can work around it by backing latest generated ID in a
persistent cache and initializing ID generator with the latest value on a
cluster restart. 

Sounds like a good solution. Anton, I do remember a discussion on the dev list in regards persistence support for data structures. Are we releasing anything related soon? Can't recall all the details.

However, one thing I could not find an answer for is if the out of the box
ID generator is split brain compliant. I cannot afford to have a duplicate
ID and want to understand if duplicate ID(s) could occur in a split-brain
scenario. If yes, what is the recommended approach to handling that
scenario?

It should be split-brain tolerant if ZooKeeper Discovery is used:

--
Denis 

On Wed, Sep 19, 2018 at 3:37 PM abatra <[hidden email]> wrote:
Hi,

I have a requirement to create a distributed cluster-unique ID generator
microservice. I have done a PoC on it using Apache Ignite ID Generator.

I created a 2 node cluster with two instances of microservices running on
each node. Nodes are in the same datacenter (in fact in the same network and
will always be deployed in the same network) and I use TCP/IP discovery to
discover cluster nodes.

So far, it looks pretty good except that it does not provide persistence out
of the box. But I can work around it by backing latest generated ID in a
persistent cache and initializing ID generator with the latest value on a
cluster restart.

However, one thing I could not find an answer for is if the out of the box
ID generator is split brain compliant. I cannot afford to have a duplicate
ID and want to understand if duplicate ID(s) could occur in a split-brain
scenario. If yes, what is the recommended approach to handling that
scenario?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Anton Vinogradov-2 Anton Vinogradov-2
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

Denis, 

As far as I understand, question is about IgniteAtomicSequence?
We fixed IgniteSet to be persisted and recovered properly.

Pavel Pereslegin, 

Could you please check whether we have the same issue with IgniteAtomicSequence?

сб, 22 сент. 2018 г. в 4:17, Denis Magda <[hidden email]>:
So far, it looks pretty good except that it does not provide persistence out
of the box. But I can work around it by backing latest generated ID in a
persistent cache and initializing ID generator with the latest value on a
cluster restart. 

Sounds like a good solution. Anton, I do remember a discussion on the dev list in regards persistence support for data structures. Are we releasing anything related soon? Can't recall all the details.

However, one thing I could not find an answer for is if the out of the box
ID generator is split brain compliant. I cannot afford to have a duplicate
ID and want to understand if duplicate ID(s) could occur in a split-brain
scenario. If yes, what is the recommended approach to handling that
scenario?

It should be split-brain tolerant if ZooKeeper Discovery is used:

--
Denis 

On Wed, Sep 19, 2018 at 3:37 PM abatra <[hidden email]> wrote:
Hi,

I have a requirement to create a distributed cluster-unique ID generator
microservice. I have done a PoC on it using Apache Ignite ID Generator.

I created a 2 node cluster with two instances of microservices running on
each node. Nodes are in the same datacenter (in fact in the same network and
will always be deployed in the same network) and I use TCP/IP discovery to
discover cluster nodes.

So far, it looks pretty good except that it does not provide persistence out
of the box. But I can work around it by backing latest generated ID in a
persistent cache and initializing ID generator with the latest value on a
cluster restart.

However, one thing I could not find an answer for is if the out of the box
ID generator is split brain compliant. I cannot afford to have a duplicate
ID and want to understand if duplicate ID(s) could occur in a split-brain
scenario. If yes, what is the recommended approach to handling that
scenario?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Pavel Pereslegin Pavel Pereslegin
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

Anton,
IgniteAtomicSequence should work fine with persistence.

Ankit,
what issue did you encounter when using IgniteAtomicSequence with enabled persistence? What version of Apache Ignite you are using?


вт, 25 сент. 2018 г. в 15:11, Anton Vinogradov <[hidden email]>:
Denis, 

As far as I understand, question is about IgniteAtomicSequence?
We fixed IgniteSet to be persisted and recovered properly.

Pavel Pereslegin, 

Could you please check whether we have the same issue with IgniteAtomicSequence?

сб, 22 сент. 2018 г. в 4:17, Denis Magda <[hidden email]>:
So far, it looks pretty good except that it does not provide persistence out
of the box. But I can work around it by backing latest generated ID in a
persistent cache and initializing ID generator with the latest value on a
cluster restart. 

Sounds like a good solution. Anton, I do remember a discussion on the dev list in regards persistence support for data structures. Are we releasing anything related soon? Can't recall all the details.

However, one thing I could not find an answer for is if the out of the box
ID generator is split brain compliant. I cannot afford to have a duplicate
ID and want to understand if duplicate ID(s) could occur in a split-brain
scenario. If yes, what is the recommended approach to handling that
scenario?

It should be split-brain tolerant if ZooKeeper Discovery is used:

--
Denis 

On Wed, Sep 19, 2018 at 3:37 PM abatra <[hidden email]> wrote:
Hi,

I have a requirement to create a distributed cluster-unique ID generator
microservice. I have done a PoC on it using Apache Ignite ID Generator.

I created a 2 node cluster with two instances of microservices running on
each node. Nodes are in the same datacenter (in fact in the same network and
will always be deployed in the same network) and I use TCP/IP discovery to
discover cluster nodes.

So far, it looks pretty good except that it does not provide persistence out
of the box. But I can work around it by backing latest generated ID in a
persistent cache and initializing ID generator with the latest value on a
cluster restart.

However, one thing I could not find an answer for is if the out of the box
ID generator is split brain compliant. I cannot afford to have a duplicate
ID and want to understand if duplicate ID(s) could occur in a split-brain
scenario. If yes, what is the recommended approach to handling that
scenario?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
abatra abatra
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

Hi Anton,

With persistence enabled, if I reboot the nodes in the cluster, ID generator
starts generating ID from 1 after the nodes are back up. My expectation is
that that the latest generated ID(s) will be persisted across cluster reboot
and the ID generator will start from the latest persisted ID.

I am using the latest version of Ignite i.e. 2.6.0.

-Ankit



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Pavel Pereslegin Pavel Pereslegin
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

Ankit,
I checked sequence and atomic on Ignite 2.6 with the following code:


try (Ignite ignite = Ignition.start("example-persistent-store.xml")) {
    ignite.active(true);

    IgniteAtomicSequence seq = ignite.atomicSequence("sequence", 0, true);
    IgniteAtomicLong atomic = ignite.atomicLong("long", 0, true);

    for (int i = 0; i < 100; i++) {
        seq.incrementAndGet();
        atomic.incrementAndGet();
    }
}

try (Ignite ignite = Ignition.start("example-persistent-store.xml")) {
    ignite.active(true);

    IgniteAtomicSequence seq = ignite.atomicSequence("sequence", 0, true);
    IgniteAtomicLong atomic = ignite.atomicLong("long", 0, true);

    assert seq.get() == 1000 : seq.get();
    assert atomic.get() == 100 : atomic.get();
}


Code works fine with enabled assertions (-ea) and cleaned persistence
directory. We have sequence value = 1000, atomic value = 100 after
node restart.
Could you tell us in more detail how you use "Ignite ID generator"?

чт, 27 сент. 2018 г. в 16:27, abatra <[hidden email]>:

>
> Hi Anton,
>
> With persistence enabled, if I reboot the nodes in the cluster, ID generator
> starts generating ID from 1 after the nodes are back up. My expectation is
> that that the latest generated ID(s) will be persisted across cluster reboot
> and the ID generator will start from the latest persisted ID.
>
> I am using the latest version of Ignite i.e. 2.6.0.
>
> -Ankit
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
abatra abatra
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

Configuration
    private static final String STORAGE_PATH = "/existingDirectory/ignite/data/storage";
    private static final String WAL_PATH = "/existingDirectory/ignite/data/wal";
    private static final String WAL_ARCHIVE_PATH = "/existingDirectory/ignite/data/wal/archive";

    @Override
    public void configure() {
        Ignite ignite = Ignition.start(createConfiguration());
        /*
         * This call is required if ignite persistence is used.
         */
        ignite.cluster().active(true);
        mIgniteLogger.info("Ignite cluster is configured.");
    }

    @Override
    public void stop() {
        Ignition.stopAll(true);
    }

    private IgniteConfiguration createConfiguration() {
        IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setDiscoverySpi(createDiscoverySpi());
        igniteConfiguration.setGridLogger(mIgniteLogger);
        igniteConfiguration.setDataStorageConfiguration(createDataStorageConfiguration());
        return igniteConfiguration;
    }

    private DiscoverySpi createDiscoverySpi() {
        checkMinimumClusterIpAddressCount();
        TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
        TcpDiscoveryVmIpFinder tcpDiscoveryVmIpFinder = new TcpDiscoveryVmIpFinder();
        tcpDiscoveryVmIpFinder.setAddresses(getConfig().getNodeIpAddresses());
        tcpDiscoverySpi.setIpFinder(tcpDiscoveryVmIpFinder);
        return tcpDiscoverySpi;
    }

    private DataStorageConfiguration createDataStorageConfiguration() {
        DataStorageConfiguration dataStorageConfiguration = new DataStorageConfiguration();
        dataStorageConfiguration.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
        dataStorageConfiguration.setStoragePath(STORAGE_PATH);
        dataStorageConfiguration.setWalPath(WAL_PATH);
        dataStorageConfiguration.setWalArchivePath(WAL_ARCHIVE_PATH);
        return dataStorageConfiguration;
    }
Usage
    @Override
    public long getCurrentValue(String key) {
        return getAtomicSequence(key).get();
    }

    @Override
    public long incrementAndGet(String key) {
        return incrementAndGet(getAtomicSequence(key));
    }

    @Override
    public List incrementAndGet(String key, int count) {
        List result = new ArrayList<>();
        IgniteAtomicSequence atomicSequence = getAtomicSequence(key);
        for (int i = 0; i < count; i++) {
            result.add(incrementAndGet(atomicSequence));
        }
        return result;
    }

    private long incrementAndGet(IgniteAtomicSequence igniteAtomicSequence) {
        return igniteAtomicSequence.incrementAndGet();
    }

    private IgniteAtomicSequence getAtomicSequence(String name) {
        return Ignition.ignite().atomicSequence(name, 1500000, true);
    }
Please let me know if more details are required.

Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Pavel Pereslegin Pavel Pereslegin
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

Ankit,
I  tried test your example with the following code:


configure();

System.out.println("Initial value: " + getCurrentValue("sequence"));

for (int i = 0; i < 10_000; i++)
    incrementAndGet("sequence");

stop();

configure();

System.out.println("Value after restart: " + getCurrentValue("sequence"));

stop();


And the output is:

Initial value: 1500000
Value after restart: 1511000


This is the expected result with cleaned persistence directory
(because on last increment was reserved (+1000 by default) values).
Maybe the problem elsewhere?

чт, 27 сент. 2018 г. в 20:14, abatra <[hidden email]>:

>
> Configuration
>
>     private static final String STORAGE_PATH = "/existingDirectory/ignite/data/storage";
>     private static final String WAL_PATH = "/existingDirectory/ignite/data/wal";
>     private static final String WAL_ARCHIVE_PATH = "/existingDirectory/ignite/data/wal/archive";
>
>     @Override
>     public void configure() {
>         Ignite ignite = Ignition.start(createConfiguration());
>         /*
>          * This call is required if ignite persistence is used.
>          */
>         ignite.cluster().active(true);
>         mIgniteLogger.info("Ignite cluster is configured.");
>     }
>
>     @Override
>     public void stop() {
>         Ignition.stopAll(true);
>     }
>
>     private IgniteConfiguration createConfiguration() {
>         IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
>         igniteConfiguration.setDiscoverySpi(createDiscoverySpi());
>         igniteConfiguration.setGridLogger(mIgniteLogger);
>         igniteConfiguration.setDataStorageConfiguration(createDataStorageConfiguration());
>         return igniteConfiguration;
>     }
>
>     private DiscoverySpi createDiscoverySpi() {
>         checkMinimumClusterIpAddressCount();
>         TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
>         TcpDiscoveryVmIpFinder tcpDiscoveryVmIpFinder = new TcpDiscoveryVmIpFinder();
>         tcpDiscoveryVmIpFinder.setAddresses(getConfig().getNodeIpAddresses());
>         tcpDiscoverySpi.setIpFinder(tcpDiscoveryVmIpFinder);
>         return tcpDiscoverySpi;
>     }
>
>     private DataStorageConfiguration createDataStorageConfiguration() {
>         DataStorageConfiguration dataStorageConfiguration = new DataStorageConfiguration();
>         dataStorageConfiguration.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
>         dataStorageConfiguration.setStoragePath(STORAGE_PATH);
>         dataStorageConfiguration.setWalPath(WAL_PATH);
>         dataStorageConfiguration.setWalArchivePath(WAL_ARCHIVE_PATH);
>         return dataStorageConfiguration;
>     }
>
> Usage
>
>     @Override
>     public long getCurrentValue(String key) {
>         return getAtomicSequence(key).get();
>     }
>
>     @Override
>     public long incrementAndGet(String key) {
>         return incrementAndGet(getAtomicSequence(key));
>     }
>
>     @Override
>     public List incrementAndGet(String key, int count) {
>         List result = new ArrayList<>();
>         IgniteAtomicSequence atomicSequence = getAtomicSequence(key);
>         for (int i = 0; i < count; i++) {
>             result.add(incrementAndGet(atomicSequence));
>         }
>         return result;
>     }
>
>     private long incrementAndGet(IgniteAtomicSequence igniteAtomicSequence) {
>         return igniteAtomicSequence.incrementAndGet();
>     }
>
>     private IgniteAtomicSequence getAtomicSequence(String name) {
>         return Ignition.ignite().atomicSequence(name, 1500000, true);
>     }
>
> Please let me know if more details are required.
> ________________________________
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
abatra abatra
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

There are only two differences in the way you tried the example:

1. I run two nodes in the cluster.
2. I do not stop and configure the server while the JVM is running. I
restart the entire node and hence the server gets restarted.

Otherwise, I am not sure how is that working out for you.

Also, I see that you are stressing on "cleaned persistence directory".
Please let me know if I am missing something there.

My server nodes run as two different docker containers on two different open
stack server nodes.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Pavel Pereslegin Pavel Pereslegin
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

Ankit,

 > 1. I run two nodes in the cluster.
I checked 2 nodes cluster restart (simultaneously, 2 processes on same
machine) with the same result - after restart the "current" value !=
"initial" value.

 > 2. I restart the entire node and hence the server gets restarted.
I gave this example only for clarity - in this case it does not matter
how you stop the server. The main thing is that the persistence
directory should not be cleared, but since you mentioned that the data
in the cache is available after restart - I believe it is not cleaned

 > Otherwise, I am not sure how is that working out for you.
Please, try run code by yourself and check results.

 > I see that you are stressing on "cleaned persistence directory".
It's not necessary - I cleaned persistence directory just to see the
expected result.

Persistence should work fine for atomics "out of the box", otherwise
we should create an issue in ASF jira. But for the time being I can
not understand in what case this issue occurs.
пт, 28 сент. 2018 г. в 10:52, abatra <[hidden email]>:

>
> There are only two differences in the way you tried the example:
>
> 1. I run two nodes in the cluster.
> 2. I do not stop and configure the server while the JVM is running. I
> restart the entire node and hence the server gets restarted.
>
> Otherwise, I am not sure how is that working out for you.
>
> Also, I see that you are stressing on "cleaned persistence directory".
> Please let me know if I am missing something there.
>
> My server nodes run as two different docker containers on two different open
> stack server nodes.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
abatra abatra
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

I tried debugging it a bit.

Now, I am running only 1 server node. I am running other node as client.
Even with single server node, the atomic sequence is reset to initial value
after the server node restarts.

I checked the IGNITE_HOME folder, on every restart, the contents are
recreated with a different node ID.

node00-46580d24-2fdb-484a-ba63-3a05248c0b8d

*ID after restart*
node00-beae3aab-6165-485a-96e3-1a2fe5b7f20b

I believe this might be the reason for the sequence to not persist across
reboots. However, I am unsure why am I running into this because I am pretty
sure that I am running only 1 server from 1 node.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
abatra abatra
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

To give more information on how I restart the server:

I run server and client inside their own respective docker container and I
issue 'docker stop' for server node container ID to stop and then restart
it.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Pavel Pereslegin Pavel Pereslegin
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

Hello Ankit,

I'm not an expert in docker, but for me it looks like you have a problem with overall cache persistence (not only with AtomicSequence), could you try to put some values into cache and check their availability after the restart?


пт, 28 сент. 2018 г. в 23:45, abatra <[hidden email]>:
To give more information on how I restart the server:

I run server and client inside their own respective docker container and I
issue 'docker stop' for server node container ID to stop and then restart
it.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
abatra abatra
Reply | Threaded
Open this post in threaded view
|

Re: Is ID generator split brain compliant?

I believe it won't work either. I will try it soon and share the result.

Could you please share some pointers that I should look at for debugging
persistence? I have org.apache.ignite=DEBUG in logs and IGNITE_QUITE set to
false. There a lot of logs, most of which I can not make sense of w.r.t.
persistence.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

RE: Is ID generator split brain compliant?

Hi,

 

If the issue is that the node is starting with a new persistent storage each time

there could some issue with file system.

 

The algorithm to choose a persistent folder is

  1. If IgniteConfiguration.consistentId is specified, use the folder of that name
  2. Otherwise, check if there are any existing folders not locked by other nodes; if one is found, use that
  3. Otherwise, generate a new consistentId and create a new folder for it

I assume that when Ignite checks your folders on step 2 they appear to be locked even though they are not.

Perhaps the file system doesn’t unlock them fast enough, or something similar happens.

Look for messages like “Unable to acquire lock to file” at the start of the nodes to see if that’s the case.

 

Try to specify consistentId explicitly – perhaps it’ll solve the issue.

 

Thanks,

Stan

 

From: [hidden email]
Sent: 1 октября 2018 г. 11:47
To: [hidden email]
Subject: Re: Is ID generator split brain compliant?

 

I believe it won't work either. I will try it soon and share the result.

 

Could you please share some pointers that I should look at for debugging

persistence? I have org.apache.ignite=DEBUG in logs and IGNITE_QUITE set to

false. There a lot of logs, most of which I can not make sense of w.r.t.

persistence.

 

 

 

--

Sent from: http://apache-ignite-users.70518.x6.nabble.com/

 

abatra abatra
Reply | Threaded
Open this post in threaded view
|

RE: Is ID generator split brain compliant?

I did not see any locks related logs.
I did set consistent ID to make sure Ignite looks for the same folder on a
restart. But it did not work. On docker restart i.e. the ID generator starts
generating a sequence from the initial value.

I am attaching the ignite logs for server restart in this message. It's a
two node deployment, 1 server and 1 client where I keep client down all the
time. For that reason, you may see an expected NoRouteException to the
client.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
abatra abatra
Reply | Threaded
Open this post in threaded view
|

RE: Is ID generator split brain compliant?

abatra abatra
Reply | Threaded
Open this post in threaded view
|

RE: Is ID generator split brain compliant?

It worked. Directory set for IGNITE_HOME was getting cleaned up because it
wasn't on a mount point or docker volume. Once I set IGNITE_HOME to a
directory that persisted between container stop and start, the ID generator
worked as expected.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/