Multiple embedded Ignite instances as replicated cache

classic Classic list List threaded Threaded
4 messages Options
kramer kramer
Reply | Threaded
Open this post in threaded view
|

Multiple embedded Ignite instances as replicated cache

Hi,

I have a case where I'm trying to use Ignite (embedded i.e.: Ignition.getOrStart(igniteConfiguration) called in a Spring bean definition) in a multi-node Spring Boot application with active-active setup. My goal is to utilize Ignite as a shared, atomic, and persistent key-value store.
For this test case I have two application nodes running the same code and below is my Spring configuration class; yet I keep getting "CacheInvalidStateException: Failed to execute cache operation (all partition owners have left the grid, partition data has been lost) [cacheName=sequenceNumberCache, part=187, key=XXX.SequenceNumberKey@f3c5d51c]".

Can anyone shed some light on what I'm doing wrong? I understand that manual topology management is preferred for rebalancing concerns but considering I'm using the in fully replicated mode with native persistence, I see no harm (though am still suspicious) in calling cluster.active(true) as soon as app inits. I really would like to avoid any manual topology management, is there a better way of doing this?

Thanks in advance,
Cihan Keser


```
@Configuration
public class SequenceNumberStoreConfig {

    public static final String CACHE_NAME = "sequenceNumberCache";

    @Bean
    public Ignite igniteInstance(
            @Value("${ignite.workDir}") String workDir
    ) {
        CacheConfiguration<Integer, SequenceNumberKey> cacheConfiguration =
                new CacheConfiguration<>(CACHE_NAME);
        cacheConfiguration.setIndexedTypes(SequenceNumberKey.class, Integer.class);
        cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheConfiguration.setBackups(1);
        cacheConfiguration.setWriteSynchronizationMode(CacheWriteSynchronizationMode.PRIMARY_SYNC);
        cacheConfiguration.setCacheMode(CacheMode.REPLICATED);
        cacheConfiguration.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
        cacheConfiguration.setReadFromBackup(true);
        cacheConfiguration.setRebalanceMode(CacheRebalanceMode.ASYNC);

        DataRegionConfiguration dataRegionConfiguration = new DataRegionConfiguration();
        dataRegionConfiguration.setPersistenceEnabled(true);

        DataStorageConfiguration dataStorageConfiguration = new DataStorageConfiguration();
        dataStorageConfiguration.setDefaultDataRegionConfiguration(dataRegionConfiguration);
        dataStorageConfiguration.setWalMode(WALMode.FSYNC);

        TcpDiscoveryVmIpFinder tcpDiscoveryVmIpFinder = new TcpDiscoveryVmIpFinder();
        tcpDiscoveryVmIpFinder.setShared(true);
        tcpDiscoveryVmIpFinder.setAddresses(ImmutableList.of("node1", "node2"));

        TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
        discoverySpi.setIpFinder(tcpDiscoveryVmIpFinder);

        IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setCacheConfiguration(cacheConfiguration);
        igniteConfiguration.setDiscoverySpi(discoverySpi);
        igniteConfiguration.setDataStorageConfiguration(dataStorageConfiguration);
        igniteConfiguration.setDeploymentMode(DeploymentMode.SHARED);
        igniteConfiguration.setWorkDirectory(workDir);
        igniteConfiguration.setGridLogger(new Slf4jLogger());
        igniteConfiguration.setClientMode(false);

        Ignite ignite = Ignition.getOrStart(igniteConfiguration);
        ignite.cluster().active(true);

        return ignite;
    }

    @Bean
    public SequenceNumberProvider sequenceNumberStore(Ignite ignite) {
        IgniteCache<SequenceNumberKey, Integer> igniteCache = ignite.getOrCreateCache(CACHE_NAME);
        return new SequenceNumberProvider(igniteCache);
    }

}
```
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Multiple embedded Ignite instances as replicated cache

Hello!

You will soon be able to use topology auto-adjust. Until then, I guess you will have to fix topology on entry of every node.
--
Ilya Kasnacheev


чт, 24 окт. 2019 г. в 14:52, Cihan Keser <[hidden email]>:
Hi,

I have a case where I'm trying to use Ignite (embedded i.e.: Ignition.getOrStart(igniteConfiguration) called in a Spring bean definition) in a multi-node Spring Boot application with active-active setup. My goal is to utilize Ignite as a shared, atomic, and persistent key-value store.
For this test case I have two application nodes running the same code and below is my Spring configuration class; yet I keep getting "CacheInvalidStateException: Failed to execute cache operation (all partition owners have left the grid, partition data has been lost) [cacheName=sequenceNumberCache, part=187, key=XXX.SequenceNumberKey@f3c5d51c]".

Can anyone shed some light on what I'm doing wrong? I understand that manual topology management is preferred for rebalancing concerns but considering I'm using the in fully replicated mode with native persistence, I see no harm (though am still suspicious) in calling cluster.active(true) as soon as app inits. I really would like to avoid any manual topology management, is there a better way of doing this?

Thanks in advance,
Cihan Keser


```
@Configuration
public class SequenceNumberStoreConfig {

    public static final String CACHE_NAME = "sequenceNumberCache";

    @Bean
    public Ignite igniteInstance(
            @Value("${ignite.workDir}") String workDir
    ) {
        CacheConfiguration<Integer, SequenceNumberKey> cacheConfiguration =
                new CacheConfiguration<>(CACHE_NAME);
        cacheConfiguration.setIndexedTypes(SequenceNumberKey.class, Integer.class);
        cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheConfiguration.setBackups(1);
        cacheConfiguration.setWriteSynchronizationMode(CacheWriteSynchronizationMode.PRIMARY_SYNC);
        cacheConfiguration.setCacheMode(CacheMode.REPLICATED);
        cacheConfiguration.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
        cacheConfiguration.setReadFromBackup(true);
        cacheConfiguration.setRebalanceMode(CacheRebalanceMode.ASYNC);

        DataRegionConfiguration dataRegionConfiguration = new DataRegionConfiguration();
        dataRegionConfiguration.setPersistenceEnabled(true);

        DataStorageConfiguration dataStorageConfiguration = new DataStorageConfiguration();
        dataStorageConfiguration.setDefaultDataRegionConfiguration(dataRegionConfiguration);
        dataStorageConfiguration.setWalMode(WALMode.FSYNC);

        TcpDiscoveryVmIpFinder tcpDiscoveryVmIpFinder = new TcpDiscoveryVmIpFinder();
        tcpDiscoveryVmIpFinder.setShared(true);
        tcpDiscoveryVmIpFinder.setAddresses(ImmutableList.of("node1", "node2"));

        TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
        discoverySpi.setIpFinder(tcpDiscoveryVmIpFinder);

        IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setCacheConfiguration(cacheConfiguration);
        igniteConfiguration.setDiscoverySpi(discoverySpi);
        igniteConfiguration.setDataStorageConfiguration(dataStorageConfiguration);
        igniteConfiguration.setDeploymentMode(DeploymentMode.SHARED);
        igniteConfiguration.setWorkDirectory(workDir);
        igniteConfiguration.setGridLogger(new Slf4jLogger());
        igniteConfiguration.setClientMode(false);

        Ignite ignite = Ignition.getOrStart(igniteConfiguration);
        ignite.cluster().active(true);

        return ignite;
    }

    @Bean
    public SequenceNumberProvider sequenceNumberStore(Ignite ignite) {
        IgniteCache<SequenceNumberKey, Integer> igniteCache = ignite.getOrCreateCache(CACHE_NAME);
        return new SequenceNumberProvider(igniteCache);
    }

}
```
kramer kramer
Reply | Threaded
Open this post in threaded view
|

Re: Multiple embedded Ignite instances as replicated cache

Thanks for the info, Ilya!

Do you think it would have any actual risk to use PartitionLossPolicy.IGNORE with my configuration? Because as I understand the data is fully replicated on each node, so it cannot really lose a particular partition; or do I misunderstand?

Is there a ticket/link where I can track the status topology auto-adjust?

> Until then, I guess you will have to fix topology on entry of every node.
And by that you mean activating the cluster (or resetting the lost-nodes) after all nodes join, right? Or is there any other way?

Cheers,
Cihan




On Thu, Oct 24, 2019, at 18:26, Ilya Kasnacheev wrote:
Hello!

You will soon be able to use topology auto-adjust. Until then, I guess you will have to fix topology on entry of every node.
--
Ilya Kasnacheev


чт, 24 окт. 2019 г. в 14:52, Cihan Keser <[hidden email]>:
Hi,

I have a case where I'm trying to use Ignite (embedded i.e.: Ignition.getOrStart(igniteConfiguration) called in a Spring bean definition) in a multi-node Spring Boot application with active-active setup. My goal is to utilize Ignite as a shared, atomic, and persistent key-value store.
For this test case I have two application nodes running the same code and below is my Spring configuration class; yet I keep getting "CacheInvalidStateException: Failed to execute cache operation (all partition owners have left the grid, partition data has been lost) [cacheName=sequenceNumberCache, part=187, key=XXX.SequenceNumberKey@f3c5d51c]".

Can anyone shed some light on what I'm doing wrong? I understand that manual topology management is preferred for rebalancing concerns but considering I'm using the in fully replicated mode with native persistence, I see no harm (though am still suspicious) in calling cluster.active(true) as soon as app inits. I really would like to avoid any manual topology management, is there a better way of doing this?

Thanks in advance,
Cihan Keser


```
@Configuration
public class SequenceNumberStoreConfig {

    public static final String CACHE_NAME = "sequenceNumberCache";

    @Bean
    public Ignite igniteInstance(
            @Value("${ignite.workDir}") String workDir
    ) {
        CacheConfiguration<Integer, SequenceNumberKey> cacheConfiguration =
                new CacheConfiguration<>(CACHE_NAME);
        cacheConfiguration.setIndexedTypes(SequenceNumberKey.class, Integer.class);
        cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheConfiguration.setBackups(1);
        cacheConfiguration.setWriteSynchronizationMode(CacheWriteSynchronizationMode.PRIMARY_SYNC);
        cacheConfiguration.setCacheMode(CacheMode.REPLICATED);
        cacheConfiguration.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
        cacheConfiguration.setReadFromBackup(true);
        cacheConfiguration.setRebalanceMode(CacheRebalanceMode.ASYNC);

        DataRegionConfiguration dataRegionConfiguration = new DataRegionConfiguration();
        dataRegionConfiguration.setPersistenceEnabled(true);

        DataStorageConfiguration dataStorageConfiguration = new DataStorageConfiguration();
        dataStorageConfiguration.setDefaultDataRegionConfiguration(dataRegionConfiguration);
        dataStorageConfiguration.setWalMode(WALMode.FSYNC);

        TcpDiscoveryVmIpFinder tcpDiscoveryVmIpFinder = new TcpDiscoveryVmIpFinder();
        tcpDiscoveryVmIpFinder.setShared(true);
        tcpDiscoveryVmIpFinder.setAddresses(ImmutableList.of("node1", "node2"));

        TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
        discoverySpi.setIpFinder(tcpDiscoveryVmIpFinder);

        IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setCacheConfiguration(cacheConfiguration);
        igniteConfiguration.setDiscoverySpi(discoverySpi);
        igniteConfiguration.setDataStorageConfiguration(dataStorageConfiguration);
        igniteConfiguration.setDeploymentMode(DeploymentMode.SHARED);
        igniteConfiguration.setWorkDirectory(workDir);
        igniteConfiguration.setGridLogger(new Slf4jLogger());
        igniteConfiguration.setClientMode(false);

        Ignite ignite = Ignition.getOrStart(igniteConfiguration);
        ignite.cluster().active(true);

        return ignite;
    }

    @Bean
    public SequenceNumberProvider sequenceNumberStore(Ignite ignite) {
        IgniteCache<SequenceNumberKey, Integer> igniteCache = ignite.getOrCreateCache(CACHE_NAME);
        return new SequenceNumberProvider(igniteCache);
    }

}
```

dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Multiple embedded Ignite instances as replicated cache

In reply to this post by kramer
Hi Cihan, 

Is there any reason why you cannot separate applications from Ignite as a store/database? The embedded mode makes sense for low-latency use cases while the standard client-server deployment option is better in terms of manageability, scalability:

-
Denis


On Thu, Oct 24, 2019 at 4:53 AM Cihan Keser <[hidden email]> wrote:
Hi,

I have a case where I'm trying to use Ignite (embedded i.e.: Ignition.getOrStart(igniteConfiguration) called in a Spring bean definition) in a multi-node Spring Boot application with active-active setup. My goal is to utilize Ignite as a shared, atomic, and persistent key-value store.
For this test case I have two application nodes running the same code and below is my Spring configuration class; yet I keep getting "CacheInvalidStateException: Failed to execute cache operation (all partition owners have left the grid, partition data has been lost) [cacheName=sequenceNumberCache, part=187, key=XXX.SequenceNumberKey@f3c5d51c]".

Can anyone shed some light on what I'm doing wrong? I understand that manual topology management is preferred for rebalancing concerns but considering I'm using the in fully replicated mode with native persistence, I see no harm (though am still suspicious) in calling cluster.active(true) as soon as app inits. I really would like to avoid any manual topology management, is there a better way of doing this?

Thanks in advance,
Cihan Keser


```
@Configuration
public class SequenceNumberStoreConfig {

    public static final String CACHE_NAME = "sequenceNumberCache";

    @Bean
    public Ignite igniteInstance(
            @Value("${ignite.workDir}") String workDir
    ) {
        CacheConfiguration<Integer, SequenceNumberKey> cacheConfiguration =
                new CacheConfiguration<>(CACHE_NAME);
        cacheConfiguration.setIndexedTypes(SequenceNumberKey.class, Integer.class);
        cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheConfiguration.setBackups(1);
        cacheConfiguration.setWriteSynchronizationMode(CacheWriteSynchronizationMode.PRIMARY_SYNC);
        cacheConfiguration.setCacheMode(CacheMode.REPLICATED);
        cacheConfiguration.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
        cacheConfiguration.setReadFromBackup(true);
        cacheConfiguration.setRebalanceMode(CacheRebalanceMode.ASYNC);

        DataRegionConfiguration dataRegionConfiguration = new DataRegionConfiguration();
        dataRegionConfiguration.setPersistenceEnabled(true);

        DataStorageConfiguration dataStorageConfiguration = new DataStorageConfiguration();
        dataStorageConfiguration.setDefaultDataRegionConfiguration(dataRegionConfiguration);
        dataStorageConfiguration.setWalMode(WALMode.FSYNC);

        TcpDiscoveryVmIpFinder tcpDiscoveryVmIpFinder = new TcpDiscoveryVmIpFinder();
        tcpDiscoveryVmIpFinder.setShared(true);
        tcpDiscoveryVmIpFinder.setAddresses(ImmutableList.of("node1", "node2"));

        TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
        discoverySpi.setIpFinder(tcpDiscoveryVmIpFinder);

        IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setCacheConfiguration(cacheConfiguration);
        igniteConfiguration.setDiscoverySpi(discoverySpi);
        igniteConfiguration.setDataStorageConfiguration(dataStorageConfiguration);
        igniteConfiguration.setDeploymentMode(DeploymentMode.SHARED);
        igniteConfiguration.setWorkDirectory(workDir);
        igniteConfiguration.setGridLogger(new Slf4jLogger());
        igniteConfiguration.setClientMode(false);

        Ignite ignite = Ignition.getOrStart(igniteConfiguration);
        ignite.cluster().active(true);

        return ignite;
    }

    @Bean
    public SequenceNumberProvider sequenceNumberStore(Ignite ignite) {
        IgniteCache<SequenceNumberKey, Integer> igniteCache = ignite.getOrCreateCache(CACHE_NAME);
        return new SequenceNumberProvider(igniteCache);
    }

}
```