Ignite-cassandra module issue

classic Classic list List threaded Threaded
10 messages Options
nash1k nash1k
Reply | Threaded
Open this post in threaded view
|

Ignite-cassandra module issue

Hi!
I'm using the cassandra as persistence store for my caches and have one issue by handling a huge data (via IgniteDataStreamer from kafka).
Ignite Configuration:
final IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setIgniteInstanceName("test");
        igniteConfiguration.setClientMode(true);
        igniteConfiguration.setGridLogger(new Slf4jLogger());
        igniteConfiguration.setMetricsLogFrequency(0);
        igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi());
final BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
        binaryConfiguration.setCompactFooter(false);
        igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
        igniteConfiguration.setPeerClassLoadingEnabled(true);
final MemoryPolicyConfiguration memoryPolicyConfiguration = new MemoryPolicyConfiguration();
        memoryPolicyConfiguration.setName("3Gb_Region_Eviction");
        memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L);
        memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L);
        memoryPolicyConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
final MemoryConfiguration memoryConfiguration = new MemoryConfiguration();
        memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration);
        igniteConfiguration.setMemoryConfiguration(memoryConfiguration);

Cache configuration:
final CacheConfiguration<String, BinaryObject> cacheConfiguration = new CacheConfiguration<>();
        cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheConfiguration.setStoreKeepBinary(true);
        cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
        cacheConfiguration.setBackups(0);
        cacheConfiguration.setStatisticsEnabled(false);
        cacheConfiguration.setName("TestCache");

        cacheConfiguration.setReadThrough(true);
        cacheConfiguration.setWriteThrough(true);

        cacheConfiguration.setWriteBehindEnabled(true);
        cacheConfiguration.setWriteBehindFlushFrequency(10000);
        cacheConfiguration.setWriteBehindFlushSize(0);
        cacheConfiguration.setWriteBehindFlushThreadCount(2);
        cacheConfiguration.setWriteBehindBatchSize(10000);


        final CassandraCacheStoreFactory<String, BinaryObject> cacheStoreFactory = new CassandraCacheStoreFactory<>();
        final DataSource dataSource = new DataSource();
        dataSource.setContactPoints(contactPoints);
        dataSource.setReadConsistency("ONE");
        dataSource.setWriteConsistency("ONE");
        dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new RoundRobinPolicy()));
        cacheStoreFactory.setDataSource(dataSource);

        final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\" table=\"%s\">" +
                "<keyPersistence class=\"%s\" strategy=\"%s\"/>" +
                "<valuePersistence class=\"%s\" strategy=\"%s\"/>" +
                "</persistence>";
        final KeyValuePersistenceSettings settings = new KeyValuePersistenceSettings(
                String.format(CASSANDRA_PERSISTENCE, "test", "test_table",
                        "java.lang.String", "PRIMITIVE",
                        "org.apache.ignite.binary.BinaryObject", "BLOB"));
        cacheStoreFactory.setPersistenceSettings(settings);
        cacheConfiguration.setCacheStoreFactory(cacheStoreFactory);

When application works some time (hour or more, may be less - from time to time) I see this exceptions on the ignite nodes:
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072]
    ^-- H/N/C [hosts=3, nodes=3, CPUs=96]
    ^-- CPU [cur=0%, avg=1.86%, GC=0%]
    ^-- PageMemory [pages=118064]
    ^-- Heap [used=4800MB, free=53.12%, comm=10240MB]
    ^-- Non heap [used=78MB, free=-1%, comm=80MB]
    ^-- Public thread pool [active=0, idle=32, qSize=0]
    ^-- System thread pool [active=0, idle=32, qSize=0]
    ^-- Outbound messages queue [size=0]
[15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList [name=null, buckets=256, dataPages=102080, reusePages=0]
[15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra session refreshed
[15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed
[15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed
[15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed
[15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra session refreshed
[15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

After this I have to restart the cluster and application or have "transactions deadlock" message.

And I checked: network is fine, no OOME in the ignite and cassandra (cassandra logs don't say anything about it). I understand that this is issue of persistence into cassandra but don't know how can I fix it. And after some attempts ignite node is stopped. I tried to change write behind parameters and disable it at all - but have the same issue. May be I have to see in the other place? May be this is important - i use IgniteDataStreamer for reading values from kafka (earlier I used the simple cache operations and didn't have this problem at all).

final IgniteDataStreamer<String, BinaryObject> streamer = ignite.dataStreamer(callCache().getName());
streamer.autoFlushFrequency(5000);
streamer.keepBinary(true);
streamer.perNodeBufferSize(5120);


And one more question about persistence settings. If I'm working with BinaryObjects and ignite doesn't know about my class types, should I use valuePersistence class =  "org.apache.ignite.binary.BinaryObject"? I did it because peer class loading doesn't work for my own classes and I don't want to add new jars into all ignite nodes and all my apps.

Thanks for help!
dkarachentsev dkarachentsev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite-cassandra module issue

Hi,

Is it possible to provide a simple reproducer?
Answering your second question, yes, you can use BinaryObject as persistence class name.

Thanks!
-Dmitry.
Tobias Eriksson Tobias Eriksson
Reply | Threaded
Open this post in threaded view
|

Re: Ignite-cassandra module issue

In reply to this post by nash1k
Hi
 Did you ever resolve this problem, cause I have it too on and off
In fact it sometimes happens just like that, e.g. I have been running my
Ignite client and then stop it, and then it takes a while and run it again,
and all by a sudden this error shows up. An that is the first thing that
happens, and there is NOT a massive amount of load on Cassandra at that
time. But I have also seen it when I hammer Ignite/Cassandra with
updates/inserts.

This is a deal-breaker for me, I need to understand how to fix this, cause
having this in production is not an option.

-Tobias




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Tobias Eriksson Tobias Eriksson
Reply | Threaded
Open this post in threaded view
|

Re: Ignite-cassandra module issue

Hi
 Sorry did not include the context when I replied
 Has anyone been able to resolve this problem, cause I have it too on and
off
In fact it sometimes happens just like that, e.g. I have been running my
Ignite client and then stop it, and then it takes a while and run it again,
and all by a sudden this error shows up. An that is the first thing that
happens, and there is NOT a massive amount of load on Cassandra at that
time. But I have also seen it when I hammer Ignite/Cassandra with
updates/inserts.

This is a deal-breaker for me, I need to understand how to fix this, cause
having this in production is not an option.

-Tobias


Hi!
I'm using the cassandra as persistence store for my caches and have one
issue by handling a huge data (via IgniteDataStreamer from kafka).
Ignite Configuration:
final IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setIgniteInstanceName("test");
        igniteConfiguration.setClientMode(true);
        igniteConfiguration.setGridLogger(new Slf4jLogger());
        igniteConfiguration.setMetricsLogFrequency(0);
        igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi());
final BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
        binaryConfiguration.setCompactFooter(false);
        igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
        igniteConfiguration.setPeerClassLoadingEnabled(true);
final MemoryPolicyConfiguration memoryPolicyConfiguration = new
MemoryPolicyConfiguration();
        memoryPolicyConfiguration.setName("3Gb_Region_Eviction");
        memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L);
        memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L);
       
memoryPolicyConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
final MemoryConfiguration memoryConfiguration = new MemoryConfiguration();
        memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration);
        igniteConfiguration.setMemoryConfiguration(memoryConfiguration);

Cache configuration:
final CacheConfiguration<String, BinaryObject> cacheConfiguration = new
CacheConfiguration<>();
        cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheConfiguration.setStoreKeepBinary(true);
        cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
        cacheConfiguration.setBackups(0);
        cacheConfiguration.setStatisticsEnabled(false);
        cacheConfiguration.setName("TestCache");

        cacheConfiguration.setReadThrough(true);
        cacheConfiguration.setWriteThrough(true);

        cacheConfiguration.setWriteBehindEnabled(true);
        cacheConfiguration.setWriteBehindFlushFrequency(10000);
        cacheConfiguration.setWriteBehindFlushSize(0);
        cacheConfiguration.setWriteBehindFlushThreadCount(2);
        cacheConfiguration.setWriteBehindBatchSize(10000);


        final CassandraCacheStoreFactory<String, BinaryObject>
cacheStoreFactory = new CassandraCacheStoreFactory<>();
        final DataSource dataSource = new DataSource();
        dataSource.setContactPoints(contactPoints);
        dataSource.setReadConsistency("ONE");
        dataSource.setWriteConsistency("ONE");
        dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new
RoundRobinPolicy()));
        cacheStoreFactory.setDataSource(dataSource);

        final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\"
table=\"%s\">" +
                "<keyPersistence class=\"%s\" strategy=\"%s\"/>" +
                "<valuePersistence class=\"%s\" strategy=\"%s\"/>" +
                "</persistence>";
        final KeyValuePersistenceSettings settings = new
KeyValuePersistenceSettings(
                String.format(CASSANDRA_PERSISTENCE, "test", "test_table",
                        "java.lang.String", "PRIMITIVE",
                        "org.apache.ignite.binary.BinaryObject", "BLOB"));
        cacheStoreFactory.setPersistenceSettings(settings);
        cacheConfiguration.setCacheStoreFactory(cacheStoreFactory);

When application works some time (hour or more, may be less - from time to
time) I see this exceptions on the ignite nodes:
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072]
    ^-- H/N/C [hosts=3, nodes=3, CPUs=96]
    ^-- CPU [cur=0%, avg=1.86%, GC=0%]
    ^-- PageMemory [pages=118064]
    ^-- Heap [used=4800MB, free=53.12%, comm=10240MB]
    ^-- Non heap [used=78MB, free=-1%, comm=80MB]
    ^-- Public thread pool [active=0, idle=32, qSize=0]
    ^-- System thread pool [active=0, idle=32, qSize=0]
    ^-- Outbound messages queue [size=0]
[15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList
[name=null, buckets=256, dataPages=102080, reusePages=0]
[15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed
[15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed
[15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed
[15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 attempt
to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

After this I have to restart the cluster and application or have
"transactions deadlock" message.

And I checked: network is fine, no OOME in the ignite and cassandra
(cassandra logs don't say anything about it). I understand that this is
issue of persistence into cassandra but don't know how can I fix it. And
after some attempts ignite node is stopped. I tried to change write behind
parameters and disable it at all - but have the same issue. May be I have to
see in the other place? May be this is important - i use IgniteDataStreamer
for reading values from kafka (earlier I used the simple cache operations
and didn't have this problem at all).

final IgniteDataStreamer<String, BinaryObject> streamer =
ignite.dataStreamer(callCache().getName());
streamer.autoFlushFrequency(5000);
streamer.keepBinary(true);
streamer.perNodeBufferSize(5120);


And one more question about persistence settings. If I'm working with
BinaryObjects and ignite doesn't know about my class types, should I use
valuePersistence class =  "org.apache.ignite.binary.BinaryObject"? I did it
because peer class loading doesn't work for my own classes and I don't want
to add new jars into all ignite nodes and all my apps.

Thanks for help!

--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Andrew Mashenkov Andrew Mashenkov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite-cassandra module issue

Hi Tobias,

What ignite version do you use? May be this was already fixed in latest one?
I see related fix inclueded in upcoming 2.3 version.

See IGNITE-5897 [1] issue. It is unobvious, but this fix session init\end logic, so session should be closed in proper way.



On Wed, Oct 25, 2017 at 11:13 AM, Tobias Eriksson <[hidden email]> wrote:
Hi
 Sorry did not include the context when I replied
 Has anyone been able to resolve this problem, cause I have it too on and
off
In fact it sometimes happens just like that, e.g. I have been running my
Ignite client and then stop it, and then it takes a while and run it again,
and all by a sudden this error shows up. An that is the first thing that
happens, and there is NOT a massive amount of load on Cassandra at that
time. But I have also seen it when I hammer Ignite/Cassandra with
updates/inserts.

This is a deal-breaker for me, I need to understand how to fix this, cause
having this in production is not an option.

-Tobias


Hi!
I'm using the cassandra as persistence store for my caches and have one
issue by handling a huge data (via IgniteDataStreamer from kafka).
Ignite Configuration:
final IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setIgniteInstanceName("test");
        igniteConfiguration.setClientMode(true);
        igniteConfiguration.setGridLogger(new Slf4jLogger());
        igniteConfiguration.setMetricsLogFrequency(0);
        igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi());
final BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
        binaryConfiguration.setCompactFooter(false);
        igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
        igniteConfiguration.setPeerClassLoadingEnabled(true);
final MemoryPolicyConfiguration memoryPolicyConfiguration = new
MemoryPolicyConfiguration();
        memoryPolicyConfiguration.setName("3Gb_Region_Eviction");
        memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L);
        memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L);

memoryPolicyConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
final MemoryConfiguration memoryConfiguration = new MemoryConfiguration();
        memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration);
        igniteConfiguration.setMemoryConfiguration(memoryConfiguration);

Cache configuration:
final CacheConfiguration<String, BinaryObject> cacheConfiguration = new
CacheConfiguration<>();
        cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheConfiguration.setStoreKeepBinary(true);
        cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
        cacheConfiguration.setBackups(0);
        cacheConfiguration.setStatisticsEnabled(false);
        cacheConfiguration.setName("TestCache");

        cacheConfiguration.setReadThrough(true);
        cacheConfiguration.setWriteThrough(true);

        cacheConfiguration.setWriteBehindEnabled(true);
        cacheConfiguration.setWriteBehindFlushFrequency(10000);
        cacheConfiguration.setWriteBehindFlushSize(0);
        cacheConfiguration.setWriteBehindFlushThreadCount(2);
        cacheConfiguration.setWriteBehindBatchSize(10000);


        final CassandraCacheStoreFactory<String, BinaryObject>
cacheStoreFactory = new CassandraCacheStoreFactory<>();
        final DataSource dataSource = new DataSource();
        dataSource.setContactPoints(contactPoints);
        dataSource.setReadConsistency("ONE");
        dataSource.setWriteConsistency("ONE");
        dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new
RoundRobinPolicy()));
        cacheStoreFactory.setDataSource(dataSource);

        final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\"
table=\"%s\">" +
                "<keyPersistence class=\"%s\" strategy=\"%s\"/>" +
                "<valuePersistence class=\"%s\" strategy=\"%s\"/>" +
                "</persistence>";
        final KeyValuePersistenceSettings settings = new
KeyValuePersistenceSettings(
                String.format(CASSANDRA_PERSISTENCE, "test", "test_table",
                        "java.lang.String", "PRIMITIVE",
                        "org.apache.ignite.binary.BinaryObject", "BLOB"));
        cacheStoreFactory.setPersistenceSettings(settings);
        cacheConfiguration.setCacheStoreFactory(cacheStoreFactory);

When application works some time (hour or more, may be less - from time to
time) I see this exceptions on the ignite nodes:
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072]
    ^-- H/N/C [hosts=3, nodes=3, CPUs=96]
    ^-- CPU [cur=0%, avg=1.86%, GC=0%]
    ^-- PageMemory [pages=118064]
    ^-- Heap [used=4800MB, free=53.12%, comm=10240MB]
    ^-- Non heap [used=78MB, free=-1%, comm=80MB]
    ^-- Public thread pool [active=0, idle=32, qSize=0]
    ^-- System thread pool [active=0, idle=32, qSize=0]
    ^-- Outbound messages queue [size=0]
[15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList
[name=null, buckets=256, dataPages=102080, reusePages=0]
[15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed
[15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed
[15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed
[15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 attempt
to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

After this I have to restart the cluster and application or have
"transactions deadlock" message.

And I checked: network is fine, no OOME in the ignite and cassandra
(cassandra logs don't say anything about it). I understand that this is
issue of persistence into cassandra but don't know how can I fix it. And
after some attempts ignite node is stopped. I tried to change write behind
parameters and disable it at all - but have the same issue. May be I have to
see in the other place? May be this is important - i use IgniteDataStreamer
for reading values from kafka (earlier I used the simple cache operations
and didn't have this problem at all).

final IgniteDataStreamer<String, BinaryObject> streamer =
ignite.dataStreamer(callCache().getName());
streamer.autoFlushFrequency(5000);
streamer.keepBinary(true);
streamer.perNodeBufferSize(5120);


And one more question about persistence settings. If I'm working with
BinaryObjects and ignite doesn't know about my class types, should I use
valuePersistence class =  "org.apache.ignite.binary.BinaryObject"? I did it
because peer class loading doesn't work for my own classes and I don't want
to add new jars into all ignite nodes and all my apps.

Thanks for help!



--
Best regards,
Andrey V. Mashenkov
Regards,
Andrew.
Tobias Eriksson Tobias Eriksson
Reply | Threaded
Open this post in threaded view
|

Re: Ignite-cassandra module issue

Hi Andrey et al

I believe I now know what the problem is, the Cassandra session is refreshed, but before it is a prepared statement is created/used and there, and so using a new session with an old prepared statement is not working.

 

The way to reproduce is

1)       Start Ignite Server Node

2)       Start client which inserts a batch of 100 elements

3)       End client

4)       Now Ignite Server Node returns the Cassandra Session to the pool

5)       Wait 5+ minutes

6)       Now Ignite Server Node has does a clean-up of the “unused” Cassandra sessions

7)       Start client which inserts a batch of 100 elements

8)       Boom ! The exception starts to happen

 

Reason is

1)       Execute is called for a BATCH

2)       Prepared-statement is reused since there is a cache of those

3)       It is about to do session().execute( batch )

4)       BUT the call to session() results in refreshing the session, and this is where the prepared statements as the old session new them are cleaned up

5)       Now it is looping over 100 times with a NEW session but with an OLD prepared statement

 

This is a bug,

 

-Tobias

 

 

From: Andrey Mashenkov <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, 25 October 2017 at 14:12
To: "[hidden email]" <[hidden email]>
Subject: Re: Ignite-cassandra module issue

 

Hi Tobias,

 

What ignite version do you use? May be this was already fixed in latest one?

I see related fix inclueded in upcoming 2.3 version.

 

See IGNITE-5897 [1] issue. It is unobvious, but this fix session init\end logic, so session should be closed in proper way.

 

 

 

On Wed, Oct 25, 2017 at 11:13 AM, Tobias Eriksson <[hidden email]> wrote:

Hi
 Sorry did not include the context when I replied
 Has anyone been able to resolve this problem, cause I have it too on and
off
In fact it sometimes happens just like that, e.g. I have been running my
Ignite client and then stop it, and then it takes a while and run it again,
and all by a sudden this error shows up. An that is the first thing that
happens, and there is NOT a massive amount of load on Cassandra at that
time. But I have also seen it when I hammer Ignite/Cassandra with
updates/inserts.

This is a deal-breaker for me, I need to understand how to fix this, cause
having this in production is not an option.

-Tobias


Hi!
I'm using the cassandra as persistence store for my caches and have one
issue by handling a huge data (via IgniteDataStreamer from kafka).
Ignite Configuration:
final IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setIgniteInstanceName("test");
        igniteConfiguration.setClientMode(true);
        igniteConfiguration.setGridLogger(new Slf4jLogger());
        igniteConfiguration.setMetricsLogFrequency(0);
        igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi());
final BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
        binaryConfiguration.setCompactFooter(false);
        igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
        igniteConfiguration.setPeerClassLoadingEnabled(true);
final MemoryPolicyConfiguration memoryPolicyConfiguration = new
MemoryPolicyConfiguration();
        memoryPolicyConfiguration.setName("3Gb_Region_Eviction");
        memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L);
        memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L);

memoryPolicyConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
final MemoryConfiguration memoryConfiguration = new MemoryConfiguration();
        memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration);
        igniteConfiguration.setMemoryConfiguration(memoryConfiguration);

Cache configuration:
final CacheConfiguration<String, BinaryObject> cacheConfiguration = new
CacheConfiguration<>();
        cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheConfiguration.setStoreKeepBinary(true);
        cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
        cacheConfiguration.setBackups(0);
        cacheConfiguration.setStatisticsEnabled(false);
        cacheConfiguration.setName("TestCache");

        cacheConfiguration.setReadThrough(true);
        cacheConfiguration.setWriteThrough(true);

        cacheConfiguration.setWriteBehindEnabled(true);
        cacheConfiguration.setWriteBehindFlushFrequency(10000);
        cacheConfiguration.setWriteBehindFlushSize(0);
        cacheConfiguration.setWriteBehindFlushThreadCount(2);
        cacheConfiguration.setWriteBehindBatchSize(10000);


        final CassandraCacheStoreFactory<String, BinaryObject>
cacheStoreFactory = new CassandraCacheStoreFactory<>();
        final DataSource dataSource = new DataSource();
        dataSource.setContactPoints(contactPoints);
        dataSource.setReadConsistency("ONE");
        dataSource.setWriteConsistency("ONE");
        dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new
RoundRobinPolicy()));
        cacheStoreFactory.setDataSource(dataSource);

        final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\"
table=\"%s\">" +
                "<keyPersistence class=\"%s\" strategy=\"%s\"/>" +
                "<valuePersistence class=\"%s\" strategy=\"%s\"/>" +
                "</persistence>";
        final KeyValuePersistenceSettings settings = new
KeyValuePersistenceSettings(
                String.format(CASSANDRA_PERSISTENCE, "test", "test_table",
                        "java.lang.String", "PRIMITIVE",
                        "org.apache.ignite.binary.BinaryObject", "BLOB"));
        cacheStoreFactory.setPersistenceSettings(settings);
        cacheConfiguration.setCacheStoreFactory(cacheStoreFactory);

When application works some time (hour or more, may be less - from time to
time) I see this exceptions on the ignite nodes:
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072]
    ^-- H/N/C [hosts=3, nodes=3, CPUs=96]
    ^-- CPU [cur=0%, avg=1.86%, GC=0%]
    ^-- PageMemory [pages=118064]
    ^-- Heap [used=4800MB, free=53.12%, comm=10240MB]
    ^-- Non heap [used=78MB, free=-1%, comm=80MB]
    ^-- Public thread pool [active=0, idle=32, qSize=0]
    ^-- System thread pool [active=0, idle=32, qSize=0]
    ^-- Outbound messages queue [size=0]
[15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList
[name=null, buckets=256, dataPages=102080, reusePages=0]
[15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed
[15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed
[15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed
[15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 attempt
to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

After this I have to restart the cluster and application or have
"transactions deadlock" message.

And I checked: network is fine, no OOME in the ignite and cassandra
(cassandra logs don't say anything about it). I understand that this is
issue of persistence into cassandra but don't know how can I fix it. And
after some attempts ignite node is stopped. I tried to change write behind
parameters and disable it at all - but have the same issue. May be I have to
see in the other place? May be this is important - i use IgniteDataStreamer
for reading values from kafka (earlier I used the simple cache operations
and didn't have this problem at all).

final IgniteDataStreamer<String, BinaryObject> streamer =
ignite.dataStreamer(callCache().getName());
streamer.autoFlushFrequency(5000);
streamer.keepBinary(true);
streamer.perNodeBufferSize(5120);


And one more question about persistence settings. If I'm working with
BinaryObjects and ignite doesn't know about my class types, should I use
valuePersistence class =  "org.apache.ignite.binary.BinaryObject"? I did it
because peer class loading doesn't work for my own classes and I don't want
to add new jars into all ignite nodes and all my apps.

Thanks for help!



 

--

Best regards,
Andrey V. Mashenkov

Mikhail Mikhail
Reply | Threaded
Open this post in threaded view
|

Re: Ignite-cassandra module issue

Hi Tobias,

Thank you for explaining how to reproduce it, I'll try your instruction. I spend several days trying to reproduce the issue,
but I thought that the reason of this is too high load and I didn't stop client during testing.
I'll check your instruction and try to fix the issue.

Thanks,
Mike.

2017-10-25 16:23 GMT+03:00 Tobias Eriksson <[hidden email]>:

Hi Andrey et al

I believe I now know what the problem is, the Cassandra session is refreshed, but before it is a prepared statement is created/used and there, and so using a new session with an old prepared statement is not working.

 

The way to reproduce is

1)       Start Ignite Server Node

2)       Start client which inserts a batch of 100 elements

3)       End client

4)       Now Ignite Server Node returns the Cassandra Session to the pool

5)       Wait 5+ minutes

6)       Now Ignite Server Node has does a clean-up of the “unused” Cassandra sessions

7)       Start client which inserts a batch of 100 elements

8)       Boom ! The exception starts to happen

 

Reason is

1)       Execute is called for a BATCH

2)       Prepared-statement is reused since there is a cache of those

3)       It is about to do session().execute( batch )

4)       BUT the call to session() results in refreshing the session, and this is where the prepared statements as the old session new them are cleaned up

5)       Now it is looping over 100 times with a NEW session but with an OLD prepared statement

 

This is a bug,

 

-Tobias

 

 

From: Andrey Mashenkov <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, 25 October 2017 at 14:12
To: "[hidden email]" <[hidden email]>
Subject: Re: Ignite-cassandra module issue

 

Hi Tobias,

 

What ignite version do you use? May be this was already fixed in latest one?

I see related fix inclueded in upcoming 2.3 version.

 

See IGNITE-5897 [1] issue. It is unobvious, but this fix session init\end logic, so session should be closed in proper way.

 

 

 

On Wed, Oct 25, 2017 at 11:13 AM, Tobias Eriksson <[hidden email]> wrote:

Hi
 Sorry did not include the context when I replied
 Has anyone been able to resolve this problem, cause I have it too on and
off
In fact it sometimes happens just like that, e.g. I have been running my
Ignite client and then stop it, and then it takes a while and run it again,
and all by a sudden this error shows up. An that is the first thing that
happens, and there is NOT a massive amount of load on Cassandra at that
time. But I have also seen it when I hammer Ignite/Cassandra with
updates/inserts.

This is a deal-breaker for me, I need to understand how to fix this, cause
having this in production is not an option.

-Tobias


Hi!
I'm using the cassandra as persistence store for my caches and have one
issue by handling a huge data (via IgniteDataStreamer from kafka).
Ignite Configuration:
final IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setIgniteInstanceName("test");
        igniteConfiguration.setClientMode(true);
        igniteConfiguration.setGridLogger(new Slf4jLogger());
        igniteConfiguration.setMetricsLogFrequency(0);
        igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi());
final BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
        binaryConfiguration.setCompactFooter(false);
        igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
        igniteConfiguration.setPeerClassLoadingEnabled(true);
final MemoryPolicyConfiguration memoryPolicyConfiguration = new
MemoryPolicyConfiguration();
        memoryPolicyConfiguration.setName("3Gb_Region_Eviction");
        memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L);
        memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L);

memoryPolicyConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
final MemoryConfiguration memoryConfiguration = new MemoryConfiguration();
        memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration);
        igniteConfiguration.setMemoryConfiguration(memoryConfiguration);

Cache configuration:
final CacheConfiguration<String, BinaryObject> cacheConfiguration = new
CacheConfiguration<>();
        cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheConfiguration.setStoreKeepBinary(true);
        cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
        cacheConfiguration.setBackups(0);
        cacheConfiguration.setStatisticsEnabled(false);
        cacheConfiguration.setName("TestCache");

        cacheConfiguration.setReadThrough(true);
        cacheConfiguration.setWriteThrough(true);

        cacheConfiguration.setWriteBehindEnabled(true);
        cacheConfiguration.setWriteBehindFlushFrequency(10000);
        cacheConfiguration.setWriteBehindFlushSize(0);
        cacheConfiguration.setWriteBehindFlushThreadCount(2);
        cacheConfiguration.setWriteBehindBatchSize(10000);


        final CassandraCacheStoreFactory<String, BinaryObject>
cacheStoreFactory = new CassandraCacheStoreFactory<>();
        final DataSource dataSource = new DataSource();
        dataSource.setContactPoints(contactPoints);
        dataSource.setReadConsistency("ONE");
        dataSource.setWriteConsistency("ONE");
        dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new
RoundRobinPolicy()));
        cacheStoreFactory.setDataSource(dataSource);

        final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\"
table=\"%s\">" +
                "<keyPersistence class=\"%s\" strategy=\"%s\"/>" +
                "<valuePersistence class=\"%s\" strategy=\"%s\"/>" +
                "</persistence>";
        final KeyValuePersistenceSettings settings = new
KeyValuePersistenceSettings(
                String.format(CASSANDRA_PERSISTENCE, "test", "test_table",
                        "java.lang.String", "PRIMITIVE",
                        "org.apache.ignite.binary.BinaryObject", "BLOB"));
        cacheStoreFactory.setPersistenceSettings(settings);
        cacheConfiguration.setCacheStoreFactory(cacheStoreFactory);

When application works some time (hour or more, may be less - from time to
time) I see this exceptions on the ignite nodes:
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072]
    ^-- H/N/C [hosts=3, nodes=3, CPUs=96]
    ^-- CPU [cur=0%, avg=1.86%, GC=0%]
    ^-- PageMemory [pages=118064]
    ^-- Heap [used=4800MB, free=53.12%, comm=10240MB]
    ^-- Non heap [used=78MB, free=-1%, comm=80MB]
    ^-- Public thread pool [active=0, idle=32, qSize=0]
    ^-- System thread pool [active=0, idle=32, qSize=0]
    ^-- Outbound messages queue [size=0]
[15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList
[name=null, buckets=256, dataPages=102080, reusePages=0]
[15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed
[15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed
[15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed
[15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 attempt
to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

After this I have to restart the cluster and application or have
"transactions deadlock" message.

And I checked: network is fine, no OOME in the ignite and cassandra
(cassandra logs don't say anything about it). I understand that this is
issue of persistence into cassandra but don't know how can I fix it. And
after some attempts ignite node is stopped. I tried to change write behind
parameters and disable it at all - but have the same issue. May be I have to
see in the other place? May be this is important - i use IgniteDataStreamer
for reading values from kafka (earlier I used the simple cache operations
and didn't have this problem at all).

final IgniteDataStreamer<String, BinaryObject> streamer =
ignite.dataStreamer(callCache().getName());
streamer.autoFlushFrequency(5000);
streamer.keepBinary(true);
streamer.perNodeBufferSize(5120);


And one more question about persistence settings. If I'm working with
BinaryObjects and ignite doesn't know about my class types, should I use
valuePersistence class =  "org.apache.ignite.binary.BinaryObject"? I did it
because peer class loading doesn't work for my own classes and I don't want
to add new jars into all ignite nodes and all my apps.

Thanks for help!



 

--

Best regards,
Andrey V. Mashenkov


dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Ignite-cassandra module issue

Hi Michael, do you have any update for the issue?

On Thu, Nov 2, 2017 at 5:14 PM, Michael Cherkasov <[hidden email]> wrote:
Hi Tobias,

Thank you for explaining how to reproduce it, I'll try your instruction. I spend several days trying to reproduce the issue,
but I thought that the reason of this is too high load and I didn't stop client during testing.
I'll check your instruction and try to fix the issue.

Thanks,
Mike.

2017-10-25 16:23 GMT+03:00 Tobias Eriksson <[hidden email]>:

Hi Andrey et al

I believe I now know what the problem is, the Cassandra session is refreshed, but before it is a prepared statement is created/used and there, and so using a new session with an old prepared statement is not working.

 

The way to reproduce is

1)       Start Ignite Server Node

2)       Start client which inserts a batch of 100 elements

3)       End client

4)       Now Ignite Server Node returns the Cassandra Session to the pool

5)       Wait 5+ minutes

6)       Now Ignite Server Node has does a clean-up of the “unused” Cassandra sessions

7)       Start client which inserts a batch of 100 elements

8)       Boom ! The exception starts to happen

 

Reason is

1)       Execute is called for a BATCH

2)       Prepared-statement is reused since there is a cache of those

3)       It is about to do session().execute( batch )

4)       BUT the call to session() results in refreshing the session, and this is where the prepared statements as the old session new them are cleaned up

5)       Now it is looping over 100 times with a NEW session but with an OLD prepared statement

 

This is a bug,

 

-Tobias

 

 

From: Andrey Mashenkov <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, 25 October 2017 at 14:12
To: "[hidden email]" <[hidden email]>
Subject: Re: Ignite-cassandra module issue

 

Hi Tobias,

 

What ignite version do you use? May be this was already fixed in latest one?

I see related fix inclueded in upcoming 2.3 version.

 

See IGNITE-5897 [1] issue. It is unobvious, but this fix session init\end logic, so session should be closed in proper way.

 

 

 

On Wed, Oct 25, 2017 at 11:13 AM, Tobias Eriksson <[hidden email]> wrote:

Hi
 Sorry did not include the context when I replied
 Has anyone been able to resolve this problem, cause I have it too on and
off
In fact it sometimes happens just like that, e.g. I have been running my
Ignite client and then stop it, and then it takes a while and run it again,
and all by a sudden this error shows up. An that is the first thing that
happens, and there is NOT a massive amount of load on Cassandra at that
time. But I have also seen it when I hammer Ignite/Cassandra with
updates/inserts.

This is a deal-breaker for me, I need to understand how to fix this, cause
having this in production is not an option.

-Tobias


Hi!
I'm using the cassandra as persistence store for my caches and have one
issue by handling a huge data (via IgniteDataStreamer from kafka).
Ignite Configuration:
final IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setIgniteInstanceName("test");
        igniteConfiguration.setClientMode(true);
        igniteConfiguration.setGridLogger(new Slf4jLogger());
        igniteConfiguration.setMetricsLogFrequency(0);
        igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi());
final BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
        binaryConfiguration.setCompactFooter(false);
        igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
        igniteConfiguration.setPeerClassLoadingEnabled(true);
final MemoryPolicyConfiguration memoryPolicyConfiguration = new
MemoryPolicyConfiguration();
        memoryPolicyConfiguration.setName("3Gb_Region_Eviction");
        memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L);
        memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L);

memoryPolicyConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
final MemoryConfiguration memoryConfiguration = new MemoryConfiguration();
        memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration);
        igniteConfiguration.setMemoryConfiguration(memoryConfiguration);

Cache configuration:
final CacheConfiguration<String, BinaryObject> cacheConfiguration = new
CacheConfiguration<>();
        cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheConfiguration.setStoreKeepBinary(true);
        cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
        cacheConfiguration.setBackups(0);
        cacheConfiguration.setStatisticsEnabled(false);
        cacheConfiguration.setName("TestCache");

        cacheConfiguration.setReadThrough(true);
        cacheConfiguration.setWriteThrough(true);

        cacheConfiguration.setWriteBehindEnabled(true);
        cacheConfiguration.setWriteBehindFlushFrequency(10000);
        cacheConfiguration.setWriteBehindFlushSize(0);
        cacheConfiguration.setWriteBehindFlushThreadCount(2);
        cacheConfiguration.setWriteBehindBatchSize(10000);


        final CassandraCacheStoreFactory<String, BinaryObject>
cacheStoreFactory = new CassandraCacheStoreFactory<>();
        final DataSource dataSource = new DataSource();
        dataSource.setContactPoints(contactPoints);
        dataSource.setReadConsistency("ONE");
        dataSource.setWriteConsistency("ONE");
        dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new
RoundRobinPolicy()));
        cacheStoreFactory.setDataSource(dataSource);

        final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\"
table=\"%s\">" +
                "<keyPersistence class=\"%s\" strategy=\"%s\"/>" +
                "<valuePersistence class=\"%s\" strategy=\"%s\"/>" +
                "</persistence>";
        final KeyValuePersistenceSettings settings = new
KeyValuePersistenceSettings(
                String.format(CASSANDRA_PERSISTENCE, "test", "test_table",
                        "java.lang.String", "PRIMITIVE",
                        "org.apache.ignite.binary.BinaryObject", "BLOB"));
        cacheStoreFactory.setPersistenceSettings(settings);
        cacheConfiguration.setCacheStoreFactory(cacheStoreFactory);

When application works some time (hour or more, may be less - from time to
time) I see this exceptions on the ignite nodes:
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072]
    ^-- H/N/C [hosts=3, nodes=3, CPUs=96]
    ^-- CPU [cur=0%, avg=1.86%, GC=0%]
    ^-- PageMemory [pages=118064]
    ^-- Heap [used=4800MB, free=53.12%, comm=10240MB]
    ^-- Non heap [used=78MB, free=-1%, comm=80MB]
    ^-- Public thread pool [active=0, idle=32, qSize=0]
    ^-- System thread pool [active=0, idle=32, qSize=0]
    ^-- Outbound messages queue [size=0]
[15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList
[name=null, buckets=256, dataPages=102080, reusePages=0]
[15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed
[15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed
[15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed
[15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 attempt
to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

After this I have to restart the cluster and application or have
"transactions deadlock" message.

And I checked: network is fine, no OOME in the ignite and cassandra
(cassandra logs don't say anything about it). I understand that this is
issue of persistence into cassandra but don't know how can I fix it. And
after some attempts ignite node is stopped. I tried to change write behind
parameters and disable it at all - but have the same issue. May be I have to
see in the other place? May be this is important - i use IgniteDataStreamer
for reading values from kafka (earlier I used the simple cache operations
and didn't have this problem at all).

final IgniteDataStreamer<String, BinaryObject> streamer =
ignite.dataStreamer(callCache().getName());
streamer.autoFlushFrequency(5000);
streamer.keepBinary(true);
streamer.perNodeBufferSize(5120);


And one more question about persistence settings. If I'm working with
BinaryObjects and ignite doesn't know about my class types, should I use
valuePersistence class =  "org.apache.ignite.binary.BinaryObject"? I did it
because peer class loading doesn't work for my own classes and I don't want
to add new jars into all ignite nodes and all my apps.

Thanks for help!



 

--

Best regards,
Andrey V. Mashenkov



Mikhail Mikhail
Reply | Threaded
Open this post in threaded view
|

Re: Ignite-cassandra module issue

In reply to this post by Tobias Eriksson
Hi Tobias,

I created a ticket to fix the issue: https://issues.apache.org/jira/browse/IGNITE-6853

2017-10-25 16:23 GMT+03:00 Tobias Eriksson <[hidden email]>:

Hi Andrey et al

I believe I now know what the problem is, the Cassandra session is refreshed, but before it is a prepared statement is created/used and there, and so using a new session with an old prepared statement is not working.

 

The way to reproduce is

1)       Start Ignite Server Node

2)       Start client which inserts a batch of 100 elements

3)       End client

4)       Now Ignite Server Node returns the Cassandra Session to the pool

5)       Wait 5+ minutes

6)       Now Ignite Server Node has does a clean-up of the “unused” Cassandra sessions

7)       Start client which inserts a batch of 100 elements

8)       Boom ! The exception starts to happen

 

Reason is

1)       Execute is called for a BATCH

2)       Prepared-statement is reused since there is a cache of those

3)       It is about to do session().execute( batch )

4)       BUT the call to session() results in refreshing the session, and this is where the prepared statements as the old session new them are cleaned up

5)       Now it is looping over 100 times with a NEW session but with an OLD prepared statement

 

This is a bug,

 

-Tobias

 

 

From: Andrey Mashenkov <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, 25 October 2017 at 14:12
To: "[hidden email]" <[hidden email]>
Subject: Re: Ignite-cassandra module issue

 

Hi Tobias,

 

What ignite version do you use? May be this was already fixed in latest one?

I see related fix inclueded in upcoming 2.3 version.

 

See IGNITE-5897 [1] issue. It is unobvious, but this fix session init\end logic, so session should be closed in proper way.

 

 

 

On Wed, Oct 25, 2017 at 11:13 AM, Tobias Eriksson <[hidden email]> wrote:

Hi
 Sorry did not include the context when I replied
 Has anyone been able to resolve this problem, cause I have it too on and
off
In fact it sometimes happens just like that, e.g. I have been running my
Ignite client and then stop it, and then it takes a while and run it again,
and all by a sudden this error shows up. An that is the first thing that
happens, and there is NOT a massive amount of load on Cassandra at that
time. But I have also seen it when I hammer Ignite/Cassandra with
updates/inserts.

This is a deal-breaker for me, I need to understand how to fix this, cause
having this in production is not an option.

-Tobias


Hi!
I'm using the cassandra as persistence store for my caches and have one
issue by handling a huge data (via IgniteDataStreamer from kafka).
Ignite Configuration:
final IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setIgniteInstanceName("test");
        igniteConfiguration.setClientMode(true);
        igniteConfiguration.setGridLogger(new Slf4jLogger());
        igniteConfiguration.setMetricsLogFrequency(0);
        igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi());
final BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
        binaryConfiguration.setCompactFooter(false);
        igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
        igniteConfiguration.setPeerClassLoadingEnabled(true);
final MemoryPolicyConfiguration memoryPolicyConfiguration = new
MemoryPolicyConfiguration();
        memoryPolicyConfiguration.setName("3Gb_Region_Eviction");
        memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L);
        memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L);

memoryPolicyConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
final MemoryConfiguration memoryConfiguration = new MemoryConfiguration();
        memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration);
        igniteConfiguration.setMemoryConfiguration(memoryConfiguration);

Cache configuration:
final CacheConfiguration<String, BinaryObject> cacheConfiguration = new
CacheConfiguration<>();
        cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheConfiguration.setStoreKeepBinary(true);
        cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
        cacheConfiguration.setBackups(0);
        cacheConfiguration.setStatisticsEnabled(false);
        cacheConfiguration.setName("TestCache");

        cacheConfiguration.setReadThrough(true);
        cacheConfiguration.setWriteThrough(true);

        cacheConfiguration.setWriteBehindEnabled(true);
        cacheConfiguration.setWriteBehindFlushFrequency(10000);
        cacheConfiguration.setWriteBehindFlushSize(0);
        cacheConfiguration.setWriteBehindFlushThreadCount(2);
        cacheConfiguration.setWriteBehindBatchSize(10000);


        final CassandraCacheStoreFactory<String, BinaryObject>
cacheStoreFactory = new CassandraCacheStoreFactory<>();
        final DataSource dataSource = new DataSource();
        dataSource.setContactPoints(contactPoints);
        dataSource.setReadConsistency("ONE");
        dataSource.setWriteConsistency("ONE");
        dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new
RoundRobinPolicy()));
        cacheStoreFactory.setDataSource(dataSource);

        final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\"
table=\"%s\">" +
                "<keyPersistence class=\"%s\" strategy=\"%s\"/>" +
                "<valuePersistence class=\"%s\" strategy=\"%s\"/>" +
                "</persistence>";
        final KeyValuePersistenceSettings settings = new
KeyValuePersistenceSettings(
                String.format(CASSANDRA_PERSISTENCE, "test", "test_table",
                        "java.lang.String", "PRIMITIVE",
                        "org.apache.ignite.binary.BinaryObject", "BLOB"));
        cacheStoreFactory.setPersistenceSettings(settings);
        cacheConfiguration.setCacheStoreFactory(cacheStoreFactory);

When application works some time (hour or more, may be less - from time to
time) I see this exceptions on the ignite nodes:
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072]
    ^-- H/N/C [hosts=3, nodes=3, CPUs=96]
    ^-- CPU [cur=0%, avg=1.86%, GC=0%]
    ^-- PageMemory [pages=118064]
    ^-- Heap [used=4800MB, free=53.12%, comm=10240MB]
    ^-- Non heap [used=78MB, free=-1%, comm=80MB]
    ^-- Public thread pool [active=0, idle=32, qSize=0]
    ^-- System thread pool [active=0, idle=32, qSize=0]
    ^-- Outbound messages queue [size=0]
[15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList
[name=null, buckets=256, dataPages=102080, reusePages=0]
[15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed
[15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed
[15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed
[15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 attempt
to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

After this I have to restart the cluster and application or have
"transactions deadlock" message.

And I checked: network is fine, no OOME in the ignite and cassandra
(cassandra logs don't say anything about it). I understand that this is
issue of persistence into cassandra but don't know how can I fix it. And
after some attempts ignite node is stopped. I tried to change write behind
parameters and disable it at all - but have the same issue. May be I have to
see in the other place? May be this is important - i use IgniteDataStreamer
for reading values from kafka (earlier I used the simple cache operations
and didn't have this problem at all).

final IgniteDataStreamer<String, BinaryObject> streamer =
ignite.dataStreamer(callCache().getName());
streamer.autoFlushFrequency(5000);
streamer.keepBinary(true);
streamer.perNodeBufferSize(5120);


And one more question about persistence settings. If I'm working with
BinaryObjects and ignite doesn't know about my class types, should I use
valuePersistence class =  "org.apache.ignite.binary.BinaryObject"? I did it
because peer class loading doesn't work for my own classes and I don't want
to add new jars into all ignite nodes and all my apps.

Thanks for help!



 

--

Best regards,
Andrey V. Mashenkov


Mikhail Mikhail
Reply | Threaded
Open this post in threaded view
|

Re: Ignite-cassandra module issue

In reply to this post by dsetrakyan
Hi Dmitriy,

I created a ticket for this:
it will be fixed in 2.4.

Thanks,
Mike.

2017-11-09 2:35 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
Hi Michael, do you have any update for the issue?

On Thu, Nov 2, 2017 at 5:14 PM, Michael Cherkasov <[hidden email]> wrote:
Hi Tobias,

Thank you for explaining how to reproduce it, I'll try your instruction. I spend several days trying to reproduce the issue,
but I thought that the reason of this is too high load and I didn't stop client during testing.
I'll check your instruction and try to fix the issue.

Thanks,
Mike.

2017-10-25 16:23 GMT+03:00 Tobias Eriksson <[hidden email]>:

Hi Andrey et al

I believe I now know what the problem is, the Cassandra session is refreshed, but before it is a prepared statement is created/used and there, and so using a new session with an old prepared statement is not working.

 

The way to reproduce is

1)       Start Ignite Server Node

2)       Start client which inserts a batch of 100 elements

3)       End client

4)       Now Ignite Server Node returns the Cassandra Session to the pool

5)       Wait 5+ minutes

6)       Now Ignite Server Node has does a clean-up of the “unused” Cassandra sessions

7)       Start client which inserts a batch of 100 elements

8)       Boom ! The exception starts to happen

 

Reason is

1)       Execute is called for a BATCH

2)       Prepared-statement is reused since there is a cache of those

3)       It is about to do session().execute( batch )

4)       BUT the call to session() results in refreshing the session, and this is where the prepared statements as the old session new them are cleaned up

5)       Now it is looping over 100 times with a NEW session but with an OLD prepared statement

 

This is a bug,

 

-Tobias

 

 

From: Andrey Mashenkov <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, 25 October 2017 at 14:12
To: "[hidden email]" <[hidden email]>
Subject: Re: Ignite-cassandra module issue

 

Hi Tobias,

 

What ignite version do you use? May be this was already fixed in latest one?

I see related fix inclueded in upcoming 2.3 version.

 

See IGNITE-5897 [1] issue. It is unobvious, but this fix session init\end logic, so session should be closed in proper way.

 

 

 

On Wed, Oct 25, 2017 at 11:13 AM, Tobias Eriksson <[hidden email]> wrote:

Hi
 Sorry did not include the context when I replied
 Has anyone been able to resolve this problem, cause I have it too on and
off
In fact it sometimes happens just like that, e.g. I have been running my
Ignite client and then stop it, and then it takes a while and run it again,
and all by a sudden this error shows up. An that is the first thing that
happens, and there is NOT a massive amount of load on Cassandra at that
time. But I have also seen it when I hammer Ignite/Cassandra with
updates/inserts.

This is a deal-breaker for me, I need to understand how to fix this, cause
having this in production is not an option.

-Tobias


Hi!
I'm using the cassandra as persistence store for my caches and have one
issue by handling a huge data (via IgniteDataStreamer from kafka).
Ignite Configuration:
final IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setIgniteInstanceName("test");
        igniteConfiguration.setClientMode(true);
        igniteConfiguration.setGridLogger(new Slf4jLogger());
        igniteConfiguration.setMetricsLogFrequency(0);
        igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi());
final BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
        binaryConfiguration.setCompactFooter(false);
        igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
        igniteConfiguration.setPeerClassLoadingEnabled(true);
final MemoryPolicyConfiguration memoryPolicyConfiguration = new
MemoryPolicyConfiguration();
        memoryPolicyConfiguration.setName("3Gb_Region_Eviction");
        memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L);
        memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L);

memoryPolicyConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
final MemoryConfiguration memoryConfiguration = new MemoryConfiguration();
        memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration);
        igniteConfiguration.setMemoryConfiguration(memoryConfiguration);

Cache configuration:
final CacheConfiguration<String, BinaryObject> cacheConfiguration = new
CacheConfiguration<>();
        cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheConfiguration.setStoreKeepBinary(true);
        cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
        cacheConfiguration.setBackups(0);
        cacheConfiguration.setStatisticsEnabled(false);
        cacheConfiguration.setName("TestCache");

        cacheConfiguration.setReadThrough(true);
        cacheConfiguration.setWriteThrough(true);

        cacheConfiguration.setWriteBehindEnabled(true);
        cacheConfiguration.setWriteBehindFlushFrequency(10000);
        cacheConfiguration.setWriteBehindFlushSize(0);
        cacheConfiguration.setWriteBehindFlushThreadCount(2);
        cacheConfiguration.setWriteBehindBatchSize(10000);


        final CassandraCacheStoreFactory<String, BinaryObject>
cacheStoreFactory = new CassandraCacheStoreFactory<>();
        final DataSource dataSource = new DataSource();
        dataSource.setContactPoints(contactPoints);
        dataSource.setReadConsistency("ONE");
        dataSource.setWriteConsistency("ONE");
        dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new
RoundRobinPolicy()));
        cacheStoreFactory.setDataSource(dataSource);

        final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\"
table=\"%s\">" +
                "<keyPersistence class=\"%s\" strategy=\"%s\"/>" +
                "<valuePersistence class=\"%s\" strategy=\"%s\"/>" +
                "</persistence>";
        final KeyValuePersistenceSettings settings = new
KeyValuePersistenceSettings(
                String.format(CASSANDRA_PERSISTENCE, "test", "test_table",
                        "java.lang.String", "PRIMITIVE",
                        "org.apache.ignite.binary.BinaryObject", "BLOB"));
        cacheStoreFactory.setPersistenceSettings(settings);
        cacheConfiguration.setCacheStoreFactory(cacheStoreFactory);

When application works some time (hour or more, may be less - from time to
time) I see this exceptions on the ignite nodes:
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072]
    ^-- H/N/C [hosts=3, nodes=3, CPUs=96]
    ^-- CPU [cur=0%, avg=1.86%, GC=0%]
    ^-- PageMemory [pages=118064]
    ^-- Heap [used=4800MB, free=53.12%, comm=10240MB]
    ^-- Non heap [used=78MB, free=-1%, comm=80MB]
    ^-- Public thread pool [active=0, idle=32, qSize=0]
    ^-- System thread pool [active=0, idle=32, qSize=0]
    ^-- Outbound messages queue [size=0]
[15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList
[name=null, buckets=256, dataPages=102080, reusePages=0]
[15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed
[15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed
[15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed
[15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 attempt
to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
        at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

After this I have to restart the cluster and application or have
"transactions deadlock" message.

And I checked: network is fine, no OOME in the ignite and cassandra
(cassandra logs don't say anything about it). I understand that this is
issue of persistence into cassandra but don't know how can I fix it. And
after some attempts ignite node is stopped. I tried to change write behind
parameters and disable it at all - but have the same issue. May be I have to
see in the other place? May be this is important - i use IgniteDataStreamer
for reading values from kafka (earlier I used the simple cache operations
and didn't have this problem at all).

final IgniteDataStreamer<String, BinaryObject> streamer =
ignite.dataStreamer(callCache().getName());
streamer.autoFlushFrequency(5000);
streamer.keepBinary(true);
streamer.perNodeBufferSize(5120);


And one more question about persistence settings. If I'm working with
BinaryObjects and ignite doesn't know about my class types, should I use
valuePersistence class =  "org.apache.ignite.binary.BinaryObject"? I did it
because peer class loading doesn't work for my own classes and I don't want
to add new jars into all ignite nodes and all my apps.

Thanks for help!



 

--

Best regards,
Andrey V. Mashenkov