Ignite-cassandra module issue

classic Classic list List threaded Threaded
2 messages Options
nash1k nash1k
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Ignite-cassandra module issue

Hi!
I'm using the cassandra as persistence store for my caches and have one issue by handling a huge data (via IgniteDataStreamer from kafka).
Ignite Configuration:
final IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setIgniteInstanceName("test");
        igniteConfiguration.setClientMode(true);
        igniteConfiguration.setGridLogger(new Slf4jLogger());
        igniteConfiguration.setMetricsLogFrequency(0);
        igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi());
final BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
        binaryConfiguration.setCompactFooter(false);
        igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
        igniteConfiguration.setPeerClassLoadingEnabled(true);
final MemoryPolicyConfiguration memoryPolicyConfiguration = new MemoryPolicyConfiguration();
        memoryPolicyConfiguration.setName("3Gb_Region_Eviction");
        memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L);
        memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L);
        memoryPolicyConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
final MemoryConfiguration memoryConfiguration = new MemoryConfiguration();
        memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration);
        igniteConfiguration.setMemoryConfiguration(memoryConfiguration);

Cache configuration:
final CacheConfiguration<String, BinaryObject> cacheConfiguration = new CacheConfiguration<>();
        cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheConfiguration.setStoreKeepBinary(true);
        cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
        cacheConfiguration.setBackups(0);
        cacheConfiguration.setStatisticsEnabled(false);
        cacheConfiguration.setName("TestCache");

        cacheConfiguration.setReadThrough(true);
        cacheConfiguration.setWriteThrough(true);

        cacheConfiguration.setWriteBehindEnabled(true);
        cacheConfiguration.setWriteBehindFlushFrequency(10000);
        cacheConfiguration.setWriteBehindFlushSize(0);
        cacheConfiguration.setWriteBehindFlushThreadCount(2);
        cacheConfiguration.setWriteBehindBatchSize(10000);


        final CassandraCacheStoreFactory<String, BinaryObject> cacheStoreFactory = new CassandraCacheStoreFactory<>();
        final DataSource dataSource = new DataSource();
        dataSource.setContactPoints(contactPoints);
        dataSource.setReadConsistency("ONE");
        dataSource.setWriteConsistency("ONE");
        dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new RoundRobinPolicy()));
        cacheStoreFactory.setDataSource(dataSource);

        final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\" table=\"%s\">" +
                "<keyPersistence class=\"%s\" strategy=\"%s\"/>" +
                "<valuePersistence class=\"%s\" strategy=\"%s\"/>" +
                "</persistence>";
        final KeyValuePersistenceSettings settings = new KeyValuePersistenceSettings(
                String.format(CASSANDRA_PERSISTENCE, "test", "test_table",
                        "java.lang.String", "PRIMITIVE",
                        "org.apache.ignite.binary.BinaryObject", "BLOB"));
        cacheStoreFactory.setPersistenceSettings(settings);
        cacheConfiguration.setCacheStoreFactory(cacheStoreFactory);

When application works some time (hour or more, may be less - from time to time) I see this exceptions on the ignite nodes:
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072]
    ^-- H/N/C [hosts=3, nodes=3, CPUs=96]
    ^-- CPU [cur=0%, avg=1.86%, GC=0%]
    ^-- PageMemory [pages=118064]
    ^-- Heap [used=4800MB, free=53.12%, comm=10240MB]
    ^-- Non heap [used=78MB, free=-1%, comm=80MB]
    ^-- Public thread pool [active=0, idle=32, qSize=0]
    ^-- System thread pool [active=0, idle=32, qSize=0]
    ^-- Outbound messages queue [size=0]
[15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList [name=null, buckets=256, dataPages=102080, reusePages=0]
[15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra session refreshed
[15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed
[15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed
[15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed
[15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?;
[15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra session refreshed
[15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance.
        at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
        at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
        at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
        at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
        at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
        at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
        at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
        at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

After this I have to restart the cluster and application or have "transactions deadlock" message.

And I checked: network is fine, no OOME in the ignite and cassandra (cassandra logs don't say anything about it). I understand that this is issue of persistence into cassandra but don't know how can I fix it. And after some attempts ignite node is stopped. I tried to change write behind parameters and disable it at all - but have the same issue. May be I have to see in the other place? May be this is important - i use IgniteDataStreamer for reading values from kafka (earlier I used the simple cache operations and didn't have this problem at all).

final IgniteDataStreamer<String, BinaryObject> streamer = ignite.dataStreamer(callCache().getName());
streamer.autoFlushFrequency(5000);
streamer.keepBinary(true);
streamer.perNodeBufferSize(5120);


And one more question about persistence settings. If I'm working with BinaryObjects and ignite doesn't know about my class types, should I use valuePersistence class =  "org.apache.ignite.binary.BinaryObject"? I did it because peer class loading doesn't work for my own classes and I don't want to add new jars into all ignite nodes and all my apps.

Thanks for help!
dkarachentsev dkarachentsev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ignite-cassandra module issue

Hi,

Is it possible to provide a simple reproducer?
Answering your second question, yes, you can use BinaryObject as persistence class name.

Thanks!
-Dmitry.
Loading...