Random CorruptedTreeException from Apache Ignite

classic Classic list List threaded Threaded
8 messages Options
SamsonLai SamsonLai
Reply | Threaded
Open this post in threaded view
|

Random CorruptedTreeException from Apache Ignite

Hello

We are using Apache Ignite 2.7.5 and we encountered the following in our
ignite. After this error occurred, it seems that the cache have been
corrupted and when we try to access this cache, the same errors occurred. We
also encountered this error with Apache Ignite 2.7.0.

The errors seems to occurred for random cache as well. Sometimes it will
happen in QA_Account cache but another times it happened for QA_Order cache,
etc. The only common denominator at the moment are the issues occurred for
the caches in the same data region and there is a schedule job at the
backend which operation involved iterate all records in the cache. (The job
run every 1 minutes). We do not have any idea what caused this as the error
seems to occur at random time for random cache and this is the only error
messages found in the ignite log.

Please kindly help. Thanks.

[15:01:53,699][SEVERE][query-#127][GridCacheDistributedQueryManager]
<QA_Account> Failed to run query [qry=GridCacheQueryInfo [loc=false,
trans=null, rdc=null, qry=GridCacheQueryAdapter [type=SCAN, clsName=null,
clause=null, filter=null, transform=null, part=null, incMeta=false,
metrics=null, pageSize=1024, timeout=0, incBackups=false, forceLocal=false,
dedup=false, prj=null, keepBinary=false,
subjId=c52f26d5-48c6-4b48-9566-41accfe995ff, taskHash=0, mvccSnapshot=null],
locFut=null, sndId=c52f26d5-48c6-4b48-9566-41accfe995ff, reqId=97142,
incMeta=false, all=false], node=b79e29cd-80ad-4191-9001-5a1454b5e207]
class org.apache.ignite.IgniteException: Runtime failure on bounds:
[lower=SearchRow [key=null, hash=0, cacheId=-703495123], upper=SearchRow
[key=null, hash=0, cacheId=-703495123]]
        at
org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:48)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:2996)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
        at
org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at
org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=SearchRow [key=null, hash=0,
cacheId=-703495123], upper=SearchRow [key=null, hash=0, cacheId=-703495123]]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1013)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:985)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2919)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2890)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2884)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2872)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.cursor(GridCacheOffheapManager.java:2075)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$6.onHasNext(IgniteCacheOffheapManagerImpl.java:968)
        at
org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
        at
org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
        ... 21 more
Caused by: java.lang.IllegalArgumentException: Invalid object type: 107
        at
org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.toKeyCacheObject(IgniteCacheObjectProcessorImpl.java:166)
        at
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toKeyCacheObject(CacheObjectBinaryProcessorImpl.java:865)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFullRow(CacheDataRowAdapter.java:333)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:167)
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:108)
        at
org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
        at
org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:92)
        at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:200)
        at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:49)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5512)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5280)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5205)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetCursor.notFound(BPlusTree.java:2899)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:306)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5615)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5600)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:159)
        at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:334)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1309)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doFind(BPlusTree.java:1276)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$8700(BPlusTree.java:90)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.find(BPlusTree.java:5289)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1005)
        ... 30 more



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Maxim.Pudov Maxim.Pudov
Reply | Threaded
Open this post in threaded view
|

Re: Random CorruptedTreeException from Apache Ignite

Hi,
From the stacktrace I can tell that there is a problem with an index in your
cache. It's hard to tell more from the stacktrace itself. Could you share
the cache config as well?
In the next release (probably 2.8) such exceptions will be better explained
in the logs (https://issues.apache.org/jira/browse/IGNITE-11750).





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
SamsonLai SamsonLai
Reply | Threaded
Open this post in threaded view
|

Re: Random CorruptedTreeException from Apache Ignite

This post was updated on .
Please find below the config we used. You mentioned that there is a problem
with the index so I guessed the data are still in the cache. Are there any
way to re-build the index to remedy the error when it occurred? Thanks.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xmlns:util="http://www.springframework.org/schema/util"
                xsi:schemaLocation="
                http://www.springframework.org/schema/beans
                http://www.springframework.org/schema/beans/spring-beans.xsd
                http://www.springframework.org/schema/util
                http://www.springframework.org/schema/util/spring-util.xsd"> 
        <bean id="grid.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
                 
                <property name="discoverySpi">
                        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                                <property name="localPort" value="48500"/>
                                <property name="localPortRange" value="10"/>
                                <property name="ipFinder">
                                        <bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">       
                                       
                                                <property name="addresses">                                               
                                                        <list>
                                                                <value>127.0.0.1:48500..48510</value>               
                                                </property>
                                        </bean>
                                </property>
                        </bean>
                </property>
               
               
                <property name="dataStorageConfiguration">
                        <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
                                <property name="defaultDataRegionConfiguration">
                                        <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                                                <property name="initialSize" value="#{1L * 1024 * 1024 * 1024}"/>
                                                <property name="maxSize" value="#{2L * 1024 * 1024 * 1024}"/>
                                                <property name="persistenceEnabled" value="true"/>
                                        </bean>
                                </property>
                                <property name="dataRegionConfigurations">
                                        <list>
                                                <bean class="org.apache.ignite.configuration.DataRegionConfiguration">                       
                                                        <property name="name" value="CC"/>
                                                        <property name="initialSize" value="#{512 * 1024 * 1024}"/>
                                                        <property name="maxSize" value="#{1L * 1024 * 1024 * 1024}"/>
                                                        <property name="persistenceEnabled" value="false"/>
                                                </bean>
                                                <bean class="org.apache.ignite.configuration.DataRegionConfiguration">                       
                                                        <property name="name" value="NP"/>
                                                        <property name="initialSize" value="#{512 * 1024 * 1024}"/>
                                                        <property name="maxSize" value="#{1L * 1024 * 1024 * 1024}"/>
                                                        <property name="persistenceEnabled" value="false"/>
                                                </bean>
                                        </list>
                                </property>
                        </bean>
                </property>
               
                <property name="communicationSpi">
                        <bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
                                <property name="slowClientQueueLimit" value="100"/>
                                <property name="messageQueueLimit" value="1024"/>
                        </bean>
                </property>
 
               
                <property name="authenticationEnabled" value="false"/>

               
                <property name="sslContextFactory">
                        <bean class="org.apache.ignite.ssl.SslContextFactory">
                                <property name="keyStoreFilePath" value="C:/temp/ignite.jks"/>
                                <property name="keyStorePassword" value="password"/>
                                <property name="protocol" value="TLS"/>
                                <property name="trustManagers">
                                        <bean class="org.apache.ignite.ssl.SslContextFactory"
factory-method="getDisabledTrustManager"/>
                                </property>
                        </bean>
                </property>

               
               
                <property name="peerClassLoadingEnabled" value="true"/>       
        </bean>       
</beans>




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Maxim.Pudov Maxim.Pudov
Reply | Threaded
Open this post in threaded view
|

Re: Random CorruptedTreeException from Apache Ignite

I don't see cacheConfiguration in the provided snippet. Do you create it
dynamically? What is your usage pattern? How do you use your cache? SQL,
cache API?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
SamsonLai SamsonLai
Reply | Threaded
Open this post in threaded view
|

Re: Random CorruptedTreeException from Apache Ignite

Yes they are created using JAVA API. The JAVA version is 1.8 and please find
below the code we used to create and get the caches. Thanks.

private IgniteCache<?, ?> getIgniteCache(String cacheName, String
groupName){
        CacheConfiguration cacheCfg = new CacheConfiguration();
        cacheCfg.setName(cacheName);
        cacheCfg.setBackups(0);
        cacheCfg.setCacheMode(CacheMode.REPLICATED);

        cacheCfg.setGroupName(groupName);
        cacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);

        return this.ignite.getOrCreateCache(cacheCfg);
}



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Maxim.Pudov Maxim.Pudov
Reply | Threaded
Open this post in threaded view
|

Re: Random CorruptedTreeException from Apache Ignite

There is a known issue with cache groups which lead to CorruptedTreeException. Try to remove 
cacheCfg.setGroupName(groupName); 
from your configuration. I'll check whether JIRA ticket was created or not for this problem.

ср, 26 июн. 2019 г. в 04:17, SamsonLai <[hidden email]>:
Yes they are created using JAVA API. The JAVA version is 1.8 and please find
below the code we used to create and get the caches. Thanks.

private IgniteCache<?, ?> getIgniteCache(String cacheName, String
groupName){
        CacheConfiguration cacheCfg = new CacheConfiguration();
        cacheCfg.setName(cacheName);
        cacheCfg.setBackups(0);
        cacheCfg.setCacheMode(CacheMode.REPLICATED);

        cacheCfg.setGroupName(groupName);
        cacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);

        return this.ignite.getOrCreateCache(cacheCfg);
}



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Maxim.Pudov Maxim.Pudov
Reply | Threaded
Open this post in threaded view
|

Re: Random CorruptedTreeException from Apache Ignite


вт, 2 июл. 2019 г. в 11:17, Maxim Pudov <[hidden email]>:
There is a known issue with cache groups which lead to CorruptedTreeException. Try to remove 
cacheCfg.setGroupName(groupName); 
from your configuration. I'll check whether JIRA ticket was created or not for this problem.

ср, 26 июн. 2019 г. в 04:17, SamsonLai <[hidden email]>:
Yes they are created using JAVA API. The JAVA version is 1.8 and please find
below the code we used to create and get the caches. Thanks.

private IgniteCache<?, ?> getIgniteCache(String cacheName, String
groupName){
        CacheConfiguration cacheCfg = new CacheConfiguration();
        cacheCfg.setName(cacheName);
        cacheCfg.setBackups(0);
        cacheCfg.setCacheMode(CacheMode.REPLICATED);

        cacheCfg.setGroupName(groupName);
        cacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);

        return this.ignite.getOrCreateCache(cacheCfg);
}



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ihalilaltun ihalilaltun
Reply | Threaded
Open this post in threaded view
|

Re: Random CorruptedTreeException from Apache Ignite

This post was updated on .
Hi Maxim,

we are facing the exact same problem :(

Is it ok/safe to remove cacheGroupName from the code that is already been
created on nodes? If so, when we start our applications from the updated
code will we be still have access to same caches or new caches will be
created on nodes?



-----
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
İbrahim Halil Altun
Senior Software Engineer @ Segmentify