And again... Failed to get page IO instance (page content is corrupted)

classic Classic list List threaded Threaded
21 messages Options
12
Sergey Sergeev Sergey Sergeev
Reply | Threaded
Open this post in threaded view
|

And again... Failed to get page IO instance (page content is corrupted)

Hi Igniters

I can't defeat the bug. It is reproducing on ignite version community 2.3.0 and on trial professional 2.3.2 editions.

The Cache has configuration like:


final CacheConfiguration<Pk, Message> cacheConfiguration

= new CacheConfiguration<>(getEntityClass().getSimpleName());

cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
cacheConfiguration.setBackups(
0);
cacheConfiguration.
setRebalanceMode(CacheRebalanceMode.ASYNC);
cacheConfiguration.
setDataRegionName("messagePartSentRegion");
cacheConfiguration.
setEagerTtl(true);
cacheConfiguration.
setManagementEnabled(true);
cacheConfiguration.
setStatisticsEnabled(true);

IgniteCache<Pk, Message> cache = ignite.getOrCreateCache(
cacheConfiguration)
.withExpiryPolicy(new CreatedExpiryPolicy(new Duration(DAYS, 7)));

The data storage has configuration:
...
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="walMode" value="LOG_ONLY" />
<property name="writeThrottlingEnabled" value="true" />
<property name="metricsEnabled" value="true" />
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="metricsEnabled" value="true" />
<property name="persistenceEnabled" value="true" />
<!-- 2Gb maximum size -->
<property name="maxSize" value="#{2L * 1024 * 1024 * 1024}"/>
</bean>
</property>
<property name="dataRegionConfigurations">
<list>
...

<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="metricsEnabled" value="true" />
<property name="name" value="messagePartSentRegion" />
<property name="persistenceEnabled" value="true" />
<!-- 2Gb maximum size -->
<property name="maxSize" value="#{2L * 1024 * 1024 * 1024}"/>
</bean>
...
</list>
</property>
</bean>
</property>
...
After 7 days, I've exception:

ERROR  GridCacheIoManager - Failed to process message [senderId=e9e71d4c-e1dc-4647-8fd4-72fb281907a4, messageType=class o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicFullUpdateRequest]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessageProcessed(GridCacheIoManager.java:1099) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1072) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) ~[ignite-core-2.3.2.jar:2.3.2]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted)
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.2.jar:2.3.2]
    ... 17 more

The description of such a problem:
http://apache-ignite-users.70518.x6.nabble.com/Partition-eviction-failed-this-can-cause-grid-hang-Caused-by-java-lang-IllegalStateException-Failed--td19122.html

Issues:
https://issues.apache.org/jira/browse/IGNITE-6423
https://issues.apache.org/jira/browse/IGNITE-7278

And... When do you plan to release the 2.4.0 version? May be fix?



--
Best regards,
Sergey S. Sergeev
exit(0);
Mikhail Mikhail
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi Sergey,

The release of 2.4 should be soon, in a week or couple, however, there's no
strong schedule for Apache releases.

Could you please share a reproducer for the issue? Might be you can share a
storage on which the issue can be reproduced?

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi Sergey,

The 2.4 release is about to be voted for. You can use the RC1 in the mean time:

D.

On Mon, Feb 19, 2018 at 6:43 AM, Mikhail <[hidden email]> wrote:
Hi Sergey,

The release of 2.4 should be soon, in a week or couple, however, there's no
strong schedule for Apache releases.

Could you please share a reproducer for the issue? Might be you can share a
storage on which the issue can be reproduced?

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Sergey Sergeev Sergey Sergeev
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi Mikhail,

Unfortunately, the problem has repeated itself on ignite-core-2.3.3

27.02.18 00:27:55 ERROR  GridCacheIoManager - Failed to process message [senderId=8f99c887-cd4b-4c38-a649-ca430040d535, messageType=class o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessageProcessed(GridCacheIoManager.java:1099) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1072) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) ~[ignite-core-2.3.3.jar:2.3.3]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted)
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
        ... 17 more


And if we are reading from cache...


27.02.18 00:27:56 ERROR MessagePartProcessingHandler - error processing incoming messagePartDeliveryReceipt=MessagePartDeliveryReceipt{id=null, abonent=7****, externalId='7586F73B'}
org.springframework.dao.DataAccessResourceFailureException: Runtime failure on bounds: [lower=null, upper=PendingRow []]; nested exception is class org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
        at com.a1s.sdp.data.dao.ignite.IgniteExceptionTranslator.translateExceptionIfPossible(IgniteExceptionTranslator.java:35) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:41) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.findOne(BaseIgniteCassandraRepository.java:136) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.common.service.TransportServiceImpl.processMessagePartDelivered(TransportServiceImpl.java:213) ~[core-common-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.common.service.TransportServiceImpl.messagePartDelivered(TransportServiceImpl.java:202) ~[core-common-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.common.transport.handler.MessagePartProcessingHandler.lambda$channelReadMessagePartDeliveryReceipt$1(MessagePartProcessingHandler.java:91) ~[core-common-1.5-SNAPSHOT.jar:3752]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_151]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:608) ~[ignite-core-2.3.3.jar:2.3.3]
        at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.lambda$findOne$7(BaseIgniteCassandraRepository.java:136) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39) ~[core-data-1.5-SNAPSHOT.jar:3752]
        ... 9 more
Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted)
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:608) ~[ignite-core-2.3.3.jar:2.3.3]
        at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.lambda$findOne$7(BaseIgniteCassandraRepository.java:136) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39) ~[core-data-1.5-SNAPSHOT.jar:3752]
        ... 9 more
       

The attachment (https://yadi.sk/d/u7fSK8Y53SpCpA) contains a part of cache obtained from node of cluster - /var/ignite/***/db/node00-66fd86f5-4e2d-4f1b-8fe1-993bf5937d99/cache-MessagePartSent.
This cache contains objects of class MessagePartSent:

@Table("message_part_sent_external")
public final class MessagePartSent {
    @Id
    private MessagePartSentPk id;
    private MessagePartPk partId;

    public MessagePartSentPk getId() {
        return id;
    }

    public void setId(MessagePartSentPk id) {
        this.id = id;
    }

    public MessagePartPk getPartId() {
        return partId;
    }

    public void setPartId(MessagePartPk partId) {
        this.partId = partId;
    }

    @Override
    public String toString() {
        final StringBuilder sb = new StringBuilder("MessagePartSent{");
        sb.append("id=").append(id);
        sb.append(", partId=").append(partId);
        sb.append('}');
        return sb.toString();
    }

    public static final class MessagePartSentPk implements Serializable {
        private Integer linkId;
        private String externalId;

        public MessagePartSentPk() {
            //
        }

        public MessagePartSentPk(Integer linkId, String externalId) {
            this.linkId = linkId;
            this.externalId = externalId;
        }

        public Integer getLinkId() {
            return linkId;
        }

        public void setLinkId(Integer linkId) {
            this.linkId = linkId;
        }

        public String getExternalId() {
            return externalId;
        }

        public void setExternalId(String externalId) {
            this.externalId = externalId;
        }

        @Override
        public boolean equals(Object o) {
            if (this == o) return true;
            if (!(o instanceof MessagePartSentPk)) return false;

            final MessagePartSentPk that = (MessagePartSentPk) o;

            if (externalId != null ? !externalId.equals(that.externalId) : that.externalId != null) return false;
            if (linkId != null ? !linkId.equals(that.linkId) : that.linkId != null) return false;

            return true;
        }

        @Override
        public int hashCode() {
            int result = linkId != null ? linkId.hashCode() : 0;
            result = 31 * result + (externalId != null ? externalId.hashCode() : 0);
            return result;
        }

        @Override
        public String toString() {
            final StringBuilder sb = new StringBuilder("MessagePartSentPk{");
            sb.append("linkId=").append(linkId);
            sb.append(", externalId='").append(externalId).append('\'');
            sb.append('}');
            return sb.toString();
        }
    }

}


public final class MessagePartPk implements Serializable {
    private UUID transactionId;
    private Short partNumber;

    public MessagePartPk() {
        //default
    }

    public MessagePartPk(UUID transactionId, Short partNumber) {
        this.transactionId = transactionId;
        this.partNumber = partNumber;
    }

    public UUID getTransactionId() {
        return transactionId;
    }

    public void setTransactionId(UUID transactionId) {
        this.transactionId = transactionId;
    }

    public Short getPartNumber() {
        return partNumber;
    }

    public void setPartNumber(Short partNumber) {
        this.partNumber = partNumber;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (!(o instanceof MessagePartPk)) return false;

        final MessagePartPk messagePartPk = (MessagePartPk) o;

        if (partNumber != null ? !partNumber.equals(messagePartPk.partNumber) :
                messagePartPk.partNumber != null) return false;
        if (transactionId != null ? !transactionId.equals(messagePartPk.transactionId) :
                messagePartPk.transactionId != null)
            return false;

        return true;
    }

    @Override
    public int hashCode() {
        int result = transactionId != null ? transactionId.hashCode() : 0;
        result = 31 * result + (partNumber != null ? partNumber.hashCode() : 0);
        return result;
    }

    @Override
    public String toString() {
        final StringBuilder sb = new StringBuilder("MessagePartPk{");
        sb.append("transactionId=").append(transactionId);
        sb.append(", partNumber=").append(partNumber);
        sb.append('}');
        return sb.toString();
}

On Thu, Mar 8, 2018 at 11:03 PM, Dmitriy Setrakyan <[hidden email]> wrote:
Hi Sergey,

The 2.4 release is about to be voted for. You can use the RC1 in the mean time:

D.

On Mon, Feb 19, 2018 at 6:43 AM, Mikhail <[hidden email]> wrote:
Hi Sergey,

The release of 2.4 should be soon, in a week or couple, however, there's no
strong schedule for Apache releases.

Could you please share a reproducer for the issue? Might be you can share a
storage on which the issue can be reproduced?

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/




--
Best regards,
Sergey S. Sergeev (aka SSSerj)
Notes: https://twitter.com/ssserj
exit(0);
arseny.kovalchuk arseny.kovalchuk
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi guys.

I've got a reproducer that can be related. See comments http://apache-ignite-users.70518.x6.nabble.com/Partition-eviction-failed-this-can-cause-grid-hang-Caused-by-java-lang-IllegalStateException-Failed--tp19122p20524.html

Sergey Sergeev, just for reference, what kind of file system do you use with Ignite's persistence? 

Arseny Kovalchuk

Senior Software Engineer at Synesis
skype: arseny.kovalchuk
mobile: +375 (29) 666-16-16

On 9 March 2018 at 12:31, Sergey Sergeev <[hidden email]> wrote:
Hi Mikhail,

Unfortunately, the problem has repeated itself on ignite-core-2.3.3

27.02.18 00:27:55 ERROR  GridCacheIoManager - Failed to process message [senderId=8f99c887-cd4b-4c38-a649-ca430040d535, messageType=class o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessageProcessed(GridCacheIoManager.java:1099) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1072) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) ~[ignite-core-2.3.3.jar:2.3.3]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted)
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
        ... 17 more


And if we are reading from cache...


27.02.18 00:27:56 ERROR MessagePartProcessingHandler - error processing incoming messagePartDeliveryReceipt=MessagePartDeliveryReceipt{id=null, abonent=7****, externalId='7586F73B'}
org.springframework.dao.DataAccessResourceFailureException: Runtime failure on bounds: [lower=null, upper=PendingRow []]; nested exception is class org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
        at com.a1s.sdp.data.dao.ignite.IgniteExceptionTranslator.translateExceptionIfPossible(IgniteExceptionTranslator.java:35) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:41) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.findOne(BaseIgniteCassandraRepository.java:136) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.common.service.TransportServiceImpl.processMessagePartDelivered(TransportServiceImpl.java:213) ~[core-common-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.common.service.TransportServiceImpl.messagePartDelivered(TransportServiceImpl.java:202) ~[core-common-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.common.transport.handler.MessagePartProcessingHandler.lambda$channelReadMessagePartDeliveryReceipt$1(MessagePartProcessingHandler.java:91) ~[core-common-1.5-SNAPSHOT.jar:3752]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_151]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:608) ~[ignite-core-2.3.3.jar:2.3.3]
        at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.lambda$findOne$7(BaseIgniteCassandraRepository.java:136) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39) ~[core-data-1.5-SNAPSHOT.jar:3752]
        ... 9 more
Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted)
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:608) ~[ignite-core-2.3.3.jar:2.3.3]
        at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.lambda$findOne$7(BaseIgniteCassandraRepository.java:136) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39) ~[core-data-1.5-SNAPSHOT.jar:3752]
        ... 9 more
       

The attachment (https://yadi.sk/d/u7fSK8Y53SpCpA) contains a part of cache obtained from node of cluster - /var/ignite/***/db/node00-66fd86f5-4e2d-4f1b-8fe1-993bf5937d99/cache-MessagePartSent.
This cache contains objects of class MessagePartSent:

@Table("message_part_sent_external")
public final class MessagePartSent {
    @Id
    private MessagePartSentPk id;
    private MessagePartPk partId;

    public MessagePartSentPk getId() {
        return id;
    }

    public void setId(MessagePartSentPk id) {
        this.id = id;
    }

    public MessagePartPk getPartId() {
        return partId;
    }

    public void setPartId(MessagePartPk partId) {
        this.partId = partId;
    }

    @Override
    public String toString() {
        final StringBuilder sb = new StringBuilder("MessagePartSent{");
        sb.append("id=").append(id);
        sb.append(", partId=").append(partId);
        sb.append('}');
        return sb.toString();
    }

    public static final class MessagePartSentPk implements Serializable {
        private Integer linkId;
        private String externalId;

        public MessagePartSentPk() {
            //
        }

        public MessagePartSentPk(Integer linkId, String externalId) {
            this.linkId = linkId;
            this.externalId = externalId;
        }

        public Integer getLinkId() {
            return linkId;
        }

        public void setLinkId(Integer linkId) {
            this.linkId = linkId;
        }

        public String getExternalId() {
            return externalId;
        }

        public void setExternalId(String externalId) {
            this.externalId = externalId;
        }

        @Override
        public boolean equals(Object o) {
            if (this == o) return true;
            if (!(o instanceof MessagePartSentPk)) return false;

            final MessagePartSentPk that = (MessagePartSentPk) o;

            if (externalId != null ? !externalId.equals(that.externalId) : that.externalId != null) return false;
            if (linkId != null ? !linkId.equals(that.linkId) : that.linkId != null) return false;

            return true;
        }

        @Override
        public int hashCode() {
            int result = linkId != null ? linkId.hashCode() : 0;
            result = 31 * result + (externalId != null ? externalId.hashCode() : 0);
            return result;
        }

        @Override
        public String toString() {
            final StringBuilder sb = new StringBuilder("MessagePartSentPk{");
            sb.append("linkId=").append(linkId);
            sb.append(", externalId='").append(externalId).append('\'');
            sb.append('}');
            return sb.toString();
        }
    }

}


public final class MessagePartPk implements Serializable {
    private UUID transactionId;
    private Short partNumber;

    public MessagePartPk() {
        //default
    }

    public MessagePartPk(UUID transactionId, Short partNumber) {
        this.transactionId = transactionId;
        this.partNumber = partNumber;
    }

    public UUID getTransactionId() {
        return transactionId;
    }

    public void setTransactionId(UUID transactionId) {
        this.transactionId = transactionId;
    }

    public Short getPartNumber() {
        return partNumber;
    }

    public void setPartNumber(Short partNumber) {
        this.partNumber = partNumber;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (!(o instanceof MessagePartPk)) return false;

        final MessagePartPk messagePartPk = (MessagePartPk) o;

        if (partNumber != null ? !partNumber.equals(messagePartPk.partNumber) :
                messagePartPk.partNumber != null) return false;
        if (transactionId != null ? !transactionId.equals(messagePartPk.transactionId) :
                messagePartPk.transactionId != null)
            return false;

        return true;
    }

    @Override
    public int hashCode() {
        int result = transactionId != null ? transactionId.hashCode() : 0;
        result = 31 * result + (partNumber != null ? partNumber.hashCode() : 0);
        return result;
    }

    @Override
    public String toString() {
        final StringBuilder sb = new StringBuilder("MessagePartPk{");
        sb.append("transactionId=").append(transactionId);
        sb.append(", partNumber=").append(partNumber);
        sb.append('}');
        return sb.toString();
}

On Thu, Mar 8, 2018 at 11:03 PM, Dmitriy Setrakyan <[hidden email]> wrote:
Hi Sergey,

The 2.4 release is about to be voted for. You can use the RC1 in the mean time:

D.

On Mon, Feb 19, 2018 at 6:43 AM, Mikhail <[hidden email]> wrote:
Hi Sergey,

The release of 2.4 should be soon, in a week or couple, however, there's no
strong schedule for Apache releases.

Could you please share a reproducer for the issue? Might be you can share a
storage on which the issue can be reproduced?

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/




--
Best regards,
Sergey S. Sergeev (aka SSSerj)
Notes: https://twitter.com/ssserj
exit(0);

Sergey Sergeev Sergey Sergeev
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi, Arseny

We are using ext4 with defaults,noatime,lazytime options on Virtuozzo Linux release 7.3.

We have refactored parts of logic and abandoned the use of any expired policies((



On Thu, Mar 15, 2018 at 7:34 PM, Arseny Kovalchuk <[hidden email]> wrote:
Hi guys.

I've got a reproducer that can be related. See comments http://apache-ignite-users.70518.x6.nabble.com/Partition-eviction-failed-this-can-cause-grid-hang-Caused-by-java-lang-IllegalStateException-Failed--tp19122p20524.html

Sergey Sergeev, just for reference, what kind of file system do you use with Ignite's persistence? 

Arseny Kovalchuk

Senior Software Engineer at Synesis
skype: arseny.kovalchuk
mobile: +375 (29) 666-16-16

On 9 March 2018 at 12:31, Sergey Sergeev <[hidden email]> wrote:
Hi Mikhail,

Unfortunately, the problem has repeated itself on ignite-core-2.3.3

27.02.18 00:27:55 ERROR  GridCacheIoManager - Failed to process message [senderId=8f99c887-cd4b-4c38-a649-ca430040d535, messageType=class o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessageProcessed(GridCacheIoManager.java:1099) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1072) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) ~[ignite-core-2.3.3.jar:2.3.3]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted)
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
        ... 17 more


And if we are reading from cache...


27.02.18 00:27:56 ERROR MessagePartProcessingHandler - error processing incoming messagePartDeliveryReceipt=MessagePartDeliveryReceipt{id=null, abonent=7****, externalId='7586F73B'}
org.springframework.dao.DataAccessResourceFailureException: Runtime failure on bounds: [lower=null, upper=PendingRow []]; nested exception is class org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
        at com.a1s.sdp.data.dao.ignite.IgniteExceptionTranslator.translateExceptionIfPossible(IgniteExceptionTranslator.java:35) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:41) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.findOne(BaseIgniteCassandraRepository.java:136) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.common.service.TransportServiceImpl.processMessagePartDelivered(TransportServiceImpl.java:213) ~[core-common-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.common.service.TransportServiceImpl.messagePartDelivered(TransportServiceImpl.java:202) ~[core-common-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.common.transport.handler.MessagePartProcessingHandler.lambda$channelReadMessagePartDeliveryReceipt$1(MessagePartProcessingHandler.java:91) ~[core-common-1.5-SNAPSHOT.jar:3752]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_151]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:608) ~[ignite-core-2.3.3.jar:2.3.3]
        at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.lambda$findOne$7(BaseIgniteCassandraRepository.java:136) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39) ~[core-data-1.5-SNAPSHOT.jar:3752]
        ... 9 more
Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted)
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:608) ~[ignite-core-2.3.3.jar:2.3.3]
        at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.lambda$findOne$7(BaseIgniteCassandraRepository.java:136) ~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39) ~[core-data-1.5-SNAPSHOT.jar:3752]
        ... 9 more
       

The attachment (https://yadi.sk/d/u7fSK8Y53SpCpA) contains a part of cache obtained from node of cluster - /var/ignite/***/db/node00-66fd86f5-4e2d-4f1b-8fe1-993bf5937d99/cache-MessagePartSent.
This cache contains objects of class MessagePartSent:

@Table("message_part_sent_external")
public final class MessagePartSent {
    @Id
    private MessagePartSentPk id;
    private MessagePartPk partId;

    public MessagePartSentPk getId() {
        return id;
    }

    public void setId(MessagePartSentPk id) {
        this.id = id;
    }

    public MessagePartPk getPartId() {
        return partId;
    }

    public void setPartId(MessagePartPk partId) {
        this.partId = partId;
    }

    @Override
    public String toString() {
        final StringBuilder sb = new StringBuilder("MessagePartSent{");
        sb.append("id=").append(id);
        sb.append(", partId=").append(partId);
        sb.append('}');
        return sb.toString();
    }

    public static final class MessagePartSentPk implements Serializable {
        private Integer linkId;
        private String externalId;

        public MessagePartSentPk() {
            //
        }

        public MessagePartSentPk(Integer linkId, String externalId) {
            this.linkId = linkId;
            this.externalId = externalId;
        }

        public Integer getLinkId() {
            return linkId;
        }

        public void setLinkId(Integer linkId) {
            this.linkId = linkId;
        }

        public String getExternalId() {
            return externalId;
        }

        public void setExternalId(String externalId) {
            this.externalId = externalId;
        }

        @Override
        public boolean equals(Object o) {
            if (this == o) return true;
            if (!(o instanceof MessagePartSentPk)) return false;

            final MessagePartSentPk that = (MessagePartSentPk) o;

            if (externalId != null ? !externalId.equals(that.externalId) : that.externalId != null) return false;
            if (linkId != null ? !linkId.equals(that.linkId) : that.linkId != null) return false;

            return true;
        }

        @Override
        public int hashCode() {
            int result = linkId != null ? linkId.hashCode() : 0;
            result = 31 * result + (externalId != null ? externalId.hashCode() : 0);
            return result;
        }

        @Override
        public String toString() {
            final StringBuilder sb = new StringBuilder("MessagePartSentPk{");
            sb.append("linkId=").append(linkId);
            sb.append(", externalId='").append(externalId).append('\'');
            sb.append('}');
            return sb.toString();
        }
    }

}


public final class MessagePartPk implements Serializable {
    private UUID transactionId;
    private Short partNumber;

    public MessagePartPk() {
        //default
    }

    public MessagePartPk(UUID transactionId, Short partNumber) {
        this.transactionId = transactionId;
        this.partNumber = partNumber;
    }

    public UUID getTransactionId() {
        return transactionId;
    }

    public void setTransactionId(UUID transactionId) {
        this.transactionId = transactionId;
    }

    public Short getPartNumber() {
        return partNumber;
    }

    public void setPartNumber(Short partNumber) {
        this.partNumber = partNumber;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (!(o instanceof MessagePartPk)) return false;

        final MessagePartPk messagePartPk = (MessagePartPk) o;

        if (partNumber != null ? !partNumber.equals(messagePartPk.partNumber) :
                messagePartPk.partNumber != null) return false;
        if (transactionId != null ? !transactionId.equals(messagePartPk.transactionId) :
                messagePartPk.transactionId != null)
            return false;

        return true;
    }

    @Override
    public int hashCode() {
        int result = transactionId != null ? transactionId.hashCode() : 0;
        result = 31 * result + (partNumber != null ? partNumber.hashCode() : 0);
        return result;
    }

    @Override
    public String toString() {
        final StringBuilder sb = new StringBuilder("MessagePartPk{");
        sb.append("transactionId=").append(transactionId);
        sb.append(", partNumber=").append(partNumber);
        sb.append('}');
        return sb.toString();
}

On Thu, Mar 8, 2018 at 11:03 PM, Dmitriy Setrakyan <[hidden email]> wrote:
Hi Sergey,

The 2.4 release is about to be voted for. You can use the RC1 in the mean time:

D.

On Mon, Feb 19, 2018 at 6:43 AM, Mikhail <[hidden email]> wrote:
Hi Sergey,

The release of 2.4 should be soon, in a week or couple, however, there's no
strong schedule for Apache releases.

Could you please share a reproducer for the issue? Might be you can share a
storage on which the issue can be reproduced?

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/




--
Best regards,
Sergey S. Sergeev (aka SSSerj)
Notes: https://twitter.com/ssserj
exit(0);




--
Best regards,
Sergey S. Sergeev (aka SSSerj)
exit(0);
Andrew Mashenkov Andrew Mashenkov
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

In reply to this post by Sergey Sergeev
Sergey, Arseny,

We've fixed a number of similar issues in 2.4 Ignite version.
However, this one, ExpiryPolicy with persistence still buggy. Here is a ticket for it [1].

The ticket has a PR that should fix this issue, but it broke backward compatibility 
as TTLManager, who responsible for expired entries eviction, can't restore data saved by previous Ignite versions.
So, TTL manager will not be able to evict persisted entries in background after restart until entry will be requested.


On Sun, Feb 18, 2018 at 6:24 PM, Sergey Sergeev <[hidden email]> wrote:
Hi Igniters

I can't defeat the bug. It is reproducing on ignite version community 2.3.0 and on trial professional 2.3.2 editions.

The Cache has configuration like:


final CacheConfiguration<Pk, Message> cacheConfiguration

= new CacheConfiguration<>(getEntityClass().getSimpleName());

cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
cacheConfiguration.setBackups(
0);
cacheConfiguration.
setRebalanceMode(CacheRebalanceMode.ASYNC);
cacheConfiguration.
setDataRegionName("messagePartSentRegion");
cacheConfiguration.
setEagerTtl(true);
cacheConfiguration.
setManagementEnabled(true);
cacheConfiguration.
setStatisticsEnabled(true);

IgniteCache<Pk, Message> cache = ignite.getOrCreateCache(
cacheConfiguration)
.withExpiryPolicy(new CreatedExpiryPolicy(new Duration(DAYS, 7)));

The data storage has configuration:
...
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="walMode" value="LOG_ONLY" />
<property name="writeThrottlingEnabled" value="true" />
<property name="metricsEnabled" value="true" />
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="metricsEnabled" value="true" />
<property name="persistenceEnabled" value="true" />
<!-- 2Gb maximum size -->
<property name="maxSize" value="#{2L * 1024 * 1024 * 1024}"/>
</bean>
</property>
<property name="dataRegionConfigurations">
<list>
...

<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="metricsEnabled" value="true" />
<property name="name" value="messagePartSentRegion" />
<property name="persistenceEnabled" value="true" />
<!-- 2Gb maximum size -->
<property name="maxSize" value="#{2L * 1024 * 1024 * 1024}"/>
</bean>
...
</list>
</property>
</bean>
</property>
...
After 7 days, I've exception:

ERROR  GridCacheIoManager - Failed to process message [senderId=e9e71d4c-e1dc-4647-8fd4-72fb281907a4, messageType=class o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicFullUpdateRequest]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessageProcessed(GridCacheIoManager.java:1099) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1072) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) ~[ignite-core-2.3.2.jar:2.3.2]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted)
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910) ~[ignite-core-2.3.2.jar:2.3.2]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.2.jar:2.3.2]
    ... 17 more

The description of such a problem:
http://apache-ignite-users.70518.x6.nabble.com/Partition-eviction-failed-this-can-cause-grid-hang-Caused-by-java-lang-IllegalStateException-Failed--td19122.html

Issues:
https://issues.apache.org/jira/browse/IGNITE-6423
https://issues.apache.org/jira/browse/IGNITE-7278

And... When do you plan to release the 2.4.0 version? May be fix?



--
Best regards,
Sergey S. Sergeev
exit(0);



--
Best regards,
Andrey V. Mashenkov
Regards,
Andrew.
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

In reply to this post by Sergey Sergeev
Hello Sergey!

Cross-posting from a different thread, the underlying problem could be fixed
in 2.4:

https://issues.apache.org/jira/browse/IGNITE-7278

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Oleks K Oleks K
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi guys,

I got similar errors in 2.4.0

First:

org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
  --> Caused by: java.lang.IllegalStateException: Failed to get page IO
instance (page content is corrupted)

Then lots of:

org.apache.ignite.IgniteException: Runtime failure on bounds
  --> Caused by: java.lang.IllegalStateException: Item not found: 3

This was reproduced when I started and stopped server nodes under the load
Topology: 2 server and 2 client nodes
Java: 1.8.0_162
OS: Windows Server 2012 R2 6.3 amd64

Cache config:
<bean class="org.apache.ignite.configuration.CacheConfiguration">
                    <property name="name" value="RefreshToken"/>
                    <property name="dataRegionName"
value="auth_durable_region"/>
                    <property name="atomicityMode" value="ATOMIC"/>
                    <property name="writeSynchronizationMode"
value="FULL_ASYNC"/>
                    <property name="cacheMode" value="PARTITIONED"/>
                    <property name="backups" value="0"/>
                    <property name="eagerTtl" value="true"/>
                </bean>

Ignite team, can you comment on this please?
How critical is the issue? What is the impact?
Any workarounds? Fix planned?

2018-06-13 00:22:30.978 [exchange-worker-#42] INFO
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander
- Starting rebalancing [mode=ASYNC,
fromNode=bdddfe24-aab3-46fa-9452-efe933783adb, partitionsCount=787,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], updateSeq=12]
2018-06-13 00:22:31.594 [ttl-cleanup-worker-#52] ERROR
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager
- Runtime error caught during grid runnable execution: GridWorker
[name=ttl-cleanup-worker, igniteInstanceName=null, finished=false,
hashCode=473353699, interrupted=false, runner=ttl-cleanup-worker-#52]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:963)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:974)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:129)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
[ignite-core-2.4.0.jar:2.4.0]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Failed to get page IO instance
(page content is corrupted)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4614)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4516)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4455)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:919)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:951)
~[ignite-core-2.4.0.jar:2.4.0]
        ... 6 more

2018-06-13 00:23:35.596 [sys-stripe-6-#7] ERROR
org.apache.ignite.internal.processors.cache.GridCacheIoManager - Failed to
process message [senderId=1dedb701-ffd1-4937-b202-7326367a92fb,
messageType=class
o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicFullUpdateRequest]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:963)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:974)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:834)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessageProcessed(GridCacheIoManager.java:1099)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1072)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
[ignite-core-2.4.0.jar:2.4.0]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Item not found: 3
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4614)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4516)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4455)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:919)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:951)
~[ignite-core-2.4.0.jar:2.4.0]
        ... 17 more



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Andrew Mashenkov Andrew Mashenkov
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi,

Possibly, it is a bug in partition eviction optimization. Ignite can skip partition eviction procedure and remove partition instantly if there is no indexes.

If it is so, you can try the latest ignite-2.5 version [1] or
as a workaround you can add index via configuring QueryEntity or via setting cacheCfg.setIndexTypes().



On Wed, Jun 13, 2018 at 1:22 AM, Oleks K <[hidden email]> wrote:
Hi guys,

I got similar errors in 2.4.0

First:

org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
  --> Caused by: java.lang.IllegalStateException: Failed to get page IO
instance (page content is corrupted)

Then lots of:

org.apache.ignite.IgniteException: Runtime failure on bounds
  --> Caused by: java.lang.IllegalStateException: Item not found: 3

This was reproduced when I started and stopped server nodes under the load
Topology: 2 server and 2 client nodes
Java: 1.8.0_162
OS: Windows Server 2012 R2 6.3 amd64

Cache config:
<bean class="org.apache.ignite.configuration.CacheConfiguration">
                    <property name="name" value="RefreshToken"/>
                    <property name="dataRegionName"
value="auth_durable_region"/>
                    <property name="atomicityMode" value="ATOMIC"/>
                    <property name="writeSynchronizationMode"
value="FULL_ASYNC"/>
                    <property name="cacheMode" value="PARTITIONED"/>
                    <property name="backups" value="0"/>
                    <property name="eagerTtl" value="true"/>
                </bean>

Ignite team, can you comment on this please?
How critical is the issue? What is the impact?
Any workarounds? Fix planned?

2018-06-13 00:22:30.978 [exchange-worker-#42] INFO
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander
- Starting rebalancing [mode=ASYNC,
fromNode=bdddfe24-aab3-46fa-9452-efe933783adb, partitionsCount=787,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], updateSeq=12]
2018-06-13 00:22:31.594 [ttl-cleanup-worker-#52] ERROR
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager
- Runtime error caught during grid runnable execution: GridWorker
[name=ttl-cleanup-worker, igniteInstanceName=null, finished=false,
hashCode=473353699, interrupted=false, runner=ttl-cleanup-worker-#52]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:963)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:974)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:129)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
[ignite-core-2.4.0.jar:2.4.0]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Failed to get page IO instance
(page content is corrupted)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4614)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4516)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4455)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:919)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:951)
~[ignite-core-2.4.0.jar:2.4.0]
        ... 6 more

2018-06-13 00:23:35.596 [sys-stripe-6-#7] ERROR
org.apache.ignite.internal.processors.cache.GridCacheIoManager - Failed to
process message [senderId=1dedb701-ffd1-4937-b202-7326367a92fb,
messageType=class
o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicFullUpdateRequest]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:963)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:974)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:834)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessageProcessed(GridCacheIoManager.java:1099)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1072)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
[ignite-core-2.4.0.jar:2.4.0]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Item not found: 3
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4614)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4516)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4455)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:919)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:951)
~[ignite-core-2.4.0.jar:2.4.0]
        ... 17 more



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/



--
Best regards,
Andrey V. Mashenkov
Regards,
Andrew.
Oleks K Oleks K
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

I'm using key/value API.
Should I define index for the key explicitly?
This sounds strange... Can you give a sample how I can do this in xml please?

Here is one of my cache configurations.
Actually I'm storing UUID per UUID here: IgniteCache<UUID, UUID>

<bean class="org.apache.ignite.configuration.CacheConfiguration">
                    <property name="name" value="RefreshToken"/>
                    <property name="dataRegionName" value="auth_durable_region"/>
                    <property name="atomicityMode" value="ATOMIC"/>
                    <property name="writeSynchronizationMode" value="FULL_ASYNC"/>
                    <property name="cacheMode" value="PARTITIONED"/>
                    <property name="backups" value="0"/>
                    <property name="eagerTtl" value="true"/> <!-- true: clean expired items in background, false: on next access -->
                </bean>


On Wed, Jun 13, 2018 at 12:18 PM, Andrey Mashenkov <[hidden email]> wrote:
Hi,

Possibly, it is a bug in partition eviction optimization. Ignite can skip partition eviction procedure and remove partition instantly if there is no indexes.

If it is so, you can try the latest ignite-2.5 version [1] or
as a workaround you can add index via configuring QueryEntity or via setting cacheCfg.setIndexTypes().



On Wed, Jun 13, 2018 at 1:22 AM, Oleks K <[hidden email]> wrote:
Hi guys,

I got similar errors in 2.4.0

First:

org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
  --> Caused by: java.lang.IllegalStateException: Failed to get page IO
instance (page content is corrupted)

Then lots of:

org.apache.ignite.IgniteException: Runtime failure on bounds
  --> Caused by: java.lang.IllegalStateException: Item not found: 3

This was reproduced when I started and stopped server nodes under the load
Topology: 2 server and 2 client nodes
Java: 1.8.0_162
OS: Windows Server 2012 R2 6.3 amd64

Cache config:
<bean class="org.apache.ignite.configuration.CacheConfiguration">
                    <property name="name" value="RefreshToken"/>
                    <property name="dataRegionName"
value="auth_durable_region"/>
                    <property name="atomicityMode" value="ATOMIC"/>
                    <property name="writeSynchronizationMode"
value="FULL_ASYNC"/>
                    <property name="cacheMode" value="PARTITIONED"/>
                    <property name="backups" value="0"/>
                    <property name="eagerTtl" value="true"/>
                </bean>

Ignite team, can you comment on this please?
How critical is the issue? What is the impact?
Any workarounds? Fix planned?

2018-06-13 00:22:30.978 [exchange-worker-#42] INFO
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander
- Starting rebalancing [mode=ASYNC,
fromNode=bdddfe24-aab3-46fa-9452-efe933783adb, partitionsCount=787,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], updateSeq=12]
2018-06-13 00:22:31.594 [ttl-cleanup-worker-#52] ERROR
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager
- Runtime error caught during grid runnable execution: GridWorker
[name=ttl-cleanup-worker, igniteInstanceName=null, finished=false,
hashCode=473353699, interrupted=false, runner=ttl-cleanup-worker-#52]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:963)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:974)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:129)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
[ignite-core-2.4.0.jar:2.4.0]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Failed to get page IO instance
(page content is corrupted)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4614)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4516)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4455)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:919)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:951)
~[ignite-core-2.4.0.jar:2.4.0]
        ... 6 more

2018-06-13 00:23:35.596 [sys-stripe-6-#7] ERROR
org.apache.ignite.internal.processors.cache.GridCacheIoManager - Failed to
process message [senderId=1dedb701-ffd1-4937-b202-7326367a92fb,
messageType=class
o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicFullUpdateRequest]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:963)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:974)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:834)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessageProcessed(GridCacheIoManager.java:1099)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1072)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
[ignite-core-2.4.0.jar:2.4.0]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Item not found: 3
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4614)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4516)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4455)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:919)
~[ignite-core-2.4.0.jar:2.4.0]
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:951)
~[ignite-core-2.4.0.jar:2.4.0]
        ... 17 more



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/



--
Best regards,
Andrey V. Mashenkov

dkarachentsev dkarachentsev
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

It would be better to upgrade to 2.5, where it is fixed.
But if you want to overcome this issue in your's version, you need to add
ignite-indexing dependency to your classpath and configure SQL indexes. For
example [1], just modify it to work with Spring in XML:
<property name="indexedTypes">
                        <list>
                            <value>org.your.KeyObject</value>
                            <value>org.your.ValueObject</value>
                        </list>
                    </property>

[1]
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-registering-indexed-types

Thanks!
-Dmitry



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Oleks K Oleks K
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Upgraded to 2.5.0 and didn't get such error so far..
Thanks!

On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <[hidden email]> wrote:
It would be better to upgrade to 2.5, where it is fixed.
But if you want to overcome this issue in your's version, you need to add
ignite-indexing dependency to your classpath and configure SQL indexes. For
example [1], just modify it to work with Spring in XML:
<property name="indexedTypes">
                        <list>
                            <value>org.your.KeyObject</value>
                            <value>org.your.ValueObject</value>
                        </list>
                    </property>

[1]
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-registering-indexed-types

Thanks!
-Dmitry

Oleks K Oleks K
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi Team,

Issue is still there in 2.5.0

Steps to reproduce:
1) start 2 servers + 2 clients topology
2) start load testing on client nodes
3) stop server 1
4) start server 1
5) stop server 1 again when rebalancing is in progress
=> and we got data corrupted here, see error below
=> we were not able to restart Ignite cluster after that and need to perform data folders cleanup...

2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR  - Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]]]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137) [ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) [ignite-core-2.5.0.jar:2.5.0]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Item not found: 2
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.5.0.j

BR, Oleksandr

On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <[hidden email]> wrote:
Upgraded to 2.5.0 and didn't get such error so far..
Thanks!

On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <[hidden email]> wrote:
It would be better to upgrade to 2.5, where it is fixed.
But if you want to overcome this issue in your's version, you need to add
ignite-indexing dependency to your classpath and configure SQL indexes. For
example [1], just modify it to work with Spring in XML:
<property name="indexedTypes">
                        <list>
                            <value>org.your.KeyObject</value>
                            <value>org.your.ValueObject</value>
                        </list>
                    </property>

[1]
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-registering-indexed-types

Thanks!
-Dmitry


Andrew Mashenkov Andrew Mashenkov
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi,

We've found and fixed few issues related to ExpiryPolicy usage.
Most likely, your issue is [1] and it is planned to ignite 2.6 release.



On Fri, Jun 22, 2018 at 8:43 AM Olexandr K <[hidden email]> wrote:
Hi Team,

Issue is still there in 2.5.0

Steps to reproduce:
1) start 2 servers + 2 clients topology
2) start load testing on client nodes
3) stop server 1
4) start server 1
5) stop server 1 again when rebalancing is in progress
=> and we got data corrupted here, see error below
=> we were not able to restart Ignite cluster after that and need to perform data folders cleanup...

2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR  - Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]]]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137) [ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) [ignite-core-2.5.0.jar:2.5.0]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Item not found: 2
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.5.0.j

BR, Oleksandr

On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <[hidden email]> wrote:
Upgraded to 2.5.0 and didn't get such error so far..
Thanks!

On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <[hidden email]> wrote:
It would be better to upgrade to 2.5, where it is fixed.
But if you want to overcome this issue in your's version, you need to add
ignite-indexing dependency to your classpath and configure SQL indexes. For
example [1], just modify it to work with Spring in XML:
<property name="indexedTypes">
                        <list>
                            <value>org.your.KeyObject</value>
                            <value>org.your.ValueObject</value>
                        </list>
                    </property>

[1]
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-registering-indexed-types

Thanks!
-Dmitry




--
Best regards,
Andrey V. Mashenkov
Regards,
Andrew.
Oleks K Oleks K
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi Andrey,

This is a critical bug.. bouncing of server node in not-a-right-time causes a catastrophe.
This mean no availability in fact - I had to clean data folders to start my cluster after that
 
BR, Oleksandr


On Fri, Jun 22, 2018 at 4:06 PM, Andrey Mashenkov <[hidden email]> wrote:
Hi,

We've found and fixed few issues related to ExpiryPolicy usage.
Most likely, your issue is [1] and it is planned to ignite 2.6 release.



On Fri, Jun 22, 2018 at 8:43 AM Olexandr K <[hidden email]> wrote:
Hi Team,

Issue is still there in 2.5.0

Steps to reproduce:
1) start 2 servers + 2 clients topology
2) start load testing on client nodes
3) stop server 1
4) start server 1
5) stop server 1 again when rebalancing is in progress
=> and we got data corrupted here, see error below
=> we were not able to restart Ignite cluster after that and need to perform data folders cleanup...

2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR  - Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]]]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137) [ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) [ignite-core-2.5.0.jar:2.5.0]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Item not found: 2
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.5.0.j

BR, Oleksandr

On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <[hidden email]> wrote:
Upgraded to 2.5.0 and didn't get such error so far..
Thanks!

On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <[hidden email]> wrote:
It would be better to upgrade to 2.5, where it is fixed.
But if you want to overcome this issue in your's version, you need to add
ignite-indexing dependency to your classpath and configure SQL indexes. For
example [1], just modify it to work with Spring in XML:
<property name="indexedTypes">
                        <list>
                            <value>org.your.KeyObject</value>
                            <value>org.your.ValueObject</value>
                        </list>
                    </property>

[1]
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-registering-indexed-types

Thanks!
-Dmitry




--
Best regards,
Andrey V. Mashenkov

Andrew Mashenkov Andrew Mashenkov
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi Oleg,

The issue you mentioned IGNITE-8659 [1] is caused by IGNITE-5874 [2] that will not a part of ignite-2.6 release.
For now, 'ExpiryPolicy with persistence' is totally broken and all it's fixes are planned to the next 2.7 release.



On Tue, Jun 26, 2018 at 11:26 PM Olexandr K <[hidden email]> wrote:
Hi Andrey,

This is a critical bug.. bouncing of server node in not-a-right-time causes a catastrophe.
This mean no availability in fact - I had to clean data folders to start my cluster after that
 
BR, Oleksandr


On Fri, Jun 22, 2018 at 4:06 PM, Andrey Mashenkov <[hidden email]> wrote:
Hi,

We've found and fixed few issues related to ExpiryPolicy usage.
Most likely, your issue is [1] and it is planned to ignite 2.6 release.



On Fri, Jun 22, 2018 at 8:43 AM Olexandr K <[hidden email]> wrote:
Hi Team,

Issue is still there in 2.5.0

Steps to reproduce:
1) start 2 servers + 2 clients topology
2) start load testing on client nodes
3) stop server 1
4) start server 1
5) stop server 1 again when rebalancing is in progress
=> and we got data corrupted here, see error below
=> we were not able to restart Ignite cluster after that and need to perform data folders cleanup...

2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR  - Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]]]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137) [ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) [ignite-core-2.5.0.jar:2.5.0]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Item not found: 2
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.5.0.j

BR, Oleksandr

On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <[hidden email]> wrote:
Upgraded to 2.5.0 and didn't get such error so far..
Thanks!

On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <[hidden email]> wrote:
It would be better to upgrade to 2.5, where it is fixed.
But if you want to overcome this issue in your's version, you need to add
ignite-indexing dependency to your classpath and configure SQL indexes. For
example [1], just modify it to work with Spring in XML:
<property name="indexedTypes">
                        <list>
                            <value>org.your.KeyObject</value>
                            <value>org.your.ValueObject</value>
                        </list>
                    </property>

[1]
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-registering-indexed-types

Thanks!
-Dmitry




--
Best regards,
Andrey V. Mashenkov



--
Best regards,
Andrey V. Mashenkov
Regards,
Andrew.
Oleks K Oleks K
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi Andrey,

Thanks for clarifying this.
We have just a single persistent cache and I reworked the code to get rid of expiration policy.
All our non-persistent caches have expiration policy but this should not be a problem, right?

BR, Oleksandr

On Thu, Jun 28, 2018 at 8:37 PM, Andrey Mashenkov <[hidden email]> wrote:
Hi Oleg,

The issue you mentioned IGNITE-8659 [1] is caused by IGNITE-5874 [2] that will not a part of ignite-2.6 release.
For now, 'ExpiryPolicy with persistence' is totally broken and all it's fixes are planned to the next 2.7 release.



On Tue, Jun 26, 2018 at 11:26 PM Olexandr K <[hidden email]> wrote:
Hi Andrey,

This is a critical bug.. bouncing of server node in not-a-right-time causes a catastrophe.
This mean no availability in fact - I had to clean data folders to start my cluster after that
 
BR, Oleksandr


On Fri, Jun 22, 2018 at 4:06 PM, Andrey Mashenkov <[hidden email]> wrote:
Hi,

We've found and fixed few issues related to ExpiryPolicy usage.
Most likely, your issue is [1] and it is planned to ignite 2.6 release.



On Fri, Jun 22, 2018 at 8:43 AM Olexandr K <[hidden email]> wrote:
Hi Team,

Issue is still there in 2.5.0

Steps to reproduce:
1) start 2 servers + 2 clients topology
2) start load testing on client nodes
3) stop server 1
4) start server 1
5) stop server 1 again when rebalancing is in progress
=> and we got data corrupted here, see error below
=> we were not able to restart Ignite cluster after that and need to perform data folders cleanup...

2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR  - Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]]]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137) [ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) [ignite-core-2.5.0.jar:2.5.0]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Item not found: 2
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.5.0.j

BR, Oleksandr

On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <[hidden email]> wrote:
Upgraded to 2.5.0 and didn't get such error so far..
Thanks!

On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <[hidden email]> wrote:
It would be better to upgrade to 2.5, where it is fixed.
But if you want to overcome this issue in your's version, you need to add
ignite-indexing dependency to your classpath and configure SQL indexes. For
example [1], just modify it to work with Spring in XML:
<property name="indexedTypes">
                        <list>
                            <value>org.your.KeyObject</value>
                            <value>org.your.ValueObject</value>
                        </list>
                    </property>

[1]
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-registering-indexed-types

Thanks!
-Dmitry




--
Best regards,
Andrey V. Mashenkov



--
Best regards,
Andrey V. Mashenkov

Andrew Mashenkov Andrew Mashenkov
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi Oleg,

Yes, page corruption issues shouldn't happened when persistence is disabled.
Please, let us know it you will face one. 


On Fri, Jun 29, 2018 at 1:56 AM Olexandr K <[hidden email]> wrote:
Hi Andrey,

Thanks for clarifying this.
We have just a single persistent cache and I reworked the code to get rid of expiration policy.
All our non-persistent caches have expiration policy but this should not be a problem, right?

BR, Oleksandr

On Thu, Jun 28, 2018 at 8:37 PM, Andrey Mashenkov <[hidden email]> wrote:
Hi Oleg,

The issue you mentioned IGNITE-8659 [1] is caused by IGNITE-5874 [2] that will not a part of ignite-2.6 release.
For now, 'ExpiryPolicy with persistence' is totally broken and all it's fixes are planned to the next 2.7 release.



On Tue, Jun 26, 2018 at 11:26 PM Olexandr K <[hidden email]> wrote:
Hi Andrey,

This is a critical bug.. bouncing of server node in not-a-right-time causes a catastrophe.
This mean no availability in fact - I had to clean data folders to start my cluster after that
 
BR, Oleksandr


On Fri, Jun 22, 2018 at 4:06 PM, Andrey Mashenkov <[hidden email]> wrote:
Hi,

We've found and fixed few issues related to ExpiryPolicy usage.
Most likely, your issue is [1] and it is planned to ignite 2.6 release.



On Fri, Jun 22, 2018 at 8:43 AM Olexandr K <[hidden email]> wrote:
Hi Team,

Issue is still there in 2.5.0

Steps to reproduce:
1) start 2 servers + 2 clients topology
2) start load testing on client nodes
3) stop server 1
4) start server 1
5) stop server 1 again when rebalancing is in progress
=> and we got data corrupted here, see error below
=> we were not able to restart Ignite cluster after that and need to perform data folders cleanup...

2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR  - Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]]]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137) [ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) [ignite-core-2.5.0.jar:2.5.0]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Item not found: 2
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.5.0.j

BR, Oleksandr

On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <[hidden email]> wrote:
Upgraded to 2.5.0 and didn't get such error so far..
Thanks!

On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <[hidden email]> wrote:
It would be better to upgrade to 2.5, where it is fixed.
But if you want to overcome this issue in your's version, you need to add
ignite-indexing dependency to your classpath and configure SQL indexes. For
example [1], just modify it to work with Spring in XML:
<property name="indexedTypes">
                        <list>
                            <value>org.your.KeyObject</value>
                            <value>org.your.ValueObject</value>
                        </list>
                    </property>

[1]
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-registering-indexed-types

Thanks!
-Dmitry




--
Best regards,
Andrey V. Mashenkov



--
Best regards,
Andrey V. Mashenkov



--
Best regards,
Andrey V. Mashenkov
Regards,
Andrew.
Oleks K Oleks K
Reply | Threaded
Open this post in threaded view
|

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi guys,

are you still planning to release 2.6.0 as patch release for 2.5.0?

BR, Oleksandr

On Fri, Jun 29, 2018 at 11:43 AM, Andrey Mashenkov <[hidden email]> wrote:
Hi Oleg,

Yes, page corruption issues shouldn't happened when persistence is disabled.
Please, let us know it you will face one. 


On Fri, Jun 29, 2018 at 1:56 AM Olexandr K <[hidden email]> wrote:
Hi Andrey,

Thanks for clarifying this.
We have just a single persistent cache and I reworked the code to get rid of expiration policy.
All our non-persistent caches have expiration policy but this should not be a problem, right?

BR, Oleksandr

On Thu, Jun 28, 2018 at 8:37 PM, Andrey Mashenkov <[hidden email]> wrote:
Hi Oleg,

The issue you mentioned IGNITE-8659 [1] is caused by IGNITE-5874 [2] that will not a part of ignite-2.6 release.
For now, 'ExpiryPolicy with persistence' is totally broken and all it's fixes are planned to the next 2.7 release.



On Tue, Jun 26, 2018 at 11:26 PM Olexandr K <[hidden email]> wrote:
Hi Andrey,

This is a critical bug.. bouncing of server node in not-a-right-time causes a catastrophe.
This mean no availability in fact - I had to clean data folders to start my cluster after that
 
BR, Oleksandr


On Fri, Jun 22, 2018 at 4:06 PM, Andrey Mashenkov <[hidden email]> wrote:
Hi,

We've found and fixed few issues related to ExpiryPolicy usage.
Most likely, your issue is [1] and it is planned to ignite 2.6 release.



On Fri, Jun 22, 2018 at 8:43 AM Olexandr K <[hidden email]> wrote:
Hi Team,

Issue is still there in 2.5.0

Steps to reproduce:
1) start 2 servers + 2 clients topology
2) start load testing on client nodes
3) stop server 1
4) start server 1
5) stop server 1 again when rebalancing is in progress
=> and we got data corrupted here, see error below
=> we were not able to restart Ignite cluster after that and need to perform data folders cleanup...

2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR  - Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]]]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137) [ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) [ignite-core-2.5.0.jar:2.5.0]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Item not found: 2
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150) ~[ignite-core-2.5.0.jar:2.5.0]
    at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.5.0.j

BR, Oleksandr

On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <[hidden email]> wrote:
Upgraded to 2.5.0 and didn't get such error so far..
Thanks!

On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <[hidden email]> wrote:
It would be better to upgrade to 2.5, where it is fixed.
But if you want to overcome this issue in your's version, you need to add
ignite-indexing dependency to your classpath and configure SQL indexes. For
example [1], just modify it to work with Spring in XML:
<property name="indexedTypes">
                        <list>
                            <value>org.your.KeyObject</value>
                            <value>org.your.ValueObject</value>
                        </list>
                    </property>

[1]
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-registering-indexed-types

Thanks!
-Dmitry




--
Best regards,
Andrey V. Mashenkov



--
Best regards,
Andrey V. Mashenkov



--
Best regards,
Andrey V. Mashenkov

12