Item not found, B+Tree is corrupted and critical error detected after add nodes to baseline topology

classic Classic list List threaded Threaded
5 messages Options
Steven Zheng Steven Zheng
Reply | Threaded
Open this post in threaded view
|

Item not found, B+Tree is corrupted and critical error detected after add nodes to baseline topology

Hi community,
Currently I have 25 nodes in my ignite cluster and all of them were added into the baseline, and I was trying to add another 5 nodes into it. The data in my cluster is about 8TB and the persistence is enabled.
At first I start all the 5 nodes; then execute in the command line:
```
bin/control.sh --baseline add ${my_node_id}
```
Meanwhile, there is still read/write workloads on the cluster.

After a few minutes, a critical error detected , several nodes crashed and emits the logs like this(json format):
```
{
  "message": "Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]]]",
  "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
  "thrown": {
    "localizedMessage": "B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]",
    "message": "B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]",
    "commonElementCount": 0,
    "cause": {
      "localizedMessage": "java.lang.IllegalStateException: Item not found: 27",
      "message": "java.lang.IllegalStateException: Item not found: 27",
      "commonElementCount": 21,
      "cause": {
        "localizedMessage": "Item not found: 27",
        "message": "Item not found: 27",
        "commonElementCount": 21,
        "name": "java.lang.IllegalStateException",
        "extendedStackTrace": [{
            "line": 351,
            "method": "findIndirectItemIndex",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 459,
            "method": "getDataOffset",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 501,
            "method": "readPayload",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 325,
            "method": "readIncomplete",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 261,
            "method": "doInitFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 160,
            "method": "initFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 131,
            "method": "initFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 277,
            "method": "createRow0",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 262,
            "method": "createRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 154,
            "method": "getLookupRow",
            "exact": false,
            "file": "AbstractH2ExtrasLeafIO.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 36,
            "method": "getLookupRow",
            "exact": false,
            "file": "AbstractH2ExtrasLeafIO.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 339,
            "method": "getRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 63,
            "method": "getRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5218,
            "method": "getRow",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 418,
            "method": "compare",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 63,
            "method": "compare",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5202,
            "method": "compare",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5121,
            "method": "findInsertionPoint",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 95,
            "method": "access$1100",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 299,
            "method": "run0",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5714,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 279,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5700,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 169,
            "method": "readPage",
            "exact": false,
            "file": "PageHandler.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 364,
            "method": "read",
            "exact": false,
            "file": "DataStructure.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.DataStructure",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5901,
            "method": "read",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2064,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2083,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2083,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 1994,
            "method": "doRemove",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          }
        ]
      },
      "name": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException",
      "extendedStackTrace": [{
          "line": 290,
          "method": "doInitFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 160,
          "method": "initFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 131,
          "method": "initFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 277,
          "method": "createRow0",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 262,
          "method": "createRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 154,
          "method": "getLookupRow",
          "exact": false,
          "file": "AbstractH2ExtrasLeafIO.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 36,
          "method": "getLookupRow",
          "exact": false,
          "file": "AbstractH2ExtrasLeafIO.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 339,
          "method": "getRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 63,
          "method": "getRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5218,
          "method": "getRow",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 418,
          "method": "compare",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 63,
          "method": "compare",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5202,
          "method": "compare",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5121,
          "method": "findInsertionPoint",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 95,
          "method": "access$1100",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 299,
          "method": "run0",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5714,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 279,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5700,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 169,
          "method": "readPage",
          "exact": false,
          "file": "PageHandler.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 364,
          "method": "read",
          "exact": false,
          "file": "DataStructure.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.DataStructure",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5901,
          "method": "read",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2064,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2083,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2083,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 1994,
          "method": "doRemove",
          "exact": true,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        }
      ]
    },
    "name": "org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException",
    "extendedStackTrace": [{
        "line": 678,
        "method": "corruptedTreeException",
        "exact": true,
        "file": "H2Tree.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2029,
        "method": "doRemove",
        "exact": true,
        "file": "BPlusTree.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1825,
        "method": "removex",
        "exact": true,
        "file": "BPlusTree.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 453,
        "method": "removex",
        "exact": true,
        "file": "H2TreeIndex.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 805,
        "method": "remove",
        "exact": true,
        "file": "GridH2Table.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.opt.GridH2Table",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 404,
        "method": "remove",
        "exact": true,
        "file": "IgniteH2Indexing.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2828,
        "method": "remove",
        "exact": true,
        "file": "GridQueryProcessor.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.GridQueryProcessor",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 440,
        "method": "remove",
        "exact": true,
        "file": "GridCacheQueryManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2695,
        "method": "finishRemove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2672,
        "method": "remove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2439,
        "method": "remove",
        "exact": true,
        "file": "GridCacheOffheapManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 615,
        "method": "remove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 4403,
        "method": "removeValue",
        "exact": true,
        "file": "GridCacheMapEntry.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.GridCacheMapEntry",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 678,
        "method": "clearInternal",
        "exact": true,
        "file": "GridDhtCacheEntry.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1175,
        "method": "clearAll",
        "exact": true,
        "file": "GridDhtLocalPartition.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 938,
        "method": "tryClear",
        "exact": true,
        "file": "GridDhtLocalPartition.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 417,
        "method": "run",
        "exact": true,
        "file": "PartitionsEvictManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 7061,
        "method": "wrapThreadLoader",
        "exact": true,
        "file": "IgniteUtils.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.util.IgniteUtils",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 827,
        "method": "body",
        "exact": true,
        "file": "GridClosureProcessor.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.closure.GridClosureProcessor$1",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 120,
        "method": "run",
        "exact": true,
        "file": "GridWorker.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.util.worker.GridWorker",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1149,
        "method": "runWorker",
        "exact": true,
        "file": "ThreadPoolExecutor.java",
        "location": "?",
        "class": "java.util.concurrent.ThreadPoolExecutor",
        "version": "1.8.0_221"
      },
      {
        "line": 624,
        "method": "run",
        "exact": true,
        "file": "ThreadPoolExecutor.java",
        "location": "?",
        "class": "java.util.concurrent.ThreadPoolExecutor$Worker",
        "version": "1.8.0_221"
      },
      {
        "line": 748,
        "method": "run",
        "exact": true,
        "file": "Thread.java",
        "location": "?",
        "class": "java.lang.Thread",
        "version": "1.8.0_221"
      }
    ]
  },
  "level": "ERROR"
}
```

Can anyone kindly help? Thanks!

Best Regards,
---------------------------------------------------------
Steven Zheng
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Item not found, B+Tree is corrupted and critical error detected after add nodes to baseline topology

Hi,

Can you attach logs in normal format? It's really hard to read it. Also, please attach full logs from nodes, not only the stacktrace.

Thanks,
Evgenii

вт, 25 авг. 2020 г. в 19:27, Steven Zheng <[hidden email]>:
Hi community,
Currently I have 25 nodes in my ignite cluster and all of them were added into the baseline, and I was trying to add another 5 nodes into it. The data in my cluster is about 8TB and the persistence is enabled.
At first I start all the 5 nodes; then execute in the command line:
```
bin/control.sh --baseline add ${my_node_id}
```
Meanwhile, there is still read/write workloads on the cluster.

After a few minutes, a critical error detected , several nodes crashed and emits the logs like this(json format):
```
{
  "message": "Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]]]",
  "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
  "thrown": {
    "localizedMessage": "B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]",
    "message": "B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]",
    "commonElementCount": 0,
    "cause": {
      "localizedMessage": "java.lang.IllegalStateException: Item not found: 27",
      "message": "java.lang.IllegalStateException: Item not found: 27",
      "commonElementCount": 21,
      "cause": {
        "localizedMessage": "Item not found: 27",
        "message": "Item not found: 27",
        "commonElementCount": 21,
        "name": "java.lang.IllegalStateException",
        "extendedStackTrace": [{
            "line": 351,
            "method": "findIndirectItemIndex",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 459,
            "method": "getDataOffset",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 501,
            "method": "readPayload",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 325,
            "method": "readIncomplete",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 261,
            "method": "doInitFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 160,
            "method": "initFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 131,
            "method": "initFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 277,
            "method": "createRow0",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 262,
            "method": "createRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 154,
            "method": "getLookupRow",
            "exact": false,
            "file": "AbstractH2ExtrasLeafIO.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 36,
            "method": "getLookupRow",
            "exact": false,
            "file": "AbstractH2ExtrasLeafIO.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 339,
            "method": "getRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 63,
            "method": "getRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5218,
            "method": "getRow",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 418,
            "method": "compare",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 63,
            "method": "compare",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5202,
            "method": "compare",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5121,
            "method": "findInsertionPoint",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 95,
            "method": "access$1100",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 299,
            "method": "run0",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5714,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 279,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5700,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 169,
            "method": "readPage",
            "exact": false,
            "file": "PageHandler.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 364,
            "method": "read",
            "exact": false,
            "file": "DataStructure.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.DataStructure",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5901,
            "method": "read",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2064,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2083,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2083,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 1994,
            "method": "doRemove",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          }
        ]
      },
      "name": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException",
      "extendedStackTrace": [{
          "line": 290,
          "method": "doInitFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 160,
          "method": "initFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 131,
          "method": "initFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 277,
          "method": "createRow0",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 262,
          "method": "createRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 154,
          "method": "getLookupRow",
          "exact": false,
          "file": "AbstractH2ExtrasLeafIO.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 36,
          "method": "getLookupRow",
          "exact": false,
          "file": "AbstractH2ExtrasLeafIO.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 339,
          "method": "getRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 63,
          "method": "getRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5218,
          "method": "getRow",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 418,
          "method": "compare",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 63,
          "method": "compare",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5202,
          "method": "compare",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5121,
          "method": "findInsertionPoint",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 95,
          "method": "access$1100",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 299,
          "method": "run0",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5714,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 279,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5700,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 169,
          "method": "readPage",
          "exact": false,
          "file": "PageHandler.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 364,
          "method": "read",
          "exact": false,
          "file": "DataStructure.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.DataStructure",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5901,
          "method": "read",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2064,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2083,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2083,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 1994,
          "method": "doRemove",
          "exact": true,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        }
      ]
    },
    "name": "org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException",
    "extendedStackTrace": [{
        "line": 678,
        "method": "corruptedTreeException",
        "exact": true,
        "file": "H2Tree.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2029,
        "method": "doRemove",
        "exact": true,
        "file": "BPlusTree.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1825,
        "method": "removex",
        "exact": true,
        "file": "BPlusTree.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 453,
        "method": "removex",
        "exact": true,
        "file": "H2TreeIndex.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 805,
        "method": "remove",
        "exact": true,
        "file": "GridH2Table.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.opt.GridH2Table",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 404,
        "method": "remove",
        "exact": true,
        "file": "IgniteH2Indexing.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2828,
        "method": "remove",
        "exact": true,
        "file": "GridQueryProcessor.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.GridQueryProcessor",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 440,
        "method": "remove",
        "exact": true,
        "file": "GridCacheQueryManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2695,
        "method": "finishRemove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2672,
        "method": "remove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2439,
        "method": "remove",
        "exact": true,
        "file": "GridCacheOffheapManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 615,
        "method": "remove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 4403,
        "method": "removeValue",
        "exact": true,
        "file": "GridCacheMapEntry.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.GridCacheMapEntry",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 678,
        "method": "clearInternal",
        "exact": true,
        "file": "GridDhtCacheEntry.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1175,
        "method": "clearAll",
        "exact": true,
        "file": "GridDhtLocalPartition.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 938,
        "method": "tryClear",
        "exact": true,
        "file": "GridDhtLocalPartition.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 417,
        "method": "run",
        "exact": true,
        "file": "PartitionsEvictManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 7061,
        "method": "wrapThreadLoader",
        "exact": true,
        "file": "IgniteUtils.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.util.IgniteUtils",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 827,
        "method": "body",
        "exact": true,
        "file": "GridClosureProcessor.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.closure.GridClosureProcessor$1",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 120,
        "method": "run",
        "exact": true,
        "file": "GridWorker.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.util.worker.GridWorker",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1149,
        "method": "runWorker",
        "exact": true,
        "file": "ThreadPoolExecutor.java",
        "location": "?",
        "class": "java.util.concurrent.ThreadPoolExecutor",
        "version": "1.8.0_221"
      },
      {
        "line": 624,
        "method": "run",
        "exact": true,
        "file": "ThreadPoolExecutor.java",
        "location": "?",
        "class": "java.util.concurrent.ThreadPoolExecutor$Worker",
        "version": "1.8.0_221"
      },
      {
        "line": 748,
        "method": "run",
        "exact": true,
        "file": "Thread.java",
        "location": "?",
        "class": "java.lang.Thread",
        "version": "1.8.0_221"
      }
    ]
  },
  "level": "ERROR"
}
```

Can anyone kindly help? Thanks!

Best Regards,
---------------------------------------------------------
Steven Zheng
Steven Zheng Steven Zheng
Reply | Threaded
Open this post in threaded view
|

Re: Item not found, B+Tree is corrupted and critical error detected after add nodes to baseline topology

Thanks evgenii, but origin log is not been saved and I can only provide json format logs. 
Best Regards,
---------------------------------------------------------
Steven Zheng


Evgenii Zhuravlev <[hidden email]> 于2020年8月28日周五 上午6:45写道:
Hi,

Can you attach logs in normal format? It's really hard to read it. Also, please attach full logs from nodes, not only the stacktrace.

Thanks,
Evgenii

вт, 25 авг. 2020 г. в 19:27, Steven Zheng <[hidden email]>:
Hi community,
Currently I have 25 nodes in my ignite cluster and all of them were added into the baseline, and I was trying to add another 5 nodes into it. The data in my cluster is about 8TB and the persistence is enabled.
At first I start all the 5 nodes; then execute in the command line:
```
bin/control.sh --baseline add ${my_node_id}
```
Meanwhile, there is still read/write workloads on the cluster.

After a few minutes, a critical error detected , several nodes crashed and emits the logs like this(json format):
```
{
  "message": "Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]]]",
  "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
  "thrown": {
    "localizedMessage": "B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]",
    "message": "B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]",
    "commonElementCount": 0,
    "cause": {
      "localizedMessage": "java.lang.IllegalStateException: Item not found: 27",
      "message": "java.lang.IllegalStateException: Item not found: 27",
      "commonElementCount": 21,
      "cause": {
        "localizedMessage": "Item not found: 27",
        "message": "Item not found: 27",
        "commonElementCount": 21,
        "name": "java.lang.IllegalStateException",
        "extendedStackTrace": [{
            "line": 351,
            "method": "findIndirectItemIndex",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 459,
            "method": "getDataOffset",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 501,
            "method": "readPayload",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 325,
            "method": "readIncomplete",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 261,
            "method": "doInitFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 160,
            "method": "initFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 131,
            "method": "initFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 277,
            "method": "createRow0",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 262,
            "method": "createRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 154,
            "method": "getLookupRow",
            "exact": false,
            "file": "AbstractH2ExtrasLeafIO.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 36,
            "method": "getLookupRow",
            "exact": false,
            "file": "AbstractH2ExtrasLeafIO.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 339,
            "method": "getRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 63,
            "method": "getRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5218,
            "method": "getRow",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 418,
            "method": "compare",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 63,
            "method": "compare",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5202,
            "method": "compare",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5121,
            "method": "findInsertionPoint",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 95,
            "method": "access$1100",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 299,
            "method": "run0",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5714,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 279,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5700,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 169,
            "method": "readPage",
            "exact": false,
            "file": "PageHandler.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 364,
            "method": "read",
            "exact": false,
            "file": "DataStructure.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.DataStructure",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5901,
            "method": "read",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2064,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2083,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2083,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 1994,
            "method": "doRemove",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          }
        ]
      },
      "name": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException",
      "extendedStackTrace": [{
          "line": 290,
          "method": "doInitFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 160,
          "method": "initFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 131,
          "method": "initFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 277,
          "method": "createRow0",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 262,
          "method": "createRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 154,
          "method": "getLookupRow",
          "exact": false,
          "file": "AbstractH2ExtrasLeafIO.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 36,
          "method": "getLookupRow",
          "exact": false,
          "file": "AbstractH2ExtrasLeafIO.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 339,
          "method": "getRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 63,
          "method": "getRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5218,
          "method": "getRow",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 418,
          "method": "compare",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 63,
          "method": "compare",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5202,
          "method": "compare",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5121,
          "method": "findInsertionPoint",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 95,
          "method": "access$1100",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 299,
          "method": "run0",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5714,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 279,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5700,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 169,
          "method": "readPage",
          "exact": false,
          "file": "PageHandler.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 364,
          "method": "read",
          "exact": false,
          "file": "DataStructure.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.DataStructure",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5901,
          "method": "read",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2064,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2083,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2083,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 1994,
          "method": "doRemove",
          "exact": true,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        }
      ]
    },
    "name": "org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException",
    "extendedStackTrace": [{
        "line": 678,
        "method": "corruptedTreeException",
        "exact": true,
        "file": "H2Tree.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2029,
        "method": "doRemove",
        "exact": true,
        "file": "BPlusTree.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1825,
        "method": "removex",
        "exact": true,
        "file": "BPlusTree.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 453,
        "method": "removex",
        "exact": true,
        "file": "H2TreeIndex.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 805,
        "method": "remove",
        "exact": true,
        "file": "GridH2Table.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.opt.GridH2Table",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 404,
        "method": "remove",
        "exact": true,
        "file": "IgniteH2Indexing.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2828,
        "method": "remove",
        "exact": true,
        "file": "GridQueryProcessor.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.GridQueryProcessor",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 440,
        "method": "remove",
        "exact": true,
        "file": "GridCacheQueryManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2695,
        "method": "finishRemove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2672,
        "method": "remove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2439,
        "method": "remove",
        "exact": true,
        "file": "GridCacheOffheapManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 615,
        "method": "remove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 4403,
        "method": "removeValue",
        "exact": true,
        "file": "GridCacheMapEntry.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.GridCacheMapEntry",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 678,
        "method": "clearInternal",
        "exact": true,
        "file": "GridDhtCacheEntry.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1175,
        "method": "clearAll",
        "exact": true,
        "file": "GridDhtLocalPartition.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 938,
        "method": "tryClear",
        "exact": true,
        "file": "GridDhtLocalPartition.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 417,
        "method": "run",
        "exact": true,
        "file": "PartitionsEvictManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 7061,
        "method": "wrapThreadLoader",
        "exact": true,
        "file": "IgniteUtils.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.util.IgniteUtils",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 827,
        "method": "body",
        "exact": true,
        "file": "GridClosureProcessor.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.closure.GridClosureProcessor$1",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 120,
        "method": "run",
        "exact": true,
        "file": "GridWorker.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.util.worker.GridWorker",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1149,
        "method": "runWorker",
        "exact": true,
        "file": "ThreadPoolExecutor.java",
        "location": "?",
        "class": "java.util.concurrent.ThreadPoolExecutor",
        "version": "1.8.0_221"
      },
      {
        "line": 624,
        "method": "run",
        "exact": true,
        "file": "ThreadPoolExecutor.java",
        "location": "?",
        "class": "java.util.concurrent.ThreadPoolExecutor$Worker",
        "version": "1.8.0_221"
      },
      {
        "line": 748,
        "method": "run",
        "exact": true,
        "file": "Thread.java",
        "location": "?",
        "class": "java.lang.Thread",
        "version": "1.8.0_221"
      }
    ]
  },
  "level": "ERROR"
}
```

Can anyone kindly help? Thanks!

Best Regards,
---------------------------------------------------------
Steven Zheng
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Item not found, B+Tree is corrupted and critical error detected after add nodes to baseline topology

Can you export logs from your system in a different format? I don't really understand how it can be analyzed by a person in this format.

Evgenii

пн, 31 авг. 2020 г. в 00:36, Steven Zheng <[hidden email]>:
Thanks evgenii, but origin log is not been saved and I can only provide json format logs. 
Best Regards,
---------------------------------------------------------
Steven Zheng


Evgenii Zhuravlev <[hidden email]> 于2020年8月28日周五 上午6:45写道:
Hi,

Can you attach logs in normal format? It's really hard to read it. Also, please attach full logs from nodes, not only the stacktrace.

Thanks,
Evgenii

вт, 25 авг. 2020 г. в 19:27, Steven Zheng <[hidden email]>:
Hi community,
Currently I have 25 nodes in my ignite cluster and all of them were added into the baseline, and I was trying to add another 5 nodes into it. The data in my cluster is about 8TB and the persistence is enabled.
At first I start all the 5 nodes; then execute in the command line:
```
bin/control.sh --baseline add ${my_node_id}
```
Meanwhile, there is still read/write workloads on the cluster.

After a few minutes, a critical error detected , several nodes crashed and emits the logs like this(json format):
```
{
  "message": "Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]]]",
  "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
  "thrown": {
    "localizedMessage": "B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]",
    "message": "B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]",
    "commonElementCount": 0,
    "cause": {
      "localizedMessage": "java.lang.IllegalStateException: Item not found: 27",
      "message": "java.lang.IllegalStateException: Item not found: 27",
      "commonElementCount": 21,
      "cause": {
        "localizedMessage": "Item not found: 27",
        "message": "Item not found: 27",
        "commonElementCount": 21,
        "name": "java.lang.IllegalStateException",
        "extendedStackTrace": [{
            "line": 351,
            "method": "findIndirectItemIndex",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 459,
            "method": "getDataOffset",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 501,
            "method": "readPayload",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 325,
            "method": "readIncomplete",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 261,
            "method": "doInitFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 160,
            "method": "initFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 131,
            "method": "initFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 277,
            "method": "createRow0",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 262,
            "method": "createRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 154,
            "method": "getLookupRow",
            "exact": false,
            "file": "AbstractH2ExtrasLeafIO.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 36,
            "method": "getLookupRow",
            "exact": false,
            "file": "AbstractH2ExtrasLeafIO.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 339,
            "method": "getRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 63,
            "method": "getRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5218,
            "method": "getRow",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 418,
            "method": "compare",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 63,
            "method": "compare",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5202,
            "method": "compare",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5121,
            "method": "findInsertionPoint",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 95,
            "method": "access$1100",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 299,
            "method": "run0",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5714,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 279,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5700,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 169,
            "method": "readPage",
            "exact": false,
            "file": "PageHandler.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 364,
            "method": "read",
            "exact": false,
            "file": "DataStructure.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.DataStructure",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5901,
            "method": "read",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2064,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2083,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2083,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 1994,
            "method": "doRemove",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          }
        ]
      },
      "name": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException",
      "extendedStackTrace": [{
          "line": 290,
          "method": "doInitFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 160,
          "method": "initFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 131,
          "method": "initFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 277,
          "method": "createRow0",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 262,
          "method": "createRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 154,
          "method": "getLookupRow",
          "exact": false,
          "file": "AbstractH2ExtrasLeafIO.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 36,
          "method": "getLookupRow",
          "exact": false,
          "file": "AbstractH2ExtrasLeafIO.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 339,
          "method": "getRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 63,
          "method": "getRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5218,
          "method": "getRow",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 418,
          "method": "compare",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 63,
          "method": "compare",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5202,
          "method": "compare",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5121,
          "method": "findInsertionPoint",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 95,
          "method": "access$1100",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 299,
          "method": "run0",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5714,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 279,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5700,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 169,
          "method": "readPage",
          "exact": false,
          "file": "PageHandler.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 364,
          "method": "read",
          "exact": false,
          "file": "DataStructure.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.DataStructure",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5901,
          "method": "read",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2064,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2083,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2083,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 1994,
          "method": "doRemove",
          "exact": true,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        }
      ]
    },
    "name": "org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException",
    "extendedStackTrace": [{
        "line": 678,
        "method": "corruptedTreeException",
        "exact": true,
        "file": "H2Tree.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2029,
        "method": "doRemove",
        "exact": true,
        "file": "BPlusTree.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1825,
        "method": "removex",
        "exact": true,
        "file": "BPlusTree.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 453,
        "method": "removex",
        "exact": true,
        "file": "H2TreeIndex.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 805,
        "method": "remove",
        "exact": true,
        "file": "GridH2Table.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.opt.GridH2Table",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 404,
        "method": "remove",
        "exact": true,
        "file": "IgniteH2Indexing.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2828,
        "method": "remove",
        "exact": true,
        "file": "GridQueryProcessor.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.GridQueryProcessor",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 440,
        "method": "remove",
        "exact": true,
        "file": "GridCacheQueryManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2695,
        "method": "finishRemove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2672,
        "method": "remove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2439,
        "method": "remove",
        "exact": true,
        "file": "GridCacheOffheapManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 615,
        "method": "remove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 4403,
        "method": "removeValue",
        "exact": true,
        "file": "GridCacheMapEntry.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.GridCacheMapEntry",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 678,
        "method": "clearInternal",
        "exact": true,
        "file": "GridDhtCacheEntry.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1175,
        "method": "clearAll",
        "exact": true,
        "file": "GridDhtLocalPartition.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 938,
        "method": "tryClear",
        "exact": true,
        "file": "GridDhtLocalPartition.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 417,
        "method": "run",
        "exact": true,
        "file": "PartitionsEvictManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 7061,
        "method": "wrapThreadLoader",
        "exact": true,
        "file": "IgniteUtils.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.util.IgniteUtils",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 827,
        "method": "body",
        "exact": true,
        "file": "GridClosureProcessor.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.closure.GridClosureProcessor$1",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 120,
        "method": "run",
        "exact": true,
        "file": "GridWorker.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.util.worker.GridWorker",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1149,
        "method": "runWorker",
        "exact": true,
        "file": "ThreadPoolExecutor.java",
        "location": "?",
        "class": "java.util.concurrent.ThreadPoolExecutor",
        "version": "1.8.0_221"
      },
      {
        "line": 624,
        "method": "run",
        "exact": true,
        "file": "ThreadPoolExecutor.java",
        "location": "?",
        "class": "java.util.concurrent.ThreadPoolExecutor$Worker",
        "version": "1.8.0_221"
      },
      {
        "line": 748,
        "method": "run",
        "exact": true,
        "file": "Thread.java",
        "location": "?",
        "class": "java.lang.Thread",
        "version": "1.8.0_221"
      }
    ]
  },
  "level": "ERROR"
}
```

Can anyone kindly help? Thanks!

Best Regards,
---------------------------------------------------------
Steven Zheng
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Item not found, B+Tree is corrupted and critical error detected after add nodes to baseline topology

In reply to this post by Steven Zheng
Hello!

This looks like a PDS corruption: an entry was expected to present in data page, but was not found.

I recommend removing PDS files of that node and re-adding it to baseline topology, if you can survive the data loss.

Regards,
--
Ilya Kasnacheev


ср, 26 авг. 2020 г. в 05:27, Steven Zheng <[hidden email]>:
Hi community,
Currently I have 25 nodes in my ignite cluster and all of them were added into the baseline, and I was trying to add another 5 nodes into it. The data in my cluster is about 8TB and the persistence is enabled.
At first I start all the 5 nodes; then execute in the command line:
```
bin/control.sh --baseline add ${my_node_id}
```
Meanwhile, there is still read/write workloads on the cluster.

After a few minutes, a critical error detected , several nodes crashed and emits the logs like this(json format):
```
{
  "message": "Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]]]",
  "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
  "thrown": {
    "localizedMessage": "B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]",
    "message": "B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011] ][ 25323111322, , 0, 1596023962011 ]]",
    "commonElementCount": 0,
    "cause": {
      "localizedMessage": "java.lang.IllegalStateException: Item not found: 27",
      "message": "java.lang.IllegalStateException: Item not found: 27",
      "commonElementCount": 21,
      "cause": {
        "localizedMessage": "Item not found: 27",
        "message": "Item not found: 27",
        "commonElementCount": 21,
        "name": "java.lang.IllegalStateException",
        "extendedStackTrace": [{
            "line": 351,
            "method": "findIndirectItemIndex",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 459,
            "method": "getDataOffset",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 501,
            "method": "readPayload",
            "exact": false,
            "file": "AbstractDataPageIO.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 325,
            "method": "readIncomplete",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 261,
            "method": "doInitFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 160,
            "method": "initFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 131,
            "method": "initFromLink",
            "exact": false,
            "file": "CacheDataRowAdapter.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 277,
            "method": "createRow0",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 262,
            "method": "createRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 154,
            "method": "getLookupRow",
            "exact": false,
            "file": "AbstractH2ExtrasLeafIO.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 36,
            "method": "getLookupRow",
            "exact": false,
            "file": "AbstractH2ExtrasLeafIO.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 339,
            "method": "getRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 63,
            "method": "getRow",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5218,
            "method": "getRow",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 418,
            "method": "compare",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 63,
            "method": "compare",
            "exact": false,
            "file": "H2Tree.java",
            "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5202,
            "method": "compare",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5121,
            "method": "findInsertionPoint",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 95,
            "method": "access$1100",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 299,
            "method": "run0",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5714,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 279,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5700,
            "method": "run",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 169,
            "method": "readPage",
            "exact": false,
            "file": "PageHandler.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 364,
            "method": "read",
            "exact": false,
            "file": "DataStructure.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.DataStructure",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 5901,
            "method": "read",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2064,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2083,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 2083,
            "method": "removeDown",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          },
          {
            "line": 1994,
            "method": "doRemove",
            "exact": false,
            "file": "BPlusTree.java",
            "location": "ignite-core-2.8.0-SNAPSHOT.jar",
            "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
            "version": "2.8.0-SNAPSHOT"
          }
        ]
      },
      "name": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException",
      "extendedStackTrace": [{
          "line": 290,
          "method": "doInitFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 160,
          "method": "initFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 131,
          "method": "initFromLink",
          "exact": false,
          "file": "CacheDataRowAdapter.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 277,
          "method": "createRow0",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 262,
          "method": "createRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 154,
          "method": "getLookupRow",
          "exact": false,
          "file": "AbstractH2ExtrasLeafIO.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 36,
          "method": "getLookupRow",
          "exact": false,
          "file": "AbstractH2ExtrasLeafIO.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 339,
          "method": "getRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 63,
          "method": "getRow",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5218,
          "method": "getRow",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 418,
          "method": "compare",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 63,
          "method": "compare",
          "exact": false,
          "file": "H2Tree.java",
          "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5202,
          "method": "compare",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5121,
          "method": "findInsertionPoint",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 95,
          "method": "access$1100",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 299,
          "method": "run0",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5714,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 279,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5700,
          "method": "run",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 169,
          "method": "readPage",
          "exact": false,
          "file": "PageHandler.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 364,
          "method": "read",
          "exact": false,
          "file": "DataStructure.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.DataStructure",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 5901,
          "method": "read",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2064,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2083,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 2083,
          "method": "removeDown",
          "exact": false,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        },
        {
          "line": 1994,
          "method": "doRemove",
          "exact": true,
          "file": "BPlusTree.java",
          "location": "ignite-core-2.8.0-SNAPSHOT.jar",
          "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
          "version": "2.8.0-SNAPSHOT"
        }
      ]
    },
    "name": "org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException",
    "extendedStackTrace": [{
        "line": 678,
        "method": "corruptedTreeException",
        "exact": true,
        "file": "H2Tree.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.database.H2Tree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2029,
        "method": "doRemove",
        "exact": true,
        "file": "BPlusTree.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1825,
        "method": "removex",
        "exact": true,
        "file": "BPlusTree.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 453,
        "method": "removex",
        "exact": true,
        "file": "H2TreeIndex.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 805,
        "method": "remove",
        "exact": true,
        "file": "GridH2Table.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.opt.GridH2Table",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 404,
        "method": "remove",
        "exact": true,
        "file": "IgniteH2Indexing.java",
        "location": "ignite-indexing-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2828,
        "method": "remove",
        "exact": true,
        "file": "GridQueryProcessor.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.query.GridQueryProcessor",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 440,
        "method": "remove",
        "exact": true,
        "file": "GridCacheQueryManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2695,
        "method": "finishRemove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2672,
        "method": "remove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 2439,
        "method": "remove",
        "exact": true,
        "file": "GridCacheOffheapManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 615,
        "method": "remove",
        "exact": true,
        "file": "IgniteCacheOffheapManagerImpl.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 4403,
        "method": "removeValue",
        "exact": true,
        "file": "GridCacheMapEntry.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.GridCacheMapEntry",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 678,
        "method": "clearInternal",
        "exact": true,
        "file": "GridDhtCacheEntry.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1175,
        "method": "clearAll",
        "exact": true,
        "file": "GridDhtLocalPartition.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 938,
        "method": "tryClear",
        "exact": true,
        "file": "GridDhtLocalPartition.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 417,
        "method": "run",
        "exact": true,
        "file": "PartitionsEvictManager.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 7061,
        "method": "wrapThreadLoader",
        "exact": true,
        "file": "IgniteUtils.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.util.IgniteUtils",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 827,
        "method": "body",
        "exact": true,
        "file": "GridClosureProcessor.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.processors.closure.GridClosureProcessor$1",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 120,
        "method": "run",
        "exact": true,
        "file": "GridWorker.java",
        "location": "ignite-core-2.8.0-SNAPSHOT.jar",
        "class": "org.apache.ignite.internal.util.worker.GridWorker",
        "version": "2.8.0-SNAPSHOT"
      },
      {
        "line": 1149,
        "method": "runWorker",
        "exact": true,
        "file": "ThreadPoolExecutor.java",
        "location": "?",
        "class": "java.util.concurrent.ThreadPoolExecutor",
        "version": "1.8.0_221"
      },
      {
        "line": 624,
        "method": "run",
        "exact": true,
        "file": "ThreadPoolExecutor.java",
        "location": "?",
        "class": "java.util.concurrent.ThreadPoolExecutor$Worker",
        "version": "1.8.0_221"
      },
      {
        "line": 748,
        "method": "run",
        "exact": true,
        "file": "Thread.java",
        "location": "?",
        "class": "java.lang.Thread",
        "version": "1.8.0_221"
      }
    ]
  },
  "level": "ERROR"
}
```

Can anyone kindly help? Thanks!

Best Regards,
---------------------------------------------------------
Steven Zheng