My cluster can't activate after restart

classic Classic list List threaded Threaded
6 messages Options
yangjiajun yangjiajun
Reply | Threaded
Open this post in threaded view
|

My cluster can't activate after restart

Hi.

I have a cluster which has 2 data nodes of version 2.6.I use it as a
database with persistence.It stores tens of millions size data and all data
are replicated.I try to recover it after one node failed but I still can not
operate on it by jdbc thin connection. So I try to do a full cluster restart
one node by one.But the cluster block in activate stage.I don't see any
error logs so I have no idea now.

What's wrong with my cluster?
According to my understanding,jdbc thin connection can fail over but it
seems not work in my application.How to correctly use  fail over feature in
jdbc thin connection?
How to restart a cluster in a safe way?

Thanks for your replies.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
yangjiajun yangjiajun
Reply | Threaded
Open this post in threaded view
|

Re: My cluster can't activate after restart

I find an exception message after retries.

class org.apache.ignite.IgniteCheckedException: Cannot find metadata for
object with compact footer: -1434421210
        at
org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7307)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:259)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:207)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:159)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:151)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2433)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2299)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.binary.BinaryObjectException: Cannot find
metadata for object with compact footer: -1434421210
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.getOrCreateSchema(BinaryReaderExImpl.java:2008)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.<init>(BinaryReaderExImpl.java:284)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.<init>(BinaryReaderExImpl.java:183)
        at
org.apache.ignite.internal.binary.BinaryObjectImpl.reader(BinaryObjectImpl.java:832)
        at
org.apache.ignite.internal.binary.BinaryObjectImpl.reader(BinaryObjectImpl.java:846)
        at
org.apache.ignite.internal.binary.BinaryObjectImpl.field(BinaryObjectImpl.java:308)
        at
org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.fieldValue(QueryBinaryProperty.java:251)
        at
org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.value(QueryBinaryProperty.java:145)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2RowDescriptor.columnValue(GridH2RowDescriptor.java:331)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap.getValue0(GridH2KeyValueRowOnheap.java:122)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap.getValue(GridH2KeyValueRowOnheap.java:106)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap.toString(GridH2KeyValueRowOnheap.java:194)
        at java.lang.String.valueOf(String.java:2994)
        at java.lang.StringBuilder.append(StringBuilder.java:131)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2119)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2066)
        at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:247)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.addToIndex(GridH2Table.java:548)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:480)
        at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:659)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:403)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1393)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(IgniteCacheOffheapManagerImpl.java:1366)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.update(GridCacheOffheapManager.java:1508)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.java:367)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyUpdate(GridCacheDatabaseSharedManager.java:2492)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:2271)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:1443)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:1183)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1132)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:712)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2419)
        ... 3 more




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
yangjiajun yangjiajun
Reply | Threaded
Open this post in threaded view
|

Re: My cluster can't activate after restart

In reply to this post by yangjiajun
class org.apache.ignite.IgniteCheckedException: Cannot find metadata for
object with compact footer: -1434421210
        at
org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7307)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:259)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:207)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:159)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:151)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2433)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2299)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.binary.BinaryObjectException: Cannot find
metadata for object with compact footer: -1434421210
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.getOrCreateSchema(BinaryReaderExImpl.java:2008)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.<init>(BinaryReaderExImpl.java:284)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.<init>(BinaryReaderExImpl.java:183)
        at
org.apache.ignite.internal.binary.BinaryObjectImpl.reader(BinaryObjectImpl.java:832)
        at
org.apache.ignite.internal.binary.BinaryObjectImpl.reader(BinaryObjectImpl.java:846)
        at
org.apache.ignite.internal.binary.BinaryObjectImpl.field(BinaryObjectImpl.java:308)
        at
org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.fieldValue(QueryBinaryProperty.java:251)
        at
org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.value(QueryBinaryProperty.java:145)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2RowDescriptor.columnValue(GridH2RowDescriptor.java:331)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap.getValue0(GridH2KeyValueRowOnheap.java:122)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap.getValue(GridH2KeyValueRowOnheap.java:106)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap.toString(GridH2KeyValueRowOnheap.java:194)
        at java.lang.String.valueOf(String.java:2994)
        at java.lang.StringBuilder.append(StringBuilder.java:131)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2119)
        at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2066)
        at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:247)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.addToIndex(GridH2Table.java:548)
        at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:480)
        at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:659)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866)
        at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:403)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1393)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(IgniteCacheOffheapManagerImpl.java:1366)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.update(GridCacheOffheapManager.java:1508)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.java:367)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyUpdate(GridCacheDatabaseSharedManager.java:2492)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:2271)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:1443)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:1183)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1132)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:712)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2419)
        ... 3 more




yangjiajun wrote

> Hi.
>
> I have a cluster which has 2 data nodes of version 2.6.I use it as a
> database with persistence.It stores tens of millions size data and all
> data
> are replicated.I try to recover it after one node failed but I still can
> not
> operate on it by jdbc thin connection. So I try to do a full cluster
> restart
> one node by one.But the cluster block in activate stage.I don't see any
> error logs so I have no idea now.
>
> What's wrong with my cluster?
> According to my understanding,jdbc thin connection can fail over but it
> seems not work in my application.How to correctly use  fail over feature
> in
> jdbc thin connection?
> How to restart a cluster in a safe way?
>
> Thanks for your replies.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: My cluster can't activate after restart

In reply to this post by yangjiajun
Hi,

What do you mean when say that

1)JDBC thin client isn't worked
2)cluster block in activate stage

How you activate your cluster? Are you use the persistence?

To get more logs try to add next option to java options of Ignite:

-DIGNITE_QUIET=false

BR,
Andrei



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
yangjiajun yangjiajun
Reply | Threaded
Open this post in threaded view
|

Re: My cluster can't activate after restart

Hi,

1)JDBC thin client becomes not work after one node fails.I think the cause
is my node uses too many heap space and jvm pause too much time.I was doing
write performance test at that time.

2)I use control.sh script to active the cluster,but it does not exit .

3)Yes.I use persistence.

4)I have already disable quiet  mode.The logs shows that ignite throws an
exception after I call control.sh to active the cluster.The exception is:
class org.apache.ignite.IgniteCheckedException: Cannot find metadata for
object with compact footer: -1434421210.And you can see more details above.

Thanks for your reply.




aealexsandrov wrote

> Hi,
>
> What do you mean when say that
>
> 1)JDBC thin client isn't worked
> 2)cluster block in activate stage
>
> How you activate your cluster? Are you use the persistence?
>
> To get more logs try to add next option to java options of Ignite:
>
> -DIGNITE_QUIET=false
>
> BR,
> Andrei
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: My cluster can't activate after restart

Hello!

The last error looks like you have lost contents of work/marshaller directory on your nodes. Did you? Do you have -1434421210.classname file or similar?

Regards,
--
Ilya Kasnacheev


пт, 9 нояб. 2018 г. в 4:12, yangjiajun <[hidden email]>:
Hi,

1)JDBC thin client becomes not work after one node fails.I think the cause
is my node uses too many heap space and jvm pause too much time.I was doing
write performance test at that time.

2)I use control.sh script to active the cluster,but it does not exit .

3)Yes.I use persistence.

4)I have already disable quiet  mode.The logs shows that ignite throws an
exception after I call control.sh to active the cluster.The exception is:
class org.apache.ignite.IgniteCheckedException: Cannot find metadata for
object with compact footer: -1434421210.And you can see more details above.

Thanks for your reply.




aealexsandrov wrote
> Hi,
>
> What do you mean when say that
>
> 1)JDBC thin client isn't worked
> 2)cluster block in activate stage
>
> How you activate your cluster? Are you use the persistence?
>
> To get more logs try to add next option to java options of Ignite:
>
> -DIGNITE_QUIET=false
>
> BR,
> Andrei
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/