Ignite 2.8.1: Database Closed error

classic Classic list List threaded Threaded
2 messages Options
Mahesh Renduchintala Mahesh Renduchintala
Reply | Threaded
Open this post in threaded view
|

Ignite 2.8.1: Database Closed error

Hi,


Any pointers on what the below error means.
There seems to be an out of memory on discoverySPI.  what can cause this?




^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=50071MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=50071MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=1, qSize=0]
[16:29:45,630][INFO][exchange-worker-#85][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=2016, minorTopVer=0], force=true, evt=DISCOVERY>
[16:30:12,877][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible too long JVM pause: 2323 milliseconds.
[16:30:12,877][INFO][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/192.168.10.137, rmtPort=39599]
[16:30:12,882][SEVERE][tcp-disco-client-message-worker-[9a0c020b 192.168.1.9:61059]-#1560][TcpDiscoverySpi] Runtime error caught during grid runnable execution: GridWorker [name=tcp-disco-client-message>
java.lang.OutOfMemoryError: Java heap space
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1855)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2068)
        at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7761)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)
[16:30:12,883][INFO][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/192.168.10.137, rmtPort=39599]
[16:30:12,883][SEVERE][query-#218652][GridMapQueryExecutor] Failed to execute local query.
class org.apache.ignite.IgniteCheckedException: Failed to execute SQL query. The database has been closed [90098-197]
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:874)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:955)
        at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:412)
        at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:241)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage(IgniteH2Indexing.java:2186)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17(IgniteH2Indexing.java:2139)
        at org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:3386)
        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
        at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.h2.jdbc.JdbcSQLException: The database has been closed [90098-197]
        at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
        at org.h2.message.DbException.get(DbException.java:179)
        at org.h2.message.DbException.get(DbException.java:155)
        at org.h2.message.DbException.get(DbException.java:144)
        at org.h2.engine.Database.checkPowerOff(Database.java:536)
        at org.h2.command.Command.executeQuery(Command.java:228)



ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.8.1: Database Closed error

Hi,

I don't think it's related to the discovery. In the log, before OOM you can see the long JVM pause: 
 Possible too long JVM pause: 2323 milliseconds.

So, probably you used more heap memory than you had. 

What version do you use? How much heap memory do you have? What do you do with a cluster(SQL, key value operations, etc.)? Also, can you share full logs?

I would recommend collecting heap dump and checking objects that use memory there.

Evgenii

чт, 19 нояб. 2020 г. в 09:16, Mahesh Renduchintala <[hidden email]>:
Hi,


Any pointers on what the below error means.
There seems to be an out of memory on discoverySPI.  what can cause this?




^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=50071MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=50071MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=1, qSize=0]
[16:29:45,630][INFO][exchange-worker-#85][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=2016, minorTopVer=0], force=true, evt=DISCOVERY>
[16:30:12,877][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible too long JVM pause: 2323 milliseconds.
[16:30:12,877][INFO][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/192.168.10.137, rmtPort=39599]
[16:30:12,882][SEVERE][tcp-disco-client-message-worker-[9a0c020b 192.168.1.9:61059]-#1560][TcpDiscoverySpi] Runtime error caught during grid runnable execution: GridWorker [name=tcp-disco-client-message>
java.lang.OutOfMemoryError: Java heap space
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1855)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2068)
        at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7761)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)
[16:30:12,883][INFO][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/192.168.10.137, rmtPort=39599]
[16:30:12,883][SEVERE][query-#218652][GridMapQueryExecutor] Failed to execute local query.
class org.apache.ignite.IgniteCheckedException: Failed to execute SQL query. The database has been closed [90098-197]
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:874)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:955)
        at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:412)
        at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:241)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage(IgniteH2Indexing.java:2186)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17(IgniteH2Indexing.java:2139)
        at org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:3386)
        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
        at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.h2.jdbc.JdbcSQLException: The database has been closed [90098-197]
        at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
        at org.h2.message.DbException.get(DbException.java:179)
        at org.h2.message.DbException.get(DbException.java:155)
        at org.h2.message.DbException.get(DbException.java:144)
        at org.h2.engine.Database.checkPowerOff(Database.java:536)
        at org.h2.command.Command.executeQuery(Command.java:228)