H2 exception. lots of queries

classic Classic list List threaded Threaded
5 messages Options
vsuslov vsuslov
Reply | Threaded
Open this post in threaded view
|

H2 exception. lots of queries

This post has NOT been accepted by the mailing list yet.
Hi, we use an application in that the iBatis engine is hardcoded and we can't to get rid of sql queries unfortunately. While we were testing the application, we got an exception that follows.
We got it after a long time. Our queries were getting slower and slower. I tested it on Ignite 1.1.3 and it was much faster but now I'm trying Ignite 1.3.3 and don't know why it could be slower.
Btw I got the exception on 1.1.3 too but simultaneusly I got out of memory error.On a local machine I'm using swap properties with eviction and it's only choice for "lots of" data otherwise my computer dies.
Also I'm using one node but I got the exception even while I was testing the application on two servers with two nodes with RAM 16GB of each server. The application is OLTP-type and I know that sql is bad choice but we can't rid of it now

javax.cache.CacheException: Failed to run reduce query locally.
        at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:556)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$2.iterator(IgniteH2Indexing.java:777)
        at org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:59)
        ..........
Caused by: javax.cache.CacheException: Failed to run map query remotely.
        at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:513)
        ... 22 more
Caused by: javax.cache.CacheException: Failed to execute map query on the node: e24e6068-6e45-4dea-978e-0ab4153ad21d, class org.apache.ignite.IgniteException:Failed to set schema for DB connection for thread [schema=EXECUTION]
        at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.fail(GridReduceQueryExecutor.java:212)
        at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.onFail(GridReduceQueryExecutor.java:202)
        at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.onMessage(GridReduceQueryExecutor.java:183)
        at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor$1.onMessage(GridReduceQueryExecutor.java:139)
        at org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:1753)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:764)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$1500(GridIoManager.java:59)
        at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:727)
        ... 3 more
---------------
Best regards,
Suslov Vadim
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: H2 exception. lots of queries

vsuslov wrote
Hi, we use an application in that the iBatis engine is hardcoded and we can't to get rid of sql queries unfortunately. While we were testing the application, we got an exception that follows.
We got it after a long time. Our queries were getting slower and slower. I tested it on Ignite 1.1.3 and it was much faster but now I'm trying Ignite 1.3.3 and don't know why it could be slower.
Btw I got the exception on 1.1.3 too but simultaneusly I got out of memory error.On a local machine I'm using swap properties with eviction and it's only choice for "lots of" data otherwise my computer dies.
Also I'm using one node but I got the exception even while I was testing the application on two servers with two nodes with RAM 16GB of each server. The application is OLTP-type and I know that sql is bad choice but we can't rid of it now

javax.cache.CacheException: Failed to run reduce query locally.
        at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:556)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$2.iterator(IgniteH2Indexing.java:777)
        at org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:59)
        ..........
Caused by: javax.cache.CacheException: Failed to run map query remotely.
        at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:513)
        ... 22 more
Caused by: javax.cache.CacheException: Failed to execute map query on the node: e24e6068-6e45-4dea-978e-0ab4153ad21d, class org.apache.ignite.IgniteException:Failed to set schema for DB connection for thread [schema=EXECUTION]
        at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.fail(GridReduceQueryExecutor.java:212)
        at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.onFail(GridReduceQueryExecutor.java:202)
        at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.onMessage(GridReduceQueryExecutor.java:183)
        at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor$1.onMessage(GridReduceQueryExecutor.java:139)
        at org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:1753)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:764)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$1500(GridIoManager.java:59)
        at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:727)
        ... 3 more
Hi Vadim,

I believe there should be an exception with the root cause in one of the server node logs. Can you check this?

-Val
vsuslov vsuslov
Reply | Threaded
Open this post in threaded view
|

Re: H2 exception. lots of queries

This post has NOT been accepted by the mailing list yet.
This post was updated on .
Yes. The volume of data was not very big in tests. Each job in cache was created by business process by put() and then it was selected from the cache using SQL query then it was executed and deleted from the cache. I started 100k business processes in several threads and there were a few jobs in the cache (about 20-25) at each period of time. The whole volume of entities in caches was about 300 entities.

[SEVERE][ignite-#150%sys-null%][GridMapQueryExecutor] Failed to execute local query: GridQueryRequest [reqId=1135549, pageSize=10
24, space=JOBS, qrys=[GridCacheSqlQuery [alias=__Z0(), qry=SELECT
    RES._KEY __C0,
    RES._VAL __C1,
    RES.TYPE_ __C2,
    RES.REV_ __C3,
    RES.DUEDATE_ __C4,
    RES.LOCK_OWNER_ __C5,
    RES.LOCK_EXP_TIME_ __C6,
    RES.EXECUTION_ID_ __C7,
    RES.PROCESS_INSTANCE_ID_ __C8,
    RES.PROC_DEF_ID_ __C9,
    RES.EXCLUSIVE_ __C10,
    RES.RETRIES_ __C11,
    RES.HANDLER_TYPE_ __C12,
    RES.HANDLER_CFG_ __C13,
    RES.EXCEPTION_STACK_ID_ __C14,
    RES.EXCEPTION_MSG_ __C15,
    RES.TENANT_ID_ __C16,
    RES.REPEAT_ __C17,
    RES.ID_ __C18
FROM JOBS.JOBDTO RES
 LEFT OUTER JOIN EXECUTIONS.EXECUTIONDTO PI
 ON PI.ID_ = RES.PROCESS_INSTANCE_ID_
WHERE ((RES.EXECUTION_ID_ IS NULL) OR (PI.SUSPENSION_STATE_ = 1)) AND ((TYPE_ = 'message') AND (((RES.LOCK_EXP_TIME_ IS NULL) OR (RES.LOCK_EXP_
TIME_ <= ?2)) AND ((RES.DUEDATE_ <= ?1) AND ((RES.RETRIES_ > 0) AND (RES.DUEDATE_ IS NOT NULL))))) LIMIT ?3, params=[1441107880948, 14411078809
48, 5]]], topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], extraSpaces=[EXECUTIONS], parts=null]
class org.apache.ignite.IgniteCheckedException: Failed to execute SQL query.
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:657)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:675)
        at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:429)
        at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onMessage(GridMapQueryExecutor.java:159)
        at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor$2.onMessage(GridMapQueryExecutor.java:134)
        at org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:1753)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:764)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$1500(GridIoManager.java:59)
        at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:727)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.h2.jdbc.JdbcSQLException: ╬°шсър эхїтрЄъш ярь Єш
Out of memory.; SQL statement:
SELECT
    RES._KEY __C0,
    RES._VAL __C1,
    RES.TYPE_ __C2,
    RES.REV_ __C3,
    RES.DUEDATE_ __C4,
    RES.LOCK_OWNER_ __C5,
    RES.LOCK_EXP_TIME_ __C6,
    RES.EXECUTION_ID_ __C7,
    RES.PROCESS_INSTANCE_ID_ __C8,
    RES.PROC_DEF_ID_ __C9,
    RES.EXCLUSIVE_ __C10,
    RES.RETRIES_ __C11,
    RES.HANDLER_TYPE_ __C12,
    RES.HANDLER_CFG_ __C13,
    RES.EXCEPTION_STACK_ID_ __C14,
    RES.EXCEPTION_MSG_ __C15,
    RES.TENANT_ID_ __C16,
    RES.REPEAT_ __C17,
    RES.ID_ __C18
FROM JOBS.JOBDTO RES
 LEFT OUTER JOIN EXECUTIONS.EXECUTIONDTO PI
 ON PI.ID_ = RES.PROCESS_INSTANCE_ID_
WHERE ((RES.EXECUTION_ID_ IS NULL) OR (PI.SUSPENSION_STATE_ = 1)) AND ((TYPE_ = 'message') AND (((RES.LOCK_EXP_TIME_ IS NULL) OR (RES.LOCK_EXP_
TIME_ <= ?2)) AND ((RES.DUEDATE_ <= ?1) AND ((RES.RETRIES_ > 0) AND (RES.DUEDATE_ IS NOT NULL))))) LIMIT ?3 [90108-175]
        at org.h2.message.DbException.getJdbcSQLException(DbException.java:332)
        at org.h2.message.DbException.get(DbException.java:161)
        at org.h2.message.DbException.convert(DbException.java:278)
        at org.h2.command.Command.executeQuery(Command.java:206)
        at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:106)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:654)
        ... 11 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

[SEVERE][grid-nio-worker-2-#197%null%][TcpCommunicationSpi] Caught unhandled exception in NIO worker thread (restart the node).
java.lang.OutOfMemoryError: GC overhead limit exceeded
---------------
Best regards,
Suslov Vadim
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: H2 exception. lots of queries

vsuslov wrote
[SEVERE][grid-nio-worker-2-#197%null%][TcpCommunicationSpi] Caught unhandled exception in NIO worker thread (restart the node).
java.lang.OutOfMemoryError: GC overhead limit exceeded
So you just ran out of memory. You should either allocate more heap memory on server nodes or switch to offheap: https://apacheignite.readme.io/docs/off-heap-memory

-Val
vsuslov vsuslov
Reply | Threaded
Open this post in threaded view
|

Re: H2 exception. lots of queries

This post has NOT been accepted by the mailing list yet.
Hi, offheap mode and swap have solved my problem.
Thanks
---------------
Best regards,
Suslov Vadim