Ignite hanging with log "TcpCommunicationSpi - TCP client created"

classic Classic list List threaded Threaded
8 messages Options
swara swara
Reply | Threaded
Open this post in threaded view
|

Ignite hanging with log "TcpCommunicationSpi - TCP client created"

Hi,


In our application suddenly ignite is getting hanged(threads blocked) for 1
to 5 hours and then all threads are releasing at the same time and working
normally.

The logs which we are getting repeatedly at that time is

log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112],
duration=10143ms]

Ignite version: 2.8.1

Please tell me the solution to avoid hanging.

Thank You
Swara




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite hanging with log "TcpCommunicationSpi - TCP client created"

Hello!

This looks like the case of half-open connections with very long retransmits. One host acknowledges that connection has failed when the other does not.

Please provide complete logs from both directions.

Regards,
--
Ilya Kasnacheev


чт, 4 мар. 2021 г. в 08:42, swara <[hidden email]>:
Hi,


In our application suddenly ignite is getting hanged(threads blocked) for 1
to 5 hours and then all threads are releasing at the same time and working
normally.

The logs which we are getting repeatedly at that time is

log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112],
duration=10143ms]

Ignite version: 2.8.1

Please tell me the solution to avoid hanging.

Thank You
Swara




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
swara swara
Reply | Threaded
Open this post in threaded view
|

Re: Ignite hanging with log "TcpCommunicationSpi - TCP client created"

Thank you for your response.

Based on your analysis we figured out that there is a cursor close issue. On
a related note could you please suggest on What is the default connection
pool limit for running queries using SqlFieldsQuery?

Does it depend on the number of cores alone?
If we have 8 core processor then how many connections will we get to ignite,
is there a way to increase the pool size to 100 as we have many users using
the ignite for join queries?

By referring
https://ignite.apache.org/docs/latest/perf-and-troubleshooting/thread-pools-tuning

/The default pool size is max(8, total number of cores). Use
IgniteConfiguration.setQueryThreadPoolSize(…) or a similar API from your
programming language to change the pool size./

Thank You
Swara




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite hanging with log "TcpCommunicationSpi - TCP client created"

Hello!

SqlFieldsQuery does not have connection pool limit, since it is handled by GridGain nodes and it is multiplexed asynchronously.

The actual SQL request handling takes place in query thread pool, which indeed has default size of max(8, total number of cores)

Regards,
--
Ilya Kasnacheev


ср, 17 мар. 2021 г. в 14:12, swara <[hidden email]>:
Thank you for your response.

Based on your analysis we figured out that there is a cursor close issue. On
a related note could you please suggest on What is the default connection
pool limit for running queries using SqlFieldsQuery?

Does it depend on the number of cores alone?
If we have 8 core processor then how many connections will we get to ignite,
is there a way to increase the pool size to 100 as we have many users using
the ignite for join queries?

By referring
https://ignite.apache.org/docs/latest/perf-and-troubleshooting/thread-pools-tuning

/The default pool size is max(8, total number of cores). Use
IgniteConfiguration.setQueryThreadPoolSize(…) or a similar API from your
programming language to change the pool size./

Thank You
Swara




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
swara swara
Reply | Threaded
Open this post in threaded view
|

Re: Ignite hanging with log "TcpCommunicationSpi - TCP client created"

Thank you for your response.

Is actual SQL request handling means cursor iterations?

Please explain what is long retransmits means?

We are not closing the cursor is it the reason for hanging ignite for a few
hours?


Thank you
Swara





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite hanging with log "TcpCommunicationSpi - TCP client created"

Hello!

It's too hard to answer since I do not have context for some of these questions.

I recommend closing cursors when they are no longer needed, then checking if you still have issues.

Regards,
--
Ilya Kasnacheev


пт, 19 мар. 2021 г. в 13:43, swara <[hidden email]>:
Thank you for your response.

Is actual SQL request handling means cursor iterations?

Please explain what is long retransmits means?

We are not closing the cursor is it the reason for hanging ignite for a few
hours?


Thank you
Swara





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
swara swara
Reply | Threaded
Open this post in threaded view
|

Re: Ignite hanging with log "TcpCommunicationSpi - TCP client created"

       
Thank you for your response.

we are querying the same cache object concurrently and executing the
cache.query() eagerly. The second request is invoking the cursor.iterator()
without the first request getting completed and all the next subsequent
requests are getting stuck.

***The first request got stuck for 4hours now with these errors.

org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:92)
org.apache.ignite.internal.processors.cache.query.RegisteredQueryCursor.iter(RegisteredQueryCursor.java:73)
org.apache.ignite.internal.processors.cache.QueryCursorImpl.iter(QueryCursorImpl.java:102)
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$7.iterator(IgniteH2Indexing.java:1731)
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:608)
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.awaitAllReplies(GridReduceQueryExecutor.java:947)
org.apache.ignite.internal.util.IgniteUtils.await(IgniteUtils.java:7697)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
sun.misc.Unsafe.park(Native Method)

***Second request logs

log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/(*.*.*.*:48112, /127.0.0.1:48112],
duration=209ms]
 log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112], duration=206ms]
 log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112], duration=208ms]
 log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112], duration=209ms]

What should we do to kill these processes in ignite as the application is
getting stuck for hours with no response and getting released on its own ??

Thank you
Swara





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite hanging with log "TcpCommunicationSpi - TCP client created"

Hello!

Can you please provide a runnable reproducer for this behavior?

Regards,
--
Ilya Kasnacheev


чт, 1 апр. 2021 г. в 12:54, swara <[hidden email]>:
       
Thank you for your response.

we are querying the same cache object concurrently and executing the
cache.query() eagerly. The second request is invoking the cursor.iterator()
without the first request getting completed and all the next subsequent
requests are getting stuck.

***The first request got stuck for 4hours now with these errors.

org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:92)
org.apache.ignite.internal.processors.cache.query.RegisteredQueryCursor.iter(RegisteredQueryCursor.java:73)
org.apache.ignite.internal.processors.cache.QueryCursorImpl.iter(QueryCursorImpl.java:102)
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$7.iterator(IgniteH2Indexing.java:1731)
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:608)
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.awaitAllReplies(GridReduceQueryExecutor.java:947)
org.apache.ignite.internal.util.IgniteUtils.await(IgniteUtils.java:7697)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
sun.misc.Unsafe.park(Native Method)

***Second request logs

log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/(*.*.*.*:48112, /127.0.0.1:48112],
duration=209ms]
 log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112], duration=206ms]
 log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112], duration=208ms]
 log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112], duration=209ms]

What should we do to kill these processes in ignite as the application is
getting stuck for hours with no response and getting released on its own ??

Thank you
Swara





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/