Ignite WebSessionFilter client threads eating up CPU usage

classic Classic list List threaded Threaded
4 messages Options
wbyeh wbyeh
Reply | Threaded
Open this post in threaded view
|

Ignite WebSessionFilter client threads eating up CPU usage

We are using Ignite 2.7 as a 4-server-node session cluster but recent weeks
we found some Ignite WebSessionFilter client threads eating up CPU usage
after days of running.
The GC is running smoothly.

How to address the root cause?
Is it a Ignite 2.x known issue with StripedExecutor?

Can experts here point out any solutions?

Attached file contained thread dump & 'top' usage.

811261.jstack
<http://apache-ignite-users.70518.x6.nabble.com/file/t645/811261.jstack>  


Thank you!

-Patrick



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite WebSessionFilter client threads eating up CPU usage

Patrick,

Could you explain a bit more, why do you think that striped pool is the reason?
Striped pool is where cache operations happen. So, if you see, that it consumes a lot of CPU, then it probably means, that a lot of cache operations are coming to the cluster.

You can record a JFR and see, where most time is spent. For more information refer to the following page: https://apacheignite.readme.io/docs/jvm-and-system-tuning#section-flightrecorder-settings

Denis

вт, 14 мая 2019 г. в 10:43, wbyeh <[hidden email]>:
We are using Ignite 2.7 as a 4-server-node session cluster but recent weeks
we found some Ignite WebSessionFilter client threads eating up CPU usage
after days of running.
The GC is running smoothly.

How to address the root cause?
Is it a Ignite 2.x known issue with StripedExecutor?

Can experts here point out any solutions?

Attached file contained thread dump & 'top' usage.

811261.jstack
<http://apache-ignite-users.70518.x6.nabble.com/file/t645/811261.jstack


Thank you!

-Patrick



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite WebSessionFilter client threads eating up CPU usage

In reply to this post by wbyeh
Hello!

I can only guess that maybe your Web Sessions are very large so there are performance implications. Can you make sure you don't hold bulky objects in Sessions?

Regards,
--
Ilya Kasnacheev


вт, 14 мая 2019 г. в 10:43, wbyeh <[hidden email]>:
We are using Ignite 2.7 as a 4-server-node session cluster but recent weeks
we found some Ignite WebSessionFilter client threads eating up CPU usage
after days of running.
The GC is running smoothly.

How to address the root cause?
Is it a Ignite 2.x known issue with StripedExecutor?

Can experts here point out any solutions?

Attached file contained thread dump & 'top' usage.

811261.jstack
<http://apache-ignite-users.70518.x6.nabble.com/file/t645/811261.jstack


Thank you!

-Patrick



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
wbyeh wbyeh
Reply | Threaded
Open this post in threaded view
|

Re: Ignite WebSessionFilter client threads eating up CPU usage

Ilya , After we switched build-in openJDK (jdk-8u181-ojdkbuild-linux-x64) to Zulu JDK (zulu8.38.0.13-ca-jdk8.0.212-linux_x64), it still has a high loading CPU usage. And we found the linux node has a low CPU frequency than other nodes w/ same configurations on CentOS 7.5. Finally, we correct the CPU frequency with cpupower idle-set -d 0 #1 2 3 4 5 6 cpupower frequency-set -g performance. The high CPU loading situation is getting well. So, this check could be a first check while system tuning. cpupower frequency-info using a supported JDK. Thank you all!

Sent from the Apache Ignite Users mailing list archive at Nabble.com.