Possible dead lock when number of jobs exceeds thread pool

classic Classic list List threaded Threaded
10 messages Options
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

Possible dead lock when number of jobs exceeds thread pool

I am testing an aspect of a POC I have written in C# to determine how well it performs when processing multiple tasks at one time. I’m using Ignite 2.2.

 

Briefly, a client node sends a request to a server node for a list of items. Each of the items requested is send back to a listener in the client (using Ignite messaging) and then processed (rendered onto a bitmap). Once all parts of the request have arrived and been rendered, a wait state is triggered allowing the request to be completed. The wait state will terminate after 2 minutes which short cuts the request in the client node.

 

This works well from a functional perspective, but I ran into problems with multithreading.

 

To test multi-threading, I essentially did this:

 

                                int nThreads = 4;

            Parallel.For(0, nThreads, x => PerformRender(<some arguments>));

 

This works well for values of nThreads of up to 7. However, once nThreads is set more than 7 then the requests start stalling, and no progress is made the first eight requests until the wait states start timing out at two minutes.

 

Looking in the log is it common to see entries like this during the time the requests are stalled:

 

             WARN  2017-10-04 13:41:57,051 252758ms IgniteKernal%Raptor                      ?                  - Possible thread pool starvation detected (no task completed in last 30000ms, is public thread pool size large enough?)

Here we see an internal monitor is warning regarding a possible deadlock or race condition meaning progress is not being made on requests.

 

INFO  2017-10-04 15:47:21,182 263409ms IgniteKernal%Raptor                      ?                  -

Metrics for local node (to disable set 'metricsLogFrequency' to 0)

    ^-- Node [id=c966b7cc, name=Raptor, uptime=00:04:10:105]

    ^-- H/N/C [hosts=1, nodes=7, CPUs=8]

    ^-- CPU [cur=0.07%, avg=1.76%, GC=0%]

    ^-- PageMemory [pages=0]

    ^-- Heap [used=103MB, free=88.58%, comm=497MB]

    ^-- Non heap [used=38MB, free=-1%, comm=39MB]

    ^-- Public thread pool [active=8, idle=0, qSize=274]

    ^-- System thread pool [active=0, idle=0, qSize=0]

    ^-- Outbound messages queue [size=0]

Here we see the pubic thread pool is set to 8 by default. This suggests when the pool becomes saturated with requests something is stalling the requests.

 

I tried increasing the number of threads in the public pool in the grid configuration and this resulted in no stalls in requests which suggests Ignite does not like fully committed thread pools.

 

Is this a known issue with Ignite?

 

Thanks,

Raymond.

 

afedotov afedotov
Reply | Threaded
Open this post in threaded view
|

Re: Possible dead lock when number of jobs exceeds thread pool

CONTENTS DELETED
The author has deleted this message.
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

RE: Possible dead lock when number of jobs exceeds thread pool

Hi Alex,

There's a fair bit of source involved, I think I'd need to build a
reproducer app. I'd like to talk it through a bit before putting in that
effort.

There are actually three parts:

Client --> Service A --> Service B

A call from Client to Service A causes Service A to request a series of
items from Service B. While Service B is calculating and returning them,
the call in Service A blocks on a signal with a timeout. Elements sent
from service B to service A are sent via Ignite messaging to a listener
established by the call in Service A.

If I understand you correctly, you are saying the 8 requests from Client
to Service A that are waiting for a signal mean that messages from Service
B to Service A never arrive because there are no available threads in the
pool (default size:8) to receive the inbound information?

If so, is there a better asynchronous pattern that could be used in
Service A?

FYI, the reason I am using the message pattern from Service B to Service A
is that the quantity of information sent between the two may be quite
large meaning returning it as a single request result is impractical.

Thanks,
Raymond.


-----Original Message-----
From: afedotov [mailto:[hidden email]]
Sent: Wednesday, October 4, 2017 10:37 PM
To: [hidden email]
Subject: Re: Possible dead lock when number of jobs exceeds thread pool

Hi,

That actually looks like a starvation in public pool where compute tasks
are processed.
Please share PerformRenderer() source.

Kind regards,
Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
afedotov afedotov
Reply | Threaded
Open this post in threaded view
|

RE: Possible dead lock when number of jobs exceeds thread pool

CONTENTS DELETED
The author has deleted this message.
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

RE: Possible dead lock when number of jobs exceeds thread pool

Alex,

Thanks for the pointer to the custom thread pools.

The ExecutorConfiguration aspect of the grid configuration does not appear
to be present in the C# API. Is this only supported in the XML
configuration file for C# clients?

Raymond.

-----Original Message-----
From: afedotov [mailto:[hidden email]]
Sent: Thursday, October 5, 2017 4:07 AM
To: [hidden email]
Subject: RE: Possible dead lock when number of jobs exceeds thread pool

Raymond,

Yes. Your understanding is correct, that should be the reason of
starvation.
Please take a look at custom thread pools
https://apacheignite.readme.io/v2.2/docs/thread-pools#section-custom-threa
d-pools.

Kind regards,
Alex




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

RE: Possible dead lock when number of jobs exceeds thread pool

In reply to this post by afedotov
It seems that the C# API does not support custom thread pools. There is no
WithExecutor() method on ICompute, and looking at the client source I
don't see any relevant executor support.

Is this due to the Async method variants supported on the C# API?

-----Original Message-----
From: Raymond Wilson [mailto:[hidden email]]
Sent: Thursday, October 5, 2017 9:23 AM
To: '[hidden email]' <[hidden email]>
Subject: RE: Possible dead lock when number of jobs exceeds thread pool

Alex,

Thanks for the pointer to the custom thread pools.

The ExecutorConfiguration aspect of the grid configuration does not appear
to be present in the C# API. Is this only supported in the XML
configuration file for C# clients?

Raymond.

-----Original Message-----
From: afedotov [mailto:[hidden email]]
Sent: Thursday, October 5, 2017 4:07 AM
To: [hidden email]
Subject: RE: Possible dead lock when number of jobs exceeds thread pool

Raymond,

Yes. Your understanding is correct, that should be the reason of
starvation.
Please take a look at custom thread pools
https://apacheignite.readme.io/v2.2/docs/thread-pools#section-custom-threa
d-pools.

Kind regards,
Alex




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

RE: Possible dead lock when number of jobs exceeds thread pool

In reply to this post by afedotov
.... I have tried using the Async variants of the, but this does not
change the number of public threads required in the Ignite server that is
the recipient of the call (which I guess is sensible when I think about
it).

-----Original Message-----
From: Raymond Wilson [mailto:[hidden email]]
Sent: Thursday, October 5, 2017 10:52 AM
To: '[hidden email]' <[hidden email]>
Subject: RE: Possible dead lock when number of jobs exceeds thread pool

It seems that the C# API does not support custom thread pools. There is no
WithExecutor() method on ICompute, and looking at the client source I
don't see any relevant executor support.

Is this due to the Async method variants supported on the C# API?

-----Original Message-----
From: Raymond Wilson [mailto:[hidden email]]
Sent: Thursday, October 5, 2017 9:23 AM
To: '[hidden email]' <[hidden email]>
Subject: RE: Possible dead lock when number of jobs exceeds thread pool

Alex,

Thanks for the pointer to the custom thread pools.

The ExecutorConfiguration aspect of the grid configuration does not appear
to be present in the C# API. Is this only supported in the XML
configuration file for C# clients?

Raymond.

-----Original Message-----
From: afedotov [mailto:[hidden email]]
Sent: Thursday, October 5, 2017 4:07 AM
To: [hidden email]
Subject: RE: Possible dead lock when number of jobs exceeds thread pool

Raymond,

Yes. Your understanding is correct, that should be the reason of
starvation.
Please take a look at custom thread pools
https://apacheignite.readme.io/v2.2/docs/thread-pools#section-custom-threa
d-pools.

Kind regards,
Alex




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
afedotov afedotov
Reply | Threaded
Open this post in threaded view
|

RE: Possible dead lock when number of jobs exceeds thread pool

CONTENTS DELETED
The author has deleted this message.
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

Re: Possible dead lock when number of jobs exceeds thread pool

Hi Alex,

Thanks for clarifying.

I have been testing increasing the public pool size and it does help.

Is there a general work programme to bring the C# client into parity with the Java client?

Thanks,
Raymond.

Sent from my iPhone

> On 6/10/2017, at 2:26 AM, afedotov <[hidden email]> wrote:
>
> Hi Raymond,
>
> You are right. As for now, C# API doesn't support WithExecutor calls.
> Please find a corresponding ticket for your reference
> https://issues.apache.org/jira/browse/IGNITE-6566
> <https://issues.apache.org/jira/browse/IGNITE-6566>  
>
> In this case. You could increase the public thread pool size according to
> the anticipated load.
> As well, you could somehow limit the number of concurrently executed calls
> to make sure
> that there are enough threads in the pool to operate without starvation.
>
> Kind regards,
> Alex
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
afedotov afedotov
Reply | Threaded
Open this post in threaded view
|

Re: Possible dead lock when number of jobs exceeds thread pool

CONTENTS DELETED
The author has deleted this message.