AdaptiveLoadBalancingSpi not removing finished tasks from taskTops map

classic Classic list List threaded Threaded
4 messages Options
chris_d chris_d
Reply | Threaded
Open this post in threaded view
|

AdaptiveLoadBalancingSpi not removing finished tasks from taskTops map

Hi,

I have a question/comment about the behaviour of adaptive load balancing
with Ignite 2.7.

I was running a load test on our systems and noticed that eventually we were
running out of heap space. Most of the heap was being taken up by the
taskTops map within AdaptiveLoadBalancingSpi.

Each new task was adding topology data to that map but nothing was ever
getting removed.

There is a GridLocalEventListener registered in AdaptiveLoadBalancingSpi
that will remove tasks from the map if events of type
EVT_TASK_FINISHED/EVT_TASK_FAILED are received.

The problem is that those event types don't seem to be recorded by default.

It seems easy enough to sort by using
IgniteConfiguration.setIncludeEventTypes(EventType.EVT_TASK_FINISHED,
EventType.EVT_TASK_FAILED) but it's not obvious that this is necessary in
the first place. It took me a fair bit of digging through the code and
debugging to work out what was going on.

Is it possible to improve the adaptive load balancing behaviour a bit?
Ensuring the finished/failed event tasks get recorded by default when
adaptive load balancing is used perhaps?

Thanks,
Chris.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: AdaptiveLoadBalancingSpi not removing finished tasks from taskTops map

Hi,

It looks like an issue that possible can be a reason for OOM exception.

Is it possible to share the top consumers class list from the heap dump (it
can be done via memory analyzer e.g Eclipse MAT)?

Also, is it possible to provide the code and configuration that was used in
your load testing?

BR,
Andrei



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
chris_d chris_d
Reply | Threaded
Open this post in threaded view
|

Re: AdaptiveLoadBalancingSpi not removing finished tasks from taskTops map

Hi Andrei,

I've attached a zip of the top consumers and the class we're using to
configure the ignite clients within our app server.

I can't really provide all the code involved as the load test was testing
quite a large chunk of our system.

If necessary I can try and create a cut-down version of the test.

855_Top_Consumers.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/t2459/855_Top_Consumers.zip>  

AbstractGridConfigurationBuilder.java
<http://apache-ignite-users.70518.x6.nabble.com/file/t2459/AbstractGridConfigurationBuilder.java>  

Thanks
Chris.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: AdaptiveLoadBalancingSpi not removing finished tasks from taskTops map

Hi,

I tested your case and looks like Ignite have the memory issue here.

I filed the ticket for it:

https://issues.apache.org/jira/browse/IGNITE-11966

BR,
Andrei

On 2019/06/03 13:04:49, chris_d <[hidden email]> wrote:
 > Hi Andrei,>
 >
 > I've attached a zip of the top consumers and the class we're using to>
 > configure the ignite clients within our app server.>
 >
 > I can't really provide all the code involved as the load test was
testing>
 > quite a large chunk of our system.>
 >
 > If necessary I can try and create a cut-down version of the test. >
 >
 > 855_Top_Consumers.zip>
 >
<http://apache-ignite-users.70518.x6.nabble.com/file/t2459/855_Top_Consumers.zip>
 >
 >
 > AbstractGridConfigurationBuilder.java>
 >
<http://apache-ignite-users.70518.x6.nabble.com/file/t2459/AbstractGridConfigurationBuilder.java>
 >
 >
 > Thanks>
 > Chris.>
 >
 >
 >
 > -->
 > Sent from: http://apache-ignite-users.70518.x6.nabble.com/>
 >