Node hangs while accessing cache from Compute Task

classic Classic list List threaded Threaded
5 messages Options
GB GB
Reply | Threaded
Open this post in threaded view
|

Node hangs while accessing cache from Compute Task

Hi,

 

I am facing issue when I try to access cache from compute task which is triggered from Continuous Query listener.

This issue happens only when listener is called back synchronously (i.e when @IgniteAsyncCallback is NOT used).

Cluster node simply hangs and no error etc is thrown.

I have attached the sample application source code to reproduce this issue.

 

 Sample application details :

 

1.  App.Java is entry point of the application. Please change the path of default-config.xml and start the application.

2. Two Caches Employee and Project Cache holding Employee and Project object.

3. Employee holds key (ProjectId) of Person object.

4. Person holds field called "status"

5. Add continuous query which will be triggered on changing value of "status" field in Project.

6. Listener will trigger compute task (Map-Reduce Job)

7. In Compute Job, get all the employees of this Project. In Map step, simply return employee salary

8. In reduce, aggregate the salaries of all the employees.

9. If in the Map job, we try to access Project Cache then Node hangs.

10.If the Listener in Step 5 is annotated with @IgniteAsyncCallback, it works without getting hanged.

 

Can you please check if am I doing something wrong or it's Ignite issue ?

According to me, Ignite should atleast throw exception.


ComputeJobIgnite.zip (33K) Download Attachment
dkarachentsev dkarachentsev
Reply | Threaded
Open this post in threaded view
|

Re: Node hangs while accessing cache from Compute Task

Hi,

This is a correct behavior. It's not allowed to use cache API in CacheEntryProcessor, query listeners, interceptors and so forth, because it leads to deadlocks. In your example it looks like (threads T1, T2):
T1 put entry -> T1 lock entry -> T1 call listener -> T1 execute job -> T1 wait for result -> T2 get entry -> T2 try to lock entry -> T2 BLOCKED by T1

@IgniteAsyncCallback is a legal way to resolve such situations. In your case it will be:
T1 put entry -> T1 lock entry -> T1 submit listener to callback pool -> T1 unlock entry.
T3 - callback thread.
T3 execute job -> T3 wait for result -> T2 get entry -> T2 lock entry -> T2 unlock entry -> T2 return compute result -> T3 release with result.

No deadlocks now.

-Dmitry.
GB GB
Reply | Threaded
Open this post in threaded view
|

Re: Node hangs while accessing cache from Compute Task

Hi Dmitry,

Thanks for the detailed explanation.

Do you know if this behavior "It's not allowed to use cache API in
CacheEntryProcessor, query listeners, interceptors and so forth, because it
leads to deadlocks" is documented somewhere?


Thanks,
Gaurav


On Thu, Feb 16, 2017 at 12:16 PM, dkarachentsev <[hidden email]> wrote:
Hi,

This is a correct behavior. It's not allowed to use cache API in
CacheEntryProcessor, query listeners, interceptors and so forth, because it
leads to deadlocks. In your example it looks like (threads T1, T2):
T1 put entry -> T1 lock entry -> T1 call listener -> T1 execute job -> T1
wait for result -> T2 get entry -> T2 try to lock entry -> T2 BLOCKED by T1

@IgniteAsyncCallback is a legal way to resolve such situations. In your case
it will be:
T1 put entry -> T1 lock entry -> T1 submit listener to callback pool -> T1
unlock entry.
T3 - callback thread.
T3 execute job -> T3 wait for result -> T2 get entry -> T2 lock entry -> T2
unlock entry -> T2 return compute result -> T3 release with result.

No deadlocks now.

-Dmitry.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Node-hangs-while-accessing-cache-from-Compute-Task-tp10637p10668.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

dkarachentsev dkarachentsev
Reply | Threaded
Open this post in threaded view
|

Re: Node hangs while accessing cache from Compute Task

I haven't found such documentation, so opened a ticket to improve it [1].

[1] https://issues.apache.org/jira/browse/IGNITE-4718

Thanks!

-Dmitry
GB GB
Reply | Threaded
Open this post in threaded view
|

Re: Node hangs while accessing cache from Compute Task

Thanks Dmitry.