Futures not being triggered?

classic Classic list List threaded Threaded
4 messages Options
Dabbo Dabbo
Reply | Threaded
Open this post in threaded view
|

Futures not being triggered?

Hi,

I'm having an issue where an Ignite cluster node is completing a job, but not triggering the future.  I'm not sure how to detail my problem, so apologies if the following doesn't make any sense.

I have a Spring web server application running a client only Apache Ignite instance that waits for at least 1 worker node to connect before the server application submits jobs to the worker node(s).  The node in my example is configured to run 6 parallel jobs. (FifoQueueCollisionSpi.parallelJobsNumber=6).

Each job involves rendering XSL/XML document into PDF and returning that result result.

Usually, after about 2,000 - 9,000 jobs have been successfully processed, the node just "stops" processing any more jobs with more on the server remaining to process on it's internal queue.  The memory looks fine and no errors are reported in the logs.  I've have placed logging statements on the worker node and I can verify every job submitted to the worker nodes are processed successfully and no communication errors are logged.

On the client only server side; I have logging on the future listeners to state when they are triggered.  Every job fires the future up until the problem occurs and the remaining jobs fail to trigger their future and therefore my server application waits for the node to finish a job whilst the node itself waits for new jobs (empty?).

Both the server and node(s) run on the same Ubuntu server (Virtual).  I have looked at garbage collection tuning when observing "network" time-outs between the server and the node.

I've also adjusted (increased) the TcpDiscoverySPi properties (ackTimeout, networkTimeout, socketTimeout and heartbeatFrequency) to allow for the timeouts during a possible GC events.

Truth is, the garbage collection isn't taking that long to complete - 1 second at worst case.  There are no errors reported by ignite or my application to state loss of connection with the node.

At the moment, I believe the issue is that the node/server are losing the future event - communication problem?  Would you have any ideas or advice on how to diagnose this problem further?

Many thanks in advance,

Darren.


Yakov Zhdanov Yakov Zhdanov
Reply | Threaded
Open this post in threaded view
|

Re: Futures not being triggered?

Please,

1. take threaddump - jstack -l PID - from each node.
2. create issue in Ignite Jira and attach threaddumpt to it
3. also attach logs from all nodes and config files.

Do you start task from job and wait synchronously for its completion? If so, this may lead to starvation.

--
Yakov Zhdanov, Director R&D
GridGain Systems

2015-07-30 17:00 GMT+03:00 Dabbo <[hidden email]>:
Hi,

I'm having an issue where an Ignite cluster node is completing a job, but
not triggering the future.  I'm not sure how to detail my problem, so
apologies if the following doesn't make any sense.

I have a Spring web server application running a client only Apache Ignite
instance that waits for at least 1 worker node to connect before the server
application submits jobs to the worker node(s).  The node in my example is
configured to run 6 parallel jobs.
(FifoQueueCollisionSpi.parallelJobsNumber=6).

Each job involves rendering XSL/XML document into PDF and returning that
result result.

Usually, after about 2,000 - 9,000 jobs have been successfully processed,
the node just "stops" processing any more jobs with more on the server
remaining to process on it's internal queue.  The memory looks fine and no
errors are reported in the logs.  I've have placed logging statements on the
worker node and I can verify every job submitted to the worker nodes are
processed successfully and no communication errors are logged.

On the client only server side; I have logging on the future listeners to
state when they are triggered.  Every job fires the future up until the
problem occurs and the remaining jobs fail to trigger their future and
therefore my server application waits for the node to finish a job whilst
the node itself waits for new jobs (empty?).

Both the server and node(s) run on the same Ubuntu server (Virtual).  I have
looked at garbage collection tuning when observing "network" time-outs
between the server and the node.

I've also adjusted (increased) the TcpDiscoverySPi properties (ackTimeout,
networkTimeout, socketTimeout and heartbeatFrequency) to allow for the
timeouts during a possible GC events.

Truth is, the garbage collection isn't taking that long to complete - 1
second at worst case.  There are no errors reported by ignite or my
application to state loss of connection with the node.

At the moment, I believe the issue is that the node/server are losing the
future event - communication problem?  Would you have any ideas or advice on
how to diagnose this problem further?

Many thanks in advance,

Darren.






--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Futures-not-being-triggered-tp773.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Dabbo Dabbo
Reply | Threaded
Open this post in threaded view
|

Re: Futures not being triggered?

Hi Yakov,

Thanks for the prompt reply, I'll get the info together into the Jira ticket for you as soon as I can.

I use Apache Ignite in the following way:

- I create an IgniteCallable<MyResultObject> object with a unique ID.
- The XML/XSL file assets are added to the "ASSETS" cache (cache stored on the nodes).
- Create an IgniteCompute object (pulling from available remotes with Async).
- Affinity call with the UUID, IgniteCallable(above) and "ASSET" cache key.
- Create a future from IgniteCompute
- Attach IgniteInClosure<IgniteFuture<MyResult>> to the the future.

The server has it's own queue system to limit the number of jobs submitted to Apache Ignite and releases more when the futures come back to release them.

I guess this would cause starvation? but I have no idea what this would mean to my application and/or if it's a bad thing?

Thanks again,

Darren.

Dabbo Dabbo
Reply | Threaded
Open this post in threaded view
|

Re: Futures not being triggered?