Ignite issue during spark job execution

classic Classic list List threaded Threaded
2 messages Options
Amol Zambare Amol Zambare
Reply | Threaded
Open this post in threaded view
|

Ignite issue during spark job execution

Hi,

We are using ignite to share in memory data across the spark jobs. When we are running the spark job we see following warnings in ignite logs. We have done garbage collection tuning as suggested in apache ignite performance tips but still we see below warning.

Please suggest.

Warning 1
[00:25:14,432][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1430 milliseconds.
 Warning 2
[14:43:44,960][WARNING][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/127.0.0
.1:47101, failureDetectionTimeout=100000]
[14:43:44,960][WARNING][tcp-comm-worker-#1][TcpCommunicationSpi] Failed to connect to a remote node (make sure that destination node is alive and operating system firew
all is disabled on local and remote hosts)

Thanks,
Amol 

aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite issue during spark job execution

Hi,

Looks like your JVM faces long garbage collection pauses. You can configure
detailed GC logs to see how much time is spent in GC:
https://apacheignite.readme.io/docs/jvm-and-system-tuning#section-detailed-garbage-collection-stats

Also, you can follow next guide and add more heap:

https://apacheignite.readme.io/docs/jvm-and-system-tuning#garbage-collection-tuning

BR,
Andrei



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/