Ignite in client Mode: outofMemory error with TCP discovery

classic Classic list List threaded Threaded
2 messages Options
maheshkr76private maheshkr76private
Reply | Threaded
Open this post in threaded view
|

Ignite in client Mode: outofMemory error with TCP discovery

Hi,

Whenever, a client loses connection with ignite cluster, it throws out an
OutOfMemory error.
Can you please see if you can explain this behavior?

The ignite client had 1G memory set on XMS and XMX... and a very few clients
active.
Now it is quite possible that ignite clients lose connection with ignite
cluster frequently.

Any insights into the issue will be deeply appreciated.




15-05-2019 15:30:35.024 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 5063 milliseconds.
15-05-2019 15:30:38.958 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 3664 milliseconds.
15-05-2019 15:30:42.783 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 3666 milliseconds.
15-05-2019 15:30:46.522 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 3630 milliseconds.
15-05-2019 15:30:50.163 [exchange-worker-#38] WARN 
org.apache.ignite.internal.diagnostic.warning - Failed to wait for partition
map exchange [topVer=AffinityTopologyVersion [topVer=751, minorTopVer=1],
node=604f46ce-3693-404c-b180-29b2b6dcf8ed]. Consider changing
TransactionConfiguration.txTimeoutOnPartitionMapSynchronization to non
default value to avoid this message. Dumping pending objects that might be
the cause:
15-05-2019 15:30:50.163 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 3591 milliseconds.
15-05-2019 15:30:50.163 [exchange-worker-#38] WARN 
org.apache.ignite.internal.diagnostic.warning - Ready affinity version:
AffinityTopologyVersion [topVer=751, minorTopVer=0]
15-05-2019 15:30:57.519 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 3648 milliseconds.
15-05-2019 15:31:04.762 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 7242 milliseconds.
15-05-2019 15:31:31.284 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 22505 milliseconds.
15-05-2019 15:31:38.924 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 11482 milliseconds.
15-05-2019 15:31:47.483 [grid-timeout-worker-#23] INFO
 org.apache.ignite.internal.IgniteKernal.info -
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=604f46ce, uptime=05:40:44.720]
    ^-- H/N/C [hosts=7, nodes=7, CPUs=56]
    ^-- CPU [cur=100%, avg=0.63%, GC=30.2%]
    ^-- PageMemory [pages=0]
    ^-- Heap [used=2025MB, free=1.08%, comm=2048MB]
    ^-- Off-heap [used=0MB, free=-1%, comm=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=1, idle=0, qSize=0]
15-05-2019 15:31:54.864 [tcp-client-disco-sock-writer-#2]
ERROR
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.error - Failed to send
message: null
java.io.IOException: Failed to get acknowledge for message:
TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=1fe38eaba61-604f46ce-3693-404c-b180-29b2b6dcf8ed,
verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
isClient=true]]
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1398)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "TcpDiscoverySpi.timer"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "ajp-nio-8009-ClientPoller-1"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "Catalina-utility-1"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "http-nio-8080-ClientPoller-1"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "AsyncFileHandlerWriter-1259475182"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "jvm-pause-detector-worker"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-tcp-comm-2-#26"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "ajp-nio-8009-ClientPoller-0"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-client-listener-1-#30"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "exchange-worker-#38"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "Catalina-utility-2"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "tcp-client-disco-sock-reader-#3"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-client-listener-0-#29"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "session-timeout-worker-#34"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "nio-acceptor-client-listener-#33"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "nio-acceptor-tcp-comm-#28"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-timeout-worker-#23"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "http-nio-8080-ClientPoller-0"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-tcp-comm-3-#27"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-client-listener-3-#32"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-client-listener-2-#31"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-tcp-comm-1-#25"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-tcp-comm-0-#24"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "tcp-comm-worker-#1"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "sys-#261"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "ignite-update-noti



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite in client Mode: outofMemory error with TCP discovery

Hi,

I don’t think that it is directly related to the discovery message itself. Even before that you have long jvm pauses, probably it was a full gc, looks like you don’t have enough heap on the client. What do you there? What kind of operations do you run? I’d suggest collecting heap dump and checking what kind of objects consumes this memory.

Evgenii

> On 15 May 2019, at 19:51, mahesh76private <[hidden email]> wrote:
>
> Hi,
>
> Whenever, a client loses connection with ignite cluster, it throws out an
> OutOfMemory error.
> Can you please see if you can explain this behavior?
>
> The ignite client had 1G memory set on XMS and XMX... and a very few clients
> active.
> Now it is quite possible that ignite clients lose connection with ignite
> cluster frequently.
>
> Any insights into the issue will be deeply appreciated.
>
>
>
>
> 15-05-2019 15:30:35.024 [jvm-pause-detector-worker] WARN
>  org.apache.ignite.internal.IgniteKernal.warning - Possible too long
> JVM pause: 5063 milliseconds.
> 15-05-2019 15:30:38.958 [jvm-pause-detector-worker] WARN
>  org.apache.ignite.internal.IgniteKernal.warning - Possible too long
> JVM pause: 3664 milliseconds.
> 15-05-2019 15:30:42.783 [jvm-pause-detector-worker] WARN
>  org.apache.ignite.internal.IgniteKernal.warning - Possible too long
> JVM pause: 3666 milliseconds.
> 15-05-2019 15:30:46.522 [jvm-pause-detector-worker] WARN
>  org.apache.ignite.internal.IgniteKernal.warning - Possible too long
> JVM pause: 3630 milliseconds.
> 15-05-2019 15:30:50.163 [exchange-worker-#38] WARN 
> org.apache.ignite.internal.diagnostic.warning - Failed to wait for partition
> map exchange [topVer=AffinityTopologyVersion [topVer=751, minorTopVer=1],
> node=604f46ce-3693-404c-b180-29b2b6dcf8ed]. Consider changing
> TransactionConfiguration.txTimeoutOnPartitionMapSynchronization to non
> default value to avoid this message. Dumping pending objects that might be
> the cause:
> 15-05-2019 15:30:50.163 [jvm-pause-detector-worker] WARN
>  org.apache.ignite.internal.IgniteKernal.warning - Possible too long
> JVM pause: 3591 milliseconds.
> 15-05-2019 15:30:50.163 [exchange-worker-#38] WARN 
> org.apache.ignite.internal.diagnostic.warning - Ready affinity version:
> AffinityTopologyVersion [topVer=751, minorTopVer=0]
> 15-05-2019 15:30:57.519 [jvm-pause-detector-worker] WARN
>  org.apache.ignite.internal.IgniteKernal.warning - Possible too long
> JVM pause: 3648 milliseconds.
> 15-05-2019 15:31:04.762 [jvm-pause-detector-worker] WARN
>  org.apache.ignite.internal.IgniteKernal.warning - Possible too long
> JVM pause: 7242 milliseconds.
> 15-05-2019 15:31:31.284 [jvm-pause-detector-worker] WARN
>  org.apache.ignite.internal.IgniteKernal.warning - Possible too long
> JVM pause: 22505 milliseconds.
> 15-05-2019 15:31:38.924 [jvm-pause-detector-worker] WARN
>  org.apache.ignite.internal.IgniteKernal.warning - Possible too long
> JVM pause: 11482 milliseconds.
> 15-05-2019 15:31:47.483 [grid-timeout-worker-#23] INFO
>  org.apache.ignite.internal.IgniteKernal.info -
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>    ^-- Node [id=604f46ce, uptime=05:40:44.720]
>    ^-- H/N/C [hosts=7, nodes=7, CPUs=56]
>    ^-- CPU [cur=100%, avg=0.63%, GC=30.2%]
>    ^-- PageMemory [pages=0]
>    ^-- Heap [used=2025MB, free=1.08%, comm=2048MB]
>    ^-- Off-heap [used=0MB, free=-1%, comm=0MB]
>    ^-- Outbound messages queue [size=0]
>    ^-- Public thread pool [active=0, idle=0, qSize=0]
>    ^-- System thread pool [active=1, idle=0, qSize=0]
> 15-05-2019 15:31:54.864 [tcp-client-disco-sock-writer-#2]
> ERROR
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.error - Failed to send
> message: null
> java.io.IOException: Failed to get acknowledge for message:
> TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
> [sndNodeId=null, id=1fe38eaba61-604f46ce-3693-404c-b180-29b2b6dcf8ed,
> verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
> isClient=true]]
>    at
> org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1398)
>    at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "TcpDiscoverySpi.timer"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "ajp-nio-8009-ClientPoller-1"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "Catalina-utility-1"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "http-nio-8080-ClientPoller-1"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "AsyncFileHandlerWriter-1259475182"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "jvm-pause-detector-worker"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "grid-nio-worker-tcp-comm-2-#26"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "ajp-nio-8009-ClientPoller-0"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "grid-nio-worker-client-listener-1-#30"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "exchange-worker-#38"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "Catalina-utility-2"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "tcp-client-disco-sock-reader-#3"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "grid-nio-worker-client-listener-0-#29"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "session-timeout-worker-#34"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "nio-acceptor-client-listener-#33"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "nio-acceptor-tcp-comm-#28"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "grid-timeout-worker-#23"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "http-nio-8080-ClientPoller-0"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "grid-nio-worker-tcp-comm-3-#27"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "grid-nio-worker-client-listener-3-#32"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "grid-nio-worker-client-listener-2-#31"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "grid-nio-worker-tcp-comm-1-#25"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "grid-nio-worker-tcp-comm-0-#24"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "tcp-comm-worker-#1"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "sys-#261"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "ignite-update-noti
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/