Timeouts possibly due to GC

classic Classic list List threaded Threaded
4 messages Options
Ognen Duzlevski Ognen Duzlevski
Reply | Threaded
Open this post in threaded view
|

Timeouts possibly due to GC

Hello, what does a message like the below mean? Is it telling me that the client that is using a cache is timing out? Or is it a problem with the cache engine itself?

Thanks!
Ognen

[10:42:47,317][WARNING][tcp-disco-msg-worker-#5%null][TcpDiscoverySpi] Timed out waiting for message delivery receipt (most probably, the reason is in long GC pauses on remote node; consider tuning GC and increasing 'ackTimeout' configuration property). Will retry to send message with increased timeout. Current timeout: 5000.
yakov yakov
Reply | Threaded
Open this post in threaded view
|

Re: Timeouts possibly due to GC

Ognen, this message is output by Discovery SPI in case if it has not received ack for discovery message from peer node. This may be caused by:
1. (most probably) GC pause on peer
2. network problem, esp if you are running in virtual env
3. (less probably) - by extremely high network and/or CPU utilization

By the way, we revised defaults for timeouts in discovery and will release that change within sprint-5. We tested new values and they allow faster failure detection.

If you want to try before please configure (for details on configuration parameters please refer to javadoc):

        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ackTimeout" value="50"/>
                <property name="socketTimeout" value="200"/>
                <property name="heartbeatFrequency" value="100"/>
            </bean>
        </property>


--Yakov

2015-05-21 18:48 GMT+03:00 Ognen Duzlevski <[hidden email]>:
Hello, what does a message like the below mean? Is it telling me that the client that is using a cache is timing out? Or is it a problem with the cache engine itself?

Thanks!
Ognen

[10:42:47,317][WARNING][tcp-disco-msg-worker-#5%null][TcpDiscoverySpi] Timed out waiting for message delivery receipt (most probably, the reason is in long GC pauses on remote node; consider tuning GC and increasing 'ackTimeout' configuration property). Will retry to send message with increased timeout. Current timeout: 5000.

Ognen Duzlevski Ognen Duzlevski
Reply | Threaded
Open this post in threaded view
|

Re: Timeouts possibly due to GC

Yakov, thanks for replying. In my case I am just testing some code so running an instance of ignite on my local machine and then running code that uses Ignite in client mode. The excerpt I posted was from the engine's work/log/.log file. I am guessing that in this case the message is actually trying to tell me that my client code is timing out?

Thanks!

On Fri, May 22, 2015 at 4:07 AM, Yakov Zhdanov <[hidden email]> wrote:
Ognen, this message is output by Discovery SPI in case if it has not received ack for discovery message from peer node. This may be caused by:
1. (most probably) GC pause on peer
2. network problem, esp if you are running in virtual env
3. (less probably) - by extremely high network and/or CPU utilization

By the way, we revised defaults for timeouts in discovery and will release that change within sprint-5. We tested new values and they allow faster failure detection.

If you want to try before please configure (for details on configuration parameters please refer to javadoc):

        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ackTimeout" value="50"/>
                <property name="socketTimeout" value="200"/>
                <property name="heartbeatFrequency" value="100"/>
            </bean>
        </property>


--Yakov

2015-05-21 18:48 GMT+03:00 Ognen Duzlevski <[hidden email]>:
Hello, what does a message like the below mean? Is it telling me that the client that is using a cache is timing out? Or is it a problem with the cache engine itself?

Thanks!
Ognen

[10:42:47,317][WARNING][tcp-disco-msg-worker-#5%null][TcpDiscoverySpi] Timed out waiting for message delivery receipt (most probably, the reason is in long GC pauses on remote node; consider tuning GC and increasing 'ackTimeout' configuration property). Will retry to send message with increased timeout. Current timeout: 5000.


yakov yakov
Reply | Threaded
Open this post in threaded view
|

Re: Timeouts possibly due to GC

It is not your code (in its direct meaning). For some reason Discovery SPI on client has not sent ack within timeout. I described possible reasons for that.

--Yakov

2015-05-22 16:01 GMT+03:00 Ognen Duzlevski <[hidden email]>:
Yakov, thanks for replying. In my case I am just testing some code so running an instance of ignite on my local machine and then running code that uses Ignite in client mode. The excerpt I posted was from the engine's work/log/.log file. I am guessing that in this case the message is actually trying to tell me that my client code is timing out?

Thanks!

On Fri, May 22, 2015 at 4:07 AM, Yakov Zhdanov <[hidden email]> wrote:
Ognen, this message is output by Discovery SPI in case if it has not received ack for discovery message from peer node. This may be caused by:
1. (most probably) GC pause on peer
2. network problem, esp if you are running in virtual env
3. (less probably) - by extremely high network and/or CPU utilization

By the way, we revised defaults for timeouts in discovery and will release that change within sprint-5. We tested new values and they allow faster failure detection.

If you want to try before please configure (for details on configuration parameters please refer to javadoc):

        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ackTimeout" value="50"/>
                <property name="socketTimeout" value="200"/>
                <property name="heartbeatFrequency" value="100"/>
            </bean>
        </property>


--Yakov

2015-05-21 18:48 GMT+03:00 Ognen Duzlevski <[hidden email]>:
Hello, what does a message like the below mean? Is it telling me that the client that is using a cache is timing out? Or is it a problem with the cache engine itself?

Thanks!
Ognen

[10:42:47,317][WARNING][tcp-disco-msg-worker-#5%null][TcpDiscoverySpi] Timed out waiting for message delivery receipt (most probably, the reason is in long GC pauses on remote node; consider tuning GC and increasing 'ackTimeout' configuration property). Will retry to send message with increased timeout. Current timeout: 5000.