BufferUnderflowException on GridRedisProtocolParser

classic Classic list List threaded Threaded
6 messages Options
Michael Fong Michael Fong
Reply | Threaded
Open this post in threaded view
|

BufferUnderflowException on GridRedisProtocolParser

Hi, all

We are evaluating Ignite compatibility with Redis protocol, and we hit an issue as the following:
Does the stacktrace look a bit like IGNITE-7153?

java.nio.BufferUnderflowException
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:151)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.redis.GridRedisProtocolParser.readBulkStr(GridRedisProtocolParser.java:111)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.redis.GridRedisProtocolParser.readArray(GridRedisProtocolParser.java:86)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestParser.decode(GridTcpRestParser.java:165)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestParser.decode(GridTcpRestParser.java:72)
        at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:114)
        at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
        at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3490)
        at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
        at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1113)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)
[2018-10-09 12:45:49,946][ERROR][grid-nio-worker-tcp-rest-1-#37][GridTcpRestProtocol] Closing NIO session because of unhandled exception.
class org.apache.ignite.internal.util.nio.GridNioException: null
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2365)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

RE: BufferUnderflowException on GridRedisProtocolParser

It definitely does.

IGNITE-7153 seems to be applicable for objects greater than 8 kb. Is that your case?

If so then I guess that has to be the same issue.

 

Stan

 

From: [hidden email]
Sent: 9 октября 2018 г. 15:54
To: [hidden email]
Subject: BufferUnderflowException on GridRedisProtocolParser

 

Hi, all

 

We are evaluating Ignite compatibility with Redis protocol, and we hit an issue as the following:

Does the stacktrace look a bit like IGNITE-7153?

 

java.nio.BufferUnderflowException

        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:151)

        at org.apache.ignite.internal.processors.rest.protocols.tcp.redis.GridRedisProtocolParser.readBulkStr(GridRedisProtocolParser.java:111)

        at org.apache.ignite.internal.processors.rest.protocols.tcp.redis.GridRedisProtocolParser.readArray(GridRedisProtocolParser.java:86)

        at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestParser.decode(GridTcpRestParser.java:165)

        at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestParser.decode(GridTcpRestParser.java:72)

        at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:114)

        at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)

        at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3490)

        at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)

        at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1113)

        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)

        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)

        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)

        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)

        at java.lang.Thread.run(Thread.java:745)

[2018-10-09 12:45:49,946][ERROR][grid-nio-worker-tcp-rest-1-#37][GridTcpRestProtocol] Closing NIO session because of unhandled exception.

class org.apache.ignite.internal.util.nio.GridNioException: null

        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2365)

        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)

        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)

        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)

        at java.lang.Thread.run(Thread.java:745)

 

 

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: BufferUnderflowException on GridRedisProtocolParser

In reply to this post by Michael Fong
Hello!

Do you see this error on every request, or on some specific ones?

Regards,
--
Ilya Kasnacheev


вт, 9 окт. 2018 г. в 15:54, Michael Fong <[hidden email]>:
Hi, all

We are evaluating Ignite compatibility with Redis protocol, and we hit an issue as the following:
Does the stacktrace look a bit like IGNITE-7153?

java.nio.BufferUnderflowException
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:151)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.redis.GridRedisProtocolParser.readBulkStr(GridRedisProtocolParser.java:111)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.redis.GridRedisProtocolParser.readArray(GridRedisProtocolParser.java:86)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestParser.decode(GridTcpRestParser.java:165)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestParser.decode(GridTcpRestParser.java:72)
        at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:114)
        at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
        at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3490)
        at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
        at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1113)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)
[2018-10-09 12:45:49,946][ERROR][grid-nio-worker-tcp-rest-1-#37][GridTcpRestProtocol] Closing NIO session because of unhandled exception.
class org.apache.ignite.internal.util.nio.GridNioException: null
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2365)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

Michael Fong Michael Fong
Reply | Threaded
Open this post in threaded view
|

Re: BufferUnderflowException on GridRedisProtocolParser

Hi, 

Thank you for your response. Not to every request; we only see this for some specific ones - when elCnt (4270) > buf.limit (4096).  We are trying to narrowing down the data set to find the root cause.

Thanks.

Regards,

On Thu, Oct 11, 2018 at 12:08 AM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Do you see this error on every request, or on some specific ones?

Regards,
--
Ilya Kasnacheev


вт, 9 окт. 2018 г. в 15:54, Michael Fong <[hidden email]>:
Hi, all

We are evaluating Ignite compatibility with Redis protocol, and we hit an issue as the following:
Does the stacktrace look a bit like IGNITE-7153?

java.nio.BufferUnderflowException
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:151)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.redis.GridRedisProtocolParser.readBulkStr(GridRedisProtocolParser.java:111)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.redis.GridRedisProtocolParser.readArray(GridRedisProtocolParser.java:86)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestParser.decode(GridTcpRestParser.java:165)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestParser.decode(GridTcpRestParser.java:72)
        at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:114)
        at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
        at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3490)
        at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
        at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1113)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)
[2018-10-09 12:45:49,946][ERROR][grid-nio-worker-tcp-rest-1-#37][GridTcpRestProtocol] Closing NIO session because of unhandled exception.
class org.apache.ignite.internal.util.nio.GridNioException: null
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2365)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

Michael Fong Michael Fong
Reply | Threaded
Open this post in threaded view
|

Re: BufferUnderflowException on GridRedisProtocolParser

Hi, 

The symptom seems very likely to IGNITE-7153, where the default tcp send buffer on that environment happens to be 4096 
[root]# cat /proc/sys/net/ipv4/tcp_wmem 
4096 

Regards,

On Thu, Oct 11, 2018 at 10:56 AM Michael Fong <[hidden email]> wrote:
Hi, 

Thank you for your response. Not to every request; we only see this for some specific ones - when elCnt (4270) > buf.limit (4096).  We are trying to narrowing down the data set to find the root cause.

Thanks.

Regards,

On Thu, Oct 11, 2018 at 12:08 AM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Do you see this error on every request, or on some specific ones?

Regards,
--
Ilya Kasnacheev


вт, 9 окт. 2018 г. в 15:54, Michael Fong <[hidden email]>:
Hi, all

We are evaluating Ignite compatibility with Redis protocol, and we hit an issue as the following:
Does the stacktrace look a bit like IGNITE-7153?

java.nio.BufferUnderflowException
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:151)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.redis.GridRedisProtocolParser.readBulkStr(GridRedisProtocolParser.java:111)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.redis.GridRedisProtocolParser.readArray(GridRedisProtocolParser.java:86)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestParser.decode(GridTcpRestParser.java:165)
        at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestParser.decode(GridTcpRestParser.java:72)
        at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:114)
        at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
        at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3490)
        at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
        at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1113)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)
[2018-10-09 12:45:49,946][ERROR][grid-nio-worker-tcp-rest-1-#37][GridTcpRestProtocol] Closing NIO session because of unhandled exception.
class org.apache.ignite.internal.util.nio.GridNioException: null
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2365)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)

Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

RE: BufferUnderflowException on GridRedisProtocolParser

Well, you need to wait for the IGNITE-7153 fix then.

Or contribute it! :)

I checked the code, and it seems to be a relatively easy fix. One needs to alter the GridRedisProtocolParser

to use ParserState in the way GridTcpRestParser::parseMemcachePacker does.

 

Stan

 

From: [hidden email]
Sent: 11 октября 2018 г. 7:15
To: [hidden email]
Subject: Re: BufferUnderflowException on GridRedisProtocolParser

 

Hi, 

 

The symptom seems very likely to IGNITE-7153, where the default tcp send buffer on that environment happens to be 4096 

[root]# cat /proc/sys/net/ipv4/tcp_wmem 

4096 

 

Regards,

 

On Thu, Oct 11, 2018 at 10:56 AM Michael Fong <[hidden email]> wrote:

Hi, 

 

Thank you for your response. Not to every request; we only see this for some specific ones - when elCnt (4270) > buf.limit (4096).  We are trying to narrowing down the data set to find the root cause.

 

Thanks.

 

Regards,

 

On Thu, Oct 11, 2018 at 12:08 AM Ilya Kasnacheev <[hidden email]> wrote:

Hello!

 

Do you see this error on every request, or on some specific ones?

 

Regards,

--

Ilya Kasnacheev

 

 

вт, 9 окт. 2018 г. в 15:54, Michael Fong <[hidden email]>:

Hi, all

 

We are evaluating Ignite compatibility with Redis protocol, and we hit an issue as the following:

Does the stacktrace look a bit like IGNITE-7153?

 

java.nio.BufferUnderflowException

        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:151)

        at org.apache.ignite.internal.processors.rest.protocols.tcp.redis.GridRedisProtocolParser.readBulkStr(GridRedisProtocolParser.java:111)

        at org.apache.ignite.internal.processors.rest.protocols.tcp.redis.GridRedisProtocolParser.readArray(GridRedisProtocolParser.java:86)

        at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestParser.decode(GridTcpRestParser.java:165)

        at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestParser.decode(GridTcpRestParser.java:72)

        at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:114)

        at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)

        at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3490)

        at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)

        at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1113)

        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)

        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)

        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)

        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)

        at java.lang.Thread.run(Thread.java:745)

[2018-10-09 12:45:49,946][ERROR][grid-nio-worker-tcp-rest-1-#37][GridTcpRestProtocol] Closing NIO session because of unhandled exception.

class org.apache.ignite.internal.util.nio.GridNioException: null

        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2365)

        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)

        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)

        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)

        at java.lang.Thread.run(Thread.java:745)