Ignite node stops on read/insert/update operations from ycsb

classic Classic list List threaded Threaded
7 messages Options
summasumma summasumma
Reply | Threaded
Open this post in threaded view
|

Ignite node stops on read/insert/update operations from ycsb

Hi,
I am trying following workload from ycsb with 2 node ignite cluster with 1
backup enabled. writesync=prim_sync.

recordcount=4000000
operationcount=4000000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true

insertproportion=0.23
updateproportion=0.46
readproportion=0.31

requestdistribution=zipfian
threadcount=180
target=180000

But in few seconds one of the ignite node stops with following error:
[19:07:36,283][SEVERE][grid-nio-worker-tcp-comm-3-#28][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=3,
bytesRcvd=2843314842, bytesSent=487323705, bytesRcvd0=0, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-3,
igniteInstanceName=null, finished=false, hashCode=170320615,
interrupted=false, runner=grid-nio-worker-tcp-comm-3-#28]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=2885472, resendCnt=0,
rcvCnt=2767636, sentCnt=2885548, reserved=true, lastAck=2767616,
nodeLeft=false, node=TcpDiscoveryNode
[id=cbc18e72-429a-44ef-84eb-df2e8a805866, addrs=[172.16.0.17],
sockAddrs=[/172.16.0.17:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1543950039896, loc=false, ver=2.6.0#20180710-sha1:669feacc,
isClient=false], connected=false, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=2885472, resendCnt=0,
rcvCnt=2767636, sentCnt=2885548, reserved=true, lastAck=2767616,
nodeLeft=false, node=TcpDiscoveryNode
[id=cbc18e72-429a-44ef-84eb-df2e8a805866, addrs=[172.16.0.17],
sockAddrs=[/172.16.0.17:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1543950039896, loc=false, ver=2.6.0#20180710-sha1:669feacc,
isClient=false], connected=false, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl
[locAddr=/172.16.0.4:41328, rmtAddr=/172.16.0.17:47100,
createTime=1543950040183, closeTime=0, bytesSent=3480594647,
bytesRcvd=3339280186, bytesSent0=0, bytesRcvd0=0,
sndSchedTime=1543950040183, lastSndTime=1543950373693,
lastRcvTime=1543950373693, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5c6f7548, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:51)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1649)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1306)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2342)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)
[19:07:36,872][SEVERE][grid-nio-worker-tcp-comm-0-#25][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0,
bytesRcvd=3228052543, bytesSent=3293205484, bytesRcvd0=0, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0,
igniteInstanceName=null, finished=false, hashCode=1267030710,
interrupted=false, runner=grid-nio-worker-tcp-comm-0-#25]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=6163 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=1672992, resendCnt=0,
rcvCnt=1673192, sentCnt=1673110, reserved=true, lastAck=1673184,
nodeLeft=false, node=TcpDiscoveryNode
[id=839c980a-5d38-416c-9b82-32ce9dfefc4c, addrs=[0:0:0:0:0:0:0:1%lo,
10.62.18.32, 127.0.0.1, 172.16.0.11], sockAddrs=[/0:0:0:0:0:0:0:1%lo:0,
/127.0.0.1:0, /172.16.0.11:0, /10.62.18.32:0], discPort=0, order=7,
intOrder=6, lastExchangeTime=1543950304173, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=true], connected=true,
connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=1672992, resendCnt=0,
rcvCnt=1673192, sentCnt=1673110, reserved=true, lastAck=1673184,
nodeLeft=false, node=TcpDiscoveryNode
[id=839c980a-5d38-416c-9b82-32ce9dfefc4c, addrs=[0:0:0:0:0:0:0:1%lo,
10.62.18.32, 127.0.0.1, 172.16.0.11], sockAddrs=[/0:0:0:0:0:0:0:1%lo:0,
/127.0.0.1:0, /172.16.0.11:0, /10.62.18.32:0], discPort=0, order=7,
intOrder=6, lastExchangeTime=1543950304173, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=true], connected=true,
connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false],
super=GridNioSessionImpl [locAddr=/172.16.0.4:47100,
rmtAddr=/172.16.0.11:34656, createTime=1543950304452, closeTime=0,
bytesSent=608945515, bytesRcvd=686391430, bytesSent0=0, bytesRcvd0=0,
sndSchedTime=1543950304452, lastSndTime=1543950400759,
lastRcvTime=1543950405400, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5c6f7548, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]]
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:51)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1649)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1306)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2342)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)
[19:07:41,332][SEVERE][tcp-disco-msg-worker-#2][TcpDiscoverySpi]
TcpDiscoverSpi's message worker thread failed abnormally. Stopping the node
in order to prevent cluster wide instability.
java.lang.OutOfMemoryError: Java heap space
        at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76)
        at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.socketStream(TcpDiscoverySpi.java:1424)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.sendMessageAcrossRing(ServerImpl.java:2962)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMetricsUpdateMessage(ServerImpl.java:5154)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2756)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2536)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6775)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2621)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
^[[A^[[A^[[5~      at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1649)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1306)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2342)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)



can i know what could be the issue in ignite side?
Also is my proportion looks correct compared to no of operations?
recordcount=4000000
operationcount=4000000   --> 4 million operations
insertproportion=0.23    - ---> 23% insert
updateproportion=0.46  - ---> 46% update
readproportion=0.31      - ---> 31% read
threadcount=180
target=180000

Please clarify.

Thanks,
...summa




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite node stops on read/insert/update operations from ycsb

Hello!

> java.lang.OutOfMemoryError

Have you tried increasing heap size of your process? Or trying to figure out why you end up using so much heap.

Regards,
--
Ilya Kasnacheev


ср, 5 дек. 2018 г. в 05:40, summasumma <[hidden email]>:
Hi,
I am trying following workload from ycsb with 2 node ignite cluster with 1
backup enabled. writesync=prim_sync.

recordcount=4000000
operationcount=4000000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true

insertproportion=0.23
updateproportion=0.46
readproportion=0.31

requestdistribution=zipfian
threadcount=180
target=180000

But in few seconds one of the ignite node stops with following error:
[19:07:36,283][SEVERE][grid-nio-worker-tcp-comm-3-#28][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=3,
bytesRcvd=2843314842, bytesSent=487323705, bytesRcvd0=0, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-3,
igniteInstanceName=null, finished=false, hashCode=170320615,
interrupted=false, runner=grid-nio-worker-tcp-comm-3-#28]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=2885472, resendCnt=0,
rcvCnt=2767636, sentCnt=2885548, reserved=true, lastAck=2767616,
nodeLeft=false, node=TcpDiscoveryNode
[id=cbc18e72-429a-44ef-84eb-df2e8a805866, addrs=[172.16.0.17],
sockAddrs=[/172.16.0.17:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1543950039896, loc=false, ver=2.6.0#20180710-sha1:669feacc,
isClient=false], connected=false, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=2885472, resendCnt=0,
rcvCnt=2767636, sentCnt=2885548, reserved=true, lastAck=2767616,
nodeLeft=false, node=TcpDiscoveryNode
[id=cbc18e72-429a-44ef-84eb-df2e8a805866, addrs=[172.16.0.17],
sockAddrs=[/172.16.0.17:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1543950039896, loc=false, ver=2.6.0#20180710-sha1:669feacc,
isClient=false], connected=false, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl
[locAddr=/172.16.0.4:41328, rmtAddr=/172.16.0.17:47100,
createTime=1543950040183, closeTime=0, bytesSent=3480594647,
bytesRcvd=3339280186, bytesSent0=0, bytesRcvd0=0,
sndSchedTime=1543950040183, lastSndTime=1543950373693,
lastRcvTime=1543950373693, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5c6f7548, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:51)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1649)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1306)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2342)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)
[19:07:36,872][SEVERE][grid-nio-worker-tcp-comm-0-#25][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0,
bytesRcvd=3228052543, bytesSent=3293205484, bytesRcvd0=0, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0,
igniteInstanceName=null, finished=false, hashCode=1267030710,
interrupted=false, runner=grid-nio-worker-tcp-comm-0-#25]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=6163 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=1672992, resendCnt=0,
rcvCnt=1673192, sentCnt=1673110, reserved=true, lastAck=1673184,
nodeLeft=false, node=TcpDiscoveryNode
[id=839c980a-5d38-416c-9b82-32ce9dfefc4c, addrs=[0:0:0:0:0:0:0:1%lo,
10.62.18.32, 127.0.0.1, 172.16.0.11], sockAddrs=[/0:0:0:0:0:0:0:1%lo:0,
/127.0.0.1:0, /172.16.0.11:0, /10.62.18.32:0], discPort=0, order=7,
intOrder=6, lastExchangeTime=1543950304173, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=true], connected=true,
connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=1672992, resendCnt=0,
rcvCnt=1673192, sentCnt=1673110, reserved=true, lastAck=1673184,
nodeLeft=false, node=TcpDiscoveryNode
[id=839c980a-5d38-416c-9b82-32ce9dfefc4c, addrs=[0:0:0:0:0:0:0:1%lo,
10.62.18.32, 127.0.0.1, 172.16.0.11], sockAddrs=[/0:0:0:0:0:0:0:1%lo:0,
/127.0.0.1:0, /172.16.0.11:0, /10.62.18.32:0], discPort=0, order=7,
intOrder=6, lastExchangeTime=1543950304173, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=true], connected=true,
connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false],
super=GridNioSessionImpl [locAddr=/172.16.0.4:47100,
rmtAddr=/172.16.0.11:34656, createTime=1543950304452, closeTime=0,
bytesSent=608945515, bytesRcvd=686391430, bytesSent0=0, bytesRcvd0=0,
sndSchedTime=1543950304452, lastSndTime=1543950400759,
lastRcvTime=1543950405400, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5c6f7548, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]]
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:51)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1649)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1306)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2342)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)
[19:07:41,332][SEVERE][tcp-disco-msg-worker-#2][TcpDiscoverySpi]
TcpDiscoverSpi's message worker thread failed abnormally. Stopping the node
in order to prevent cluster wide instability.
java.lang.OutOfMemoryError: Java heap space
        at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76)
        at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.socketStream(TcpDiscoverySpi.java:1424)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.sendMessageAcrossRing(ServerImpl.java:2962)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMetricsUpdateMessage(ServerImpl.java:5154)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2756)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2536)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6775)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2621)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
^[[A^[[A^[[5~      at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1649)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1306)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2342)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)



can i know what could be the issue in ignite side?
Also is my proportion looks correct compared to no of operations?
recordcount=4000000
operationcount=4000000   --> 4 million operations
insertproportion=0.23    - ---> 23% insert
updateproportion=0.46  - ---> 46% update
readproportion=0.31      - ---> 31% read
threadcount=180
target=180000

Please clarify.

Thanks,
...summa




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
summasumma summasumma
Reply | Threaded
Open this post in threaded view
|

Re: Ignite node stops on read/insert/update operations from ycsb

Hi Ilya,

The outofmemory error is coming after 5 seconds from the following 1st
error: So i suspect the heap is increased only after initially hitting the
following issue.

"[19:07:36,283][SEVERE][grid-nio-worker-tcp-comm-3-#28][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker.... "

Also one more update is if i enable connectionpair, this issue goes away
with exactly same setup/parameters. So trying to find the connection.

Thanks,
...summa



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite node stops on read/insert/update operations from ycsb

Hello!

Maybe you queue too much data on one side so that it cannot be transferred to destination in time. If you have connection pair, there's less congestion and maybe you are able to free that data. You could try to fix this by limiting queueing:

"Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides."

This is by setting tcpCommunicationSpi.messageQueueLimit.

Regards,
--
Ilya Kasnacheev


чт, 6 дек. 2018 г. в 10:19, summasumma <[hidden email]>:
Hi Ilya,

The outofmemory error is coming after 5 seconds from the following 1st
error: So i suspect the heap is increased only after initially hitting the
following issue.

"[19:07:36,283][SEVERE][grid-nio-worker-tcp-comm-3-#28][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker.... "

Also one more update is if i enable connectionpair, this issue goes away
with exactly same setup/parameters. So trying to find the connection.

Thanks,
...summa



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
summasumma summasumma
Reply | Threaded
Open this post in threaded view
|

Re: Ignite node stops on read/insert/update operations from ycsb

Thanks for the details, Ilya.

Even with connpair enabled, i got same issue when i pump in more data from
ycsb.
Then enabled MessageQueue=1024 as you have mentioned and now the issue is
gone.

Not sure how to arrive at correct number for messagequeue though? can u pls
provide some inputs here since CPU also not utilized much.?

One more clarification:
I see the network bandwidth is the limit getting hit in ignite. so wanted to
scale horizontally and added one more interface with 1Gbps. But when i added
following property under tcpcommunucationspi its throwing error and ignite
is not starting:
<property name=&quot;localAddress&quot; value=&lt;ip2>>

Is it not possible to use Ignite node to start listening on 2 IP addresses
on 2 different interface in a VM?
Actually i have 3 interfaces in my VM, but the 1st interface is mgmt-port.
So wanted to bind on remaining 2 interfaces. Is it possible?
Note: I do not want to start another node with localaddress property set to
another IP since it will again occupy same amount of cache (in-memory) which
will warrant to increase ram again.

Please clarify.

Thanks,
...summa



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite node stops on read/insert/update operations from ycsb

Hello!

You have to try different values of queue length, see what works best for you.

Ignite will not listen 2 different IPs. I'm pretty sure there are ways to turn 2 different physical interfaces into one logical one and bind on that.
By setting local address you will specify which one you want to bind on.

Regards,
--
Ilya Kasnacheev


чт, 6 дек. 2018 г. в 22:24, summasumma <[hidden email]>:
Thanks for the details, Ilya.

Even with connpair enabled, i got same issue when i pump in more data from
ycsb.
Then enabled MessageQueue=1024 as you have mentioned and now the issue is
gone.

Not sure how to arrive at correct number for messagequeue though? can u pls
provide some inputs here since CPU also not utilized much.?

One more clarification:
I see the network bandwidth is the limit getting hit in ignite. so wanted to
scale horizontally and added one more interface with 1Gbps. But when i added
following property under tcpcommunucationspi its throwing error and ignite
is not starting:
<property name=&quot;localAddress&quot; value=&lt;ip2>>

Is it not possible to use Ignite node to start listening on 2 IP addresses
on 2 different interface in a VM?
Actually i have 3 interfaces in my VM, but the 1st interface is mgmt-port.
So wanted to bind on remaining 2 interfaces. Is it possible?
Note: I do not want to start another node with localaddress property set to
another IP since it will again occupy same amount of cache (in-memory) which
will warrant to increase ram again.

Please clarify.

Thanks,
...summa



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
summasumma summasumma
Reply | Threaded
Open this post in threaded view
|

Re: Ignite node stops on read/insert/update operations from ycsb

Thanks for the clarifications, Ilya. Will try bonding, may be.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/