Replace or Put after PutAsync causes Ignite to hang

classic Classic list List threaded Threaded
7 messages Options
Barney Pippin Barney Pippin
Reply | Threaded
Open this post in threaded view
|

Replace or Put after PutAsync causes Ignite to hang

This post was updated on .
Hi,

I'm working on a .Net Core (2+) application that uses Ignite. I've noticed some odd behaviour and have narrowed it down to a Put or a Replace call hanging after a PutAsync.

I started looking at the issue when one of our nodes was randomly logging a possible starvation in striped pool warning, hence the reduction of StripedThreadPoolSize to 1 to reproduce. However, I can't be totally sure that the hang I'm seeing in my test app is the same as with the simple configuration I'm using I'm not sure how to specify quiet mode to false.

I only see the issue being reproduced when I compile and run the test app leaving it running, then run a second instance of the same application.

The code below has no transactions specified as I stripped them out to keep the example minimal but we do need to use them in our actual application, hence the CacheAtomicityMode.Transactional setting.

I'm working with Visual Studio 2017 and Ignite 2.6. C# 7.3.

Can anyone help as to why this might be hanging? Is the test just nonsense as I've set the StripedThreadPoolSize to 1? If so why?

using System;
using System.Threading;
using System.Threading.Tasks;
using Apache.Ignite.Core;
using Apache.Ignite.Core.Cache.Configuration;

namespace IgniteSimple
{
    public static class Program
    {
        public static async Task Main(string[] args)
        {
            using (var ignite = Ignition.Start(new IgniteConfiguration() { StripedThreadPoolSize = 1 }))
            {
                var cache = ignite.GetOrCreateCache<int, string>(
                    new CacheConfiguration("TestCache")
                    {
                        AtomicityMode = CacheAtomicityMode.Transactional,
                        CacheMode = CacheMode.Partitioned
                    });

                Console.WriteLine("PutAsync");
                await cache.PutAsync(1, "Test");

                Console.WriteLine("Replace");
                cache.Replace(1, "Testing"); // Hangs here

                Console.WriteLine("Wait");
                await Task.Delay(Timeout.Infinite);
            }
        }
    }
}

Thanks.

Barney Pippin Barney Pippin
Reply | Threaded
Open this post in threaded view
|

Re: Replace or Put after PutAsync causes Ignite to hang

After upgrading to Ignite 2.7 (via nuget) I get the following output
alongside the hang (I suspect this extra logging info is due to
IGNITE-6587):

PutAsync

Replace

Apr 12, 2019 8:56:09 AM org.apache.ignite.logger.java.JavaLogger error

SEVERE: Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=sys-stripe-0, blockedFor=11s]

Apr 12, 2019 8:56:09 AM java.util.logging.LogManager$RootLogger log

SEVERE: Critical system error detected. Will be handled accordingly to
configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false,
timeout=0, super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=sys-stripe-0, igniteInstanceName=null, finished=false,
heartbeatTs=1555055758004]]]

class org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-0,
igniteInstanceName=null, finished=false, heartbeatTs=1555055758004]
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
        at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
        at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Alexandr Shapkin Alexandr Shapkin
Reply | Threaded
Open this post in threaded view
|

Re: Replace or Put after PutAsync causes Ignite to hang

In reply to this post by Barney Pippin
Hi,

I took a look at the reproducer and it works just fine with different Ignite
and .net versions.

Is there just a single Ignite server with the default config?









--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Barney Pippin Barney Pippin
Reply | Threaded
Open this post in threaded view
|

RE: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

This post was updated on .
Hi,

Thanks for looking. You need to run two instances of the reproducer. Let the first run until you can see "Wait" on the console then run the second. For me the second instance won't get past the Replace call in either 2.6 or 2.7.

It's using the default config with nothing else set up over and above the code you can see on the first post.

Thanks

-----Original Message-----
From: Alexandr Shapkin <lexwert@gmail.com> 
Sent: 15 April 2019 11:46
To: user@ignite.apache.org
Subject: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Hi,

I took a look at the reproducer and it works just fine with different Ignite and .net versions.

Is there just a single Ignite server with the default config?









--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


Barney Pippin Barney Pippin
Reply | Threaded
Open this post in threaded view
|

RE: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Hi, is anyone else seeing this or able to reproduce?

Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Hello!

Yes, I can reproduce this scenario.

I attribute it to async/await, where there is a .Net callback which is invoked from cache operation and which calls cache operations, and this leads to attempts of scheduling striped pool operations to same stripe as cuirrently occupied.

Pavel, what do you think of that? Stack trace below:

"sys-stripe-0-#1" #12 prio=5 os_prio=0 tid=0x000055cecf9e2800 nid=0x3358 waiting on condition [0x00007f25b85c7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2470)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2468)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4233)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2468)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter.replace(GridCacheAdapter.java:2896)
        at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.replace(IgniteCacheProxyImpl.java:1294)
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.replace(GatewayProtectedCacheProxy.java:1012)
        at org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInStreamOutLong(PlatformCache.java:483)
        at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67)
        at org.apache.ignite.internal.processors.platform.callback.PlatformCallbackUtils.inLongOutLong(Native Method)
        at org.apache.ignite.internal.processors.platform.callback.PlatformCallbackGateway.futureNullResult(PlatformCallbackGateway.java:643)
        at org.apache.ignite.internal.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:208)
        at org.apache.ignite.internal.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:189)
        at org.apache.ignite.internal.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:382)
        at org.apache.ignite.internal.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:377)
        at org.apache.ignite.internal.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:215)
        at org.apache.ignite.internal.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:179)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
        at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
        at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
        at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1.apply(GridCacheAdapter.java:5022)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1.apply(GridCacheAdapter.java:5017)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
        at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
        at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
        at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
        at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
        at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
        at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
        at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
        at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
        at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
        at org.apache.ignite.internal.processors.cache.GridCacheCompoundIdentityFuture.onDone(GridCacheCompoundIdentityFuture.java:56)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
        at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.onDone(GridNearTxFinishFuture.java:359)
        - locked <0x00000005cbbcbda0> (a org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture)
        at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.onDone(GridNearTxFinishFuture.java:75)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
        at org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
        at org.apache.ignite.internal.util.future.GridCompoundFuture.markInitialized(GridCompoundFuture.java:276)
        at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.doFinish(GridNearTxFinishFuture.java:510)
        at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.finish(GridNearTxFinishFuture.java:425)
        at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal$25.apply(GridNearTxLocal.java:3788)
        at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal$25.apply(GridNearTxLocal.java:3782)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
        at org.apache.ignite.internal.processors.cache.GridCacheCompoundFuture.onDone(GridCacheCompoundFuture.java:56)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
        at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.onComplete(GridNearOptimisticTxPrepareFuture.java:307)
        at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.onDone(GridNearOptimisticTxPrepareFuture.java:285)
        at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.onDone(GridNearOptimisticTxPrepareFuture.java:79)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
        at org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
        at org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
        at org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
        at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture$MiniFuture.onResult(GridNearOptimisticTxPrepareFuture.java:1046)
        at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.onResult(GridNearOptimisticTxPrepareFuture.java:195)
        at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareResponse(IgniteTxHandler.java:744)
        at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$100(IgniteTxHandler.java:119)
        at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$2.apply(IgniteTxHandler.java:205)
        at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$2.apply(IgniteTxHandler.java:203)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

Regards,
--
Ilya Kasnacheev


чт, 25 апр. 2019 г. в 12:59, Barney Pippin <[hidden email]>:
Hi, is anyone else seeing this or able to reproduce?

Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Barney Pippin Barney Pippin
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Thanks for the response Ilya. Did you get a chance to look at this Pavel?
Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/