Call to getCache never returns on CLIENT_NODE_RECONNECTED event

classic Classic list List threaded Threaded
2 messages Options
DanieleBosetti DanieleBosetti
Reply | Threaded
Open this post in threaded view
|

Call to getCache never returns on CLIENT_NODE_RECONNECTED event

Hi,

I am trying to implement a mechanism so that clients can recover from a full
cluster shutdown.
Basically on CLIENT_RECONNECTION, my code closes a QueryCursor and then
tries issuing another query.

The reconnection handling code is executed by a "disco-event-worker-#..."
thread; and when executing IgniteKernal.getOrCreateCache (from the discovery
thread); the call never returns. (stacktrace below).

If the very same code is executed asynchronously (not in this
"discovery-worker" thread), then the IgniteKernal.getOrCreateCache executes
normally.

I suspect the cause of this problem is that there some sort of deadlock,
caused by calling getOrCreateCache from this thread before the reconnection
event (that we are processing) is forwarded to all other listeners.

Would this make sense/is that expected?


Thanks!


*Register to the reconnection event*

    KeepAliveQueryListener keepAliveListener = new
KeepAliveQueryListener(ignite, queryBuilder, listener);
    ignite.events().localListen(keepAliveListener,
EventType.EVT_CLIENT_NODE_RECONNECTED);

*event handling:*

class KeepAliveQueryListener { ...
  @Override
  public boolean apply(DiscoveryEvent de) {
    exec(() -> {
        logger.info("on event: type={} ", de.name());
        if (de.type() == EventType.EVT_CLIENT_NODE_RECONNECTED) {
          logger.warn("restarting CQ on client reconnection..");
          startQuery();
          logger.warn("restarted CQ");
        }
    });
    return true;
  }
}
  private static final Executor es = Executors.newFixedThreadPool(4);
  private void exec(Runnable r) {
    // asynch vs sync
    r.run();
    //es.execute(r);
  }

class QueryBuilder { ...
  public IgniteCache<BinaryObject, BinaryObject> getCache() {
    logger.info("getCache name={} called..", table);
    IgniteCache<BinaryObject, BinaryObject> res =
ignite.getOrCreateCache(table).withKeepBinary();
    logger.info("getCache name={} done", table);
    return res;
  }
}

*stacktrace*
"disco-event-worker-#122%listener-client%" #167 prio=5 os_prio=0
tid=0x000000002019f000 nid=0x7934 waiting on condition [0x0000000040fbe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3107)
        at cachelistener.impl.ignite.QueryBuilder.getCache(QueryBuilder.java:85)
        at
cachelistener.impl.ignite.KeepAliveQueryListener.startQuery(KeepAliveQueryListener.java:43)
        at
cachelistener.impl.ignite.KeepAliveQueryListener.lambda$0(KeepAliveQueryListener.java:67)
        at
cachelistener.impl.ignite.KeepAliveQueryListener$$Lambda$28/1019213101.run(Unknown
Source)
        at
cachelistener.impl.ignite.KeepAliveQueryListener.exec(KeepAliveQueryListener.java:56)
        at
cachelistener.impl.ignite.KeepAliveQueryListener.apply(KeepAliveQueryListener.java:62)
        at
cachelistener.impl.ignite.KeepAliveQueryListener.apply(KeepAliveQueryListener.java:1)
        at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager$UserListenerWrapper.onEvent(GridEventStorageManager.java:1477)
        at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:873)
        at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:858)
        at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record0(GridEventStorageManager.java:341)
        at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:307)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:2703)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2920)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2732)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Call to getCache never returns on CLIENT_NODE_RECONNECTED event

Hi,

Yes, it is a bad idea to do such cache operations in discovery thread and this may lead to the deadlock. You can start a different thread if you want to do these operations.

Evgenii

ср, 10 окт. 2018 г. в 14:35, DanieleBosetti <[hidden email]>:
Hi,

I am trying to implement a mechanism so that clients can recover from a full
cluster shutdown.
Basically on CLIENT_RECONNECTION, my code closes a QueryCursor and then
tries issuing another query.

The reconnection handling code is executed by a "disco-event-worker-#..."
thread; and when executing IgniteKernal.getOrCreateCache (from the discovery
thread); the call never returns. (stacktrace below).

If the very same code is executed asynchronously (not in this
"discovery-worker" thread), then the IgniteKernal.getOrCreateCache executes
normally.

I suspect the cause of this problem is that there some sort of deadlock,
caused by calling getOrCreateCache from this thread before the reconnection
event (that we are processing) is forwarded to all other listeners.

Would this make sense/is that expected?


Thanks!


*Register to the reconnection event*

    KeepAliveQueryListener keepAliveListener = new
KeepAliveQueryListener(ignite, queryBuilder, listener);
    ignite.events().localListen(keepAliveListener,
EventType.EVT_CLIENT_NODE_RECONNECTED);

*event handling:*

class KeepAliveQueryListener { ...
  @Override
  public boolean apply(DiscoveryEvent de) {
    exec(() -> {
        logger.info("on event: type={} ", de.name());
        if (de.type() == EventType.EVT_CLIENT_NODE_RECONNECTED) {
          logger.warn("restarting CQ on client reconnection..");
          startQuery();
          logger.warn("restarted CQ");
        }
    });
    return true;
  }
}
  private static final Executor es = Executors.newFixedThreadPool(4);
  private void exec(Runnable r) {
    // asynch vs sync
    r.run();
    //es.execute(r);
  }

class QueryBuilder { ...
  public IgniteCache<BinaryObject, BinaryObject> getCache() {
    logger.info("getCache name={} called..", table);
    IgniteCache<BinaryObject, BinaryObject> res =
ignite.getOrCreateCache(table).withKeepBinary();
    logger.info("getCache name={} done", table);
    return res;
  }
}

*stacktrace*
"disco-event-worker-#122%listener-client%" #167 prio=5 os_prio=0
tid=0x000000002019f000 nid=0x7934 waiting on condition [0x0000000040fbe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3107)
        at cachelistener.impl.ignite.QueryBuilder.getCache(QueryBuilder.java:85)
        at
cachelistener.impl.ignite.KeepAliveQueryListener.startQuery(KeepAliveQueryListener.java:43)
        at
cachelistener.impl.ignite.KeepAliveQueryListener.lambda$0(KeepAliveQueryListener.java:67)
        at
cachelistener.impl.ignite.KeepAliveQueryListener$$Lambda$28/1019213101.run(Unknown
Source)
        at
cachelistener.impl.ignite.KeepAliveQueryListener.exec(KeepAliveQueryListener.java:56)
        at
cachelistener.impl.ignite.KeepAliveQueryListener.apply(KeepAliveQueryListener.java:62)
        at
cachelistener.impl.ignite.KeepAliveQueryListener.apply(KeepAliveQueryListener.java:1)
        at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager$UserListenerWrapper.onEvent(GridEventStorageManager.java:1477)
        at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:873)
        at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:858)
        at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record0(GridEventStorageManager.java:341)
        at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:307)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:2703)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2920)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2732)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/