Ignite client node raises "Sequence was removed from cache" after Ignite Server node restarts

classic Classic list List threaded Threaded
8 messages Options
xmw45688 xmw45688
Reply | Threaded
Open this post in threaded view
|

Ignite client node raises "Sequence was removed from cache" after Ignite Server node restarts

*USE CASE *- use IgniteAtomicLong for table sequence generation (may not be
correct approach in a distributed environment).

*Ignite Server *(start Ignite as server mode) - apache-ignite-2.8.0.20190215
daily build
*Ignite Service* (start Ignite as client mode) - use Ignite Spring to
initialize the sequence, see code snippet below.  
*code snippet*

IgniteAtomicLong userSeq;

@Autowired
UserRepository userRepository;

@Autowired
Ignite igniteInstance;

@PostConstruct
@Override
public void initSequence() {
Long maxId = userRepository.getMaxId();
if (maxId == null)

{ maxId = 0L; }
LOG.info("Max User id: {}", maxId);
userSeq = igniteInstance.atomicLong("userSeq", maxId, true);
userSeq.getAndSet(maxId);
}

@Override
public Long getNextSequence() {
return userSeq.incrementAndGet();
}

*Exception*
This code works well until the Ignite Server restarted (Ignite Service was
not restarted).  It raised "Sequence was removed from cache" after Ignite
Server node restarted.

020-08-11 16:14:46 [http-nio-8282-exec-3] ERROR
c.p.c.p.service.PersistenceService - Error while saving entity:
java.lang.IllegalStateException: Sequence was removed from cache: userSeq
at
org.apache.ignite.internal.processors.datastructures.AtomicDataStructureProxy.removedError(AtomicDataStructureProxy.java:145)
at
org.apache.ignite.internal.processors.datastructures.AtomicDataStructureProxy.checkRemoved(AtomicDataStructureProxy.java:116)
at
org.apache.ignite.internal.processors.datastructures.GridCacheAtomicLongImpl.incrementAndGet(GridCacheAtomicLongImpl.java:94)

*Tried to reinitialize when the server node is down. But raises another
exception - "cannot start/stop cache within lock or transaction"*

How to solve such issues?  Any suggestions are appreciated.

@Override
public Long getNextSequence() {
if (useSeq == null || userSeq.removed())

{ initSeqence(); }
return userSeq.incrementAndGet();
}






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Mikhail Mikhail
Reply | Threaded
Open this post in threaded view
|

Re: Ignite client node raises "Sequence was removed from cache" after Ignite Server node restarts

Hi, as I remember AtomicInteger uses the default data region as storage, so if you don't have persistence for it, then it will be lost after restart. Try to enable persistence and check it again.

ср, 26 авг. 2020 г. в 18:08, xmw45688 <[hidden email]>:
*USE CASE *- use IgniteAtomicLong for table sequence generation (may not be
correct approach in a distributed environment).

*Ignite Server *(start Ignite as server mode) - apache-ignite-2.8.0.20190215
daily build
*Ignite Service* (start Ignite as client mode) - use Ignite Spring to
initialize the sequence, see code snippet below. 
*code snippet*

IgniteAtomicLong userSeq;

@Autowired
UserRepository userRepository;

@Autowired
Ignite igniteInstance;

@PostConstruct
@Override
public void initSequence() {
Long maxId = userRepository.getMaxId();
if (maxId == null)

{ maxId = 0L; }
LOG.info("Max User id: {}", maxId);
userSeq = igniteInstance.atomicLong("userSeq", maxId, true);
userSeq.getAndSet(maxId);
}

@Override
public Long getNextSequence() {
return userSeq.incrementAndGet();
}

*Exception*
This code works well until the Ignite Server restarted (Ignite Service was
not restarted).  It raised "Sequence was removed from cache" after Ignite
Server node restarted.

020-08-11 16:14:46 [http-nio-8282-exec-3] ERROR
c.p.c.p.service.PersistenceService - Error while saving entity:
java.lang.IllegalStateException: Sequence was removed from cache: userSeq
at
org.apache.ignite.internal.processors.datastructures.AtomicDataStructureProxy.removedError(AtomicDataStructureProxy.java:145)
at
org.apache.ignite.internal.processors.datastructures.AtomicDataStructureProxy.checkRemoved(AtomicDataStructureProxy.java:116)
at
org.apache.ignite.internal.processors.datastructures.GridCacheAtomicLongImpl.incrementAndGet(GridCacheAtomicLongImpl.java:94)

*Tried to reinitialize when the server node is down. But raises another
exception - "cannot start/stop cache within lock or transaction"*

How to solve such issues?  Any suggestions are appreciated.

@Override
public Long getNextSequence() {
if (useSeq == null || userSeq.removed())

{ initSeqence(); }
return userSeq.incrementAndGet();
}






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
xmw45688 xmw45688
Reply | Threaded
Open this post in threaded view
|

Re: Ignite client node raises "Sequence was removed from cache" after Ignite Server node restarts

How do we enable the persistence for this atomicLong? My understanding is
that atomicLong or atomicReference can't be persisted.  Even if it can be
persisted, isn't next value reset and started from the initial value.  

My issue is that the cached sequence was removed from Spring bean.  How do I
re-initialize this sequence once the Ignite server node restarted while the
client node was still running.  

The initialization of the sequence is from the client side using
@PostConstruct.  We need a way to re-initialize with the max value from DB
when the ignite server restarted and the client node was connected to the
ignite server.

@PostConstruct
@Override
public void initSequence() {
    Long maxId = userRepository.getMaxId();
    if (maxId == null) { maxId = 0L; }
    LOG.info("Max User id: {}", maxId);
    userSeq = igniteInstance.atomicLong("userSeq", maxId, true);
    userSeq.getAndSet(maxId);
}
 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite client node raises "Sequence was removed from cache" after Ignite Server node restarts

Hello!

If this is not too much of a slowdown, you can just remove the userSeq field, and always use "igniteInstance.atomicLong("userSeq", maxId, true)" in its place, to make sure you always hold a fresh one.

Regards,
--
Ilya Kasnacheev


пт, 28 авг. 2020 г. в 04:47, xmw45688 <[hidden email]>:
How do we enable the persistence for this atomicLong? My understanding is
that atomicLong or atomicReference can't be persisted.  Even if it can be
persisted, isn't next value reset and started from the initial value. 

My issue is that the cached sequence was removed from Spring bean.  How do I
re-initialize this sequence once the Ignite server node restarted while the
client node was still running. 

The initialization of the sequence is from the client side using
@PostConstruct.  We need a way to re-initialize with the max value from DB
when the ignite server restarted and the client node was connected to the
ignite server.

@PostConstruct
@Override
public void initSequence() {
    Long maxId = userRepository.getMaxId();
    if (maxId == null) { maxId = 0L; }
    LOG.info("Max User id: {}", maxId);
    userSeq = igniteInstance.atomicLong("userSeq", maxId, true);
    userSeq.getAndSet(maxId);
}




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
xmw45688 xmw45688
Reply | Threaded
Open this post in threaded view
|

Re: Ignite client node raises "Sequence was removed from cache" after Ignite Server node restarts

Hi Ilya,

Tried your suggestion, removing the userSeq field from the class.  It still
raise the same exception -
    org.apache.ignite.IgniteException: Cannot start/stop cache within lock
or transaction.
 
The error is thrown when trying to get igniteInstance.atomicLong("userSeq",
maxId, true);

Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite client node raises "Sequence was removed from cache" after Ignite Server node restarts

Hello!

You need to do this outside of your transaction.

Regards,
Ilya.
--
Ilya Kasnacheev


сб, 12 сент. 2020 г. в 02:42, xmw45688 <[hidden email]>:
Hi Ilya,

Tried your suggestion, removing the userSeq field from the class.  It still
raise the same exception -
    org.apache.ignite.IgniteException: Cannot start/stop cache within lock
or transaction.

The error is thrown when trying to get igniteInstance.atomicLong("userSeq",
maxId, true);

Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
xmw45688 xmw45688
Reply | Threaded
Open this post in threaded view
|

Re: Ignite client node raises "Sequence was removed from cache" after Ignite Server node restarts

This post was updated on .
In reply to this post by xmw45688
Thanks.  
We tried it outside the transaction when the ignite instance restarted with EventType.EVT_CLIENT_NODE_RECONNECTED. We listen EVT_CLIENT_NODE_RECONNECTED event in the client side and recreate atomicLong when the client gets this event notified.  However, it's stuck at
igniteInstance.atomicLong(....), and the call never returns.
    // when the ignite instance restarted, it hangs here
    userSeq = igniteInstance.atomicLong("userSeq", maxId, true);
    userSeq.getAndSet(maxId);

We know that the sequence was removed, does the client keep the reference to
the old Ignite server?  How do we clear this reference and reset?  Or we did
it incorrectly.

Some related information on this topic.  We are using ignite 2.8.0 night build.  This issue also happens in Ignite 2.8.0, 2.8.1 as well

http://apache-ignite-users.70518.x6.nabble.com/in-a-four-node-ignite-cluster-use-atomicLong-and-process-is-stuck-in-countdown-latch-td23091.html

https://issues.apache.org/jira/browse/IGNITE-8987


Thanks,
Xinmin




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite client node raises "Sequence was removed from cache" after Ignite Server node restarts

Hello!

I recommend setting clientReconnectDisabled setting to true, and then do your reconnects manually. I have heard of the issue where restarting a client would interfere with data structures.

If you have a reproducer for this problem, I could review it and file a ticket against Apache Ignite JIRA.

Regards,
--
Ilya Kasnacheev


пт, 18 сент. 2020 г. в 02:48, xmw45688 <[hidden email]>:
Thanks. 
We tried it outside the transaction when the ignite instance restarted with
EventType.EVT_CLIENT_NODE_RECONNECTED.  However, it is hang at
igniteInstance.atomicLong(....).
    // when the ignite instance restarted, it hangs here
    userSeq = igniteInstance.atomicLong("userSeq", maxId, true);
    userSeq.getAndSet(maxId);

We know that the sequence was removed, does the client keep the reference to
the old Ignite server?  How do we clear this reference and reset?  Or we did
it incorrectly.

Thanks,
Xinmin




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/