2.8.1 : Server cluster node startup issue

classic Classic list List threaded Threaded
6 messages Options
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

2.8.1 : Server cluster node startup issue

Hello,

Our set up :
Server nodes : 3
Writer application : 1

A event storage spi that writes the cache_put events that have been
generated due to a dbeaver update into a ignite cache.

We restarted one of the servers yesterday during regular business hours and
noticed that the server was not starting properly. I took a thread dump and
noticed that the following two threads were in locked state .

1. event storage spi : The thread that gets the ignite using
Ignition.ignite(ignitenamedinstance )
2. The server : The thread that starts ignite using Ignition.start(..).

Am I right in assuming that the above two events have locked against each
other ?

How can I avoid this ?In the event storage spi before I do Ignition.ignite ,
is there any way I can check if the ignite in the current server has been
started ?

The thread dump has been attached.
threaddump.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t2757/threaddump.txt>  

regards,
Veena.




                                                                                                             




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
akorensh akorensh
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : Server cluster node startup issue

Hi,
  Going by the thread dump, the event api thread is holding a lock others
need to proceed,
  while event storage thread is itself waiting for another condition to
proceed.


  The method you are calling determined that you are not accessing ignite
from the same thread
  that started it and is waiting for a signal that the Ignite has fully
started from the starting thread.

https://github.com/apache/ignite/blob/f4b30f7f1e736845ffa8eaf2d8aa171700a928eb/modules/core/src/main/java/org/apache/ignite/internal/IgnitionEx.java#L1685

          IgniteKernal grid() {
            if (starterThread != Thread.currentThread())
                U.awaitQuiet(startLatch);

            return grid;
        }


 
  Is there a reason you are using Igintion.ignite(..) inside the
eventstorage api?
  Try using Ignition.localIgnite, it might help your use case, if you need a
handle to the Ignite kernal.

 
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/Ignition.html#localIgnite--


  It is best to redesign your event storage system to not call these
methods.
Thanks, Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : Server cluster node startup issue

Hi Alex,



1. How is the storage spi record method getting invoked on this node if the
ignite has not even fully started on this node ? Which ignite instance is it
using to invoke the record method ?

I have enabled only these events :
        <property name="includeEventTypes">
            <list>

                <util:constant
static-field="org.apache.ignite.events.EventType.EVT_TX_STARTED"/>
                <util:constant
                       
static-field="org.apache.ignite.events.EventType.EVT_CLIENT_NODE_RECONNECTED"/>
               
                <util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_PUT"/>
                <util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_REMOVED"/>
            </list>
        </property>


2. >>  Is there a reason you are using Igintion.ignite(..) inside the
eventstorage api?

I use it in the eventstorage spi to get a handle to the audit cache( which
is a ignite cache ) , where i can store the events generated from a cache
modification that was done on dbeaver.


3. >> It is best to redesign your event storage system to not call these
methods.

What is the recommended design to capture and store the event generated from
dbeaver cache update.

regards,
Veena



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
akorensh akorensh
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : Server cluster node startup issue

Veena,
  I am not getting the same issue/thread dump when I run a similar use case.

In your case it looks like Ignite has not fully started before
Ignition.ignite() has been called.
The check inside that function is specifically designed to prevent such
cases.

Here is the function you are calling: -- starterThread -- means the thread
that started Ignite -- usually main
        IgniteKernal grid() {
            if (starterThread != Thread.currentThread())
                U.awaitQuiet(startLatch);

            return grid;
        }

If Ignite hasn't fully started, it will wait( U.awaitQuiet(startLatch); )
until it has, before returning the instance.
see:
https://github.com/apache/ignite/blob/40179116ab85446740a5b7025cd6cfef05bdefe6/modules/core/src/main/java/org/apache/ignite/internal/IgnitionEx.java#L1685


take a look at:
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/eventstorage/memory/MemoryEventStorageSpi.html

and at: https://apacheignite.readme.io/docs/events
also the examples here:

https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/events/EventsExample.java

https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheEventsExample.java


If this doesn't help, include a reproducer and I will take a look.

Thanks, Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : Server cluster node startup issue

This post was updated on .
Thanks Alex,
Will try and see if I can create a reproducer.

>>If Ignite hasn't fully started, it will wait( U.awaitQuiet(startLatch); )
until it has, before returning the instance.

Does the Ignition.start wait for the storage spi to return from record() method, to
start ? since the storage spi is waiting on startLatch and the main thread
which is waiting for storage spi to return, there is a deadlock ?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
akorensh akorensh
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : Server cluster node startup issue

Veena,
  Ignite shouldn't be waiting for the storage spi. It first starts all its
components and when an
  Event is ready to be recorded, the storage SPI is invoked. This scenario
tests out ok.
  Send a reproducer and I'll take a look.
Thanks, Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/