listening for events on transactions

classic Classic list List threaded Threaded
9 messages Options
limabean limabean
Reply | Threaded
Open this post in threaded view
|

listening for events on transactions

Hi,

The Ignite documentation implies that listeners can be set on transactions:

"IgniteTransactions interface contains functionality for starting and completing transactions, as well as subscribing listeners or getting metrics."
http://apacheignite.gridgain.org/v1.6/docs/transactions#ignitetransactions


However, when I look at the interfaces/classes, it is not clear to me how to set transaction listeners.  
The behavior I am looking for is the following:

I'd like to make a call like this:
IgniteTransactionSubsystem.registerListener(myListener...)

When a transaction commits, I would like to have the listener notified (callback) with the full contents of the transaction, even if it spanned multiple caches....or at a minimum provide the cache keys and cache names that were updated in a transaction.

I looked at the CacheJdbcStoreSessionListener example and saw this piece of code in the sample persistence code:
// Configure JDBC session listener.
            cacheCfg.setCacheStoreSessionListenerFactories(new Factory<CacheStoreSessionListener>() {
                @Override public CacheStoreSessionListener create() {
                    CacheJdbcStoreSessionListener lsnr = new CacheJdbcStoreSessionListener();

                    lsnr.setDataSource(JdbcConnectionPool.create("jdbc:h2:tcp://localhost/mem:ExampleDb", "sa", ""));

                    return lsnr;
                }
            });

but it wasn't clear to me how these were supposed to work if they were the correct way to listen for transaction.commits.

Continuous queries appear to offer event capability, but it isn't clear how they work with transactions.

Thanks for the clarifications/help on how to do transaction listeners.
Alexei Scherbakov Alexei Scherbakov
Reply | Threaded
Open this post in threaded view
|

Re: listening for events on transactions

Hi,

I think the documentation is not very precise on tx topic.
Currently where is no way to listen for transaction events.
The only exception is CacheStoreSessionListener interface which works in conjunction with CacheStore.
I suppose you can implement "fake" CacheStore and track transaction activity in corresponding methods.
What's your final goal?


2016-05-31 22:49 GMT+03:00 limabean <[hidden email]>:
Hi,

The Ignite documentation implies that listeners can be set on transactions:

"IgniteTransactions interface contains functionality for starting and
completing transactions, as well as subscribing listeners or getting
metrics."
http://apacheignite.gridgain.org/v1.6/docs/transactions#ignitetransactions


However, when I look at the interfaces/classes, it is not clear to me how to
set transaction listeners.
The behavior I am looking for is the following:

I'd like to make a call like this:
IgniteTransactionSubsystem.registerListener(myListener...)

When a transaction commits, I would like to have the listener notified
(callback) with the full contents of the transaction, even if it spanned
multiple caches....or at a minimum provide the cache keys and cache names
that were updated in a transaction.

I looked at the CacheJdbcStoreSessionListener example and saw this piece of
code in the sample persistence code:
// Configure JDBC session listener.
            cacheCfg.setCacheStoreSessionListenerFactories(new
Factory<CacheStoreSessionListener>() {
                @Override public CacheStoreSessionListener create() {
                    CacheJdbcStoreSessionListener lsnr = new
CacheJdbcStoreSessionListener();


lsnr.setDataSource(JdbcConnectionPool.create("jdbc:h2:tcp://localhost/mem:ExampleDb",
"sa", ""));

                    return lsnr;
                }
            });

but it wasn't clear to me how these were supposed to work if they were the
correct way to listen for transaction.commits.

Continuous queries appear to offer event capability, but it isn't clear how
they work with transactions.

Thanks for the clarifications/help on how to do transaction listeners.




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--

Best regards,
Alexei Scherbakov
limabean limabean
Reply | Threaded
Open this post in threaded view
|

Re: listening for events on transactions

Hi Alexi,

Thank you for the clarification.

My final goal is to notify other processes not related to the grid application about changes to the data caches.  For example, it would be nice to have a Kafka publisher registered as a transaction listener and then when it gets transaction events, publish this information to a Kafka topic.

Ignite is good at pulling data in, but it needs to be equally good at sharing data to work well in my environment.

What do you think ?

Alexei Scherbakov Alexei Scherbakov
Reply | Threaded
Open this post in threaded view
|

Re: listening for events on transactions

You can easily implement such a thing as a part of transaction logic.


2016-06-01 14:52 GMT+03:00 limabean <[hidden email]>:
Hi Alexi,

Thank you for the clarification.

My final goal is to notify other processes not related to the grid
application about changes to the data caches.  For example, it would be nice
to have a Kafka publisher registered as a transaction listener and then when
it gets transaction events, publish this information to a Kafka topic.

Ignite is good at pulling data in, but it needs to be equally good at
sharing data to work well in my environment.

What do you think ?





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346p5357.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--

Best regards,
Alexei Scherbakov
limabean limabean
Reply | Threaded
Open this post in threaded view
|

Re: listening for events on transactions

I can - but apparently not with a separate process that gets notifications - which is the design point I was seeking.
Thanks again.

On Wed, Jun 1, 2016 at 12:55 PM, Alexei Scherbakov <[hidden email]> wrote:
You can easily implement such a thing as a part of transaction logic.


2016-06-01 14:52 GMT+03:00 limabean <[hidden email]>:
Hi Alexi,

Thank you for the clarification.

My final goal is to notify other processes not related to the grid
application about changes to the data caches.  For example, it would be nice
to have a Kafka publisher registered as a transaction listener and then when
it gets transaction events, publish this information to a Kafka topic.

Ignite is good at pulling data in, but it needs to be equally good at
sharing data to work well in my environment.

What do you think ?





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346p5357.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--

Best regards,
Alexei Scherbakov

Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: listening for events on transactions

Hi David,

I would implement this using custom events in the following way:

- introduce your custom event - TransactionEvent and send it every time to subscribers a transaction succeeds using IgniteEvents.recordLocal(…) method. The event can contain any info you need.

- the separate process can subscribe for your custom TransactionEvents, process them and publish to Kafka.

This will let you to decouple Ignite logic from Kafka logic.

Below is a code snippet that demonstrates how to use custom events in general:

Will this work for you?

Denis

On Jun 1, 2016, at 9:09 PM, David Robinson <[hidden email]> wrote:

I can - but apparently not with a separate process that gets notifications - which is the design point I was seeking.
Thanks again.

On Wed, Jun 1, 2016 at 12:55 PM, Alexei Scherbakov <[hidden email]> wrote:
You can easily implement such a thing as a part of transaction logic.


2016-06-01 14:52 GMT+03:00 limabean <[hidden email]>:
Hi Alexi,

Thank you for the clarification.

My final goal is to notify other processes not related to the grid
application about changes to the data caches.  For example, it would be nice
to have a Kafka publisher registered as a transaction listener and then when
it gets transaction events, publish this information to a Kafka topic.

Ignite is good at pulling data in, but it needs to be equally good at
sharing data to work well in my environment.

What do you think ?





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346p5357.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--

Best regards,
Alexei Scherbakov


alexey.goncharuk alexey.goncharuk
Reply | Threaded
Open this post in threaded view
|

Re: listening for events on transactions

David,

Have you considered using continuous queries for your use-case [1]? Even if there were such a thing as a transaction event, I do not see how you can reliably (read - in a proper order) publish this information to Kafka. 

Say, you have 2 clients and 2 server nodes. First client executes a transaction on keys (1, 2) and second client executes a transaction on keys (2, 3). These events would have been fired on client nodes because only client knows when the whole transaction is committed on all nodes. Now you get two unordered events (1, 2) and (2, 3) which gives you an undefined state for key (2), or you need to add some sort of custom logic to order these updates.

The solution would be to listen for updates on primary nodes, which is exactly what continuous queries do. They guarantee that:
  • Updates for each key will be received in a proper order
  • Events are fault-tolerant, meaning that even if primary node crashed, you would still receive an update from backup node


2016-06-01 13:16 GMT-07:00 Denis Magda <[hidden email]>:
Hi David,

I would implement this using custom events in the following way:

- introduce your custom event - TransactionEvent and send it every time to subscribers a transaction succeeds using IgniteEvents.recordLocal(…) method. The event can contain any info you need.

- the separate process can subscribe for your custom TransactionEvents, process them and publish to Kafka.

This will let you to decouple Ignite logic from Kafka logic.

Below is a code snippet that demonstrates how to use custom events in general:

Will this work for you?

Denis

On Jun 1, 2016, at 9:09 PM, David Robinson <[hidden email]> wrote:

I can - but apparently not with a separate process that gets notifications - which is the design point I was seeking.
Thanks again.

On Wed, Jun 1, 2016 at 12:55 PM, Alexei Scherbakov <[hidden email]> wrote:
You can easily implement such a thing as a part of transaction logic.


2016-06-01 14:52 GMT+03:00 limabean <[hidden email]>:
Hi Alexi,

Thank you for the clarification.

My final goal is to notify other processes not related to the grid
application about changes to the data caches.  For example, it would be nice
to have a Kafka publisher registered as a transaction listener and then when
it gets transaction events, publish this information to a Kafka topic.

Ignite is good at pulling data in, but it needs to be equally good at
sharing data to work well in my environment.

What do you think ?





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346p5357.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--

Best regards,
Alexei Scherbakov



limabean limabean
Reply | Threaded
Open this post in threaded view
|

Re: listening for events on transactions

In reply to this post by Denis Magda
Hey Denis,

Thank you for this suggestion.  I plan to take a serious look at what you suggest to
see if it will work for me and will let you know.
limabean limabean
Reply | Threaded
Open this post in threaded view
|

Re: listening for events on transactions

In reply to this post by alexey.goncharuk
Hi Alexey,

I did poke around on continuous queries before.
Based on your recommendation I will take another look to see if they solve my architecture pattern.

Transaction listeners/messages are a common pattern with other technologies.

Here is a suggestion of how Ignite might evolve to be better in this area:

I envision an Ignite server in the grid acting as a transaction coordinator.
(I have Cassandra behavior in mind when thinking through this).

The client code may issue the final "commit" API call, but this would simply
send an indicator to the coordinator in the grid which would handle the commit
across the grid with the data owners.  Then, this coordinator could generate
a transaction message to any registered listeners AND return the transaction status
back to the client.

Guessing about how Ignite works today, there appears to be an assumption
that the client...even the end user client code...has to be the coordinator of
the transaction.  But I don't think it has to remain this way.  A model more
similar to Cassandra behavior (I'm not talking Cassandra transactions - it only
has lightweight, query specific transactions - but more the model of the way
a client interacts with the Cassandras cluster),  might be an interesting model
for Ignite to evolve towards.