what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache

classic Classic list List threaded Threaded
7 messages Options
krishd krishd
Reply | Threaded
Open this post in threaded view
|

what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache

We have a requirement where Changes to data from Multiple DBs need to be periodically & automatically Pushed (not sure how) into various Ignite Caches
Once the Data is available in the Ignite Caches, it will be persisted using Ignite Native Persistence, so that in the event of a crash, the Data can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .  

What is the Best technique for having Changes to data from Multiple DBs to be automatically put into the Ignite Caches ?

While searching for this solution i came across this link :
http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :
==============
Ignite does not provide such integration out of the box, however there a commercial offering from GridGain for that: 
https://docs.gridgain.com/docs/goldengate-replication
==============

Was wondering whether we still need to use GoldenGate for such replications OR whether newer versions of Ignite are now supporting such asynchronous sync-ups with underlying DB changes

Thanks

=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you

Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache

There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then you can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the external DB.

But the recommended way to do it is to perform writing on Ignite.
Cache store with write-through enabled will take care of writing the data into the external DB.

Denis

вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <[hidden email]>:
We have a requirement where Changes to data from Multiple DBs need to be periodically & automatically Pushed (not sure how) into various Ignite Caches
Once the Data is available in the Ignite Caches, it will be persisted using Ignite Native Persistence, so that in the event of a crash, the Data can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .  

What is the Best technique for having Changes to data from Multiple DBs to be automatically put into the Ignite Caches ?

While searching for this solution i came across this link :
http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :
==============
Ignite does not provide such integration out of the box, however there a commercial offering from GridGain for that: 
https://docs.gridgain.com/docs/goldengate-replication
==============

Was wondering whether we still need to use GoldenGate for such replications OR whether newer versions of Ignite are now supporting such asynchronous sync-ups with underlying DB changes

Thanks

=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you

krishd krishd
Reply | Threaded
Open this post in threaded view
|

Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache

Thanks Denis.

But as I mentioned in earlier mail, the Caches are meant to be Read-Only (only to be used by Microservices for fetching data).  
The Databases are updated by backoffice legacy systems.  Hence we cannot do a Write-through to the DBs via the CacheStore API.

If anyone has used the Gridgain GoldenGate Adapter, then we would be glad to hear about any challenges/short-comings if any.

Regards



From:        Denis Mekhanikov <[hidden email]>
To:        [hidden email]
Date:        06-08-2018 13:18
Subject:        Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache




There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then you can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the external DB.

But the recommended way to do it is to perform writing on Ignite.
Cache store with write-through enabled will take care of writing the data into the external DB.

Denis

вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <deepa.kolwalkar@...>:
We have a requirement where Changes to data from Multiple DBs need to be periodically & automatically Pushed (not sure how) into various Ignite Caches
Once the Data is available in the Ignite Caches, it will be persisted using Ignite Native Persistence, so that in the event of a crash, the Data can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .  


What is the Best technique for having Changes to data from Multiple DBs to be automatically put into the Ignite Caches ?


While searching for this solution i came across this link :

http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :

==============

Ignite does not provide such integration out of the box, however there a commercial offering from GridGain for that: 

https://docs.gridgain.com/docs/goldengate-replication
==============


Was wondering whether we still need to use GoldenGate for such replications OR whether newer versions of Ignite are now supporting such asynchronous sync-ups with underlying DB changes


Thanks

=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you

prasadbhalerao1983 prasadbhalerao1983
Reply | Threaded
Open this post in threaded view
|

Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache

Can this back office legacy system send you a DB update message or can you make this back office system to send you DB update message?

If yes then you can have the Id/primary key, DB operation and table name in this DB update message.

In your application you use this information to refresh your cache using read through mechanism.

Thanks,
Prasad

On Mon, Aug 6, 2018, 3:02 PM Deepa Kolwalkar <[hidden email]> wrote:
Thanks Denis.

But as I mentioned in earlier mail, the Caches are meant to be Read-Only (only to be used by Microservices for fetching data).  
The Databases are updated by backoffice legacy systems.  Hence we cannot do a Write-through to the DBs via the CacheStore API.

If anyone has used the Gridgain GoldenGate Adapter, then we would be glad to hear about any challenges/short-comings if any.

Regards



From:        Denis Mekhanikov <[hidden email]>
To:        [hidden email]
Date:        06-08-2018 13:18
Subject:        Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache




There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then you can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the external DB.

But the recommended way to do it is to perform writing on Ignite.
Cache store with write-through enabled will take care of writing the data into the external DB.

Denis

вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <[hidden email]>:
We have a requirement where Changes to data from Multiple DBs need to be periodically & automatically Pushed (not sure how) into various Ignite Caches
Once the Data is available in the Ignite Caches, it will be persisted using Ignite Native Persistence, so that in the event of a crash, the Data can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .  


What is the Best technique for having Changes to data from Multiple DBs to be automatically put into the Ignite Caches ?


While searching for this solution i came across this link :

http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :

==============

Ignite does not provide such integration out of the box, however there a commercial offering from GridGain for that: 

https://docs.gridgain.com/docs/goldengate-replication
==============


Was wondering whether we still need to use GoldenGate for such replications OR whether newer versions of Ignite are now supporting such asynchronous sync-ups with underlying DB changes


Thanks

=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you

krishd krishd
Reply | Threaded
Open this post in threaded view
|

Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache

Thanks Prasad for your suggestions.

The Legacy sytems are on different platforms and some of them are products .. so there is no way of implementing any custom logic in such products to send update messages. The Legacy Systems remain a black-box to us with only its DB which is accessible for viewing..

Regards





From:        "Prasad Bhalerao" <[hidden email]>
To:        [hidden email]
Date:        06-08-2018 16:51
Subject:        Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache




Can this back office legacy system send you a DB update message or can you make this back office system to send you DB update message?

If yes then you can have the Id/primary key, DB operation and table name in this DB update message.

In your application you use this information to refresh your cache using read through mechanism.

Thanks,
Prasad

On Mon, Aug 6, 2018, 3:02 PM Deepa Kolwalkar <deepa.kolwalkar@...> wrote:
Thanks Denis.

But as I mentioned in earlier mail, the Caches are meant to be Read-Only (only to be used by Microservices for fetching data).  

The Databases are updated by backoffice legacy systems.  Hence we cannot do a Write-through to the DBs via the CacheStore API.


If anyone has used the Gridgain GoldenGate Adapter, then we would be glad to hear about any challenges/short-comings if any.


Regards




From:        
Denis Mekhanikov <dmekhanikov@...>
To:        
user@...
Date:        
06-08-2018 13:18
Subject:        
Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache




There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then you can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the external DB.

But the recommended way to do it is to perform writing on Ignite.
Cache store with
write-through enabled will take care of writing the data into the external DB.

Denis

вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <
deepa.kolwalkar@...>:
We have a requirement where Changes to data from Multiple DBs need to be periodically & automatically Pushed (not sure how) into various Ignite Caches

Once the Data is available in the Ignite Caches, it will be persisted using Ignite Native Persistence, so that in the event of a crash, the Data can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .  

What is the Best technique for having Changes to data from Multiple DBs to be automatically put into the Ignite Caches ?


While searching for this solution i came across this link :

http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :

==============

Ignite does not provide such integration out of the box, however there a commercial offering from GridGain for that: 

https://docs.gridgain.com/docs/goldengate-replication
==============


Was wondering whether we still need to use GoldenGate for such replications OR whether newer versions of Ignite are now supporting such asynchronous sync-ups with underlying DB changes


Thanks

=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you

Juan Rodríguez Hortalá Juan Rodríguez Hortalá
Reply | Threaded
Open this post in threaded view
|

Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache

That looks like something you could do with Kafka connect https://www.confluent.io/product/connectors/ using the jdbc source and the ignite sink. Just my 2 cents 

On Mon, Aug 6, 2018, 07:19 Deepa Kolwalkar <[hidden email]> wrote:
Thanks Prasad for your suggestions.

The Legacy sytems are on different platforms and some of them are products .. so there is no way of implementing any custom logic in such products to send update messages. The Legacy Systems remain a black-box to us with only its DB which is accessible for viewing..

Regards





From:        "Prasad Bhalerao" <[hidden email]>
To:        [hidden email]
Date:        06-08-2018 16:51
Subject:        Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache




Can this back office legacy system send you a DB update message or can you make this back office system to send you DB update message?

If yes then you can have the Id/primary key, DB operation and table name in this DB update message.

In your application you use this information to refresh your cache using read through mechanism.

Thanks,
Prasad

On Mon, Aug 6, 2018, 3:02 PM Deepa Kolwalkar <[hidden email]> wrote:
Thanks Denis.

But as I mentioned in earlier mail, the Caches are meant to be Read-Only (only to be used by Microservices for fetching data).  

The Databases are updated by backoffice legacy systems.  Hence we cannot do a Write-through to the DBs via the CacheStore API.


If anyone has used the Gridgain GoldenGate Adapter, then we would be glad to hear about any challenges/short-comings if any.


Regards




From:        
Denis Mekhanikov <[hidden email]>
To:        
[hidden email]
Date:        
06-08-2018 13:18
Subject:        
Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache




There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then you can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the external DB.

But the recommended way to do it is to perform writing on Ignite.
Cache store with
write-through enabled will take care of writing the data into the external DB.

Denis

вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <
[hidden email]>:
We have a requirement where Changes to data from Multiple DBs need to be periodically & automatically Pushed (not sure how) into various Ignite Caches

Once the Data is available in the Ignite Caches, it will be persisted using Ignite Native Persistence, so that in the event of a crash, the Data can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .  

What is the Best technique for having Changes to data from Multiple DBs to be automatically put into the Ignite Caches ?


While searching for this solution i came across this link :

http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :

==============

Ignite does not provide such integration out of the box, however there a commercial offering from GridGain for that: 

https://docs.gridgain.com/docs/goldengate-replication
==============


Was wondering whether we still need to use GoldenGate for such replications OR whether newer versions of Ignite are now supporting such asynchronous sync-ups with underlying DB changes


Thanks

=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you

krishd krishd
Reply | Threaded
Open this post in threaded view
|

Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache

Thanks Juan ..  I will check on the same




From:        "Juan Rodríguez Hortalá" <[hidden email]>
To:        [hidden email]
Date:        07-08-2018 08:44
Subject:        Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache




That looks like something you could do with Kafka connect https://www.confluent.io/product/connectors/ using the jdbc source and the ignite sink. Just my 2 cents 

On Mon, Aug 6, 2018, 07:19 Deepa Kolwalkar <deepa.kolwalkar@...> wrote:
Thanks Prasad for your suggestions.

The Legacy sytems are on different platforms and some of them are products .. so there is no way of implementing any custom logic in such products to send update messages. The Legacy Systems remain a black-box to us with only its DB which is accessible for viewing..


Regards






From:        
"Prasad Bhalerao" <prasadbhalerao1983@...>
To:        
user@...
Date:        
06-08-2018 16:51
Subject:        
Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache




Can this back office legacy system send you a DB update message or can you make this back office system to send you DB update message?

If yes then you can have the Id/primary key, DB operation and table name in this DB update message.

In your application you use this information to refresh your cache using read through mechanism.

Thanks,
Prasad

On Mon, Aug 6, 2018, 3:02 PM Deepa Kolwalkar <
deepa.kolwalkar@...> wrote:
Thanks Denis.


But as I mentioned in earlier mail, the Caches are meant to be Read-Only (only to be used by Microservices for fetching data).  

The Databases are updated by backoffice legacy systems.  Hence we cannot do a Write-through to the DBs via the CacheStore API.


If anyone has used the Gridgain GoldenGate Adapter, then we would be glad to hear about any challenges/short-comings if any.

Regards




From:        
Denis Mekhanikov <dmekhanikov@...>
To:        
user@...
Date:        
06-08-2018 13:18
Subject:        
Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache




There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then you can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the external DB.

But the recommended way to do it is to perform writing on Ignite.
Cache store with
write-through enabled will take care of writing the data into the external DB.

Denis

вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <
deepa.kolwalkar@...>:
We have a requirement where Changes to data from Multiple DBs need to be periodically & automatically Pushed (not sure how) into various Ignite Caches

Once the Data is available in the Ignite Caches, it will be persisted using Ignite Native Persistence, so that in the event of a crash, the Data can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .  

What is the Best technique for having Changes to data from Multiple DBs to be automatically put into the Ignite Caches ?


While searching for this solution i came across this link :

http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :

==============

Ignite does not provide such integration out of the box, however there a commercial offering from GridGain for that: 

https://docs.gridgain.com/docs/goldengate-replication
==============


Was wondering whether we still need to use GoldenGate for such replications OR whether newer versions of Ignite are now supporting such asynchronous sync-ups with underlying DB changes


Thanks

=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you