Can Ignite native persistence used with 3rd party persistence?

classic Classic list List threaded Threaded
6 messages Options
Ray Ray
Reply | Threaded
Open this post in threaded view
|

Can Ignite native persistence used with 3rd party persistence?

This post was updated on .
Ignite native persistence provides on-disk sql query and quick cluster
startup without data loading, so we definitely want to use it.
But we have a legacy hbase served as persistence layer and there are some
business rely on it.
So can Ignite native persistence used with 3rd party persistence?

Basically we want data to be persisted both on native persistence and
hbase when new entries goes into Ignite.
And when user queries data not in memory, Ignite will query its native
persistence.

Is this design supported by Ignite?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ray Ray
Reply | Threaded
Open this post in threaded view
|

Re: Can Ignite native persistence used with 3rd party persistence?

http://apache-ignite-users.70518.x6.nabble.com/Two-persistent-data-stores-for-a-single-Ignite-cluster-RDBMS-and-Ignite-native-td18463.html

Found a similar case here, I think I'll try Slava's suggestions first.

One more question here, how can a update or new inserts back-propagate to
Ignite when another application(not ignite) writes to persistence(hbase)?

For example, Ignite and hbase both have one entry for now.
When another application adds an entry to hbase, now hbase has two entries.
Can Ignite be notified and load the newly added entry automatically?

From the document, it looks like the data can only be propagated from Ignite
to persistence, not the other way around.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Andrew Mashenkov Andrew Mashenkov
Reply | Threaded
Open this post in threaded view
|

Re: Can Ignite native persistence used with 3rd party persistence?

Hi Ray,
 
One more question here, how can a update or new inserts back-propagate to
Ignite when another application(not ignite) writes to persistence(hbase)?
 
It is not supported.



On Fri, Dec 1, 2017 at 12:08 PM, Ray <[hidden email]> wrote:
http://apache-ignite-users.70518.x6.nabble.com/Two-persistent-data-stores-for-a-single-Ignite-cluster-RDBMS-and-Ignite-native-td18463.html

Found a similar case here, I think I'll try Slava's suggestions first.

One more question here, how can a update or new inserts back-propagate to
Ignite when another application(not ignite) writes to persistence(hbase)?

For example, Ignite and hbase both have one entry for now.
When another application adds an entry to hbase, now hbase has two entries.
Can Ignite be notified and load the newly added entry automatically?

From the document, it looks like the data can only be propagated from Ignite
to persistence, not the other way around.



--
Best regards,
Andrey V. Mashenkov
Regards,
Andrew.
alexey.goncharuk alexey.goncharuk
Reply | Threaded
Open this post in threaded view
|

Re: Can Ignite native persistence used with 3rd party persistence?

Ray,

With the current API it is impossible to get a reliable integration of Ignite native persistence with 3rd party persistence. The reason is that first, CacheStore interface does not have methods for 2-phase commit, second, it would require significant changes to the persistence layer itself to make a consistent crash recovery.

We could allow setting the cache store interface with write-through from primary nodes, but in this case, it would be a user's responsibility to verify that the cache store is consistent with the Ignite cluster. We will try to enable and document it in ignite 2.4.

--AG

2017-12-01 14:13 GMT+03:00 Andrey Mashenkov <[hidden email]>:
Hi Ray,
 
One more question here, how can a update or new inserts back-propagate to
Ignite when another application(not ignite) writes to persistence(hbase)?
 
It is not supported.



On Fri, Dec 1, 2017 at 12:08 PM, Ray <[hidden email]> wrote:
http://apache-ignite-users.70518.x6.nabble.com/Two-persistent-data-stores-for-a-single-Ignite-cluster-RDBMS-and-Ignite-native-td18463.html

Found a similar case here, I think I'll try Slava's suggestions first.

One more question here, how can a update or new inserts back-propagate to
Ignite when another application(not ignite) writes to persistence(hbase)?

For example, Ignite and hbase both have one entry for now.
When another application adds an entry to hbase, now hbase has two entries.
Can Ignite be notified and load the newly added entry automatically?

From the document, it looks like the data can only be propagated from Ignite
to persistence, not the other way around.



--
Best regards,
Andrey V. Mashenkov

alexey.goncharuk alexey.goncharuk
Reply | Threaded
Open this post in threaded view
|

Re: Can Ignite native persistence used with 3rd party persistence?


2017-12-15 16:16 GMT+03:00 Alexey Goncharuk <[hidden email]>:
Ray,

With the current API it is impossible to get a reliable integration of Ignite native persistence with 3rd party persistence. The reason is that first, CacheStore interface does not have methods for 2-phase commit, second, it would require significant changes to the persistence layer itself to make a consistent crash recovery.

We could allow setting the cache store interface with write-through from primary nodes, but in this case, it would be a user's responsibility to verify that the cache store is consistent with the Ignite cluster. We will try to enable and document it in ignite 2.4.

--AG

2017-12-01 14:13 GMT+03:00 Andrey Mashenkov <[hidden email]>:
Hi Ray,
 
One more question here, how can a update or new inserts back-propagate to
Ignite when another application(not ignite) writes to persistence(hbase)?
 
It is not supported.



On Fri, Dec 1, 2017 at 12:08 PM, Ray <[hidden email]> wrote:
http://apache-ignite-users.70518.x6.nabble.com/Two-persistent-data-stores-for-a-single-Ignite-cluster-RDBMS-and-Ignite-native-td18463.html

Found a similar case here, I think I'll try Slava's suggestions first.

One more question here, how can a update or new inserts back-propagate to
Ignite when another application(not ignite) writes to persistence(hbase)?

For example, Ignite and hbase both have one entry for now.
When another application adds an entry to hbase, now hbase has two entries.
Can Ignite be notified and load the newly added entry automatically?

From the document, it looks like the data can only be propagated from Ignite
to persistence, not the other way around.



--
Best regards,
Andrey V. Mashenkov


dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Can Ignite native persistence used with 3rd party persistence?

Cross-sending to dev@.

Alexey,

This issue is marked to be fixed for 2.4 which is planned to be released in a couple of weeks. Do you think you will be able to close this issue before the release?

D.


On Mon, Dec 18, 2017 at 9:51 AM, Alexey Goncharuk <[hidden email]> wrote:

2017-12-15 16:16 GMT+03:00 Alexey Goncharuk <[hidden email]>:
Ray,

With the current API it is impossible to get a reliable integration of Ignite native persistence with 3rd party persistence. The reason is that first, CacheStore interface does not have methods for 2-phase commit, second, it would require significant changes to the persistence layer itself to make a consistent crash recovery.

We could allow setting the cache store interface with write-through from primary nodes, but in this case, it would be a user's responsibility to verify that the cache store is consistent with the Ignite cluster. We will try to enable and document it in ignite 2.4.

--AG

2017-12-01 14:13 GMT+03:00 Andrey Mashenkov <[hidden email]>:
Hi Ray,
 
One more question here, how can a update or new inserts back-propagate to
Ignite when another application(not ignite) writes to persistence(hbase)?
 
It is not supported.



On Fri, Dec 1, 2017 at 12:08 PM, Ray <[hidden email]> wrote:
http://apache-ignite-users.70518.x6.nabble.com/Two-persistent-data-stores-for-a-single-Ignite-cluster-RDBMS-and-Ignite-native-td18463.html

Found a similar case here, I think I'll try Slava's suggestions first.

One more question here, how can a update or new inserts back-propagate to
Ignite when another application(not ignite) writes to persistence(hbase)?

For example, Ignite and hbase both have one entry for now.
When another application adds an entry to hbase, now hbase has two entries.
Can Ignite be notified and load the newly added entry automatically?

From the document, it looks like the data can only be propagated from Ignite
to persistence, not the other way around.



--
Best regards,
Andrey V. Mashenkov