Ignite Partitioned Cache / Use for an in-memory transaction-log

classic Classic list List threaded Threaded
4 messages Options
Johannes Lichtenberger-2 Johannes Lichtenberger-2
Reply | Threaded
Open this post in threaded view
|

Ignite Partitioned Cache / Use for an in-memory transaction-log

Hi,

can I somehow query on which cluster-nodes a partitioned cache stores
values for specific keys? I might want to use a cache for replicating an
in-memory transaction log (which does not have to be persisted) to
replicate a document of a NoSQL document store to a few nodes.

Thus, instead of a local cache I'd simply write into an ignite cache and
then would like to query which cluster-nodes have stored a specific key
(for instance the document name). So for instance I could tell a load
balancer for reads to read documents from one of the backup replicas.

During a transaction commit I would also need this information to know
where to send an event to... to commit the transaction. And somehow I'd
love to wait for a quorum of nodes if the transaction really has
committed or it needs to roll back.

kind regards

Johannes

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Partitioned Cache / Use for an in-memory transaction-log

Hello!

You can use ignite.affinity(cacheName).mapKeyToNode(key): it returns ClusterNode which is primary for this key.

Not sure that I understand you about the quorum.

Regards,
--
Ilya Kasnacheev


вт, 6 авг. 2019 г. в 15:47, Johannes Lichtenberger <[hidden email]>:
Hi,

can I somehow query on which cluster-nodes a partitioned cache stores
values for specific keys? I might want to use a cache for replicating an
in-memory transaction log (which does not have to be persisted) to
replicate a document of a NoSQL document store to a few nodes.

Thus, instead of a local cache I'd simply write into an ignite cache and
then would like to query which cluster-nodes have stored a specific key
(for instance the document name). So for instance I could tell a load
balancer for reads to read documents from one of the backup replicas.

During a transaction commit I would also need this information to know
where to send an event to... to commit the transaction. And somehow I'd
love to wait for a quorum of nodes if the transaction really has
committed or it needs to roll back.

kind regards

Johannes

Johannes Lichtenberger-2 Johannes Lichtenberger-2
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Partitioned Cache / Use for an in-memory transaction-log

Hi,

if I want to replicate data from one node to N-nodes (that is only a subset of nodes in the whole cluster), could I also specify that not only the primary is updated in synchronous mode, but some of the backup nodes, too?

I also would like to somehow guarantee read-your-own-writes semantics. My idea is for replicating a storage / database system to have a single writer through which all writes are propagated.

I'd probably use something like oldest node is the master (which can be done using Apache Ignite).

Then let's say someone decides to create a database and a first resource. I'd have to send an event round robin to create the resource on let's say three of 10 cluster nodes. Maybe I could use a distributed semaphore which just accepts 3-times acquiring the semaphore (name is database + resource concatenated). Once the initial bootstrapping of creating directories and files happened on all 3 nodes a write-transaction can be opened, which writes into a distributed cache. However as I'm thinking about it the cache for this resource must be shared on only the same 3 cluster-nodes. That's maybe already where I have no clue how to achieve this.

Maybe I'm looking into how other data stores are replicated and partitioned, first ;)

There's also BookKeeper for distributing a transaction-log. But I'm not sure if it's overhead if I don't have to persist the changes before applying it to the real data-files (to provide consistency -- I don't need persisted write-ahead logging).

kind regards

Johannes

On 06.08.19 15:20, Ilya Kasnacheev wrote:
Hello!

You can use ignite.affinity(cacheName).mapKeyToNode(key): it returns ClusterNode which is primary for this key.

Not sure that I understand you about the quorum.

Regards,
--
Ilya Kasnacheev


вт, 6 авг. 2019 г. в 15:47, Johannes Lichtenberger <[hidden email]>:
Hi,

can I somehow query on which cluster-nodes a partitioned cache stores
values for specific keys? I might want to use a cache for replicating an
in-memory transaction log (which does not have to be persisted) to
replicate a document of a NoSQL document store to a few nodes.

Thus, instead of a local cache I'd simply write into an ignite cache and
then would like to query which cluster-nodes have stored a specific key
(for instance the document name). So for instance I could tell a load
balancer for reads to read documents from one of the backup replicas.

During a transaction commit I would also need this information to know
where to send an event to... to commit the transaction. And somehow I'd
love to wait for a quorum of nodes if the transaction really has
committed or it needs to roll back.

kind regards

Johannes

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Partitioned Cache / Use for an in-memory transaction-log

Hello!

You can have any number of backups, but you can either update all of them in sync, or primary only. You can't have a subset of backups updated, despite such possibility being discussed.

Regards,
--
Ilya Kasnacheev


ср, 7 авг. 2019 г. в 13:29, Johannes Lichtenberger <[hidden email]>:

Hi,

if I want to replicate data from one node to N-nodes (that is only a subset of nodes in the whole cluster), could I also specify that not only the primary is updated in synchronous mode, but some of the backup nodes, too?

I also would like to somehow guarantee read-your-own-writes semantics. My idea is for replicating a storage / database system to have a single writer through which all writes are propagated.

I'd probably use something like oldest node is the master (which can be done using Apache Ignite).

Then let's say someone decides to create a database and a first resource. I'd have to send an event round robin to create the resource on let's say three of 10 cluster nodes. Maybe I could use a distributed semaphore which just accepts 3-times acquiring the semaphore (name is database + resource concatenated). Once the initial bootstrapping of creating directories and files happened on all 3 nodes a write-transaction can be opened, which writes into a distributed cache. However as I'm thinking about it the cache for this resource must be shared on only the same 3 cluster-nodes. That's maybe already where I have no clue how to achieve this.

Maybe I'm looking into how other data stores are replicated and partitioned, first ;)

There's also BookKeeper for distributing a transaction-log. But I'm not sure if it's overhead if I don't have to persist the changes before applying it to the real data-files (to provide consistency -- I don't need persisted write-ahead logging).

kind regards

Johannes

On 06.08.19 15:20, Ilya Kasnacheev wrote:
Hello!

You can use ignite.affinity(cacheName).mapKeyToNode(key): it returns ClusterNode which is primary for this key.

Not sure that I understand you about the quorum.

Regards,
--
Ilya Kasnacheev


вт, 6 авг. 2019 г. в 15:47, Johannes Lichtenberger <[hidden email]>:
Hi,

can I somehow query on which cluster-nodes a partitioned cache stores
values for specific keys? I might want to use a cache for replicating an
in-memory transaction log (which does not have to be persisted) to
replicate a document of a NoSQL document store to a few nodes.

Thus, instead of a local cache I'd simply write into an ignite cache and
then would like to query which cluster-nodes have stored a specific key
(for instance the document name). So for instance I could tell a load
balancer for reads to read documents from one of the backup replicas.

During a transaction commit I would also need this information to know
where to send an event to... to commit the transaction. And somehow I'd
love to wait for a quorum of nodes if the transaction really has
committed or it needs to roll back.

kind regards

Johannes