Can Ignite be configured to provide an array like replicated and durable map?

classic Classic list List threaded Threaded
9 messages Options
mfronczyk mfronczyk
Reply | Threaded
Open this post in threaded view
|

Can Ignite be configured to provide an array like replicated and durable map?

I need a data structure that holds 1 billion of objects. Each object will consist of 4 integers. This will be used to model a large 2-dimensional grid.

The structure is initially empty (or can be full but have objects with all fields set to 0), but will be filled in with elements inserted at random places. The data structure is immutable in the sense that each index can be written only once. This invariant should be guaranteed even when there are multiple concurrent clients using it - writing to an already written index should return an error. All writes should be fast and have low latency.

Another important feature is that the memory layout of the structure matches the array indexes - element with index n is before the element n + 1. There will be a need to read 16384 consecutive elements quickly and that memory layout will be very cache friendly for such operation.

Elements written to the structure need to be durable. The plan is to replicate the writes on 2 nodes. Writes to about 1% of the indexes are more important than others and it should be guaranteed that acknowledged writes for that indexes survive a complete crash of one node. There should be no time window during which such writes are lost. But it's OK for the rest of 99% other writes to have a chance of data loss for performance reasons. So we need async replication with a possibility to enable synchronous replication for certain puts.

This all sounds like a replicated, durable array which isn't provided by Ignite, but I can imagine implementing this as a specialised map/key-value store, with ints as keys, disabled hashing of keys and the storage allocated off-heap to guarantee that keys that are close to each other have consecutive locations in RAM. Ideally, it should also be possible to specify whether synchronous replication is needed or not for each write operation. Is it possible to achieve using Apache Ignite?
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Can Ignite be configured to provide an array like replicated and durable map?

Hi,

It looks like you will have to to store 16384 elements that are expected to be read at once as one cache entry. This is the only way to make sure that all these elements are stored on one node and in consecutive memory.

To update or read only one element you can use EntryProcessor and IgniteCache.invoke(..) method (see [1]).

Will this work for you?

[1] https://apacheignite.readme.io/docs/jcache#entryprocessor

-Val
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Can Ignite be configured to provide an array like replicated and durable map?

In reply to this post by mfronczyk
mfronczyk wrote
I need a data structure that holds 1 billion of objects. Each object will consist of 4 integers. This will be used to model a large 2-dimensional grid.
To keep a data structure this big in memory, you should consider using Ignite Off-Heap Memory support [1]. Also make sure that you have enough memory on your server to fit the whole data set.

[1] http://apacheignite.gridgain.org/docs/off-heap-memory

mfronczyk wrote
Another important feature is that the memory layout of the structure matches the array indexes - element with index n is before the element n + 1. There will be a need to read 16384 consecutive elements quickly and that memory layout will be very cache friendly for such operation.
If you remove this consecutive memory layout requirement, then you can actually store 1 billion of objects separately, using array index as a key. In this case, you can use full IgniteCache API, access each element separately, and update different elements concurrently. Depending on your requirements, this approach may be easier to use. However, this approach will also consume more memory than the approach suggested by Valentin.

mfronczyk wrote
Elements written to the structure need to be durable. The plan is to replicate the writes on 2 nodes. Writes to about 1% of the indexes are more important than others and it should be guaranteed that acknowledged writes for that indexes survive a complete crash of one node. There should be no time window during which such writes are lost. But it's OK for the rest of 99% other writes to have a chance of data loss for performance reasons. So we need async replication with a possibility to enable synchronous replication for certain puts.
Synchronous vs. asynchronous backups can be configured on per-cache level via FULL_SYNC and PRIMARY_SYNC modes [2]. So if you really need to separate synchronous updates from asynchronous updates, you will have to split your data structure into 2 parts and store it in 2 different caches. My preference would be that you start off with one cache in FULL_SYNC mode, and introduce another one in PRIMARY_SYNC mode only if you run into performance issue.

[2] http://apacheignite.gridgain.org/v1.3/docs/primary-and-backup-copies#synchronous-and-asynchronous-backups
mfronczyk mfronczyk
Reply | Threaded
Open this post in threaded view
|

Re: Can Ignite be configured to provide an array like replicated and durable map?

Do I need to use EntryProcessor and invoke if I'll guarantee that all data is directly accessible by the client application? My idea is to run Ignite embedded in the client, instead of as a remote cluster, to minimise the latency, and to have a single backup node to which the data will be replicated. Invoke seems like an API to execute operations on the cluster.

Storing 1 billion elements in separate keys isn't probably the way to go. As you said, it'll take much more memory and fetching 16k of objects will be slow. If I store 16384 array elements as one cache entry, how will it work when a single element is modified? Will it have to replicate the whole array to the backup node and will it cause writing of the whole array to the permanent data store?


On Thu, Sep 10, 2015 at 4:23 AM, dsetrakyan [via Apache Ignite Users] <[hidden email]> wrote:
mfronczyk wrote
I need a data structure that holds 1 billion of objects. Each object will consist of 4 integers. This will be used to model a large 2-dimensional grid.
To keep a data structure this big in memory, you should consider using Ignite Off-Heap Memory support [1]. Also make sure that you have enough memory on your server to fit the whole data set.

[1] http://apacheignite.gridgain.org/docs/off-heap-memory

mfronczyk wrote
Another important feature is that the memory layout of the structure matches the array indexes - element with index n is before the element n + 1. There will be a need to read 16384 consecutive elements quickly and that memory layout will be very cache friendly for such operation.
If you remove this consecutive memory layout requirement, then you can actually store 1 billion of objects separately, using array index as a key. In this case, you can use full IgniteCache API, access each element separately, and update different elements concurrently. Depending on your requirements, this approach may be easier to use. However, this approach will also consume more memory than the approach suggested by Valentin.

mfronczyk wrote
Elements written to the structure need to be durable. The plan is to replicate the writes on 2 nodes. Writes to about 1% of the indexes are more important than others and it should be guaranteed that acknowledged writes for that indexes survive a complete crash of one node. There should be no time window during which such writes are lost. But it's OK for the rest of 99% other writes to have a chance of data loss for performance reasons. So we need async replication with a possibility to enable synchronous replication for certain puts.
Synchronous vs. asynchronous backups can be configured on per-cache level via FULL_SYNC and PRIMARY_SYNC modes [2]. So if you really need to separate synchronous updates from asynchronous updates, you will have to split your data structure into 2 parts and store it in 2 different caches. My preference would be that you start off with one cache in FULL_SYNC mode, and introduce another one in PRIMARY_SYNC mode only if you run into performance issue.

[2] http://apacheignite.gridgain.org/v1.3/docs/primary-and-backup-copies#synchronous-and-asynchronous-backups


To unsubscribe from Can Ignite be configured to provide an array like replicated and durable map?, click here.
NAML



--
Michał Fronczyk
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Can Ignite be configured to provide an array like replicated and durable map?

mfronczyk wrote
Do I need to use EntryProcessor and invoke if I'll guarantee that all data
is directly accessible by the client application? My idea is to run Ignite
embedded in the client, instead of as a remote cluster, to minimise the
latency, and to have a single backup node to which the data will be
replicated. Invoke seems like an API to execute operations on the cluster.
I would still recommend to use invoke, because otherwise you will send the whole updated value to backup(s) which is going to be slow in case you store all 16384 elements as one entry. Also it looks like you need to configure cache in replicated mode (see [1]).

[1] https://apacheignite.readme.io/docs/cache-modes#replicated-mode

mfronczyk wrote
Storing 1 billion elements in separate keys isn't probably the way to go.
As you said, it'll take much more memory and fetching 16k of objects will
be slow. If I store 16384 array elements as one cache entry, how will it
work when a single element is modified? Will it have to replicate the whole
array to the backup node and will it cause writing of the whole array to
the permanent data store?
It depends on cache atomicity mode (see [2]). In ATOMIC mode the whole value is sent to backup nodes even if invoke method is used to update it. But in TRANSACTIONAL mode this never happens and only EntryProcessor instance is sent across network, so I think you should go with transactional cache. Note that updates in transactional caches are generally slower that in atomic caches, but it sounds like you're going to have more reads than updates, so transactional mode should fit your use case.

[2] https://apacheignite.readme.io/docs/transactions

Let us know if you have more questions.

-Val
mfronczyk mfronczyk
Reply | Threaded
Open this post in threaded view
|

Re: Can Ignite be configured to provide an array like replicated and durable map?

This post has NOT been accepted by the mailing list yet.
I see now. Using invoke with TRANSACTIONAL will only send EntryProcessor over the network. Thanks.

How will it work with Persistent Store configured with write-through or write-behind? Won't the whole value (16384 element array) be persisted each time a single entry of the array is modified?

On Thu, Sep 10, 2015 at 9:16 PM, vkulichenko [via Apache Ignite Users] <[hidden email]> wrote:
mfronczyk wrote
Do I need to use EntryProcessor and invoke if I'll guarantee that all data
is directly accessible by the client application? My idea is to run Ignite
embedded in the client, instead of as a remote cluster, to minimise the
latency, and to have a single backup node to which the data will be
replicated. Invoke seems like an API to execute operations on the cluster.
I would still recommend to use invoke, because otherwise you will send the whole updated value to backup(s) which is going to be slow in case you store all 16384 elements as one entry. Also it looks like you need to configure cache in replicated mode (see [1]).

[1] https://apacheignite.readme.io/docs/cache-modes#replicated-mode

mfronczyk wrote
Storing 1 billion elements in separate keys isn't probably the way to go.
As you said, it'll take much more memory and fetching 16k of objects will
be slow. If I store 16384 array elements as one cache entry, how will it
work when a single element is modified? Will it have to replicate the whole
array to the backup node and will it cause writing of the whole array to
the permanent data store?
It depends on cache atomicity mode (see [2]). In ATOMIC mode the whole value is sent to backup nodes even if invoke method is used to update it. But in TRANSACTIONAL mode this never happens and only EntryProcessor instance is sent across network, so I think you should go with transactional cache. Note that updates in transactional caches are generally slower that in atomic caches, but it sounds like you're going to have more reads than updates, so transactional mode should fit your use case.

[2] https://apacheignite.readme.io/docs/transactions

Let us know if you have more questions.

-Val


To unsubscribe from Can Ignite be configured to provide an array like replicated and durable map?, click here.
NAML



--
Michał Fronczyk
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Can Ignite be configured to provide an array like replicated and durable map?

mfronczyk wrote
How will it work with Persistent Store configured with write-through or
write-behind? Won't the whole value (16384 element array) be persisted each
time a single entry of the array is modified?
The whole entry will be passed to CacheStore.write(..) method and it's up to your implementation what is actually persisted to the database. The only issue I see here is that you need to know which elements were updated when the store is called. I think you should add a transient field into the value that will contain this information, update it inside entry processor and then use inside the store to optimize persistence.

-Val
mfronczyk mfronczyk
Reply | Threaded
Open this post in threaded view
|

Re: Can Ignite be configured to provide an array like replicated and durable map?

Makes sense. CacheStore needs to clear that transient information after writing the data to the persistent store. I'm assuming it doesn't require any synchronization with the entry processor in the write-through scenario because the key is still locked in CacheStore. 

How will it work in the write-behind scenario? Are writes blocked during persisting in the write-behind mode and it's safe to modify the value passed to the store after persisting it?

On Fri, Sep 11, 2015 at 8:29 PM, vkulichenko [via Apache Ignite Users] <[hidden email]> wrote:
mfronczyk wrote
How will it work with Persistent Store configured with write-through or
write-behind? Won't the whole value (16384 element array) be persisted each
time a single entry of the array is modified?
The whole entry will be passed to CacheStore.write(..) method and it's up to your implementation what is actually persisted to the database. The only issue I see here is that you need to know which elements were updated when the store is called. I think you should add a transient field into the value that will contain this information, update it inside entry processor and then use inside the store to optimize persistence.

-Val


To unsubscribe from Can Ignite be configured to provide an array like replicated and durable map?, click here.
NAML



--
Michał Fronczyk
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Can Ignite be configured to provide an array like replicated and durable map?

mfronczyk wrote
Makes sense. CacheStore needs to clear that transient information after
writing the data to the persistent store. I'm assuming it doesn't require
any synchronization with the entry processor in the write-through scenario
because the key is still locked in CacheStore.
That's right. Actually, clean up is not required, because a copy of the value will be created each time you read it, so entry processor will not get the same instance that is stored in cache. BTW, if you want to override this behavior for performance reasons, set CacheConfiguration.setCopyOnRead(false) property.
mfronczyk wrote
How will it work in the write-behind scenario? Are writes blocked during
persisting in the write-behind mode and it's safe to modify the value
passed to the store after persisting it?
With write-behind writes to persistence store happen asynchronously in background, so they don't block any cache operations. If copyOnRead=true (default value), it should work for you, because the independent copy will be written to the store. copyOnRead=false should not be used with write-behind at all, because the value written to store will be likely updated by that time.

-Val