Apache Ignite with Caching : How do I sure that cache(s) has all datas?

classic Classic list List threaded Threaded
9 messages Options
monstereo monstereo
Reply | Threaded
Open this post in threaded view
|

Apache Ignite with Caching : How do I sure that cache(s) has all datas?

I'm looking for an answer this question.
Here is the scenario

- I have datas from the database  over 100.000
- I have created cache node(node1) then, I have loaded  100.000 datas to the
cache memory.(REPLICATED mode)
- Now I will create another node(s) which is/are served for client to
consume these datas. However first I have to be sure that all datas from
node1 fully copy in node2(node2, node3... )
- Question how can be sure that all datas are also in node2

- I have seen some people says that "use countlatchdown".



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
monstereo monstereo
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite with Caching : How do I sure that cache(s) has all datas?

Is there any one who can answer my question?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Amir Akhmedov Amir Akhmedov
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite with Caching : How do I sure that cache(s) has all datas?

If you want to make sure all your data had been copied into backup nodes then you need to change synchronization mode to FULL_SYNC [1]

On Sat, Jul 7, 2018 at 8:14 AM monstereo <[hidden email]> wrote:
Is there any one who can answer my question?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
monstereo monstereo
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite with Caching : How do I sure that cache(s) has all datas?

This post was updated on .
thank you for your comment,
I also found that there is a data rebalance in ignite.
What do you think about this? Which one should I use?

Here is the data rebalance link  here
<https://apacheignite.readme.io/docs/rebalancing

For SYNCH mode in rebalance says that :::

"Synchronous rebalancing mode. Distributed caches will not start until all
necessary data is loaded from other available grid nodes. This means that
any call to cache public API will be blocked until rebalancing is finished."





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Amir Akhmedov wrote
If you want to make sure all your data had been copied into backup nodes
then you need to change synchronization mode to FULL_SYNC [1]

Thanks,
Amir

[1]
https://apacheignite.readme.io/docs/primary-and-backup-copies#section-synchronous-and-asynchronous-backups

On Sat, Jul 7, 2018 at 8:14 AM monstereo <[hidden email]> wrote:

> Is there any one who can answer my question?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Amir Akhmedov Amir Akhmedov
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite with Caching : How do I sure that cache(s) has all datas?

Rebalancing is a process when a node joins or leaves (in case backup is turned on) a cluster, data will be rebalanced within a cluster to make a fair distribution. It's applicable only for partitioned caches. But you have replicated cache and it's out of your case.

Thanks,
Amir

On Jul 8, 2018 4:02 AM, "monstereo" <[hidden email]> wrote:
thank you for your comment,
I also found that there is a data rebalance in ignite.
What do you think about this? Which one should I use?

Here is the data rebalance link  here
<https://apacheignite.readme.io/docs/rebalancing

For SYNCH mode in rebalance says that :::

"Synchronous rebalancing mode. Distributed caches will not start until all
necessary data is loaded from other available grid nodes. This means that
any call to cache public API will be blocked until rebalancing is finished."

monstereo monstereo
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite with Caching : How do I sure that cache(s) has all datas?

I haven't noticed that.
thank you,
Amir Akhmedov wrote
> Rebalancing is a process when a node joins or leaves (in case backup is
> turned on) a cluster, data will be rebalanced within a cluster to make a
> fair distribution. It's applicable only for partitioned caches. But you
> have replicated cache and it's out of your case.
>
> Thanks,
> Amir
>
> On Jul 8, 2018 4:02 AM, "monstereo" &lt;

> mehmetozanguven@

> &gt; wrote:
>
> thank you for your comment,
> I also found that there is a data rebalance in ignite.
> What do you think about this? Which one should I use?
>
> Here is the data rebalance link  here
> &lt;https://apacheignite.readme.io/docs/rebalancing&gt;
>
> For SYNCH mode in rebalance says that :::
>
> "Synchronous rebalancing mode. Distributed caches will not start until all
> necessary data is loaded from other available grid nodes. This means that
> any call to cache public API will be blocked until rebalancing is
> finished."
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
slava.koptilin slava.koptilin
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite with Caching : How do I sure that cache(s) has all datas?

In reply to this post by Amir Akhmedov
Hello,

I think rebalancing makes sense for all types of caches. It does not matter what type of cache you use.
Long story short, a replicated cache is a partitioned cache with the number of backups equals to the number of nodes minus 1.

Let's assume that you ingested all data in the cluster, and after that, added a new node.
In that case, data will be transferred/copied to a new node during rebalancing.

Perhaps, EVT_CACHE_REBALANCE_STARTED and EVT_CACHE_REBALANCE_STOPPED will be useful to track rebalancing.

Thanks,
Slava.

вс, 8 июл. 2018 г. в 15:47, Amir Akhmedov <[hidden email]>:
Rebalancing is a process when a node joins or leaves (in case backup is turned on) a cluster, data will be rebalanced within a cluster to make a fair distribution. It's applicable only for partitioned caches. But you have replicated cache and it's out of your case.

Thanks,
Amir

On Jul 8, 2018 4:02 AM, "monstereo" <[hidden email]> wrote:
thank you for your comment,
I also found that there is a data rebalance in ignite.
What do you think about this? Which one should I use?

Here is the data rebalance link  here
<https://apacheignite.readme.io/docs/rebalancing

For SYNCH mode in rebalance says that :::

"Synchronous rebalancing mode. Distributed caches will not start until all
necessary data is loaded from other available grid nodes. This means that
any call to cache public API will be blocked until rebalancing is finished."

monstereo monstereo
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite with Caching : How do I sure that cache(s) has all datas?

then may ask you what is difference between
https://apacheignite.readme.io/docs/primary-and-backup-copies#section-synchronous-and-asynchronous-backups
this and rebalance mode

thanks,


slava.koptilin wrote

> Hello,
>
> I think rebalancing makes sense for all types of caches. It does not
> matter
> what type of cache you use.
> Long story short, a replicated cache is a partitioned cache with the
> number
> of backups equals to the number of nodes minus 1.
>
> Let's assume that you ingested all data in the cluster, and after that,
> added a new node.
> In that case, data will be transferred/copied to a new node during
> rebalancing.
>
> Perhaps, EVT_CACHE_REBALANCE_STARTED and EVT_CACHE_REBALANCE_STOPPED will
> be useful to track rebalancing.
>
> Thanks,
> Slava.
>
> вс, 8 июл. 2018 г. в 15:47, Amir Akhmedov &lt;

> amir.akhmedov@

> &gt;:
>
>> Rebalancing is a process when a node joins or leaves (in case backup is
>> turned on) a cluster, data will be rebalanced within a cluster to make a
>> fair distribution. It's applicable only for partitioned caches. But you
>> have replicated cache and it's out of your case.
>>
>> Thanks,
>> Amir
>>
>> On Jul 8, 2018 4:02 AM, "monstereo" &lt;

> mehmetozanguven@

> &gt; wrote:
>>
>> thank you for your comment,
>> I also found that there is a data rebalance in ignite.
>> What do you think about this? Which one should I use?
>>
>> Here is the data rebalance link  here
>> &lt;https://apacheignite.readme.io/docs/rebalancing&gt;
>>
>> For SYNCH mode in rebalance says that :::
>>
>> "Synchronous rebalancing mode. Distributed caches will not start until
>> all
>> necessary data is loaded from other available grid nodes. This means that
>> any call to cache public API will be blocked until rebalancing is
>> finished."
>>
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>>
>>





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
slava.koptilin slava.koptilin
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite with Caching : How do I sure that cache(s) has all datas?

Hi,

CacheWriteSynchronizationMode defines the behavior of cache during modification operations. I mean IgniteCache#put(), #remove() etc.
For example, in case of 'PRIMARY_SYNC', a thread that executes modification operation will wait for completion of operation on primary node only, backups will be updated asynchronously. If 'FULL_SYNC' is used than a thread will wait for responses from all participated nodes (from primary node and all backups).

hope this helps.

Best regards,
Slava.

вс, 8 июл. 2018 г. в 17:24, monstereo <[hidden email]>:
then may ask you what is difference between
https://apacheignite.readme.io/docs/primary-and-backup-copies#section-synchronous-and-asynchronous-backups
this and rebalance mode

thanks,


slava.koptilin wrote
> Hello,
>
> I think rebalancing makes sense for all types of caches. It does not
> matter
> what type of cache you use.
> Long story short, a replicated cache is a partitioned cache with the
> number
> of backups equals to the number of nodes minus 1.
>
> Let's assume that you ingested all data in the cluster, and after that,
> added a new node.
> In that case, data will be transferred/copied to a new node during
> rebalancing.
>
> Perhaps, EVT_CACHE_REBALANCE_STARTED and EVT_CACHE_REBALANCE_STOPPED will
> be useful to track rebalancing.
>
> Thanks,
> Slava.
>
> вс, 8 июл. 2018 г. в 15:47, Amir Akhmedov &lt;

> amir.akhmedov@

> &gt;:
>
>> Rebalancing is a process when a node joins or leaves (in case backup is
>> turned on) a cluster, data will be rebalanced within a cluster to make a
>> fair distribution. It's applicable only for partitioned caches. But you
>> have replicated cache and it's out of your case.
>>
>> Thanks,
>> Amir
>>
>> On Jul 8, 2018 4:02 AM, "monstereo" &lt;

> mehmetozanguven@

> &gt; wrote:
>>
>> thank you for your comment,
>> I also found that there is a data rebalance in ignite.
>> What do you think about this? Which one should I use?
>>
>> Here is the data rebalance link  here
>> &lt;https://apacheignite.readme.io/docs/rebalancing&gt;
>>
>> For SYNCH mode in rebalance says that :::
>>
>> "Synchronous rebalancing mode. Distributed caches will not start until
>> all
>> necessary data is loaded from other available grid nodes. This means that
>> any call to cache public API will be blocked until rebalancing is
>> finished."
>>
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>>
>>





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/