Some question about the data loading

classic Classic list List threaded Threaded
5 messages Options
kcheng.mvp kcheng.mvp
Reply | Threaded
Open this post in threaded view
|

Some question about the data loading


There is a section data-loading
I have a question about it.

it address that

In case of partitioned caches, keys that are not mapped to this node, either as primary or backups, will be automatically discarded by the cache.

my understanding about this is that all the node will execute the same query, but only the keys mapped to this node will be saved. am I right?

if like this, if I want to improve system performance, how can I just make the query take a parameter to make the sql query only return the data fullfill a node.


alexey.goncharuk alexey.goncharuk
Reply | Threaded
Open this post in threaded view
|

Re: Some question about the data loading

Yes, you can do that if you store the partition ID alongside with the record in the database. I have updated the data-loading configuration page (see the partition-aware data loading section). Please let me know if it makes sense or if I need to add more information to that page. 

--Alexey

2015-08-11 6:42 GMT-07:00 kcheng.mvp <[hidden email]>:

There is a section  data-loading
<https://apacheignite.readme.io/docs/data-loading>
I have a question about it.

it address that

*In case of partitioned caches, keys that are not mapped to this node,
either as primary or backups, will be automatically discarded by the cache.*

my understanding about this is that all the node will execute the same
query, but only the keys mapped to this node will be saved. am I right?

if like this, if I want to improve system performance, how can I just make
the query take a parameter to make the sql query only return the data
fullfill a node.






--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Some-question-about-the-data-loading-tp907.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

kcheng.mvp kcheng.mvp
Reply | Threaded
Open this post in threaded view
|

Re: Some question about the data loading

In fact I have another question about the data loading.

1:
Right now I am going to implement the data loading in the LifecycleBean call back method.

@Override
        public void onLifecycleEvent(LifecycleEventType evt) throws IgniteException {
                if(evt == LifecycleEventType.AFTER_NODE_START){
                        // do data loading here
                }
               
        }


when I start the node one by one(I have four nodes), with the node starts then the node cluster topology will change accordingly.

I guess the partition ID will be changing with the topology changes. In this case there are some data would not be loaded into cache, right?

if the LifecycleBeanis not the ideal place to preload the data, so what's the best practice to do pre data loading




2: there are two method to trigger data loading IgniteCache.loadCache() and IgniteCache.localLoadCache()

suppose there are 4 nodes(1 client nodes + 4 server nodes), when call IgniteCache.loadCache()
from server node, client node would not load the data right?

3:
In fact I am running into issue when try to do data load, and I fired a bug
https://issues.apache.org/jira/browse/IGNITE-1234


 
kcheng.mvp kcheng.mvp
Reply | Threaded
Open this post in threaded view
|

Re: Some question about the data loading

This post was updated on .
In reply to this post by alexey.goncharuk
CONTENTS DELETED
The author has deleted this message.
alexey.goncharuk alexey.goncharuk
Reply | Threaded
Open this post in threaded view
|

Re: Some question about the data loading

1. When new nodes join a cluster, partition-to-node assignment changes. Let's assume you have one backup. When you have just one node, it is responsible for all partitions, so your first node will try to load all the partitions. When second node joins grid, it will try to re-balance existing data from the first node and at the same time it will try to load the same set of partitions since we assumed you have one backup. When third node joins, it again will rebalance existing data from two first nodes and will try to load from the database 2/3s of partitions, and so on. In this scenario all nodes but the last one will try to load more partitions than necessary. The best approach here is to wait until your topology has enough nodes (you can create an event listener for node join/leave event) and then call loadCache().
2. It should not run on client node because it does not make sense, I will be surprised if it is.
3. In the ticket you have created your XML configuration does not have a name, but in the code you do have cache name. So in your cluster you end up with two caches, one with cache store (defined in the XML) and one without cache store, so it looks like you end up calling loadCache on the cache without store.

2015-08-11 11:31 GMT-07:00 kcheng.mvp <[hidden email]>:
In fact I have another question about the data loading.

1:
Right now I am going to implement the data loading in the LifecycleBean call
back method.

@Override
        public void onLifecycleEvent(LifecycleEventType evt) throws IgniteException
{
                if(evt == LifecycleEventType.AFTER_NODE_START){
                        // do data loading here
                }

        }


when I start the node one by one(I have four nodes), with the node starts
then the node cluster topology will change accordingly.

I guess the *partition ID* will be changing with the topology changes. In
this case there are some data would not be loaded into cache, right?

if the *LifecycleBean*is not the ideal place to preload the data, so what's
the best practice to do pre data loading




2: there are two method to trigger data loading *IgniteCache.loadCache()*
and *IgniteCache.localLoadCache()*

suppose there are 5 nodes(1 client nodes + 4 server nodes), when call
*IgniteCache.loadCache()*
from server node, client node would not load the data right?

3:
In fact I am running into issue when try to do data load, and I fired a bug
https://issues.apache.org/jira/browse/IGNITE-1234






--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Some-question-about-the-data-loading-tp907p912.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.