Ignite cache partition size

classic Classic list List threaded Threaded
8 messages Options
colinc colinc
Reply | Threaded
Open this post in threaded view
|

Ignite cache partition size

Normally when storing data in Ignite using the default
RendezVousAffinityFunction, data is distributed reasonably evenly over the
available nodes. When increasing the cluster size (up to 24 nodes in my
case), I'm finding that the data is not so well distributed. Some nodes have
more than twice as much data as others.

The solution I found is to quadruple the number of cache partitions up to
8192. This reduces the spread significantly - to about 25%.

Is there any down-side to configuring a large number of partitions for a
cache? Specifically:
* Am I right in observing that is slows down partition map exchange? Is this
likely to cause any problems?
* When using cross-cache data affinity, do all participating caches need to
have the same number of partitions configured for this to work?

Is there any rule of thumb for choosing the best configuration? 1000 times
the number of nodes, perhaps?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite cache partition size

Hello!

* 8192 partitions have been tried with success, as far as I know. I think
you can give it a shot, see where it gets you. It might slow down partition
exchange a bit, but it should not hang just from increased amount of
information (as opposed to stuck operations)

* Yes,they need to have the same number of partitions for caches with
collocated keys. Come to think of it, if all of 2N partitions were
collocated with 1N partitions, what you will really have is just 1N
partitions. Partitions should be able to part in order to work.

For optimal number of partitions, consider 256 primary partitions per node.
This means 8192 is OK for you until you move past 32 nodes.

Some additional hints:
* Increasing number of partitions may increase number of open files if using
persistence. Plan accordingly, keep ulimits in mind.
* You can broadcast ignite.affinity(cacheName).primaryPartitions(localNode)
and ignite.cache(cacheName).localSize(CachePeekMode.PRIMARY) to data nodes
to get hold of how even partition distribution is in your case (are keys
distributed well between partitions? are partitions between nodes?
* There was a research on how well partitions are distributed between nodes
with Rendevouz affinity, see non-"balanced" PNGs (that's per # of nodes) in
https://issues.apache.org/jira/browse/IGNITE-3018 - I think you can observe
serious deviations from average number of partitions in those images, so
it's possible that your 2x difference is no accident.
* The community is currently working on faster and fairer affinity functions
:)

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
codeboyyong codeboyyong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite cache partition size

In reply to this post by colinc
Hi can you please tell me how to config this ?
"number of cache partitions"

Thank You
Yong



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
a.dolmatov a.dolmatov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite cache partition size

Try to implement you're own affinity function. If it map any key to numbers from 1 to n, you have n partitions

On Sat, Oct 26, 2019, 09:46 codeboyyong <[hidden email]> wrote:
Hi can you please tell me how to config this ?
"number of cache partitions"

Thank You
Yong



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
prasadbhalerao1983 prasadbhalerao1983
Reply | Threaded
Open this post in threaded view
|

Re: Ignite cache partition size

Please check this link...

https://apacheignite.readme.io/docs/affinity-collocation#section-affinity-function

Example from the ignite doc.

// Preparing Apache Ignite node configuration.
IgniteConfiguration cfg = new IgniteConfiguration();
        
// Creating a cache configuration.
CacheConfiguration cacheCfg = new CacheConfiguration("myCache");

// Creating the affinity function with custom setting.
RendezvousAffinityFunction affFunc = new RendezvousAffinityFunction();
        
affFunc.setExcludeNeighbors(true);
        
affFunc.setPartitions(2048);

// Applying the affinity function configuration.
cacheCfg.setAffinity(affFunc);
        
// Setting the cache configuration.
cfg.setCacheConfiguration(cacheCfg);


On Sat 26 Oct, 2019, 7:42 PM Andrey Dolmatov <[hidden email] wrote:
Try to implement you're own affinity function. If it map any key to numbers from 1 to n, you have n partitions

On Sat, Oct 26, 2019, 09:46 codeboyyong <[hidden email]> wrote:
Hi can you please tell me how to config this ?
"number of cache partitions"

Thank You
Yong



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
codeboyyong codeboyyong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite cache partition size


Cool, thanks .
I have a quite big cluster with 100 nodes, so I guess 25600 will be good partition?
I am using ignite as database.
I am creating a table with 20 million rows,each row has 1450 columns. 
Is this a good idea? 

Thank you!

On Sat, Oct 26, 2019 at 9:10 AM Prasad Bhalerao <[hidden email]> wrote:
Please check this link...

https://apacheignite.readme.io/docs/affinity-collocation#section-affinity-function

Example from the ignite doc.

// Preparing Apache Ignite node configuration.
IgniteConfiguration cfg = new IgniteConfiguration();
        
// Creating a cache configuration.
CacheConfiguration cacheCfg = new CacheConfiguration("myCache");

// Creating the affinity function with custom setting.
RendezvousAffinityFunction affFunc = new RendezvousAffinityFunction();
        
affFunc.setExcludeNeighbors(true);
        
affFunc.setPartitions(2048);

// Applying the affinity function configuration.
cacheCfg.setAffinity(affFunc);
        
// Setting the cache configuration.
cfg.setCacheConfiguration(cacheCfg);


On Sat 26 Oct, 2019, 7:42 PM Andrey Dolmatov <[hidden email] wrote:
Try to implement you're own affinity function. If it map any key to numbers from 1 to n, you have n partitions

On Sat, Oct 26, 2019, 09:46 codeboyyong <[hidden email]> wrote:
Hi can you please tell me how to config this ?
"number of cache partitions"

Thank You
Yong



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
prasadbhalerao1983 prasadbhalerao1983
Reply | Threaded
Open this post in threaded view
|

Re: Ignite cache partition size

I have heard people creating 4096 but not more that. 
What's you reasoning behind creating such huge partitions.
I have a table which contains 40+ million rows, but it has only 5 columns and I have kept the default partitions and getting good results out of it.
You have 1450 columns, an interesting use case.

1) Is your app already in production?
2) Did you calculate the average size of single row.
3) How do you manage and monitor your big cluster.
4) Do you have any scripts or tools to stop,start, restart or rolling reastart the cluster or node of your cluster?

I have cluster of 7 nodes, each node has a data region size of 40-50 Gb but whenever a segmentation happens i.e when Node goes of out cluster I have found it difficult to debug or investigate such issues. Most of the times I could not figure out the  exact reason for sengmentation and always ended up with blaming network issue for the segmentation.

If you have already figured out all these things then well and good, please share your ideas to tools with usergroup if possible.

If you are doing POC and if you want 24/7 availability then I would suggest to figure out these things as well. 

Thanks,
Prasad


On Sun 27 Oct, 2019, 6:53 AM Yong Zhao <[hidden email] wrote:

Cool, thanks .
I have a quite big cluster with 100 nodes, so I guess 25600 will be good partition?
I am using ignite as database.
I am creating a table with 20 million rows,each row has 1450 columns. 
Is this a good idea? 

Thank you!

On Sat, Oct 26, 2019 at 9:10 AM Prasad Bhalerao <[hidden email]> wrote:
Please check this link...

https://apacheignite.readme.io/docs/affinity-collocation#section-affinity-function

Example from the ignite doc.

// Preparing Apache Ignite node configuration.
IgniteConfiguration cfg = new IgniteConfiguration();
        
// Creating a cache configuration.
CacheConfiguration cacheCfg = new CacheConfiguration("myCache");

// Creating the affinity function with custom setting.
RendezvousAffinityFunction affFunc = new RendezvousAffinityFunction();
        
affFunc.setExcludeNeighbors(true);
        
affFunc.setPartitions(2048);

// Applying the affinity function configuration.
cacheCfg.setAffinity(affFunc);
        
// Setting the cache configuration.
cfg.setCacheConfiguration(cacheCfg);


On Sat 26 Oct, 2019, 7:42 PM Andrey Dolmatov <[hidden email] wrote:
Try to implement you're own affinity function. If it map any key to numbers from 1 to n, you have n partitions

On Sat, Oct 26, 2019, 09:46 codeboyyong <[hidden email]> wrote:
Hi can you please tell me how to config this ?
"number of cache partitions"

Thank You
Yong



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite cache partition size

It's best to have the number of partitions being a power of two, so better to go with 32768 I think.
There are big clusters with hundreds of nodes out there, and they do use large partition numbers sometimes - as large as 16k or 32k.

Note that it will bring some overhead on the metadata being stored, etc. You may need to tune your heap to store the partition information.
And as Prasad has mentioned, you need to make sure you know how your system behaves when a node fails - how much time does rebalancing takes, do your remaining node have enough memory to restore the backups number, etc.

Stan

On Sun, Oct 27, 2019 at 7:21 PM Prasad Bhalerao <[hidden email]> wrote:
I have heard people creating 4096 but not more that. 
What's you reasoning behind creating such huge partitions.
I have a table which contains 40+ million rows, but it has only 5 columns and I have kept the default partitions and getting good results out of it.
You have 1450 columns, an interesting use case.

1) Is your app already in production?
2) Did you calculate the average size of single row.
3) How do you manage and monitor your big cluster.
4) Do you have any scripts or tools to stop,start, restart or rolling reastart the cluster or node of your cluster?

I have cluster of 7 nodes, each node has a data region size of 40-50 Gb but whenever a segmentation happens i.e when Node goes of out cluster I have found it difficult to debug or investigate such issues. Most of the times I could not figure out the  exact reason for sengmentation and always ended up with blaming network issue for the segmentation.

If you have already figured out all these things then well and good, please share your ideas to tools with usergroup if possible.

If you are doing POC and if you want 24/7 availability then I would suggest to figure out these things as well. 

Thanks,
Prasad


On Sun 27 Oct, 2019, 6:53 AM Yong Zhao <[hidden email] wrote:

Cool, thanks .
I have a quite big cluster with 100 nodes, so I guess 25600 will be good partition?
I am using ignite as database.
I am creating a table with 20 million rows,each row has 1450 columns. 
Is this a good idea? 

Thank you!

On Sat, Oct 26, 2019 at 9:10 AM Prasad Bhalerao <[hidden email]> wrote:
Please check this link...

https://apacheignite.readme.io/docs/affinity-collocation#section-affinity-function

Example from the ignite doc.

// Preparing Apache Ignite node configuration.
IgniteConfiguration cfg = new IgniteConfiguration();
        
// Creating a cache configuration.
CacheConfiguration cacheCfg = new CacheConfiguration("myCache");

// Creating the affinity function with custom setting.
RendezvousAffinityFunction affFunc = new RendezvousAffinityFunction();
        
affFunc.setExcludeNeighbors(true);
        
affFunc.setPartitions(2048);

// Applying the affinity function configuration.
cacheCfg.setAffinity(affFunc);
        
// Setting the cache configuration.
cfg.setCacheConfiguration(cacheCfg);


On Sat 26 Oct, 2019, 7:42 PM Andrey Dolmatov <[hidden email] wrote:
Try to implement you're own affinity function. If it map any key to numbers from 1 to n, you have n partitions

On Sat, Oct 26, 2019, 09:46 codeboyyong <[hidden email]> wrote:
Hi can you please tell me how to config this ?
"number of cache partitions"

Thank You
Yong



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/