Customized affinity function

classic Classic list List threaded Threaded
6 messages Options
prasadbhalerao1983 prasadbhalerao1983
Reply | Threaded
Open this post in threaded view
|

Customized affinity function

Hi,

I have following case.

I have around 10000 subscriptions. Each subscription has data varying from 1 to 10 million.
Currently the affinity key is set on subscriptionId.

e.g:

Subscription1: 2 million rows
Subscription2: 10 million rows
Subscription3: 50 million rows
Subscription5: 30 million rows
Subscription6: 20000 rows

Now I want make sure that all big subscription should be evenly distributed across all nodes.

I meant Subscription2,Subscription3,Subscription4,Subscription5 should not land on same node based on their affinity id.

Is there any way with which I can distribute all big subscriptions across all node evenly or in round robin fashion?


Thanks,
Prasad
Mikael Mikael
Reply | Threaded
Open this post in threaded view
|

Re: Customized affinity function

Hi!

The @AffinityKeyMapped annotation can be used on a field or a method, so you can have a method returning anything you want (based on size), would that not work ?

Mikael


Den 2018-04-11 kl. 07:59, skrev Prasad Bhalerao:
Hi,

I have following case.

I have around 10000 subscriptions. Each subscription has data varying from 1 to 10 million.
Currently the affinity key is set on subscriptionId.

e.g:

Subscription1: 2 million rows
Subscription2: 10 million rows
Subscription3: 50 million rows
Subscription5: 30 million rows
Subscription6: 20000 rows

Now I want make sure that all big subscription should be evenly distributed across all nodes.

I meant Subscription2,Subscription3,Subscription4,Subscription5 should not land on same node based on their affinity id.

Is there any way with which I can distribute all big subscriptions across all node evenly or in round robin fashion?


Thanks,
Prasad

aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Customized affinity function

In reply to this post by prasadbhalerao1983
Hi Prasad,

Affinity function does not directly map keys to nodes, it maps keys to
partitions. So it takes care about entities balance between nodes. To read
more please take a look here:

https://apacheignite.readme.io/docs/affinity-collocation#affinity-function

In case if for every subscriber you are going to create it's own cache then
you can try to setup the node filters for every subscriber:

https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/CacheConfiguration.html#setNodeFilter-org.apache.ignite.lang.IgnitePredicate-

If you have the same cache for all subscribers then try to setup your own
AffinityKey (as descibed above):

            IgniteCache<EntityKey, String> cache =
ignite.getOrCreateCache(cfg);

            for (int i = 0 ; i < 1_000_000; i++) {
                cache.put(new EntityKey(2*i, 1L), "Value1" + i); //sub id =
1
                cache.put(new EntityKey(2*i + 1, 2L), "Value2" + i); //sub
id =2
            }

where:

    public static class EntityKey {
        public EntityKey(long id, long subscriberId) {
            this.id = id;
            this.subscriberId = subscriberId;
        }

        private long id;

        // Subscriber ID which will be used for affinity.
        @AffinityKeyMapped
        private long subscriberId;

        public long getSubscriberId() {
            return subscriberId;
        }

        public long getId() {
            return id;
        }
    }

Thank you,
Andrei



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Customized affinity function

In reply to this post by prasadbhalerao1983
Prasad,

Are you collocating data by subscription? Do you have any actual issue with
data distribution? With 10000 subscriptions, I don't see why would you have
one with default settings.

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
prasadbhalerao1983 prasadbhalerao1983
Reply | Threaded
Open this post in threaded view
|

Re: Customized affinity function

Hi Val,

Out 10000 subscriptions, only 10-20 subscriptions are very big. I just want to make sure that these big subscriptions do not land up in partitions which are located on same node.





On Thu, Apr 12, 2018 at 7:44 AM, vkulichenko <[hidden email]> wrote:
Prasad,

Are you collocating data by subscription? Do you have any actual issue with
data distribution? With 10000 subscriptions, I don't see why would you have
one with default settings.

-Val

vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Customized affinity function

Statistically it's very unlikely that all big subscriptions (or many of them)
will end up together. So I would check if there is an issue first and go
from there.

Technically you can implement your own AffinityFunction to customize
distribution. But that's not a trivial task as you will have to consider all
the cases of nodes going out of topology, scaling out, etc.

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/