Affinity Key Mapping

classic Classic list List threaded Threaded
18 messages Options
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Affinity Key Mapping

If I have two domain objects lets say A, B, do I need to put them in the same cache to use Affinity key mapping? Or can it work with different caches?
vdpyatkov vdpyatkov
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

Hello,
You can put objects in a single cache and use affinity collocation.
But I would recommend using a different caches, but do not forget, they should be equally configured (same number of partitions and same affinity function).

On Thu, May 26, 2016 at 6:40 AM, amitpa <[hidden email]> wrote:
If I have two domain objects lets say A, B, do I need to put them in the same
cache to use Affinity key mapping? Or can it work with different caches?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--
Vladislav Pyatkov
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

Hey Thanks for the reply.

I am copying the example here :-

/ Instantiate person keys with the same company ID which is used as affinity key.
Object personKey1 = new PersonKey("myPersonId1", "myCompanyId");
Object personKey2 = new PersonKey("myPersonId2", "myCompanyId");
 
Person p1 = new Person(personKey1, ...);
Person p2 = new Person(personKey2, ...);
 
// Both, the company and the person objects will be cached on the same node.
cache.put("myCompanyId", new Company(...));
cache.put(personKey1, p1);
cache.put(personKey2, p2);

From the example it doesnt appear that we can put it in different caches. See both are putting to the same cache.

Is there any exampel of using different caches for storing related domain objects, how does Ignite figure out colocation in that case? Sorry for the questions, but it got me confused.

Also how do you configure partitions for a cache?
Vladislav Pyatkov Vladislav Pyatkov
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

Hello,

>> From the example it doesnt appear that we can put it in different caches.
Yes, from the example in Java is not clear, but see example on Scala. This should work well (in different caches), it clear from implementation.

>> how does Ignite figure out colocation in that case?
Ignite does this with the affinity function. Annotation @AffinitiKeyMapped means that the evaluation of the partition (for store key-value pair) will be on a particular field and not on the whole key.

>>how do you configure partitions for a cache?
You can set the number of partitions to the constructor affinity function for a particular cache.
<property name="affinity">
<bean class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
<constructor-arg value="true"/>
<constructor-arg value="1024"/>
</bean>
</property>

On Fri, May 27, 2016 at 6:33 AM, amitpa <[hidden email]> wrote:
Hey Thanks for the reply.

I am copying the example here :-

/ Instantiate person keys with the same company ID which is used as affinity
key.
Object personKey1 = new PersonKey("myPersonId1", "myCompanyId");
Object personKey2 = new PersonKey("myPersonId2", "myCompanyId");

Person p1 = new Person(personKey1, ...);
Person p2 = new Person(personKey2, ...);

// Both, the company and the person objects will be cached on the same node.
cache.put("myCompanyId", new Company(...));
cache.put(personKey1, p1);
cache.put(personKey2, p2);

From the example it doesnt appear that we can put it in different caches.
See both are putting to the same cache.

Is there any exampel of using different caches for storing related domain
objects, how does Ignite figure out colocation in that case? Sorry for the
questions, but it got me confused.

Also how do you configure partitions for a cache?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5265.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

In reply to this post by amitpa
I’ve fixed the examples in the docs [1]. Thanks pointing out to this!

Also you may refer to CacheQueryExample.initialized() method to see how it works in practice. The example is delivered as a part of Ignite release bundles. [2]

Feel free to ask if anythings is still unclear.


Denis

On May 27, 2016, at 6:33 AM, amitpa <[hidden email]> wrote:

Hey Thanks for the reply.

I am copying the example here :-

/ Instantiate person keys with the same company ID which is used as affinity
key.
Object personKey1 = new PersonKey("myPersonId1", "myCompanyId");
Object personKey2 = new PersonKey("myPersonId2", "myCompanyId");

Person p1 = new Person(personKey1, ...);
Person p2 = new Person(personKey2, ...);

// Both, the company and the person objects will be cached on the same node.
cache.put("myCompanyId", new Company(...));
cache.put(personKey1, p1);
cache.put(personKey2, p2);

From the example it doesnt appear that we can put it in different caches.
See both are putting to the same cache.

Is there any exampel of using different caches for storing related domain
objects, how does Ignite figure out colocation in that case? Sorry for the
questions, but it got me confused.

Also how do you configure partitions for a cache?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5265.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

Hey , many thanks for the quick and awesome response. I have an idea now about how to use Affinity mapping.

The doc is clear now. I will ask some questions later on about Affinity mapping.

But does increasing partitions for a cache help with Transaction performance? I am struggling to increase increase transaction performance under my usecase (due to some mis understanding on my part may be).

I have already submitted a thread, where I will submit the code. However it will still be helpful if I can get some tips? I wanted to configure Affinity mapping for throughput of transactions. Any other tips are welcome
Vladislav Pyatkov Vladislav Pyatkov
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

Number of partition does not affect the performance of the transaction. 
In general changing the number of partitions is done to evenly distribute data over nodes.

On Fri, May 27, 2016 at 12:06 PM, amitpa <[hidden email]> wrote:
Hey , many thanks for the quick and awesome response. I have an idea now
about how to use Affinity mapping.

The doc is clear now. I will ask some questions later on about Affinity
mapping.

But does increasing partitions for a cache help with Transaction
performance? I am struggling to increase increase transaction performance
under my usecase (due to some mis understanding on my part may be).

I have already submitted a thread, where I will submit the code. However it
will still be helpful if I can get some tips? I wanted to configure Affinity
mapping for throughput of transactions. Any other tips are welcome



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5272.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

Does Affinity Key imporve performance of multiple cache puts in the same transaction?

Alexei Scherbakov Alexei Scherbakov
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

1. Increasing number of partitions might improve throughput of Ignite cluster, because more partitions means less contention on cache locks.
It's application dependant.

2. Affinity mapping will improve performance if puts are invoked from affinity node ( using affinityCall [1] ), because puts need not to be transferred over wire ( except for backups ).

3. If you have unexpected performance problems with Ignite, you can share you code and someone from community may give you a hint.

[1] https://apacheignite.readme.io/docs/affinity-collocation#collocating-compute-with-data

2016-05-28 12:03 GMT+03:00 amitpa <[hidden email]>:
Does Affinity Key imporve performance of multiple cache puts in the same
transaction?





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5290.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--

Best regards,
Alexei Scherbakov
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

disruptor.zip

Uploading the code.

I have tried to mimic my code as much as possible. However I have used no DTO's (only one). In my actual business code there are lots of caches with Key  Long and Value DTO. I understand the performance will be slower in that code due to that. But that is extremely slow though too.
The main code is in Disruptor server.
IgniteMain starts another node.

The App is basically client code.
Once I run two nodes I am getting a TPS of 1100.

I have used OFF_HEAP tiered and ONHEAP tiered. The offheap seems to give more stable performance.

However I would like to improve the performance to at least 3000 TPS. I get over 10000 TPS on a single node (when I run only Disruptor server).

My config is i7 6700 with 32 GB ram. When Ignite boots up both nodes it shows 14 gb memory.

Can some one provide some performance tips on this?



vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

I responded regarding the performance in this thread: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Transaction-performance-td5085.html

-Val
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

Val,

Excellent feedback. I will update and let you guys know. However is there any way to use transactions within https://apacheignite.readme.io/docs/affinity-collocation#collocating-compute-with-data ...
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

Ignore my latest question . This seems to work fine.

I have one kind of feature request though.

It seems with transaction, Ignite always tries to get the lock for the keys accessed.
However some caches can be read only. Multiple transactions can access the same key, but they will never  modify it.

Is there any way to mark a cache as read only and hence exempt for transaction lock acquiring, this I believe is a common case for things like reference data.  This will also help in throughput for most applications
Alexei Scherbakov Alexei Scherbakov
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

Ignite's lock acquisition depends on the transaction isolation.
AFAIK, read_commited and optimistic modes do not lock entries on read.
Ignte has no meaning of read-only cache and I doubt it will have in future,
but you can try to ask on the dev list.

As a workaround, you can create local ConcurrentHashMap and populate it with static data using the broadcast job [1]
on cluster startup.

[1] See javadoc for ignite.compute().broadcast()

2016-05-30 17:52 GMT+03:00 amitpa <[hidden email]>:
Ignore my latest question . This seems to work fine.

I have one kind of feature request though.

It seems with transaction, Ignite always tries to get the lock for the keys
accessed.
However some caches can be read only. Multiple transactions can access the
same key, but they will never  modify it.

Is there any way to mark a cache as read only and hence exempt for
transaction lock acquiring, this I believe is a common case for things like
reference data.  This will also help in throughput for most applications



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5321.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--

Best regards,
Alexei Scherbakov
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

This post was updated on .
Alexei,

yes I understand with Ignite Optmistic means lock acquisition on write for READCOMMITED.

With PESSIMISTIC it would be on read.

However consider a cache like Country (contrived example )...many people may live in USA. But if someone tries to do a transaction for a entry on write all will lock on the same entry.

This is a contrived example, and I can ofcourse find multiple ways to circumvent this particular scenario.

But for some cases this can not be done.

And if there is any cache config which says a cache is read only that should be fast enough.
Though I am not sure but may be for read_only caches I can use Atomic caches and that way transaction participation can be avoided? Is this correct ?
However I wil try your work around.

Many thanks for your replies, it is really helpful.
Alexei Scherbakov Alexei Scherbakov
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

I have another idea.
Try to define country cache as atomic.
It shouldn't block on read when.
Of course it can't be used with transactional semantics.

2016-05-30 18:50 GMT+03:00 amitpa <[hidden email]>:
Alexei,

yes I understand with Ignite Optmistic means lock acquisition on write for
READCOMMITED.

With PESSIMISTIC it would be on read.

However consider a cache like Country (contrived example )...many people may
live in USA. But if someone tries to do a transaction for a entry on write
all will lock on the same entry.

This is a contrived example, and I can ofcourse find multiple ways to
circumvent this particular scenario.

But for some cases this can not be done.

And if there is any cache config which says a cache is read only that should
be fast enough.

However I wil try your work around.

Many thanks for your replies, it is really helpful.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5327.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--

Best regards,
Alexei Scherbakov
Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

In reply to this post by amitpa
Hi,

There is already a solution for your use case - you can use IgniteCache.getAllOutTx(…). No locks will be acquired for the keys passed to this method.

Alternatively, as it’s already suggested, you can make such caches ATOMIC if you don’t consider to use them in transactional mode.


Denis

> On May 30, 2016, at 6:50 PM, amitpa <[hidden email]> wrote:
>
> Alexei,
>
> yes I understand with Ignite Optmistic means lock acquisition on write for
> READCOMMITED.
>
> With PESSIMISTIC it would be on read.
>
> However consider a cache like Country (contrived example )...many people may
> live in USA. But if someone tries to do a transaction for a entry on write
> all will lock on the same entry.
>
> This is a contrived example, and I can ofcourse find multiple ways to
> circumvent this particular scenario.
>
> But for some cases this can not be done.
>
> And if there is any cache config which says a cache is read only that should
> be fast enough.
>
> However I wil try your work around.
>
> Many thanks for your replies, it is really helpful.
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5327.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Affinity Key Mapping

Thanks everyone.

I got around this with Ignite Atomic cache.


However with replication my transactions still seem to be too slow.
Is there anything I can do to make it faster?