Fanout related query

classic Classic list List threaded Threaded
5 messages Options
deep0mal deep0mal
Reply | Threaded
Open this post in threaded view
|

Fanout related query

Hi Community members,

I am looking for how to design fanout related problem in Ignite

We have two cache - product and items.
Items has product name and details along with item details.

Every time when there is product related update, item cache should be
updated for all the  items with respective product details. Please note the
items can be in millions.

What is best way to achieve this usecase in Ignite?
I see there is StreamVisitor  (
https://apacheignite.readme.io/v2.4/docs/data-streamers#streamvisitor)
implementation that might fit this kind of usecase but want to know how
feasible it is and is there better way to achieve this usecase?



Regards,
_DM

Roman Guseinov Roman Guseinov
Reply | Threaded
Open this post in threaded view
|

Re: Fanout related query

Hi,

It isn't a good idea to update millions of items every time product details
are changed.

I guess the main purpose of this schema is to store product details close to
items.

I suggest you add an additional cache for product details and make it
replicated. It seems that products count is much less than the items one.
Another way is configuring the affinity collocation [1]. You can specify
indexes and use SQL to merge items data with product details [2]. It is
going to work fast and you don't need to update millions of items every time
product details are changed.

Will it work for you?

Best Regards,
Roman

[1] https://apacheignite.readme.io/v2.3/docs/affinity-collocation

[2]
https://apacheignite-sql.readme.io/docs/distributed-joins#section-collocated-joins



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Fanout related query

This post was updated on .
In reply to this post by deep0mal
First of all, it would be a good idea to collocate data in your caches, i.e.
make items be stored on the same nodes with corresponding products. Take a
look at the documentation [1]


When you need to update some product, you can perform an  affinityRun [2], that will send a task to the node, where the corresponding product data is
stored. In that affinity run you can perform a  local query [3], that will update
all needed items. In these settings you will execute the query on a single
node only.

[1] https://apacheignite.readme.io/docs/affinity-collocation#collocate-data-with-data
[2] https://apacheignite.readme.io/docs/affinity-collocation#section-collocating-compute-with-data
[3] https://apacheignite-sql.readme.io/docs/local-queries

--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
deep0mal deep0mal
Reply | Threaded
Open this post in threaded view
|

Re: Fanout related query

In reply to this post by Roman Guseinov
Hi Roman, Denis,

Thanks for your responses.

You are right. The primary usecase is to keep product data along with items in cache to quickly fetch item in single lookup for display purpose. Therefore, each item is updated at write time with product data instead of doing it at read time. We do create product cache replicated but then StreamVisitor is used to update product data in items.

I notice that affinity solution is still going to update millions of items but the updates are local instead of cluster-wide. Please let me know if my interpretation is wrong.

I see Ignite also support Outer joins. Does keeping product cache and item cache separate and join at get query time will be efficient?

Regards,
Deepesh



On Tue, Mar 27, 2018 at 11:20 AM, Roman Guseinov <[hidden email]> wrote:
Hi,

It isn't a good idea to update millions of items every time product details
are changed.

I guess the main purpose of this schema is to store product details close to
items.

I suggest you add an additional cache for product details and make it
replicated. It seems that products count is much less than the items one.
Another way is configuring the affinity collocation [1]. You can specify
indexes and use SQL to merge items data with product details [2]. It is
going to work fast and you don't need to update millions of items every time
product details are changed.

Will it work for you?

Best Regards,
Roman

[1] https://apacheignite.readme.io/v2.3/docs/affinity-collocation

[2]
https://apacheignite-sql.readme.io/docs/distributed-joins#section-collocated-joins



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/



--
_Deepesh
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Fanout related query



On Tue, Mar 27, 2018 at 10:53 PM, Deepesh Malviya <[hidden email]> wrote:

I notice that affinity solution is still going to update millions of items but the updates are local instead of cluster-wide. Please let me know if my interpretation is wrong.

Yes.
 
I see Ignite also support Outer joins. Does keeping product cache and item cache separate and join at get query time will be efficient?

Yes, but only if you use affinity collocation between product and item caches.