Pre-calculated JOIN queries as map in ignite

classic Classic list List threaded Threaded
4 messages Options
dharamthacker dharamthacker
Reply | Threaded
Open this post in threaded view
|

Pre-calculated JOIN queries as map in ignite

This post has NOT been accepted by the mailing list yet.
CONTENTS DELETED
The author has deleted this message.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Pre-calculated JOIN queries as map in ignite

Hi Dharam,

Can you please properly subscribe to the mailing list so that the community can receive email notifications? Here is the instruction: http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1

dharamthacker wrote
I am new to ignite and POCing currently. I have a question regarding ways to store/load data in map. It's bit tricky and strange requirement.

Example:

I have Employee, Department, Project [Tables in database] + [Entity classes in application]. But I don't want to store each of these in a separate map in memory but rather I want to store pre-calculated join results in a designated map.

Dynamic Query :
select employeeId,employeeName,departmentName,projectName,projectStart,projectEnd from Employee,Department,Project where $JOIN

I know at least before hand that, what would be key fields and what would be value fields. From above example, I can denote my "Map" as shown below,

(Key=Set<Tuple>, Value = List<Tuple>)

Key : Set (employeeId,departmentId) , Value : List [(employeeName,value),(departmentName,value),(projectName,value),(projectStart,value),(projectEnd,value)]

So you can see with every pair of (employeeId,departmentId) I would be having multiple values associates with it. But dilemma is I don't have domain model/entity pojos before hand. Such dynamic views/maps can be added flexibly so that we don't have to go and change domain/entity model every time. We don't want to do joins/calculations every time for thousands of such client request on every call.

Is it possible to fire such join queries using MapLoader or by any other means? I can think of Map with (Key=Set<Tuple>, Value = List<Tuple>) as data structure to store final results.Any other better alternative?

Could there be any performance issues while retrieving values from such map based on keys? Any memory optimizations I should take care of?

Thanks, Dharam
You are not required to use SQL queries. It's fine to use Ignite as a simple caching mechanism for DB query results. Each time a query is executed, save the result in IgniteCache and then use this cached result is the same query is requested. You can also use expirations [1] and/or evictions [2] to make sure that you don't have too much data in the cache and don't run out of memory.

[1] https://apacheignite.readme.io/docs/expiry-policies
[2] https://apacheignite.readme.io/docs/evictions

-Val
dharamthacker dharamthacker
Reply | Threaded
Open this post in threaded view
|

Re: Pre-calculated JOIN queries as map in ignite

CONTENTS DELETED
The author has deleted this message.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Pre-calculated JOIN queries as map in ignite

dharamthacker wrote
But that's what I want to avoid [Each time DB query].
I meant that you go to the DB only if the data is not in the cache (was not cached yet or was already evicted). If it's already there, you just use IgniteCache.get().
dharamthacker wrote
But I don't want to have predefined POJO to store it into cache. I should be able to add dynamic query each resulting into separate map in distributed cache.
Ignite stores all the data in binary format which already supports dynamic schema changes. It also provides the builder for binary objects which you can use in case you don't have classes. See this page for details: https://apacheignite.readme.io/docs/binary-marshaller

-Val