Cache Preload from database using IgniteDatastreamer
I have searched a lot to find out how to use IgniteDataStreamer to preload
cache from database, but all i couldn't find any.
So I tried to do it myslef. Following is my approach:
1. A spring boot application having java based server configuration for
cache - Ignite server application
2. A spring boot application having java based client configuration for
cache - Ignite client application
3. I start the server app and then start the client app.
4. on startup of client app, after context is refreshed, i am loading ignite
cache by fetching data from database using spring data jpa.
I am using this process to load all the cache.There are total 14 caches
created and loaded like this.
The tables are having around 600000 records.And using spring data JPA gives
me gc overhead and jvm pause issues.
I have read about data streamer that it is performance wise better, but i
don't know whether i am using it correctly.
Also i am not sure that loadcache() approach is better than datastreamer
Can anyone please help me on this issue.
Re: Cache Preload from database using IgniteDatastreamer
There is no so called 'silver bullet' for data streamer parameters that will
fit in all or most cases, you should try tuning individual streamer
parameters to find out how it affects performance. Also it would be useful
to understand is there anything that could be done against GC pressure
(tuning GC on server nodes, tuning heap parameters etc.). There are just too
many unknown values to say what could be done exactly.
>And using spring data JPA gives me gc overhead and jvm pause issues.
Not entirely sure how to understand this, do you have GC issues on client
I am trying to implement / follow the same approach for my POC to understand
how that can be applied to a real time scenario's.
I googled but couldn't find a complete implementation in spring boot.
Is it possible to share the source code location, it would help me a lot.
Sent from: http://apache-ignite-users.70518.x6.nabble.com/