Cache Preload from database using IgniteDatastreamer

classic Classic list List threaded Threaded
3 messages Options
Om Thacker Om Thacker
Reply | Threaded
Open this post in threaded view
|

Cache Preload from database using IgniteDatastreamer

I have searched a lot to find out how to use IgniteDataStreamer to preload
cache from database, but all i couldn't find any.
So I tried to do it myslef. Following is my approach:
1. A spring boot application having java based server configuration  for
cache - Ignite server application
2. A spring boot application having java based client configuration  for
cache - Ignite client application
3. I start the server app and then start the client app.
4. on startup of client app, after context is refreshed, i am loading ignite
cache by fetching data from database using spring data jpa.


@Autowired
private Ignite ignite

public void preload(){
   try(IgniteDataStremer<Long,Users> stmr =
ignite.dataStremer("UsersCache")){
       List<Users> usersList = userRepository.preload();
       for(Users users: usersList){
           stmr.addData(users.getUsersKey(),users);
       }
   }
}


I am using this process to load all the cache.There are total 14 caches
created and loaded like this.

The tables are having around 600000 records.And using spring data JPA gives
me gc overhead and jvm pause issues.
I have read about data streamer that it is performance wise better, but i
don't know whether i am using it correctly.

Also i am not sure that loadcache() approach is better than datastreamer
approach.
Can anyone please help me on this issue.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
akurbanov akurbanov
Reply | Threaded
Open this post in threaded view
|

Re: Cache Preload from database using IgniteDatastreamer

There is no so called 'silver bullet' for data streamer parameters that will
fit in all or most cases, you should try tuning individual streamer
parameters to find out how it affects performance. Also it would be useful
to understand is there anything that could be done against GC pressure
(tuning GC on server nodes, tuning heap parameters etc.). There are just too
many unknown values to say what could be done exactly.

>And using spring data JPA gives me gc overhead and jvm pause issues.
Not entirely sure how to understand this, do you have GC issues on client
node?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Om Thacker Om Thacker
Reply | Threaded
Open this post in threaded view
|

Re: Cache Preload from database using IgniteDatastreamer

Thanks for the reply..I was just wanted to confirm that the way I am using
datastreamer to load the data in cache is correct or not.

Regarding gc overhead- I was running it in my local machine with limited
heap.so that's no the exact problem.please ignore that part.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/