Loading Data from RDMS to ignite using Data Streamer

classic Classic list List threaded Threaded
3 messages Options
nithin91 nithin91
Reply | Threaded
Open this post in threaded view
|

Loading Data from RDMS to ignite using Data Streamer

Hi ,

I am new to Ignite. i am able to load data from oracle into ignite using
Cache Pojo Store method ignite.cache.("cachename").loadcache(null) . But in
the documentation it was mentioned that Data Steamers will provide better
performance when loading data into ignite cache.

*Documentation Reference:*

https://apacheignite.readme.io/docs/data-streamers

Following is the example provided for Data Streamers in the documentation

// Get the data streamer reference and stream data.
try (IgniteDataStreamer<Integer, String> stmr =
ignite.dataStreamer("myStreamCache")) {    
    // Stream entries.
    for (int i = 0; i < 100000; i++)
        stmr.addData(i, Integer.toString(i));
}

I have following questions by taking  this example as reference to stream
data from oracle table to ignite
cache.

1. Is an external streamer needed to stream data from oracle to ignite to
use data streaming technique
as mentioned in above example.

2. If an external streamer is not required, then do we need to loop through
the jdbc result set and add data to the cache.Please find below for the
pseudo code implementation of the same.

try (IgniteDataStreamer<Key, Obj> stmr = ignite.dataStreamer("CacheName")) {    
    // Stream entries.


    while(rs.hasNext()){-->rs is JDBC result set

        Key keyobj=new Key(rs.getString(1));
        Value valueobj=new
Value(rs.getString(2),rs.getString(3),rs.getString(4));
        stmr.addData(keyobj , valueobj);
    }
}

When i tried to load data into ignite cache from oracle using this  method
it is taking lot of time which is as expected as we are looping through each
and every record.Is this right method of implementing Data Streamers to load
data from Oracle to Ignite.If this is not right way,can anyone share sample
code to stream data from any RDBMS table to Ignite cache.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
akurbanov akurbanov
Reply | Threaded
Open this post in threaded view
|

Re: Loading Data from RDMS to ignite using Data Streamer

Since this feels like as a performance question, I would start with
identifying the bottleneck in this case.

How much data are you going to load from Oracle?

What is consuming the most time while streaming, reading data from Oracle or
streaming itself? How high the resource usage while streaming (CPU/memory
consumption)? This will define what should be tuned/changed.

What current numbers are for loadCache(null) vs DataStreamer?

Best regards,
Anton



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
nithin91 nithin91
Reply | Threaded
Open this post in threaded view
|

Re: Loading Data from RDMS to ignite using Data Streamer

Hi

Currently i am able to load 5.5 million data from Oracle Table to Ignite in
1 hr using JDBC Pojo Configuration.So i was curious that how much time will
it take to load the data using data streamers as a result of which i tried
to load 10,000 records using the Data Streamers and it took almost half an
hour.
So wanted to know whether technique i implemented for  Data Streamers to
load Data from Oracle to Ignite is correct or wrong. PFB for the psuedo code
i implemented for  Data Streamers to load Data from Oracle to Ignite.
Using this  method it is taking lot of time which is as expected as we are
looping through each
and every record .

try (IgniteDataStreamer<Key, Obj> stmr = ignite.dataStreamer("CacheName")) {    
    // Stream entries.


    while(rs.hasNext()){-->rs is JDBC result set

        Key keyobj=new Key(rs.getString(1));
        Value valueobj=new
Value(rs.getString(2),rs.getString(3),rs.getString(4));
        stmr.addData(keyobj , valueobj);
    }
}

Can you please share a sample code streaming data  from any RDBMS to ignite.
Also can you please let me know if an external streamer is needed to stream
data from RDBMS to ignite



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/