Data Loading Options

classic Classic list List threaded Threaded
11 messages Options
ksatya ksatya
Reply | Threaded
Open this post in threaded view
|

Data Loading Options

This post has NOT been accepted by the mailing list yet.
Hi,
    I am new to Apche Ignite. I want to know what are the options available in Ignite to load the fast data from different sources into Data Grid? We have a use case where in data that comes very fast need to be loaded into Data Grid and leverage Compute Grid to execute our custom business logic in parallel across different nodes.

We were initially planning to use some ETL too to load the data, but wanted to check if there are better options available to load around 1000 TPS and process them.

Thanks.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Options

ksatya wrote
Hi,
    I am new to Apche Ignite. I want to know what are the options available in Ignite to load the fast data from different sources into Data Grid? We have a use case where in data that comes very fast need to be loaded into Data Grid and leverage Compute Grid to execute our custom business logic in parallel across different nodes.

We were initially planning to use some ETL too to load the data, but wanted to check if there are better options available to load around 1000 TPS and process them.

Thanks.
Hi,

Essentially there are two options: via IgniteDataStreamer API and by implementing CacheStore.loadCache() method. The latter can be used only for initial load while nothing else is happening in the grid, so it looks like streamer fits your use case better (I may be wrong).

Please refer to this documentation page for more information about data loading and let us know if you have any follow up questions: https://apacheignite.readme.io/docs/data-loading

-Val
ksatya ksatya
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Options

This post has NOT been accepted by the mailing list yet.
Thanks Val for your response. I looked in at the examples in GitHub and could not find any based on DataStreamers, unless I am not looking at the right location. Can you point me to some examples, if there are any?

Thanks...
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Options

ksatya wrote
Thanks Val for your response. I looked in at the examples in GitHub and could not find any based on DataStreamers, unless I am not looking at the right location. Can you point me to some examples, if there are any?

Thanks...
Take a look at these:

https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheDataStreamerExample.java

https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/streaming

-Val
ksatya ksatya
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Options

This post has NOT been accepted by the mailing list yet.
Thanks Val. Do you know if I can use compute fabric for processing code built in c++?
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Options

ksatya wrote
Thanks Val. Do you know if I can use compute fabric for processing code built in c++?
I think you can execute C++ code using JNI call or start external process and wait synchronously for its completion inside a job. Is this what you're looking for?

-Val
ksatya ksatya
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Options

This post has NOT been accepted by the mailing list yet.
I am looking more of distributing my c++ functionality across the different nodes to achieve better performance. Hoping I can use Compute Fabric for that.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Options

ksatya wrote
I am looking more of distributing my c++ functionality across the different nodes to achieve better performance. Hoping I can use Compute Fabric for that.
In the first place, your algorithms implemented in C++ (or any other language) have to be splittable into parts, so that you can execute these parts in parallel and then reduce to get final result. If this is the case, you will just have to implement ComputeTask interface and provide proper configuration - Ignite will do the rest.

-Val
ksatya ksatya
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Options

This post has NOT been accepted by the mailing list yet.
OK Thanks for the response. I am assuming that I can find some examples on Compute Grid in github that I can refer to get an idea.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Options

ksatya wrote
OK Thanks for the response. I am assuming that I can find some examples on Compute Grid in github that I can refer to get an idea.
Here are some Compute Grid examples: https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/computegrid

-Val
ksatya ksatya
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Options

This post has NOT been accepted by the mailing list yet.
Thanks for the information.