Data Isolation during Ignite Cache load

classic Classic list List threaded Threaded
6 messages Options
nithin91 nithin91
Reply | Threaded
Open this post in threaded view
|

Data Isolation during Ignite Cache load

Hi ,

We have Created a cache in Ignite that gets loaded with Data from
Oracle.Cache is getting loaded using JDBC POJO Store method
Ignite.Cache("Cache Name").loadCache(null).Can anyone explain how will
ignite  handle the below scenario when loading data using the method
Ignite.Cache("Cache Name").loadCache(null).

*Scenario:*
 
We have built a rest service on top this cache and sends the data in the
cache as response after applying some filters.Now During the time of Data
load, if the user sends a get request and when we try to fetch the data from
ignite cache will the filters be applied only the data that is loaded so far
or does the filters be applied on the data that is available prior to the
start of data load.
*Please note we have set the Atomicity mode is set as Atomic for cache.*


In case if  the filters are applied only on the data that is loaded so far,
then is there a property that i need to set so that until the cache load is
loaded , any query on the cache should fetch only the data that is available
prior to the start of data load.Currently as per my observation,  i am able
to see that the filters are applied only on the data that is loaded so far.

Also can anyone help me the possible ways to schedule the Ignite cache
refresh such that the ignite cache refresh job has to be triggered on the
success condition of another unix shell script job.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Data Isolation during Ignite Cache load

Hi,

> will the filters be applied 
What kind of filters? Do you mean SQL queries?

Evgenii

вт, 17 мар. 2020 г. в 07:47, nithin91 <[hidden email]>:
Hi ,

We have Created a cache in Ignite that gets loaded with Data from
Oracle.Cache is getting loaded using JDBC POJO Store method
Ignite.Cache("Cache Name").loadCache(null).Can anyone explain how will
ignite  handle the below scenario when loading data using the method
Ignite.Cache("Cache Name").loadCache(null).

*Scenario:*

We have built a rest service on top this cache and sends the data in the
cache as response after applying some filters.Now During the time of Data
load, if the user sends a get request and when we try to fetch the data from
ignite cache will the filters be applied only the data that is loaded so far
or does the filters be applied on the data that is available prior to the
start of data load.
*Please note we have set the Atomicity mode is set as Atomic for cache.*


In case if  the filters are applied only on the data that is loaded so far,
then is there a property that i need to set so that until the cache load is
loaded , any query on the cache should fetch only the data that is available
prior to the start of data load.Currently as per my observation,  i am able
to see that the filters are applied only on the data that is loaded so far.

Also can anyone help me the possible ways to schedule the Ignite cache
refresh such that the ignite cache refresh job has to be triggered on the
success condition of another unix shell script job.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
nithin91 nithin91
Reply | Threaded
Open this post in threaded view
|

Re: Data Isolation during Ignite Cache load

Hi ,

Filter in the sense,  SQL Fields Query with where clause is executed. I hope
the following example helps you understand my query.

*Scenario:*

For example there is a cache named Products which has 1 million records with
two distinct product categories(i.e. Furniture,Electronics) among these 1
million records.

Now once the Data load is started , if i execute a sql fields query like
below, will i get the count as 1 million or only the count of records that
are loaded so far.

select count(1) from Products where category='Electronics'



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Data Isolation during Ignite Cache load

Yes, SQL queries use data that was already preloaded only.

Evgenii

ср, 18 мар. 2020 г. в 01:23, nithin91 <[hidden email]>:
Hi ,

Filter in the sense,  SQL Fields Query with where clause is executed. I hope
the following example helps you understand my query.

*Scenario:*

For example there is a cache named Products which has 1 million records with
two distinct product categories(i.e. Furniture,Electronics) among these 1
million records.

Now once the Data load is started , if i execute a sql fields query like
below, will i get the count as 1 million or only the count of records that
are loaded so far.

select count(1) from Products where category='Electronics'



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
nithin91 nithin91
Reply | Threaded
Open this post in threaded view
|

Re: Data Isolation during Ignite Cache load

Hi ,

But it is not behaving that way. I have tested it  using the following
scenario.

Scenario:


Initially loading 1000 rows from oracle table with the following filter
where rownum<=1000 into ignite cache.
During the cache load process, i checked the count using sql fields query i
got the response as number of records that are loaded into the cache till
that point i.e  100,200 and finally 1000.

Again started the load into ignite cache  from oracle table with the same
filter where rownum<=1000.
Now during the cache load time, till 1000 rows are loaded it is showing the
count as 1000 but once the count crosses 1000 it is showing the count as
1001,1002 and finally 2000.

Can you please help me fix this issue (i.e While loading Ignite Cache using
JDBC POJO Ignite.cache.loadcache method,  ignite should display the data
prior to the start of data load instead of displaying the data that is
loaded till that point when a query is submitted during the load process. As
we are using the ignite cache for front-end websites,there might be
scenarios where the  user queries the cache against a specific value for
which  he might get an empty response in case if the requested data is not
yet loaded when the cache load process is running. Instead the problem will
be resolved if ignite always shows the data prior to start of data load if
the cache is queried during data load process).









--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Data Isolation during Ignite Cache load

Hi,

> ignite should display the data prior to data load instead of displaying the data that isloaded till that point
What do you mean? How it will help with a problem that you describing after that? 
> he might get an empty response in case if the requested data is notyet loaded when the cache load process is running

There is no data in Ignite, that's why it will not return any value. I don't really understand, why in this case UI will allow user to query entry which was not loaded yet.

Evgenii

пн, 23 мар. 2020 г. в 08:45, nithin91 <[hidden email]>:
Hi ,

But it is not behaving that way. I have tested it  using the following
scenario.

Scenario:


Initially loading 1000 rows from oracle table with the following filter
where rownum<=1000 into ignite cache.
During the cache load process, i checked the count using sql fields query i
got the response as number of records that are loaded into the cache till
that point i.e  100,200 and finally 1000.

Again started the load into ignite cache  from oracle table with the same
filter where rownum<=1000.
Now during the cache load time, till 1000 rows are loaded it is showing the
count as 1000 but once the count crosses 1000 it is showing the count as
1001,1002 and finally 2000.

Can you please help me fix this issue (i.e While loading Ignite Cache using
JDBC POJO Ignite.cache.loadcache method,  ignite should display the data
prior to the start of data load instead of displaying the data that is
loaded till that point when a query is submitted during the load process. As
we are using the ignite cache for front-end websites,there might be
scenarios where the  user queries the cache against a specific value for
which  he might get an empty response in case if the requested data is not
yet loaded when the cache load process is running. Instead the problem will
be resolved if ignite always shows the data prior to start of data load if
the cache is queried during data load process).









--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/