C# CacheStoreAdapter - Customizing Load, LoadCache methods

classic Classic list List threaded Threaded
23 messages Options
12
abd abd
Reply | Threaded
Open this post in threaded view
|

C# CacheStoreAdapter - Customizing Load, LoadCache methods

CONTENTS DELETED
The author has deleted this message.
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Hello,

> (1) Regarding the Load method, I was under the impression that calling Get(key) would inherently call the Load(key) method from my OracleStore.cs

It would if you set CacheConfiguration.ReadThrough = true. Similarly, enable WriteThrough if you want Ignite to call OracleStore.Write on cache.Put.


(2) With the LoadCache method, I essentially would be doing the same thing as the Load method, but this time, I would be querying 10,000 entries from the database and putting it into cache
>  am unsure about what to do for the Action<TK, TV> parameter

You are supposed to call this action for every entry to be loaded into cache, e.g.:

while(reader.Read())
            {
                /*
                 * 1) Instantiate Associate object
                 * 2) Put Associate object in cache
                 */
                  var associate = new Associate(reader);
                  act(associate.Id, associate);
            }


On Fri, Nov 20, 2020 at 8:23 PM ABDumalagan <[hidden email]> wrote:

Hello,

I have some confusion about implementing my own Load and LoadCache methods -

I am trying to load data from an underlying Oracle database into cache, and the entries are key-value pairs where the key is the Associate ID and the value is the entire Associate object.

(1) Regarding the Load method, I was under the impression that calling Get(key) would inherently call the Load(key) method from my OracleStore.cs file and load the key and corresponding object from the database into cache.

Under the Load method in OracleStore.cs, I connect to the underlying Oracle database and execute a query which grabs the data that correlates to the Associate ID.

However, after running my Program.cs file, I received an error saying "System.Collections.Generic.KeyNotFoundException: 'The given key was not present in the cache: 18755" -- was I under the wrong impression that the Load(key) would just put the data into cache?

I know that the Load method returns an object, and in my case, it returns an Associate object. However, do I need to do implement something else in order to put the object returned from the Load method into cache?

(2) With the LoadCache method, I essentially would be doing the same thing as the Load method, but this time, I would be querying 10,000 entries from the database and putting it into cache. I also hold the same assumption that calling the LoadCache method will just load the 10,000 entries in cache.

Also, I am unsure about what to do for the Action<TK, TV> parameter in LoadCache(Action<TK, TV>, Object[])--would I need to define a method to pass as a parameter, and would that method be responsible for putting the entries in cache? If so, how would I go about it?

Files attached below:

Inherits CacheStoreAdapter: OracleStore.cs

Inherits IFactory: OracleStoreFactory.cs

Establishes Oracle database connection: DBConnection.cs

Defines Associate object: Associate.cs

Runs program (creates and starts node): Program.cs



Sent from the Apache Ignite Users mailing list archive at Nabble.com.

abd abd
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

CONTENTS DELETED
The author has deleted this message.
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Looks like we mixed up two different delegates:

1. ICache.LoadCache method takes an optional filter delegate in case you want to skip some of the entries from cache store. You can pass null there when filtering is not required.
2. ICacheStore.LoadCache method gives you (an implementor of the interface) a delegate that you should call for every entry to be loaded

On Sat, Nov 21, 2020 at 11:05 PM ABDumalagan <[hidden email]> wrote:

Got (1) to work - thank you!

However, for (2), I tried to implement a test Action delegate that writes the value of the entry in the console and I got an error stating:

Argument 1: cannot convert from 'System.Action<int, IgniteTest.Associate>' to 'Apache.Ignite.Core.Cache.ICacheEntryFilter<int, IgniteTest.Associate>'

This error occurred at line cache.Load(action, null)

Am I going about this implementation the wrong way? If so, what would I need to do to correct this?


Updated files:

Runs program; test Action delegate:Program.cs

Inherits CacheStoreAdapter: OracleStore.cs



Sent from the Apache Ignite Users mailing list archive at Nabble.com.

abd abd
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

CONTENTS DELETED
The author has deleted this message.
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Your code seems to be correct. It works for me in a simplified form:

1. Can you try the program above - does it work for you?
2. Can you confirm that the Oracle query returns a non-empty result set?


On Mon, Nov 23, 2020 at 3:00 AM ABDumalagan <[hidden email]> wrote:

I see - am I dealing with 1.) in my case?

When I hover over the method LoadCache(Action<int, Associate>, params object[] args) in Visual Studio, it says that "...This method is called whenever Apache.Ignite.Core.Cache.ICache<TK,TV>.LocalLoadCache(Apache.Ignite.Core.Cache.ICacheEntryFilter<TK,TV>, params object[]) method is invoked which is usually to preload the cache from persistent storage".

From the .NET docs, it says that for LocalLoadCache(ICacheEntryFilter<TK, TV>, Object[]), the loaded values will then be given to the optionally passed in predicate, and, if the predicate returns true, will be stored in cache. If predicate is null, then all loaded values will be stored in cache.

In my case, I call cache.LocalLoadCache(null) in Program.cs, where it then calls on LoadCache(Action<int, Associate> act, params object[] args) in OracleStore.cs and the Associate IDs are being printed to console.

However, when I try to print the cache size in console, it returns 0 - does this mean that the values are not being stored in cache?

Do I need to do something else to make sure the values are loaded into cache? I thought that LocalLoadCache would just put all the values queried from the underlying storage into cache.



Sent from the Apache Ignite Users mailing list archive at Nabble.com.

abd abd
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

CONTENTS DELETED
The author has deleted this message.
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

1. How many Ignite nodes do you have?
2. What if you add Thread.Sleep(5000) before the last Console.WriteLine? Does the resulting number change?

On Mon, Nov 23, 2020 at 6:01 PM ABDumalagan <[hidden email]> wrote:

1. Your program worked for me!


2. I added something to my LoadCache(Action<int, Associate>, params object[] args) method in OracleStore.cs. I added the following 3 lines after the while loop:

reader.Dispose();

cmd.Dispose();

con.Dispose();

Console returned a non-zero cache size of 5136, however, the queries I wanted to do and the queries counted by the method is 10,000 - I was wondering what happened to the other ~5000 queries and why they aren't in cache?



Sent from the Apache Ignite Users mailing list archive at Nabble.com.

abd abd
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

CONTENTS DELETED
The author has deleted this message.
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Ok, here is the problem, per documentation [1]:
"In case of partitioned caches, keys that are not mapped to this node, either as primary or backups, will be automatically discarded by the cache."

Since you have two nodes in the cluster, but call localLoadCache only on one node, part of the cache entries that belong to the other nodes are discarded.
You can do one of the following:
1. Call LoadCache instead of LocalLoadCache, so that cache store is invoked on every node. This will perform the same Oracle query multiple times, which may be suboptimal.
2. Use DataStreamer [2] instead of LoadCache to load all the data from a single node

On Tue, Nov 24, 2020 at 1:25 AM ABDumalagan <[hidden email]> wrote:

1. I currently just have 2 Ignite nodes--first one remotely to start the cluster and the second one (this one) started programmatically with C#.

2. Adding Thread.Sleep(5000) doesn't change the result, unfortunately.



Sent from the Apache Ignite Users mailing list archive at Nabble.com.
abd abd
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

CONTENTS DELETED
The author has deleted this message.
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

DataStreamer is the best approach for loading large amounts of data into Ignite

On Tue, Nov 24, 2020 at 3:50 PM ABDumalagan <[hidden email]> wrote:

I see, that makes sense - I actually have a question regarding that:

If I wanted to store a whole table from an underlying database as key-value pairs in cache, but the database had over 500,000 entries, how would you suggest going about that?



Sent from the Apache Ignite Users mailing list archive at Nabble.com.

abd abd
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

In reply to this post by ptupitsyn
CONTENTS DELETED
The author has deleted this message.
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

NullPointerException usually indicates a bug. Please provide full exception stack trace.

On Tue, Nov 24, 2020 at 4:03 PM ABDumalagan <[hidden email]> wrote:

Also, I tried changing the code from cache.LocalLoadCache(null) to cache.LoadCache(null) and I received a NullPointerException.

Am I supposed to pass a non-null parameter into LoadCache?



Sent from the Apache Ignite Users mailing list archive at Nabble.com.

abd abd
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

CONTENTS DELETED
The author has deleted this message.
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

1) Yes
2) On my local machine with 2 nodes I can stream 500 000 entries in 2 seconds (I tried to recreate the Associate class with some fake data).
Are you sure that Oracle is not a bottleneck? Can you try a local scenario with fake data - how much time does it take?

On Tue, Nov 24, 2020 at 5:46 PM ABDumalagan <[hidden email]> wrote:

I see - I'm actually going to scrap the LoadCache approach and implement your suggestion of DataStream - I just have a couple of questions regarding that:

1) Is it possible to use both DataStream and CacheStoreFactory together?

2) Right now, my Program.cs of DataStream is naïve and loads 1000 entries in ~7 seconds, 10000 entries in ~55 seconds, and 100000 entries in 9 mins and 12 seconds

Is this a normal time for loading bulk entries?

Is there another way I can implement or configure DataStream to load bulk entries faster? I think at this rate, loading > 500,000 entries will be pushing around an hour.



Sent from the Apache Ignite Users mailing list archive at Nabble.com.

abd abd
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

CONTENTS DELETED
The author has deleted this message.
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

> how would I check to see if the underlying database is a bottleneck
Check how much time it takes to read 500 000 rows from there, without loading into Ignite.
Or run a profiler.

On Tue, Nov 24, 2020 at 6:25 PM ABDumalagan <[hidden email]> wrote:
I'll try a local scenario and stream with fake data right now, but how would I check to see if the underlying database is a bottleneck?

Sent from the Apache Ignite Users mailing list archive at Nabble.com.
abd abd
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

CONTENTS DELETED
The author has deleted this message.
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

The code looks ok to me.
Since the query is so simple, most likely your connection to the DB is the bottleneck.

On Tue, Nov 24, 2020 at 8:30 PM ABDumalagan <[hidden email]> wrote:

So I am currently running this code to just pull all the rows from the underlying database, and the diagnostic session is hitting 20 mins and counting -

Is this definitively a database issue or is it in the way I've written the code?



Sent from the Apache Ignite Users mailing list archive at Nabble.com.

12