Data taking more time to load using GetDataStreamer

classic Classic list List threaded Threaded
8 messages Options
siva siva
Reply | Threaded
Open this post in threaded view
|

Data taking more time to load using GetDataStreamer

This post was updated on .
Hi,
I am using Apache Ignite 2.7.5 .Net client and Server.And getting data
streamer from Cache.
I have around 50000 thousand of records in SQL Server DB table.And i am
trying to load using GetDataStreamer.

here is the code:
public async void loadData(string configPath,List<JObject> dataList)
        {
            var _ignite = Ignition.Start(configPath);

            var cacheName = "myTenantCache";

            var parsedObjectList = dataList;//records list

            var keyArray = new string[] {"EmpId"};
            Employee emp = new Employee();
            using (var ldr = _ignite.GetDataStreamer<string, Employee>(cacheName))
            {
                //here if i am adding auto flush then method each record getting executed in 1 sec,
                //but when i perform query on sqlline ,getting no of records count is less than no of executed records here.

                ldr.AutoFlushFrequency = 1000;

                foreach (var item in parsedObjectList)
                {
                    try
                    {
                        JObject keyObj = new JObject();
                        foreach (var keyName in keyArray)
                        {
                            keyObj[keyName.ToString()] = item[keyName.ToString()];
                        }

                        var serializerSettings = new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore };

                        JsonConvert.PopulateObject(item.ToString(), emp, serializerSettings);

                        string json = JsonConvert.SerializeObject(keyObj, Formatting.None);

                        string base64EncodedKey = Convert.ToBase64String(Encoding.UTF8.GetBytes(json));

                        await ldr.AddData(base64EncodedKey, emp);//from here not return to next line for long long time                     

                    }
                    catch (Exception ex)
                    {
                        Console.WriteLine(ex);
                    }
                }
                
            }
        }

How to achive loading data fast into cache.
Any one suggest me.
 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Data taking more time to load using GetDataStreamer

Hello!

You do not need to `await' on futures returned to AddData. They are provided for listen() functionality only. What you need to do is close DataStreamer after data load is complete, which you already do.

Regards,
--
Ilya Kasnacheev


чт, 27 июн. 2019 г. в 18:35, siva <[hidden email]>:
Hi,
I am using Apache Ignite 7.5 .Net client and Server.And getting data
streamer from Cache.
I have around 50000 thousand of records in SQL Server DB table.And i am
trying to load using GetDataStreamer.

here is the code:


How to achive loading data fast into cache.
Any one suggest me.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
siva siva
Reply | Threaded
Open this post in threaded view
|

Re: Data taking more time to load using GetDataStreamer

If I'm not using await then sqlline console result showing "0 row selected".



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Data taking more time to load using GetDataStreamer

Hello!

This is strange. Can you throw together a reproducer project that I can run and validate this behavior?

Regards,
--
Ilya Kasnacheev


пт, 28 июн. 2019 г. в 07:33, siva <[hidden email]>:
If I'm not using await then sqlline console result showing "0 row selected".



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
siva siva
Reply | Threaded
Open this post in threaded view
|

Re: Data taking more time to load using GetDataStreamer

Hi ilya.kasnacheev,i have created git repo code .Net core console app with
client and server.

please find the repository link.
https://github.com/CP-Ram/ApacheIgniteClientServerTestDotNetCoreConsoleApp

Cache Name : "myTenantCache"

query field TableName : EMPLOYEE

So when i performing Sqlline query select count(*) from
"myTenantCache".EMPLOYEE;

Expected Result count 10 but getting 1 entry.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Data taking more time to load using GetDataStreamer

Hello!

It seems that you are inserting the same key over and over again - eyJFbXBJZCI6IjAwMDAwMDAwLTAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMCJ9

Regards,
--
Ilya Kasnacheev


пт, 28 июн. 2019 г. в 19:50, siva <[hidden email]>:
Hi ilya.kasnacheev,i have created git repo code .Net core console app with
client and server.

please find the repository link.
https://github.com/CP-Ram/ApacheIgniteClientServerTestDotNetCoreConsoleApp

Cache Name : "myTenantCache"

query field TableName : EMPLOYEE

So when i performing Sqlline query select count(*) from
"myTenantCache".EMPLOYEE;

Expected Result count 10 but getting 1 entry.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
siva siva
Reply | Threaded
Open this post in threaded view
|

Re: Data taking more time to load using GetDataStreamer

Thanks for reply ilya.kasnacheev.

No.
I thing confussion between foreach loop iteration there I'm taking EmpId
value from Employee Object and setting into new JObject,because of that it's
every time new key.

I am not taking same key ....could you little explain where same key getting
inserted or  it's any formatting issue ?

Please see below image,how is the key getting generated .
<http://apache-ignite-users.70518.x6.nabble.com/file/t1379/Screenshot_1.png>



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Data taking more time to load using GetDataStreamer

Hello!

I have no idea why, but when I have added Console.WriteLine, it's the same value over and over. I think that's in code outside of Apache Ignite.

Regards,
--
Ilya Kasnacheev


вт, 2 июл. 2019 г. в 15:49, siva <[hidden email]>:
Thanks for reply ilya.kasnacheev.

No.
I thing confussion between foreach loop iteration there I'm taking EmpId
value from Employee Object and setting into new JObject,because of that it's
every time new key.

I am not taking same key ....could you little explain where same key getting
inserted or  it's any formatting issue ?

Please see below image,how is the key getting generated .
<http://apache-ignite-users.70518.x6.nabble.com/file/t1379/Screenshot_1.png>



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/