Ignite server not equally distribution load

classic Classic list List threaded Threaded
10 messages Options
Charlin Charlin
Reply | Threaded
Open this post in threaded view
|

Ignite server not equally distribution load

Hi,

I'm running an ASP.Net application with ignite 2.9.1  and node setup as 2 server nodes and 11 client nodes.
We are seeing most of the load on one server only it's supposed to distribute load between two servers.
How can we distributed load be equally distributed on both servers?


Thanks & Regards,
Charlin
stephendarlington stephendarlington
Reply | Threaded
Open this post in threaded view
|

Re: Ignite server not equally distribution load

What are you doing with Ignite? Are you sending compute tasks, cache operations, both? What’s your configuration?

> On 31 Mar 2021, at 12:31, Charlin S <[hidden email]> wrote:
>
> Hi,
>
> I'm running an ASP.Net application with ignite 2.9.1  and node setup as 2 server nodes and 11 client nodes.
> We are seeing most of the load on one server only it's supposed to distribute load between two servers.
> How can we distributed load be equally distributed on both servers?
>
>
> Thanks & Regards,
> Charlin


Charlin Charlin
Reply | Threaded
Open this post in threaded view
|

Re: Ignite server not equally distribution load

Hello,
Thanks for your reply. I have attached my configuration files here.
Below code has been used to start ignite.

IgniteConfiguration igniteGridIg = new IgniteConfiguration();
igniteGridIg.SpringConfigUrl = Path.Combine(igniteXmlfilePath);
Ignition.Start(igniteGridIg);

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 17:44, Stephen Darlington <[hidden email]> wrote:
What are you doing with Ignite? Are you sending compute tasks, cache operations, both? What’s your configuration?

> On 31 Mar 2021, at 12:31, Charlin S <[hidden email]> wrote:
>
> Hi,
>
> I'm running an ASP.Net application with ignite 2.9.1  and node setup as 2 server nodes and 11 client nodes.
> We are seeing most of the load on one server only it's supposed to distribute load between two servers.
> How can we distributed load be equally distributed on both servers?
>
>
> Thanks & Regards,
> Charlin



client_tmc_staticcache_client_config.txt (2K) Download Attachment
server_static.txt (3K) Download Attachment
Mikael-2 Mikael-2
Reply | Threaded
Open this post in threaded view
|

Re: Ignite server not equally distribution load

Hi!

What is it that is not distributed well, is it cache data/memory load or computations/cpu load ?

There are no caches in the config so I assume this is created by the application (if there are any), do you use any custom collocation ?

Mikael

Do you use any custom collocation of any kind

On 2021-03-31 15:52, Charlin S wrote:
Hello,
Thanks for your reply. I have attached my configuration files here.
Below code has been used to start ignite.

IgniteConfiguration igniteGridIg = new IgniteConfiguration();
igniteGridIg.SpringConfigUrl = Path.Combine(igniteXmlfilePath);
Ignition.Start(igniteGridIg);

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 17:44, Stephen Darlington <[hidden email]> wrote:
What are you doing with Ignite? Are you sending compute tasks, cache operations, both? What’s your configuration?

> On 31 Mar 2021, at 12:31, Charlin S <[hidden email]> wrote:
>
> Hi,
>
> I'm running an ASP.Net application with ignite 2.9.1  and node setup as 2 server nodes and 11 client nodes.
> We are seeing most of the load on one server only it's supposed to distribute load between two servers.
> How can we distributed load be equally distributed on both servers?
>
>
> Thanks & Regards,
> Charlin


Charlin Charlin
Reply | Threaded
Open this post in threaded view
|

Re: Ignite server not equally distribution load

Hello,
Thanks for the reply.
Based on our findings in logs, loads are going to one server.
The server 10.212.120.66 is struggling to do its work: there are lots of "Possible too long JVM pause", "Query execution is too long" and "Long running query". 
On the other hand server 10.212.120.67 has a clean log without those messages.  
Yes, caches(165 cache models)are created by other windows applications and ASP.Net API for only reading data except few cache. 

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 20:00, Mikael <[hidden email]> wrote:

Hi!

What is it that is not distributed well, is it cache data/memory load or computations/cpu load ?

There are no caches in the config so I assume this is created by the application (if there are any), do you use any custom collocation ?

Mikael

Do you use any custom collocation of any kind

On 2021-03-31 15:52, Charlin S wrote:
Hello,
Thanks for your reply. I have attached my configuration files here.
Below code has been used to start ignite.

IgniteConfiguration igniteGridIg = new IgniteConfiguration();
igniteGridIg.SpringConfigUrl = Path.Combine(igniteXmlfilePath);
Ignition.Start(igniteGridIg);

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 17:44, Stephen Darlington <[hidden email]> wrote:
What are you doing with Ignite? Are you sending compute tasks, cache operations, both? What’s your configuration?

> On 31 Mar 2021, at 12:31, Charlin S <[hidden email]> wrote:
>
> Hi,
>
> I'm running an ASP.Net application with ignite 2.9.1  and node setup as 2 server nodes and 11 client nodes.
> We are seeing most of the load on one server only it's supposed to distribute load between two servers.
> How can we distributed load be equally distributed on both servers?
>
>
> Thanks & Regards,
> Charlin


stephendarlington stephendarlington
Reply | Threaded
Open this post in threaded view
|

Re: Ignite server not equally distribution load

I can see two likely possibilities:

Your two server nodes are not forming a cluster. What do you see in the logs? You should see something like this:

[18:42:46] Topology snapshot [ver=9, locNode=a4b47587, servers=2, clients=0, state=ACTIVE, CPUs=8, offheap=12.0GB, heap=4.0GB]

Please confirm that it’s using the correct number of servers.

Alternatively, your keys (or affinity keys) are not very random. What kind of field are you using as a key? You’ve not shared your cache configuration so this is difficult to validate.

On 31 Mar 2021, at 17:04, Charlin S <[hidden email]> wrote:

Hello,
Thanks for the reply.
Based on our findings in logs, loads are going to one server.
The server 10.212.120.66 is struggling to do its work: there are lots of "Possible too long JVM pause", "Query execution is too long" and "Long running query". 
On the other hand server 10.212.120.67 has a clean log without those messages.  
Yes, caches(165 cache models)are created by other windows applications and ASP.Net API for only reading data except few cache. 

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 20:00, Mikael <[hidden email]> wrote:

Hi!

What is it that is not distributed well, is it cache data/memory load or computations/cpu load ?

There are no caches in the config so I assume this is created by the application (if there are any), do you use any custom collocation ?

Mikael

Do you use any custom collocation of any kind

On 2021-03-31 15:52, Charlin S wrote:
Hello,
Thanks for your reply. I have attached my configuration files here.
Below code has been used to start ignite.

IgniteConfiguration igniteGridIg = new IgniteConfiguration();
igniteGridIg.SpringConfigUrl = Path.Combine(igniteXmlfilePath);
Ignition.Start(igniteGridIg);

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 17:44, Stephen Darlington <[hidden email]> wrote:
What are you doing with Ignite? Are you sending compute tasks, cache operations, both? What’s your configuration?

> On 31 Mar 2021, at 12:31, Charlin S <[hidden email]> wrote:
>
> Hi,
>
> I'm running an ASP.Net application with ignite 2.9.1  and node setup as 2 server nodes and 11 client nodes.
> We are seeing most of the load on one server only it's supposed to distribute load between two servers.
> How can we distributed load be equally distributed on both servers?
>
>
> Thanks & Regards,
> Charlin




Charlin Charlin
Reply | Threaded
Open this post in threaded view
|

Re: Ignite server not equally distribution load

Hello,
Thanks for the reply.
Two servers are formed as cluster and the details as below
Topology snapshot [ver=262, locNode=99bfa50b, servers=2, clients=14, state=ACTIVE, CPUs=188, offheap=64.0GB, heap=97.0GB]

My code snippet in cache creation application

Step 1  - Ignite grid instance creation on client node
IgniteConfiguration igniteGridIg = new IgniteConfiguration();
igniteGridIg.SpringConfigUrl = Path.Combine(igniteXmlfilePath);
 IIgnite IgniteInstance =  Ignition.Start( igniteGridIg );

Step 2  - Cache creation with following  Cache configuration
 CacheConfiguration cg = null;
 cg = new CacheConfiguration(cacheName, new QueryEntity(typeof(string), type));
 cg.CacheMode = CacheMode.Replicated;
cg.EnableStatistics = true;
Apache.Ignite.Core.Cache.ICache<string, object> igniteModel = null;
 igniteModel = IgniteInstance.GetOrCreateCache<string, object>(cg);

Step 3 - Cache data store/push in cache
using (var cacheDataStreamer = IgniteInstance.GetDataStreamer<string, object>("cacheName"))
{
   object dbObj = class model data
   cacheDataStreamer.AddData(modelCacheKey, dbObj);
}

Kindly let me know if I need to add/set any more property for this.

Thanks & Regards,
Charlin

On Wed, 31 Mar 2021 at 22:01, Stephen Darlington <[hidden email]> wrote:
I can see two likely possibilities:

Your two server nodes are not forming a cluster. What do you see in the logs? You should see something like this:

[18:42:46] Topology snapshot [ver=9, locNode=a4b47587, servers=2, clients=0, state=ACTIVE, CPUs=8, offheap=12.0GB, heap=4.0GB]

Please confirm that it’s using the correct number of servers.

Alternatively, your keys (or affinity keys) are not very random. What kind of field are you using as a key? You’ve not shared your cache configuration so this is difficult to validate.

On 31 Mar 2021, at 17:04, Charlin S <[hidden email]> wrote:

Hello,
Thanks for the reply.
Based on our findings in logs, loads are going to one server.
The server 10.212.120.66 is struggling to do its work: there are lots of "Possible too long JVM pause", "Query execution is too long" and "Long running query". 
On the other hand server 10.212.120.67 has a clean log without those messages.  
Yes, caches(165 cache models)are created by other windows applications and ASP.Net API for only reading data except few cache. 

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 20:00, Mikael <[hidden email]> wrote:

Hi!

What is it that is not distributed well, is it cache data/memory load or computations/cpu load ?

There are no caches in the config so I assume this is created by the application (if there are any), do you use any custom collocation ?

Mikael

Do you use any custom collocation of any kind

On 2021-03-31 15:52, Charlin S wrote:
Hello,
Thanks for your reply. I have attached my configuration files here.
Below code has been used to start ignite.

IgniteConfiguration igniteGridIg = new IgniteConfiguration();
igniteGridIg.SpringConfigUrl = Path.Combine(igniteXmlfilePath);
Ignition.Start(igniteGridIg);

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 17:44, Stephen Darlington <[hidden email]> wrote:
What are you doing with Ignite? Are you sending compute tasks, cache operations, both? What’s your configuration?

> On 31 Mar 2021, at 12:31, Charlin S <[hidden email]> wrote:
>
> Hi,
>
> I'm running an ASP.Net application with ignite 2.9.1  and node setup as 2 server nodes and 11 client nodes.
> We are seeing most of the load on one server only it's supposed to distribute load between two servers.
> How can we distributed load be equally distributed on both servers?
>
>
> Thanks & Regards,
> Charlin




ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite server not equally distribution load

Hello!

What is your baseline topology? Are both of your server nodes on it?

Regards,
--
Ilya Kasnacheev


чт, 1 апр. 2021 г. в 07:02, Charlin S <[hidden email]>:
Hello,
Thanks for the reply.
Two servers are formed as cluster and the details as below
Topology snapshot [ver=262, locNode=99bfa50b, servers=2, clients=14, state=ACTIVE, CPUs=188, offheap=64.0GB, heap=97.0GB]

My code snippet in cache creation application

Step 1  - Ignite grid instance creation on client node
IgniteConfiguration igniteGridIg = new IgniteConfiguration();
igniteGridIg.SpringConfigUrl = Path.Combine(igniteXmlfilePath);
 IIgnite IgniteInstance =  Ignition.Start( igniteGridIg );

Step 2  - Cache creation with following  Cache configuration
 CacheConfiguration cg = null;
 cg = new CacheConfiguration(cacheName, new QueryEntity(typeof(string), type));
 cg.CacheMode = CacheMode.Replicated;
cg.EnableStatistics = true;
Apache.Ignite.Core.Cache.ICache<string, object> igniteModel = null;
 igniteModel = IgniteInstance.GetOrCreateCache<string, object>(cg);

Step 3 - Cache data store/push in cache
using (var cacheDataStreamer = IgniteInstance.GetDataStreamer<string, object>("cacheName"))
{
   object dbObj = class model data
   cacheDataStreamer.AddData(modelCacheKey, dbObj);
}

Kindly let me know if I need to add/set any more property for this.

Thanks & Regards,
Charlin

On Wed, 31 Mar 2021 at 22:01, Stephen Darlington <[hidden email]> wrote:
I can see two likely possibilities:

Your two server nodes are not forming a cluster. What do you see in the logs? You should see something like this:

[18:42:46] Topology snapshot [ver=9, locNode=a4b47587, servers=2, clients=0, state=ACTIVE, CPUs=8, offheap=12.0GB, heap=4.0GB]

Please confirm that it’s using the correct number of servers.

Alternatively, your keys (or affinity keys) are not very random. What kind of field are you using as a key? You’ve not shared your cache configuration so this is difficult to validate.

On 31 Mar 2021, at 17:04, Charlin S <[hidden email]> wrote:

Hello,
Thanks for the reply.
Based on our findings in logs, loads are going to one server.
The server 10.212.120.66 is struggling to do its work: there are lots of "Possible too long JVM pause", "Query execution is too long" and "Long running query". 
On the other hand server 10.212.120.67 has a clean log without those messages.  
Yes, caches(165 cache models)are created by other windows applications and ASP.Net API for only reading data except few cache. 

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 20:00, Mikael <[hidden email]> wrote:

Hi!

What is it that is not distributed well, is it cache data/memory load or computations/cpu load ?

There are no caches in the config so I assume this is created by the application (if there are any), do you use any custom collocation ?

Mikael

Do you use any custom collocation of any kind

On 2021-03-31 15:52, Charlin S wrote:
Hello,
Thanks for your reply. I have attached my configuration files here.
Below code has been used to start ignite.

IgniteConfiguration igniteGridIg = new IgniteConfiguration();
igniteGridIg.SpringConfigUrl = Path.Combine(igniteXmlfilePath);
Ignition.Start(igniteGridIg);

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 17:44, Stephen Darlington <[hidden email]> wrote:
What are you doing with Ignite? Are you sending compute tasks, cache operations, both? What’s your configuration?

> On 31 Mar 2021, at 12:31, Charlin S <[hidden email]> wrote:
>
> Hi,
>
> I'm running an ASP.Net application with ignite 2.9.1  and node setup as 2 server nodes and 11 client nodes.
> We are seeing most of the load on one server only it's supposed to distribute load between two servers.
> How can we distributed load be equally distributed on both servers?
>
>
> Thanks & Regards,
> Charlin




Charlin Charlin
Reply | Threaded
Open this post in threaded view
|

Re: Ignite server not equally distribution load

Hello,
Thanks for your response,
Yes, both the servers are on.

Thanks & Regards,
Charlin


On Thu, 1 Apr 2021 at 20:29, Ilya Kasnacheev <[hidden email]> wrote:
Hello!

What is your baseline topology? Are both of your server nodes on it?

Regards,
--
Ilya Kasnacheev


чт, 1 апр. 2021 г. в 07:02, Charlin S <[hidden email]>:
Hello,
Thanks for the reply.
Two servers are formed as cluster and the details as below
Topology snapshot [ver=262, locNode=99bfa50b, servers=2, clients=14, state=ACTIVE, CPUs=188, offheap=64.0GB, heap=97.0GB]

My code snippet in cache creation application

Step 1  - Ignite grid instance creation on client node
IgniteConfiguration igniteGridIg = new IgniteConfiguration();
igniteGridIg.SpringConfigUrl = Path.Combine(igniteXmlfilePath);
 IIgnite IgniteInstance =  Ignition.Start( igniteGridIg );

Step 2  - Cache creation with following  Cache configuration
 CacheConfiguration cg = null;
 cg = new CacheConfiguration(cacheName, new QueryEntity(typeof(string), type));
 cg.CacheMode = CacheMode.Replicated;
cg.EnableStatistics = true;
Apache.Ignite.Core.Cache.ICache<string, object> igniteModel = null;
 igniteModel = IgniteInstance.GetOrCreateCache<string, object>(cg);

Step 3 - Cache data store/push in cache
using (var cacheDataStreamer = IgniteInstance.GetDataStreamer<string, object>("cacheName"))
{
   object dbObj = class model data
   cacheDataStreamer.AddData(modelCacheKey, dbObj);
}

Kindly let me know if I need to add/set any more property for this.

Thanks & Regards,
Charlin

On Wed, 31 Mar 2021 at 22:01, Stephen Darlington <[hidden email]> wrote:
I can see two likely possibilities:

Your two server nodes are not forming a cluster. What do you see in the logs? You should see something like this:

[18:42:46] Topology snapshot [ver=9, locNode=a4b47587, servers=2, clients=0, state=ACTIVE, CPUs=8, offheap=12.0GB, heap=4.0GB]

Please confirm that it’s using the correct number of servers.

Alternatively, your keys (or affinity keys) are not very random. What kind of field are you using as a key? You’ve not shared your cache configuration so this is difficult to validate.

On 31 Mar 2021, at 17:04, Charlin S <[hidden email]> wrote:

Hello,
Thanks for the reply.
Based on our findings in logs, loads are going to one server.
The server 10.212.120.66 is struggling to do its work: there are lots of "Possible too long JVM pause", "Query execution is too long" and "Long running query". 
On the other hand server 10.212.120.67 has a clean log without those messages.  
Yes, caches(165 cache models)are created by other windows applications and ASP.Net API for only reading data except few cache. 

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 20:00, Mikael <[hidden email]> wrote:

Hi!

What is it that is not distributed well, is it cache data/memory load or computations/cpu load ?

There are no caches in the config so I assume this is created by the application (if there are any), do you use any custom collocation ?

Mikael

Do you use any custom collocation of any kind

On 2021-03-31 15:52, Charlin S wrote:
Hello,
Thanks for your reply. I have attached my configuration files here.
Below code has been used to start ignite.

IgniteConfiguration igniteGridIg = new IgniteConfiguration();
igniteGridIg.SpringConfigUrl = Path.Combine(igniteXmlfilePath);
Ignition.Start(igniteGridIg);

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 17:44, Stephen Darlington <[hidden email]> wrote:
What are you doing with Ignite? Are you sending compute tasks, cache operations, both? What’s your configuration?

> On 31 Mar 2021, at 12:31, Charlin S <[hidden email]> wrote:
>
> Hi,
>
> I'm running an ASP.Net application with ignite 2.9.1  and node setup as 2 server nodes and 11 client nodes.
> We are seeing most of the load on one server only it's supposed to distribute load between two servers.
> How can we distributed load be equally distributed on both servers?
>
>
> Thanks & Regards,
> Charlin




ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite server not equally distribution load

Hello!

I recommend collecting (java) thread dumps on both server nodes and comparing them, under load.

Then you can infer what is the difference in the load that they handle.

Regards,
--
Ilya Kasnacheev


пн, 5 апр. 2021 г. в 07:20, Charlin S <[hidden email]>:
Hello,
Thanks for your response,
Yes, both the servers are on.

Thanks & Regards,
Charlin


On Thu, 1 Apr 2021 at 20:29, Ilya Kasnacheev <[hidden email]> wrote:
Hello!

What is your baseline topology? Are both of your server nodes on it?

Regards,
--
Ilya Kasnacheev


чт, 1 апр. 2021 г. в 07:02, Charlin S <[hidden email]>:
Hello,
Thanks for the reply.
Two servers are formed as cluster and the details as below
Topology snapshot [ver=262, locNode=99bfa50b, servers=2, clients=14, state=ACTIVE, CPUs=188, offheap=64.0GB, heap=97.0GB]

My code snippet in cache creation application

Step 1  - Ignite grid instance creation on client node
IgniteConfiguration igniteGridIg = new IgniteConfiguration();
igniteGridIg.SpringConfigUrl = Path.Combine(igniteXmlfilePath);
 IIgnite IgniteInstance =  Ignition.Start( igniteGridIg );

Step 2  - Cache creation with following  Cache configuration
 CacheConfiguration cg = null;
 cg = new CacheConfiguration(cacheName, new QueryEntity(typeof(string), type));
 cg.CacheMode = CacheMode.Replicated;
cg.EnableStatistics = true;
Apache.Ignite.Core.Cache.ICache<string, object> igniteModel = null;
 igniteModel = IgniteInstance.GetOrCreateCache<string, object>(cg);

Step 3 - Cache data store/push in cache
using (var cacheDataStreamer = IgniteInstance.GetDataStreamer<string, object>("cacheName"))
{
   object dbObj = class model data
   cacheDataStreamer.AddData(modelCacheKey, dbObj);
}

Kindly let me know if I need to add/set any more property for this.

Thanks & Regards,
Charlin

On Wed, 31 Mar 2021 at 22:01, Stephen Darlington <[hidden email]> wrote:
I can see two likely possibilities:

Your two server nodes are not forming a cluster. What do you see in the logs? You should see something like this:

[18:42:46] Topology snapshot [ver=9, locNode=a4b47587, servers=2, clients=0, state=ACTIVE, CPUs=8, offheap=12.0GB, heap=4.0GB]

Please confirm that it’s using the correct number of servers.

Alternatively, your keys (or affinity keys) are not very random. What kind of field are you using as a key? You’ve not shared your cache configuration so this is difficult to validate.

On 31 Mar 2021, at 17:04, Charlin S <[hidden email]> wrote:

Hello,
Thanks for the reply.
Based on our findings in logs, loads are going to one server.
The server 10.212.120.66 is struggling to do its work: there are lots of "Possible too long JVM pause", "Query execution is too long" and "Long running query". 
On the other hand server 10.212.120.67 has a clean log without those messages.  
Yes, caches(165 cache models)are created by other windows applications and ASP.Net API for only reading data except few cache. 

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 20:00, Mikael <[hidden email]> wrote:

Hi!

What is it that is not distributed well, is it cache data/memory load or computations/cpu load ?

There are no caches in the config so I assume this is created by the application (if there are any), do you use any custom collocation ?

Mikael

Do you use any custom collocation of any kind

On 2021-03-31 15:52, Charlin S wrote:
Hello,
Thanks for your reply. I have attached my configuration files here.
Below code has been used to start ignite.

IgniteConfiguration igniteGridIg = new IgniteConfiguration();
igniteGridIg.SpringConfigUrl = Path.Combine(igniteXmlfilePath);
Ignition.Start(igniteGridIg);

Thanks & Regards,
Charlin



On Wed, 31 Mar 2021 at 17:44, Stephen Darlington <[hidden email]> wrote:
What are you doing with Ignite? Are you sending compute tasks, cache operations, both? What’s your configuration?

> On 31 Mar 2021, at 12:31, Charlin S <[hidden email]> wrote:
>
> Hi,
>
> I'm running an ASP.Net application with ignite 2.9.1  and node setup as 2 server nodes and 11 client nodes.
> We are seeing most of the load on one server only it's supposed to distribute load between two servers.
> How can we distributed load be equally distributed on both servers?
>
>
> Thanks & Regards,
> Charlin