Ignite.Net configuration for storing large cache data

classic Classic list List threaded Threaded
7 messages Options
SudhirPatil SudhirPatil
Reply | Threaded
Open this post in threaded view
|

Ignite.Net configuration for storing large cache data

Hi All,

What should be configuration changes required for storing large data e.g. 10 million records of custom class objects having around 30- 50 properties?

I am using basic default configuration for Ignite.net.
JvmOptions used are like
-Djava.net.preferIPv4Stack=true
-Xms512m
-XX:+UseG1GC
-XX:+DisabkeExplicitGC

Regards,
Sudhir



--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: Ignite.Net configuration for storing large cache data

Hi Sudhir,

There are 3 things to consider:

1) Capacity planning - how much memory do you need for your data?
Depending on type, 30-50 properties can take very different amount of memory to store.
- n+4+1 for strings, where n is size in utf8 encoding
- 4+1 bytes for int
- 8+1 bytes for long
and so on
(every field takes extra byte for type code)


2) JVM settings
Ignite does not store your data in Java Heap, but some memory is still required for processing


3) Ignite DataRegionConfiguration
Ignite stores cache data in so-called Data Regions, in unmanaged heap (aka "offheap").
By default, only one region is configured with max size set to 20% of available RAM.
You may want to increase this by changing IgniteConfiguration.DataStorageConfiguration.DefaultDataRegionConfiguration


On Wed, Nov 13, 2019 at 4:13 AM Sudhir Patil <[hidden email]> wrote:
Hi All,

What should be configuration changes required for storing large data e.g. 10 million records of custom class objects having around 30- 50 properties?

I am using basic default configuration for Ignite.net.
JvmOptions used are like
-Djava.net.preferIPv4Stack=true
-Xms512m
-XX:+UseG1GC
-XX:+DisabkeExplicitGC

Regards,
Sudhir



--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.
SudhirPatil SudhirPatil
Reply | Threaded
Open this post in threaded view
|

Re: Ignite.Net configuration for storing large cache data

Thanks Pavel. 
1) i will look into this.

2) ok.

3) DefaultDataRegionConfiguration - 
What is default unit used  for different properties to set size values ? Is it byte??

For - By default, only one region is configured with max size set to 20% of available RAM.

What is the way to set max size ? MaxSize property? But documentation says it can be set maximum to 10 mb ??

How to set multiple regions?

Regards
Sudhir

On Wednesday, November 13, 2019, Pavel Tupitsyn <[hidden email]> wrote:
Hi Sudhir,

There are 3 things to consider:

1) Capacity planning - how much memory do you need for your data?
Depending on type, 30-50 properties can take very different amount of memory to store.
- n+4+1 for strings, where n is size in utf8 encoding
- 4+1 bytes for int
- 8+1 bytes for long
and so on
(every field takes extra byte for type code)


2) JVM settings
Ignite does not store your data in Java Heap, but some memory is still required for processing


3) Ignite DataRegionConfiguration
Ignite stores cache data in so-called Data Regions, in unmanaged heap (aka "offheap").
By default, only one region is configured with max size set to 20% of available RAM.
You may want to increase this by changing IgniteConfiguration.DataStorageConfiguration.DefaultDataRegionConfiguration


On Wed, Nov 13, 2019 at 4:13 AM Sudhir Patil <[hidden email]> wrote:
Hi All,

What should be configuration changes required for storing large data e.g. 10 million records of custom class objects having around 30- 50 properties?

I am using basic default configuration for Ignite.net.
JvmOptions used are like
-Djava.net.preferIPv4Stack=true
-Xms512m
-XX:+UseG1GC
-XX:+DisabkeExplicitGC

Regards,
Sudhir



--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.


--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: Ignite.Net configuration for storing large cache data

Yes, region sizes are in bytes (thanks for pointing this out, I'll update the XMLDoc)
Yes, MaxSize is the right property. Docs say it should not be less than 10 mb.
Here is an example of C# config with multiple regions:
var cfg = new IgniteConfiguration
{
DataStorageConfiguration = new DataStorageConfiguration
{
DefaultDataRegionConfiguration = new DataRegionConfiguration
{
MaxSize = 16L * 1024 * 1024 * 1024,
Name = "default_region"
},
DataRegionConfigurations = new[]
{
new DataRegionConfiguration
{
MaxSize = 8L * 1024 * 1024 * 1024,
Name = "custom_region"
}
}
},
CacheConfiguration = new []
{
new CacheConfiguration
{
Name = "cache_in_default_region"
},
new CacheConfiguration
{
Name = "cache_in_custom_region",
DataRegionName = "custom_region"
},
}
};

var ignite = Ignition.Start(cfg);

As a result, in the Ignite log we can see:
[12:27:23] Data Regions Configured:
[12:27:23]   ^-- default_region [initSize=256.0 MiB, maxSize=16.0 GiB, persistence=false]
[12:27:23]   ^-- custom_region [initSize=256.0 MiB, maxSize=8.0 GiB, persistence=false]

On Wed, Nov 13, 2019 at 11:25 AM Sudhir Patil <[hidden email]> wrote:
Thanks Pavel. 
1) i will look into this.

2) ok.

3) DefaultDataRegionConfiguration - 
What is default unit used  for different properties to set size values ? Is it byte??

For - By default, only one region is configured with max size set to 20% of available RAM.

What is the way to set max size ? MaxSize property? But documentation says it can be set maximum to 10 mb ??

How to set multiple regions?

Regards
Sudhir

On Wednesday, November 13, 2019, Pavel Tupitsyn <[hidden email]> wrote:
Hi Sudhir,

There are 3 things to consider:

1) Capacity planning - how much memory do you need for your data?
Depending on type, 30-50 properties can take very different amount of memory to store.
- n+4+1 for strings, where n is size in utf8 encoding
- 4+1 bytes for int
- 8+1 bytes for long
and so on
(every field takes extra byte for type code)


2) JVM settings
Ignite does not store your data in Java Heap, but some memory is still required for processing


3) Ignite DataRegionConfiguration
Ignite stores cache data in so-called Data Regions, in unmanaged heap (aka "offheap").
By default, only one region is configured with max size set to 20% of available RAM.
You may want to increase this by changing IgniteConfiguration.DataStorageConfiguration.DefaultDataRegionConfiguration


On Wed, Nov 13, 2019 at 4:13 AM Sudhir Patil <[hidden email]> wrote:
Hi All,

What should be configuration changes required for storing large data e.g. 10 million records of custom class objects having around 30- 50 properties?

I am using basic default configuration for Ignite.net.
JvmOptions used are like
-Djava.net.preferIPv4Stack=true
-Xms512m
-XX:+UseG1GC
-XX:+DisabkeExplicitGC

Regards,
Sudhir



--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.


--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.
SudhirPatil SudhirPatil
Reply | Threaded
Open this post in threaded view
|

Re: Ignite.Net configuration for storing large cache data

Outside this question, such large data will not work with thin client???

Regards,
Sudhir

On Wednesday, November 13, 2019, Pavel Tupitsyn <[hidden email]> wrote:
Yes, region sizes are in bytes (thanks for pointing this out, I'll update the XMLDoc)
Yes, MaxSize is the right property. Docs say it should not be less than 10 mb.
Here is an example of C# config with multiple regions:
var cfg = new IgniteConfiguration
{
DataStorageConfiguration = new DataStorageConfiguration
{
DefaultDataRegionConfiguration = new DataRegionConfiguration
{
MaxSize = 16L * 1024 * 1024 * 1024,
Name = "default_region"
},
DataRegionConfigurations = new[]
{
new DataRegionConfiguration
{
MaxSize = 8L * 1024 * 1024 * 1024,
Name = "custom_region"
}
}
},
CacheConfiguration = new []
{
new CacheConfiguration
{
Name = "cache_in_default_region"
},
new CacheConfiguration
{
Name = "cache_in_custom_region",
DataRegionName = "custom_region"
},
}
};

var ignite = Ignition.Start(cfg);

As a result, in the Ignite log we can see:
[12:27:23] Data Regions Configured:
[12:27:23]   ^-- default_region [initSize=256.0 MiB, maxSize=16.0 GiB, persistence=false]
[12:27:23]   ^-- custom_region [initSize=256.0 MiB, maxSize=8.0 GiB, persistence=false]

On Wed, Nov 13, 2019 at 11:25 AM Sudhir Patil <[hidden email]> wrote:
Thanks Pavel. 
1) i will look into this.

2) ok.

3) DefaultDataRegionConfiguration - 
What is default unit used  for different properties to set size values ? Is it byte??

For - By default, only one region is configured with max size set to 20% of available RAM.

What is the way to set max size ? MaxSize property? But documentation says it can be set maximum to 10 mb ??

How to set multiple regions?

Regards
Sudhir

On Wednesday, November 13, 2019, Pavel Tupitsyn <[hidden email]> wrote:
Hi Sudhir,

There are 3 things to consider:

1) Capacity planning - how much memory do you need for your data?
Depending on type, 30-50 properties can take very different amount of memory to store.
- n+4+1 for strings, where n is size in utf8 encoding
- 4+1 bytes for int
- 8+1 bytes for long
and so on
(every field takes extra byte for type code)


2) JVM settings
Ignite does not store your data in Java Heap, but some memory is still required for processing


3) Ignite DataRegionConfiguration
Ignite stores cache data in so-called Data Regions, in unmanaged heap (aka "offheap").
By default, only one region is configured with max size set to 20% of available RAM.
You may want to increase this by changing IgniteConfiguration.DataStorageConfiguration.DefaultDataRegionConfiguration


On Wed, Nov 13, 2019 at 4:13 AM Sudhir Patil <[hidden email]> wrote:
Hi All,

What should be configuration changes required for storing large data e.g. 10 million records of custom class objects having around 30- 50 properties?

I am using basic default configuration for Ignite.net.
JvmOptions used are like
-Djava.net.preferIPv4Stack=true
-Xms512m
-XX:+UseG1GC
-XX:+DisabkeExplicitGC

Regards,
Sudhir



--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.


--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.


--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.
Alexandr Shapkin Alexandr Shapkin
Reply | Threaded
Open this post in threaded view
|

RE: Re: Ignite.Net configuration for storing large cache data

Thin client does not store data and is not a part of a cluster, instead, you can think of it as an interface of accessing and manipulating your data.

Please, note, that you can not configure these properties for IgniteClientConfiguration.

The data region configuration makes sense only for server nodes.

 

From: [hidden email]
Sent: Wednesday, November 13, 2019 1:17 PM
To: [hidden email]
Subject: Re: Ignite.Net configuration for storing large cache data

 

Outside this question, such large data will not work with thin client???

 

Regards,

Sudhir

On Wednesday, November 13, 2019, Pavel Tupitsyn <[hidden email]> wrote:

Yes, region sizes are in bytes (thanks for pointing this out, I'll update the XMLDoc)

Yes, MaxSize is the right property. Docs say it should not be less than 10 mb.

Here is an example of C# config with multiple regions:

var cfg = new IgniteConfiguration
{
    DataStorageConfiguration =
new DataStorageConfiguration
   
{
        DefaultDataRegionConfiguration =
new DataRegionConfiguration
       
{
            MaxSize = 16L * 1024 * 1024 * 1024,
            Name =
"default_region"
       
},
        DataRegionConfigurations =
new[]
        {
           
new DataRegionConfiguration
           
{
                MaxSize = 8L * 1024 * 1024 * 1024,
                Name =
"custom_region"
           
}
        }
    },
    CacheConfiguration =
new []
    {
       
new CacheConfiguration
       
{
            Name =
"cache_in_default_region"
       
},
       
new CacheConfiguration
       
{
            Name =
"cache_in_custom_region",
            DataRegionName =
"custom_region"
       
},
   
}
};

var ignite = Ignition.Start(cfg);
 
As a result, in the Ignite log we can see:
[12:27:23] Data Regions Configured:
[12:27:23]   ^-- default_region [initSize=256.0 MiB, maxSize=16.0 GiB, persistence=false]
[12:27:23]   ^-- custom_region [initSize=256.0 MiB, maxSize=8.0 GiB, persistence=false]

 

On Wed, Nov 13, 2019 at 11:25 AM Sudhir Patil <[hidden email]> wrote:

Thanks Pavel. 

1) i will look into this.

 

2) ok.

 

3) DefaultDataRegionConfiguration - 

What is default unit used  for different properties to set size values ? Is it byte??

 

For - By default, only one region is configured with max size set to 20% of available RAM.

 

What is the way to set max size ? MaxSize property? But documentation says it can be set maximum to 10 mb ??

 

How to set multiple regions?

 

Regards

Sudhir

On Wednesday, November 13, 2019, Pavel Tupitsyn <[hidden email]> wrote:

Hi Sudhir,

 

There are 3 things to consider:

 

1) Capacity planning - how much memory do you need for your data?

Depending on type, 30-50 properties can take very different amount of memory to store.

- n+4+1 for strings, where n is size in utf8 encoding

- 4+1 bytes for int

- 8+1 bytes for long

and so on

(every field takes extra byte for type code)

 

 

2) JVM settings

Ignite does not store your data in Java Heap, but some memory is still required for processing

 

 

3) Ignite DataRegionConfiguration

Ignite stores cache data in so-called Data Regions, in unmanaged heap (aka "offheap").

By default, only one region is configured with max size set to 20% of available RAM.

You may want to increase this by changing IgniteConfiguration.DataStorageConfiguration.DefaultDataRegionConfiguration

 

 

On Wed, Nov 13, 2019 at 4:13 AM Sudhir Patil <[hidden email]> wrote:

Hi All,

 

What should be configuration changes required for storing large data e.g. 10 million records of custom class objects having around 30- 50 properties?

 

I am using basic default configuration for Ignite.net.

JvmOptions used are like

-Djava.net.preferIPv4Stack=true

-Xms512m

-XX:+UseG1GC

-XX:+DisabkeExplicitGC

 

Regards,

Sudhir

 



--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.



--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.



--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.

 

Alex Shapkin
SudhirPatil SudhirPatil
Reply | Threaded
Open this post in threaded view
|

Re: Ignite.Net configuration for storing large cache data

Ok. Thanks Alexandr.

On Wednesday, November 13, 2019, Alexandr Shapkin <[hidden email]> wrote:

Thin client does not store data and is not a part of a cluster, instead, you can think of it as an interface of accessing and manipulating your data.

Please, note, that you can not configure these properties for IgniteClientConfiguration.

The data region configuration makes sense only for server nodes.

 

From: [hidden email]
Sent: Wednesday, November 13, 2019 1:17 PM
To: [hidden email]
Subject: Re: Ignite.Net configuration for storing large cache data

 

Outside this question, such large data will not work with thin client???

 

Regards,

Sudhir

On Wednesday, November 13, 2019, Pavel Tupitsyn <[hidden email]> wrote:

Yes, region sizes are in bytes (thanks for pointing this out, I'll update the XMLDoc)

Yes, MaxSize is the right property. Docs say it should not be less than 10 mb.

Here is an example of C# config with multiple regions:

var cfg = new IgniteConfiguration
{
    DataStorageConfiguration =
new DataStorageConfiguration
   
{
        DefaultDataRegionConfiguration =
new DataRegionConfiguration
       
{
            MaxSize = 16L * 1024 * 1024 * 1024,
            Name =
"default_region"
       
},
        DataRegionConfigurations =
new[]
        {
           
new DataRegionConfiguration
           
{
                MaxSize = 8L * 1024 * 1024 * 1024,
                Name =
"custom_region"
           
}
        }
    },
    CacheConfiguration =
new []
    {
       
new CacheConfiguration
       
{
            Name =
"cache_in_default_region"
       
},
       
new CacheConfiguration
       
{
            Name =
"cache_in_custom_region",
            DataRegionName =
"custom_region"
       
},
   
}
};

var ignite = Ignition.Start(cfg);
 
As a result, in the Ignite log we can see:
[12:27:23] Data Regions Configured:
[12:27:23]   ^-- default_region [initSize=256.0 MiB, maxSize=16.0 GiB, persistence=false]
[12:27:23]   ^-- custom_region [initSize=256.0 MiB, maxSize=8.0 GiB, persistence=false]

 

On Wed, Nov 13, 2019 at 11:25 AM Sudhir Patil <[hidden email]> wrote:

Thanks Pavel. 

1) i will look into this.

 

2) ok.

 

3) DefaultDataRegionConfiguration - 

What is default unit used  for different properties to set size values ? Is it byte??

 

For - By default, only one region is configured with max size set to 20% of available RAM.

 

What is the way to set max size ? MaxSize property? But documentation says it can be set maximum to 10 mb ??

 

How to set multiple regions?

 

Regards

Sudhir

On Wednesday, November 13, 2019, Pavel Tupitsyn <[hidden email]> wrote:

Hi Sudhir,

 

There are 3 things to consider:

 

1) Capacity planning - how much memory do you need for your data?

Depending on type, 30-50 properties can take very different amount of memory to store.

- n+4+1 for strings, where n is size in utf8 encoding

- 4+1 bytes for int

- 8+1 bytes for long

and so on

(every field takes extra byte for type code)

 

 

2) JVM settings

Ignite does not store your data in Java Heap, but some memory is still required for processing

 

 

3) Ignite DataRegionConfiguration

Ignite stores cache data in so-called Data Regions, in unmanaged heap (aka "offheap").

By default, only one region is configured with max size set to 20% of available RAM.

You may want to increase this by changing IgniteConfiguration.DataStorageConfiguration.DefaultDataRegionConfiguration

 

 

On Wed, Nov 13, 2019 at 4:13 AM Sudhir Patil <[hidden email]> wrote:

Hi All,

 

What should be configuration changes required for storing large data e.g. 10 million records of custom class objects having around 30- 50 properties?

 

I am using basic default configuration for Ignite.net.

JvmOptions used are like

-Djava.net.preferIPv4Stack=true

-Xms512m

-XX:+UseG1GC

-XX:+DisabkeExplicitGC

 

Regards,

Sudhir

 



--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.



--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.



--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.

 



--
Thanks & Regards,
Sudhir Patil,
+91 9881095647.