Setting Cache Size using Max Records

classic Classic list List threaded Threaded
10 messages Options
Victor Victor
Reply | Threaded
Open this post in threaded view
|

Setting Cache Size using Max Records

Hi,

I am looking to set the cache size for off-heap via number of max records
instead of max bytes. Similar to how LRU eviction policy for on-heap can set
it.

Haven't found anything around that. Is that doable, if yes, how do i go
about it?

Thanks,
Vic



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Setting Cache Size using Max Records

Hi Vic,

That’s unsupported for the off-heap memory. You can only limit memory usage based on the size in bytes.

Please share more details of why this type of records eviction is needed. We might suggest an alternate solution.

Denis

On Wednesday, June 24, 2020, Victor <[hidden email]> wrote:
Hi,

I am looking to set the cache size for off-heap via number of max records
instead of max bytes. Similar to how LRU eviction policy for on-heap can set
it.

Haven't found anything around that. Is that doable, if yes, how do i go
about it?

Thanks,
Vic



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


--
-
Denis

Victor Victor
Reply | Threaded
Open this post in threaded view
|

Re: Setting Cache Size using Max Records

Hi Denis,

For our product we are using an in-house solution where we expose the size
in records. Now we are looking at replacing it with Ignite. So we wanted to
make the experience seamless and let the end user's continue to set the same
configuration they are familiar with, rather than adding a new learning
curve.

Does any of the expiry policies except record size?

Thanks,
Vic



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Setting Cache Size using Max Records

Hello!

You can have a data region configured for this specific cache, with max size (in bytes) and page eviction on.

Regards,
--
Ilya Kasnacheev


чт, 25 июн. 2020 г. в 07:16, Victor <[hidden email]>:
Hi Denis,

For our product we are using an in-house solution where we expose the size
in records. Now we are looking at replacing it with Ignite. So we wanted to
make the experience seamless and let the end user's continue to set the same
configuration they are familiar with, rather than adding a new learning
curve.

Does any of the expiry policies except record size?

Thanks,
Vic



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Victor Victor
Reply | Threaded
Open this post in threaded view
|

Re: Setting Cache Size using Max Records

Iiya/ Denis,

I am aware of the Data Region with max size as bytes. I was looking for way
to do this via number of records.

Anyway, seems there is no way in Ignite to do this today.

I am looking at couple of other options to achieve this, for that same i am
looking at a way to calculate the storage size per record in cache/table. Is
there an api available to retrieve that?

Thanks,
Vic



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Setting Cache Size using Max Records

Victor, 

Is it correct to assume that the users define this configuration first and only then you span up an Ignite cluster for them? If it's so, then you can translate a user-defined number of records to final data region size. The formula might be as follows - "data_region_size = provided_number_of_records * avg_record_size * avg_ignite_overhead_per_record * K" where:
This spreadsheet calculator can be helpful with K: https://apacheignite.readme.io/docs/capacity-planning#example

-
Denis


On Thu, Jun 25, 2020 at 9:44 AM Victor <[hidden email]> wrote:
Iiya/ Denis,

I am aware of the Data Region with max size as bytes. I was looking for way
to do this via number of records.

Anyway, seems there is no way in Ignite to do this today.

I am looking at couple of other options to achieve this, for that same i am
looking at a way to calculate the storage size per record in cache/table. Is
there an api available to retrieve that?

Thanks,
Vic



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Victor Victor
Reply | Threaded
Open this post in threaded view
|

Re: Setting Cache Size using Max Records

Yes, user upfront defines a config and the idea indeed is to convert the
number of records to total bytes.

Followup around this. Given i want to calculate the exact row entry size
upfront, even before the data is added. Couple of queries,

1. Is there a way(api) just based on the Cache (table), be able to inspect
the capacity a row can hold based on the column datatypes?
2. Can the Dataregion be set post the cache is created?

Thanks,
Victor



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Setting Cache Size using Max Records

1. There is no any public API for that but you can follow a workaround suggested here. If you have a POJO class of your domain object, then create an instance, convert it to BinaryObjectImpl and get the array size (as suggested on StackOverflow).

2. No, you have to set the size in advance. However, the combination of initial and max size might do the trick for you. Use the initial size for after you calculate the value for an expected number of records the cache has to store while the max size can give you an extra space to deal with the calculation/estimations errors.
  
-
Denis


On Tue, Jun 30, 2020 at 9:08 PM Victor <[hidden email]> wrote:
Yes, user upfront defines a config and the idea indeed is to convert the
number of records to total bytes.

Followup around this. Given i want to calculate the exact row entry size
upfront, even before the data is added. Couple of queries,

1. Is there a way(api) just based on the Cache (table), be able to inspect
the capacity a row can hold based on the column datatypes?
2. Can the Dataregion be set post the cache is created?

Thanks,
Victor



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Victor Victor
Reply | Threaded
Open this post in threaded view
|

Re: Setting Cache Size using Max Records

Thanks Denis this helps.

Btw, one thing to check, if you create a 25MB region and multiple cache's
are associated to that region, do all the caches share the region capacity
or just share the configuration but each cache will be allocated a 25MB
capacity?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Victor Victor
Reply | Threaded
Open this post in threaded view
|

Re: Setting Cache Size using Max Records

From what i have read so far, seems like if i have 2 caches each to be
configured with 50MB, then i need to define 2 identical regions with unique
names and apply to each of the caches.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/