Using Ignite as blob store?

classic Classic list List threaded Threaded
7 messages Options
shaneduan shaneduan
Reply | Threaded
Open this post in threaded view
|

Using Ignite as blob store?

Hi Igniters, is it a good idea to use Ignite(with persistence) as a blob store? I did run some testing with a small dataset, and it looks performing okay, even with a small off-heap mem for the data region. 

Thanks!

Shane
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Using Ignite as blob store?

Hello!

How large are these blobs? Ignite is going to divide blobs into <4k chunks. We have no special optimizations for storing large key-value pairs.

Regards,
--
Ilya Kasnacheev


чт, 22 авг. 2019 г. в 02:53, Shane Duan <[hidden email]>:
Hi Igniters, is it a good idea to use Ignite(with persistence) as a blob store? I did run some testing with a small dataset, and it looks performing okay, even with a small off-heap mem for the data region. 

Thanks!

Shane
shaneduan shaneduan
Reply | Threaded
Open this post in threaded view
|

Re: Using Ignite as blob store?

Thanks, Ilya. The blob size varies from a few KBs to a few MBs.

Cheers,
Shane



On Thu, Aug 22, 2019 at 5:02 AM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

How large are these blobs? Ignite is going to divide blobs into <4k chunks. We have no special optimizations for storing large key-value pairs.

Regards,
--
Ilya Kasnacheev


чт, 22 авг. 2019 г. в 02:53, Shane Duan <[hidden email]>:
Hi Igniters, is it a good idea to use Ignite(with persistence) as a blob store? I did run some testing with a small dataset, and it looks performing okay, even with a small off-heap mem for the data region. 

Thanks!

Shane
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Using Ignite as blob store?

How about setting page size to more KBs or MBs based on the average value? That should work perfectly fine.

-
Denis


On Thu, Aug 22, 2019 at 8:11 AM Shane Duan <[hidden email]> wrote:
Thanks, Ilya. The blob size varies from a few KBs to a few MBs.

Cheers,
Shane



On Thu, Aug 22, 2019 at 5:02 AM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

How large are these blobs? Ignite is going to divide blobs into <4k chunks. We have no special optimizations for storing large key-value pairs.

Regards,
--
Ilya Kasnacheev


чт, 22 авг. 2019 г. в 02:53, Shane Duan <[hidden email]>:
Hi Igniters, is it a good idea to use Ignite(with persistence) as a blob store? I did run some testing with a small dataset, and it looks performing okay, even with a small off-heap mem for the data region. 

Thanks!

Shane
colinc colinc
Reply | Threaded
Open this post in threaded view
|

Re: Using Ignite as blob store?

I understand from this post:
https://stackoverflow.com/questions/50116444/unable-to-increase-pagesize/50121410#50121410

that the maximum page size is 16K. Is that still true?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Pavel Kovalenko Pavel Kovalenko
Reply | Threaded
Open this post in threaded view
|

Re: Using Ignite as blob store?

In reply to this post by dmagda
Denis,

You can't set page size greater than 16Kb due to our page memory limitations.

чт, 22 авг. 2019 г. в 22:34, Denis Magda <[hidden email]>:
How about setting page size to more KBs or MBs based on the average value? That should work perfectly fine.

-
Denis


On Thu, Aug 22, 2019 at 8:11 AM Shane Duan <[hidden email]> wrote:
Thanks, Ilya. The blob size varies from a few KBs to a few MBs.

Cheers,
Shane



On Thu, Aug 22, 2019 at 5:02 AM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

How large are these blobs? Ignite is going to divide blobs into <4k chunks. We have no special optimizations for storing large key-value pairs.

Regards,
--
Ilya Kasnacheev


чт, 22 авг. 2019 г. в 02:53, Shane Duan <[hidden email]>:
Hi Igniters, is it a good idea to use Ignite(with persistence) as a blob store? I did run some testing with a small dataset, and it looks performing okay, even with a small off-heap mem for the data region. 

Thanks!

Shane
colinc colinc
Reply | Threaded
Open this post in threaded view
|

Re: Using Ignite as blob store?

From anecdotal experience of storing larger objects (up to say 10MB) in
Ignite, I find that the overall access performance is significantly better
than storing lots of small objects. The main thing to watch out for is that
very large objects can cause unbalanced data distribution. Similar to
over-use of affinity.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/