Multiple indexes.

classic Classic list List threaded Threaded
6 messages Options
javadevmtl javadevmtl
Reply | Threaded
Open this post in threaded view
|

Multiple indexes.

Using 1.3.0

I'm testing insertion performance. My model requires 12 secondary indexes. I.e: 12 x @QuerySqlField(index = true)

At 6 indexes (meaning I comment out 6 of the 12 indexes)
I can get about 11,000 inserts per second with avg 1ms (includes jmeter latency) using data streamer and tested up to 14 million records.

At 8 indexes
I can get about 7,900 inserts per second with avg 8ms (includes jmeter latency) using data streamer and tested up to 14 million records.

At 12 indexes it starts at 11K and slowly degrades to 5,000 and less inserts per second and of course higher latency.

JMeter always send the same amount of data.

Just wondering if it worth separating my model into separate objects and do 2 inserts to 2 separate caches and do cross cache joins.


vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes.

javadevmtl wrote
Just wondering if it worth separating my model into separate objects and do 2 inserts to 2 separate caches and do cross cache joins.
Personally I don't think that such separation can affect performance, but you can try and see if it changes.
javadevmtl wrote
At 12 indexes it starts at 11K and slowly degrades to 5,000 and less inserts per second and of course higher latency.
This looks like a GC issue. Please make sure that you have enough memory to store the data.

-Val
javadevmtl javadevmtl
Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes.

I think I know what it is...

It's not GC everything is off-heap and I start VM with XMX4g which is more then enough heap to handle the incoming HTTP requests parse the json, convert to object and serialize in the off-heap cache.

Basically, the model has 12 fields. 2 of those fields are "home" address fields e.g: 1234 Street, Apt: 9, New York... And they can be up to 240 characters, so 480 characters total.

I removed those 2 fields for now and I can index 10 fields (most of which are still strings of 60 chars) at 11K requests per second with no slowdown.

So the 2 fields either 2 long for an index or affecting serialization to the cache???



Sergi Vladykin Sergi Vladykin
Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes.

Yes, this can be related to serialization. I think we should test this ourselves to make sure.
Igniters, can someone add such a benchmark to our performance suite and check what happens?

Sergi

2015-08-13 17:24 GMT+03:00 javadevmtl <[hidden email]>:
I think I know what it is...

It's not GC everything is off-heap and I start VM with XMX4g which is more
then enough heap to handle the incoming HTTP requests parse the json,
convert to object and serialize in the off-heap cache.

Basically, the model has 12 fields. 2 of those fields are "home" address
fields e.g: 1234 Street, Apt: 9, New York... And they can be up to 240
characters, so 480 characters total.

I removed those 2 fields for now and I can index 10 fields (most of which
are still strings of 60 chars) at 11K requests per second with no slowdown.

So the 2 fields either 2 long for an index or affecting serialization to the
cache???







--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Multiple-indexes-tp936p945.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes.

@javadevmtl - Do you have a test we can use to reproduce it? You can either provide a github repo link or paste your code into a Gist or Pastebin.

D.

On Thu, Aug 13, 2015 at 1:37 PM, Sergi Vladykin <[hidden email]> wrote:
Yes, this can be related to serialization. I think we should test this ourselves to make sure.
Igniters, can someone add such a benchmark to our performance suite and check what happens?

Sergi

2015-08-13 17:24 GMT+03:00 javadevmtl <[hidden email]>:
I think I know what it is...

It's not GC everything is off-heap and I start VM with XMX4g which is more
then enough heap to handle the incoming HTTP requests parse the json,
convert to object and serialize in the off-heap cache.

Basically, the model has 12 fields. 2 of those fields are "home" address
fields e.g: 1234 Street, Apt: 9, New York... And they can be up to 240
characters, so 480 characters total.

I removed those 2 fields for now and I can index 10 fields (most of which
are still strings of 60 chars) at 11K requests per second with no slowdown.

So the 2 fields either 2 long for an index or affecting serialization to the
cache???







--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Multiple-indexes-tp936p945.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


Konstantin Boudnik Konstantin Boudnik
Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes.

Oh better yet - open a JIRA and contribute a patch, and improve your karma;)

On August 13, 2015 1:41:57 PM PDT, Dmitriy Setrakyan <[hidden email]> wrote:

>@javadevmtl - Do you have a test we can use to reproduce it? You can
>either
>provide a github repo link or paste your code into a Gist or Pastebin.
>
>D.
>
>On Thu, Aug 13, 2015 at 1:37 PM, Sergi Vladykin
><[hidden email]>
>wrote:
>
>> Yes, this can be related to serialization. I think we should test
>this
>> ourselves to make sure.
>> Igniters, can someone add such a benchmark to our performance suite
>and
>> check what happens?
>>
>> Sergi
>>
>> 2015-08-13 17:24 GMT+03:00 javadevmtl <[hidden email]>:
>>
>>> I think I know what it is...
>>>
>>> It's not GC everything is off-heap and I start VM with XMX4g which
>is more
>>> then enough heap to handle the incoming HTTP requests parse the
>json,
>>> convert to object and serialize in the off-heap cache.
>>>
>>> Basically, the model has 12 fields. 2 of those fields are "home"
>address
>>> fields e.g: 1234 Street, Apt: 9, New York... And they can be up to
>240
>>> characters, so 480 characters total.
>>>
>>> I removed those 2 fields for now and I can index 10 fields (most of
>which
>>> are still strings of 60 chars) at 11K requests per second with no
>>> slowdown.
>>>
>>> So the 2 fields either 2 long for an index or affecting
>serialization to
>>> the
>>> cache???
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>>
>http://apache-ignite-users.70518.x6.nabble.com/Multiple-indexes-tp936p945.html
>>> Sent from the Apache Ignite Users mailing list archive at
>Nabble.com.
>>>
>>
>>