Issue with large number of Continuous Queries

classic Classic list List threaded Threaded
2 messages Options
zork zork
Reply | Threaded
Open this post in threaded view
|

Issue with large number of Continuous Queries

*Topology*:

Server-1 --> Cache myCache, holds continuously updating data like market
data(prices, status, tradetime etc) for instruments. InstrumentId is the key
for this cache.
                   Server-1 running with following jvm params: -Xms1g,-Xmx6g

Client-1 --> Pushing continuous updates on the cache

Client - 2 & 3 --> Listening updates on myCache using separate Continuous
query on every key (i.e. one continuous query per instrumentId).

The Cache Configuration is as follows:
          cacheMode    PARTITIONED
          atomicityMode    ATOMIC
          backups    2
          readFromBackup    true
          copyOnRead    true
          statisticsEnabled    true
          managementEnabled    true

System hardware: 8 core, 32gb RAM
For now all servers and client below run on same machine.

---------------------------------------------------------

*Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
updates per second on every item. Lets say both client-2 and client-3 have
1000 different continuous queries open to listen to every update.

With the above scenario, we are observing the server-1 alone taking 60-70%
CPU and about 4GB memory.
In this case when high number continuous queries machine reaches 100% CPU
utilization.

*Thinking to fix as*: Use single continuous query per client to listen to
all the updates. i.e. there would be just one continuous query and it would
listen to all the updates.

<http://apache-ignite-users.70518.x6.nabble.com/file/t2754/Untitled.png>

But now the problem is that both the clients do not need to listen to
updates in all the keys in cache. So we are thinking of adding another
column to the ignite cache using which we can filter the updates by checking
if the client column of the updated row contains the client name for which
filter is being checked. e.g. the new table would look like-

<http://apache-ignite-users.70518.x6.nabble.com/file/t2754/Untitled2.png>

Would this be the correct way to achieve what we are trying to achieve? Or
could this be done some other better way in ignite?

Follow up Question:
How many continuous queries can ignite handle at once with the configuration
we mentioned or is there any such benchmark values available on any
particular configuration? Is it fine to have as many as 1000 (or even more)
continuous queries running at once? If yes, how can we make it less CPU
intensive and more performant?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Issue with large number of Continuous Queries

Hello,

How do you decide about a number of continuous queries to be started? I see on one of your pictures that a query is started per key. Usually, a single query per cache per client should be enough as long as the filter allows to exclude records of no interest for a client. Probably, you can put client-specific logic into the filter.

As for 1000s of continuous queries for a single node cluster, this sounds like overkill as long as the server node needs to process all the 1000s queries for every data change.

-
Denis


On Tue, Feb 11, 2020 at 3:00 AM zork <[hidden email]> wrote:
*Topology*:

Server-1 --> Cache myCache, holds continuously updating data like market
data(prices, status, tradetime etc) for instruments. InstrumentId is the key
for this cache.
                   Server-1 running with following jvm params: -Xms1g,-Xmx6g

Client-1 --> Pushing continuous updates on the cache

Client - 2 & 3 --> Listening updates on myCache using separate Continuous
query on every key (i.e. one continuous query per instrumentId).

The Cache Configuration is as follows:
          cacheMode    PARTITIONED
          atomicityMode    ATOMIC
          backups    2
          readFromBackup    true
          copyOnRead    true
          statisticsEnabled    true
          managementEnabled    true

System hardware: 8 core, 32gb RAM
For now all servers and client below run on same machine.

---------------------------------------------------------

*Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
updates per second on every item. Lets say both client-2 and client-3 have
1000 different continuous queries open to listen to every update.

With the above scenario, we are observing the server-1 alone taking 60-70%
CPU and about 4GB memory.
In this case when high number continuous queries machine reaches 100% CPU
utilization.

*Thinking to fix as*: Use single continuous query per client to listen to
all the updates. i.e. there would be just one continuous query and it would
listen to all the updates.

<http://apache-ignite-users.70518.x6.nabble.com/file/t2754/Untitled.png>

But now the problem is that both the clients do not need to listen to
updates in all the keys in cache. So we are thinking of adding another
column to the ignite cache using which we can filter the updates by checking
if the client column of the updated row contains the client name for which
filter is being checked. e.g. the new table would look like-

<http://apache-ignite-users.70518.x6.nabble.com/file/t2754/Untitled2.png>

Would this be the correct way to achieve what we are trying to achieve? Or
could this be done some other better way in ignite?

Follow up Question:
How many continuous queries can ignite handle at once with the configuration
we mentioned or is there any such benchmark values available on any
particular configuration? Is it fine to have as many as 1000 (or even more)
continuous queries running at once? If yes, how can we make it less CPU
intensive and more performant?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/