Read/query TPS is decreasing after enabling mix load i.e. write services

classic Classic list List threaded Threaded
12 messages Options
Kamlesh Joshi Kamlesh Joshi
Reply | Threaded
Open this post in threaded view
|

Read/query TPS is decreasing after enabling mix load i.e. write services

Hi Igniters,

 

                We were performing PT for ignite cluster and observed that after enabling write services, query service TPS drops more than 1500 TPS. While analyzing thread dump we found that only one thread is writing to disk (native persistence enabled) and rest all are in WAITING state. Please find below analysis of the same (we are using default Thread pool configuration) :

 

sys-stripe-43-#44%EDIFCustomer% thread obtained java.util.concurrent.locks.ReentrantLock$NonfairSync's lock & did not release it for writing to file.

sys-stripe-43 thread is performing IO operations for persistency, due to that all other threads are in WAITING state as shown below.

 

 

 

System details as follow:

  1. CPU 48 core
  2. RAM 190 GB
  3. We have separate disk for WAL and DATASTORE
  4. HDD is striped

 

Now the concern is, is this the expected behavior? Or we can change this by using different thread pools ?

 

Kindly help !

 

Thanks and Regards,

Kamlesh Joshi

 


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

Maxim.Pudov Maxim.Pudov
Reply | Threaded
Open this post in threaded view
|

Re: Read/query TPS is decreasing after enabling mix load i.e. write services

Hi, could you share a reproducer of your problem? We are missing a lot of
information here. The configuration of your nodes, cache configurations,
what API you use to query and update the data.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Kamlesh Joshi Kamlesh Joshi
Reply | Threaded
Open this post in threaded view
|

RE: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

Hi,

Unfortunately, I cant share the reproducer.
We are using Ignite Binary Objects for performing operations on the cluster (Get/Put). We have exposed these operations to other application (i.e. TIBCO BW services) which performs operations on the cluster.

Thanks and Regards,
Kamlesh Joshi

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Wednesday, April 17, 2019 8:18 PM
To: [hidden email]
Subject: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hi, could you share a reproducer of your problem? We are missing a lot of information here. The configuration of your nodes, cache configurations, what API you use to query and update the data.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email.
and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

Hello!

We recommend using SSD and not HDD with Ignite. Otherwise, try to increase size of Checkpoint Page Buffer - if you run out of this buffer, all activity will stop until checkpoint is finished. Maybe you also need to decrease time between checkpoints.

Regards,
--
Ilya Kasnacheev


вт, 7 мая 2019 г. в 13:01, Kamlesh Joshi <[hidden email]>:
Hi,

Unfortunately, I cant share the reproducer.
We are using Ignite Binary Objects for performing operations on the cluster (Get/Put). We have exposed these operations to other application (i.e. TIBCO BW services) which performs operations on the cluster.

Thanks and Regards,
Kamlesh Joshi

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Wednesday, April 17, 2019 8:18 PM
To: [hidden email]
Subject: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hi, could you share a reproducer of your problem? We are missing a lot of information here. The configuration of your nodes, cache configurations, what API you use to query and update the data.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email.
and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

Kamlesh Joshi Kamlesh Joshi
Reply | Threaded
Open this post in threaded view
|

RE: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

Hi Ilya,

 

                Looking at the logs, every time checkpoint starts due to ‘timeout’ reason not because the buffer is full. So still do we need to increase checkpointing buffer or only changing checkpointing frequency will help ?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 4:19 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

We recommend using SSD and not HDD with Ignite. Otherwise, try to increase size of Checkpoint Page Buffer - if you run out of this buffer, all activity will stop until checkpoint is finished. Maybe you also need to decrease time between checkpoints.

 

Regards,

--

Ilya Kasnacheev

 

 

вт, 7 мая 2019 г. в 13:01, Kamlesh Joshi <[hidden email]>:

Hi,

Unfortunately, I cant share the reproducer.
We are using Ignite Binary Objects for performing operations on the cluster (Get/Put). We have exposed these operations to other application (i.e. TIBCO BW services) which performs operations on the cluster.

Thanks and Regards,
Kamlesh Joshi

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Wednesday, April 17, 2019 8:18 PM
To: [hidden email]
Subject: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hi, could you share a reproducer of your problem? We are missing a lot of information here. The configuration of your nodes, cache configurations, what API you use to query and update the data.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email.
and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

Hello!

If it is 'timeout' then it is likely not the reason of your issues.

What is your walMode? Is it LOG_ONLY? If it's not, try to use LOG_ONLY here.

Regards,
--
Ilya Kasnacheev


ср, 22 мая 2019 г. в 16:30, Kamlesh Joshi <[hidden email]>:

Hi Ilya,

 

                Looking at the logs, every time checkpoint starts due to ‘timeout’ reason not because the buffer is full. So still do we need to increase checkpointing buffer or only changing checkpointing frequency will help ?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 4:19 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

We recommend using SSD and not HDD with Ignite. Otherwise, try to increase size of Checkpoint Page Buffer - if you run out of this buffer, all activity will stop until checkpoint is finished. Maybe you also need to decrease time between checkpoints.

 

Regards,

--

Ilya Kasnacheev

 

 

вт, 7 мая 2019 г. в 13:01, Kamlesh Joshi <[hidden email]>:

Hi,

Unfortunately, I cant share the reproducer.
We are using Ignite Binary Objects for performing operations on the cluster (Get/Put). We have exposed these operations to other application (i.e. TIBCO BW services) which performs operations on the cluster.

Thanks and Regards,
Kamlesh Joshi

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Wednesday, April 17, 2019 8:18 PM
To: [hidden email]
Subject: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hi, could you share a reproducer of your problem? We are missing a lot of information here. The configuration of your nodes, cache configurations, what API you use to query and update the data.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email.
and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

Kamlesh Joshi Kamlesh Joshi
Reply | Threaded
Open this post in threaded view
|

RE: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

Hi ilya,

 

We tried with LOG_ONLY and BACKGROUND both but still behavior remains same. Any other parameter tweak which would help?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 9:14 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

If it is 'timeout' then it is likely not the reason of your issues.

 

What is your walMode? Is it LOG_ONLY? If it's not, try to use LOG_ONLY here.

 

Regards,

--

Ilya Kasnacheev

 

 

ср, 22 мая 2019 г. в 16:30, Kamlesh Joshi <[hidden email]>:

Hi Ilya,

 

                Looking at the logs, every time checkpoint starts due to ‘timeout’ reason not because the buffer is full. So still do we need to increase checkpointing buffer or only changing checkpointing frequency will help ?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 4:19 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

We recommend using SSD and not HDD with Ignite. Otherwise, try to increase size of Checkpoint Page Buffer - if you run out of this buffer, all activity will stop until checkpoint is finished. Maybe you also need to decrease time between checkpoints.

 

Regards,

--

Ilya Kasnacheev

 

 

вт, 7 мая 2019 г. в 13:01, Kamlesh Joshi <[hidden email]>:

Hi,

Unfortunately, I cant share the reproducer.
We are using Ignite Binary Objects for performing operations on the cluster (Get/Put). We have exposed these operations to other application (i.e. TIBCO BW services) which performs operations on the cluster.

Thanks and Regards,
Kamlesh Joshi

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Wednesday, April 17, 2019 8:18 PM
To: [hidden email]
Subject: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hi, could you share a reproducer of your problem? We are missing a lot of information here. The configuration of your nodes, cache configurations, what API you use to query and update the data.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email.
and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

Hello!

I recommend gathering thread dumps from cluster as it has performance issues, sharing these dumps with us.

Regards,
--
Ilya Kasnacheev


чт, 23 мая 2019 г. в 10:52, Kamlesh Joshi <[hidden email]>:

Hi ilya,

 

We tried with LOG_ONLY and BACKGROUND both but still behavior remains same. Any other parameter tweak which would help?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 9:14 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

If it is 'timeout' then it is likely not the reason of your issues.

 

What is your walMode? Is it LOG_ONLY? If it's not, try to use LOG_ONLY here.

 

Regards,

--

Ilya Kasnacheev

 

 

ср, 22 мая 2019 г. в 16:30, Kamlesh Joshi <[hidden email]>:

Hi Ilya,

 

                Looking at the logs, every time checkpoint starts due to ‘timeout’ reason not because the buffer is full. So still do we need to increase checkpointing buffer or only changing checkpointing frequency will help ?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 4:19 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

We recommend using SSD and not HDD with Ignite. Otherwise, try to increase size of Checkpoint Page Buffer - if you run out of this buffer, all activity will stop until checkpoint is finished. Maybe you also need to decrease time between checkpoints.

 

Regards,

--

Ilya Kasnacheev

 

 

вт, 7 мая 2019 г. в 13:01, Kamlesh Joshi <[hidden email]>:

Hi,

Unfortunately, I cant share the reproducer.
We are using Ignite Binary Objects for performing operations on the cluster (Get/Put). We have exposed these operations to other application (i.e. TIBCO BW services) which performs operations on the cluster.

Thanks and Regards,
Kamlesh Joshi

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Wednesday, April 17, 2019 8:18 PM
To: [hidden email]
Subject: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hi, could you share a reproducer of your problem? We are missing a lot of information here. The configuration of your nodes, cache configurations, what API you use to query and update the data.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email.
and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

Kamlesh Joshi Kamlesh Joshi
Reply | Threaded
Open this post in threaded view
|

RE: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

Hi Ilya,

 

PFA thread dumps and JFRs from the given cluster. Any observation and recommendation would be of great help!

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Thursday, May 23, 2019 7:35 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

I recommend gathering thread dumps from cluster as it has performance issues, sharing these dumps with us.

 

Regards,

--

Ilya Kasnacheev

 

 

чт, 23 мая 2019 г. в 10:52, Kamlesh Joshi <[hidden email]>:

Hi ilya,

 

We tried with LOG_ONLY and BACKGROUND both but still behavior remains same. Any other parameter tweak which would help?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 9:14 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

If it is 'timeout' then it is likely not the reason of your issues.

 

What is your walMode? Is it LOG_ONLY? If it's not, try to use LOG_ONLY here.

 

Regards,

--

Ilya Kasnacheev

 

 

ср, 22 мая 2019 г. в 16:30, Kamlesh Joshi <[hidden email]>:

Hi Ilya,

 

                Looking at the logs, every time checkpoint starts due to ‘timeout’ reason not because the buffer is full. So still do we need to increase checkpointing buffer or only changing checkpointing frequency will help ?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 4:19 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

We recommend using SSD and not HDD with Ignite. Otherwise, try to increase size of Checkpoint Page Buffer - if you run out of this buffer, all activity will stop until checkpoint is finished. Maybe you also need to decrease time between checkpoints.

 

Regards,

--

Ilya Kasnacheev

 

 

вт, 7 мая 2019 г. в 13:01, Kamlesh Joshi <[hidden email]>:

Hi,

Unfortunately, I cant share the reproducer.
We are using Ignite Binary Objects for performing operations on the cluster (Get/Put). We have exposed these operations to other application (i.e. TIBCO BW services) which performs operations on the cluster.

Thanks and Regards,
Kamlesh Joshi

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Wednesday, April 17, 2019 8:18 PM
To: [hidden email]
Subject: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hi, could you share a reproducer of your problem? We are missing a lot of information here. The configuration of your nodes, cache configurations, what API you use to query and update the data.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email.
and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


PT.Ignite.ThreadDumps&JFRs.zip (3M) Download Attachment
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

Hello!

The relevant stack trace as follows:

"sys-stripe-43-#44%EDIFCustomer%" #70 prio=5 os_prio=0 tid=0x00007fc68e274800 nid=0x7ebb runnable [0x00007fc4898d6000]
   java.lang.Thread.State: RUNNABLE
        at java.nio.MappedByteBuffer.force0(Native Method)
        at java.nio.MappedByteBuffer.force(MappedByteBuffer.java:203)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.close(FileWriteAheadLogManager.java:2798)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.access$600(FileWriteAheadLogManager.java:2438)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.rollOver(FileWriteAheadLogManager.java:1064)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.log(FileWriteAheadLogManager.java:753)
        at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.beforeReleaseWrite(PageMemoryImpl.java:1611)
        at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlockPage(PageMemoryImpl.java:1478)
        at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:449)
        at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:443)
        at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writeUnlock(PageHandler.java:377)
        at org.apache.ignite.internal.processors.cache.persistence.DataStructure.writeUnlock(DataStructure.java:210)
        at org.apache.ignite.internal.processors.cache.persistence.DataStructure.writeUnlock(DataStructure.java:181)
        at org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.put(PagesList.java:684)

It seems that syncing WAL file to disk takes prolonged period of time, which is indicative of not being able to write to disks fast enough. I think that you have saturated your storage capacity.

Maybe it will make sense to upgrade to 2.7 and disable WAL for the period of data ingestion, if you can separate it. Otherwise, look int beefing up your storage. What might also help is fewer WAL segments of larger size.

Regards,
--
Ilya Kasnacheev


пт, 24 мая 2019 г. в 09:27, Kamlesh Joshi <[hidden email]>:

Hi Ilya,

 

PFA thread dumps and JFRs from the given cluster. Any observation and recommendation would be of great help!

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Thursday, May 23, 2019 7:35 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

I recommend gathering thread dumps from cluster as it has performance issues, sharing these dumps with us.

 

Regards,

--

Ilya Kasnacheev

 

 

чт, 23 мая 2019 г. в 10:52, Kamlesh Joshi <[hidden email]>:

Hi ilya,

 

We tried with LOG_ONLY and BACKGROUND both but still behavior remains same. Any other parameter tweak which would help?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 9:14 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

If it is 'timeout' then it is likely not the reason of your issues.

 

What is your walMode? Is it LOG_ONLY? If it's not, try to use LOG_ONLY here.

 

Regards,

--

Ilya Kasnacheev

 

 

ср, 22 мая 2019 г. в 16:30, Kamlesh Joshi <[hidden email]>:

Hi Ilya,

 

                Looking at the logs, every time checkpoint starts due to ‘timeout’ reason not because the buffer is full. So still do we need to increase checkpointing buffer or only changing checkpointing frequency will help ?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 4:19 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

We recommend using SSD and not HDD with Ignite. Otherwise, try to increase size of Checkpoint Page Buffer - if you run out of this buffer, all activity will stop until checkpoint is finished. Maybe you also need to decrease time between checkpoints.

 

Regards,

--

Ilya Kasnacheev

 

 

вт, 7 мая 2019 г. в 13:01, Kamlesh Joshi <[hidden email]>:

Hi,

Unfortunately, I cant share the reproducer.
We are using Ignite Binary Objects for performing operations on the cluster (Get/Put). We have exposed these operations to other application (i.e. TIBCO BW services) which performs operations on the cluster.

Thanks and Regards,
Kamlesh Joshi

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Wednesday, April 17, 2019 8:18 PM
To: [hidden email]
Subject: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hi, could you share a reproducer of your problem? We are missing a lot of information here. The configuration of your nodes, cache configurations, what API you use to query and update the data.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email.
and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

Kamlesh Joshi Kamlesh Joshi
Reply | Threaded
Open this post in threaded view
|

RE: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

Thanks for your analysis on this Ilya.

 

Have some queries/observations on this as below:

 

  1. Disabling WAL won’t be possible as it’s the current scenario that we are having in Production system. Also, currently difficult to upgrade (may be an least wanted option).
  2. I have seen that there is still a disk space available on the drive (not actually saturated).
  3. Also, we have tried with altering WAL size to 2GB (instead of default of 64 MB) and default chunks of it i.e. 10 but still there was no visible change.
  4. Lastly, if we replace HDD with SSDs will this problem occur or it will really help ?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Monday, May 27, 2019 7:49 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

The relevant stack trace as follows:

 

"sys-stripe-43-#44%EDIFCustomer%" #70 prio=5 os_prio=0 tid=0x00007fc68e274800 nid=0x7ebb runnable [0x00007fc4898d6000]
   java.lang.Thread.State: RUNNABLE
        at java.nio.MappedByteBuffer.force0(Native Method)
        at java.nio.MappedByteBuffer.force(MappedByteBuffer.java:203)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.close(FileWriteAheadLogManager.java:2798)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.access$600(FileWriteAheadLogManager.java:2438)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.rollOver(FileWriteAheadLogManager.java:1064)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.log(FileWriteAheadLogManager.java:753)
        at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.beforeReleaseWrite(PageMemoryImpl.java:1611)
        at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlockPage(PageMemoryImpl.java:1478)
        at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:449)
        at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:443)
        at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writeUnlock(PageHandler.java:377)
        at org.apache.ignite.internal.processors.cache.persistence.DataStructure.writeUnlock(DataStructure.java:210)
        at org.apache.ignite.internal.processors.cache.persistence.DataStructure.writeUnlock(DataStructure.java:181)
        at org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.put(PagesList.java:684)

 

It seems that syncing WAL file to disk takes prolonged period of time, which is indicative of not being able to write to disks fast enough. I think that you have saturated your storage capacity.

 

Maybe it will make sense to upgrade to 2.7 and disable WAL for the period of data ingestion, if you can separate it. Otherwise, look int beefing up your storage. What might also help is fewer WAL segments of larger size.

 

Regards,

--

Ilya Kasnacheev

 

 

пт, 24 мая 2019 г. в 09:27, Kamlesh Joshi <[hidden email]>:

Hi Ilya,

 

PFA thread dumps and JFRs from the given cluster. Any observation and recommendation would be of great help!

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Thursday, May 23, 2019 7:35 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

I recommend gathering thread dumps from cluster as it has performance issues, sharing these dumps with us.

 

Regards,

--

Ilya Kasnacheev

 

 

чт, 23 мая 2019 г. в 10:52, Kamlesh Joshi <[hidden email]>:

Hi ilya,

 

We tried with LOG_ONLY and BACKGROUND both but still behavior remains same. Any other parameter tweak which would help?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 9:14 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

If it is 'timeout' then it is likely not the reason of your issues.

 

What is your walMode? Is it LOG_ONLY? If it's not, try to use LOG_ONLY here.

 

Regards,

--

Ilya Kasnacheev

 

 

ср, 22 мая 2019 г. в 16:30, Kamlesh Joshi <[hidden email]>:

Hi Ilya,

 

                Looking at the logs, every time checkpoint starts due to ‘timeout’ reason not because the buffer is full. So still do we need to increase checkpointing buffer or only changing checkpointing frequency will help ?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 4:19 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

We recommend using SSD and not HDD with Ignite. Otherwise, try to increase size of Checkpoint Page Buffer - if you run out of this buffer, all activity will stop until checkpoint is finished. Maybe you also need to decrease time between checkpoints.

 

Regards,

--

Ilya Kasnacheev

 

 

вт, 7 мая 2019 г. в 13:01, Kamlesh Joshi <[hidden email]>:

Hi,

Unfortunately, I cant share the reproducer.
We are using Ignite Binary Objects for performing operations on the cluster (Get/Put). We have exposed these operations to other application (i.e. TIBCO BW services) which performs operations on the cluster.

Thanks and Regards,
Kamlesh Joshi

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Wednesday, April 17, 2019 8:18 PM
To: [hidden email]
Subject: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hi, could you share a reproducer of your problem? We are missing a lot of information here. The configuration of your nodes, cache configurations, what API you use to query and update the data.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email.
and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

Hello!

I'm not talking about running out of disk, but about reaching maximum throughput.

Upgrading to SSL will definitely help.Even if you must hold the bulk of data on HDD, at least store your WAL on SSD.

Regards,
--
Ilya Kasnacheev


вт, 28 мая 2019 г. в 09:14, Kamlesh Joshi <[hidden email]>:

Thanks for your analysis on this Ilya.

 

Have some queries/observations on this as below:

 

  1. Disabling WAL won’t be possible as it’s the current scenario that we are having in Production system. Also, currently difficult to upgrade (may be an least wanted option).
  2. I have seen that there is still a disk space available on the drive (not actually saturated).
  3. Also, we have tried with altering WAL size to 2GB (instead of default of 64 MB) and default chunks of it i.e. 10 but still there was no visible change.
  4. Lastly, if we replace HDD with SSDs will this problem occur or it will really help ?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Monday, May 27, 2019 7:49 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

The relevant stack trace as follows:

 

"sys-stripe-43-#44%EDIFCustomer%" #70 prio=5 os_prio=0 tid=0x00007fc68e274800 nid=0x7ebb runnable [0x00007fc4898d6000]
   java.lang.Thread.State: RUNNABLE
        at java.nio.MappedByteBuffer.force0(Native Method)
        at java.nio.MappedByteBuffer.force(MappedByteBuffer.java:203)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.close(FileWriteAheadLogManager.java:2798)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.access$600(FileWriteAheadLogManager.java:2438)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.rollOver(FileWriteAheadLogManager.java:1064)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.log(FileWriteAheadLogManager.java:753)
        at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.beforeReleaseWrite(PageMemoryImpl.java:1611)
        at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlockPage(PageMemoryImpl.java:1478)
        at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:449)
        at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:443)
        at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writeUnlock(PageHandler.java:377)
        at org.apache.ignite.internal.processors.cache.persistence.DataStructure.writeUnlock(DataStructure.java:210)
        at org.apache.ignite.internal.processors.cache.persistence.DataStructure.writeUnlock(DataStructure.java:181)
        at org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.put(PagesList.java:684)

 

It seems that syncing WAL file to disk takes prolonged period of time, which is indicative of not being able to write to disks fast enough. I think that you have saturated your storage capacity.

 

Maybe it will make sense to upgrade to 2.7 and disable WAL for the period of data ingestion, if you can separate it. Otherwise, look int beefing up your storage. What might also help is fewer WAL segments of larger size.

 

Regards,

--

Ilya Kasnacheev

 

 

пт, 24 мая 2019 г. в 09:27, Kamlesh Joshi <[hidden email]>:

Hi Ilya,

 

PFA thread dumps and JFRs from the given cluster. Any observation and recommendation would be of great help!

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Thursday, May 23, 2019 7:35 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

I recommend gathering thread dumps from cluster as it has performance issues, sharing these dumps with us.

 

Regards,

--

Ilya Kasnacheev

 

 

чт, 23 мая 2019 г. в 10:52, Kamlesh Joshi <[hidden email]>:

Hi ilya,

 

We tried with LOG_ONLY and BACKGROUND both but still behavior remains same. Any other parameter tweak which would help?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 9:14 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

If it is 'timeout' then it is likely not the reason of your issues.

 

What is your walMode? Is it LOG_ONLY? If it's not, try to use LOG_ONLY here.

 

Regards,

--

Ilya Kasnacheev

 

 

ср, 22 мая 2019 г. в 16:30, Kamlesh Joshi <[hidden email]>:

Hi Ilya,

 

                Looking at the logs, every time checkpoint starts due to ‘timeout’ reason not because the buffer is full. So still do we need to increase checkpointing buffer or only changing checkpointing frequency will help ?

 

Thanks and Regards,

Kamlesh Joshi

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, May 22, 2019 4:19 PM
To: [hidden email]
Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

 

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hello!

 

We recommend using SSD and not HDD with Ignite. Otherwise, try to increase size of Checkpoint Page Buffer - if you run out of this buffer, all activity will stop until checkpoint is finished. Maybe you also need to decrease time between checkpoints.

 

Regards,

--

Ilya Kasnacheev

 

 

вт, 7 мая 2019 г. в 13:01, Kamlesh Joshi <[hidden email]>:

Hi,

Unfortunately, I cant share the reproducer.
We are using Ignite Binary Objects for performing operations on the cluster (Get/Put). We have exposed these operations to other application (i.e. TIBCO BW services) which performs operations on the cluster.

Thanks and Regards,
Kamlesh Joshi

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Wednesday, April 17, 2019 8:18 PM
To: [hidden email]
Subject: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services

The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.

Hi, could you share a reproducer of your problem? We are missing a lot of information here. The configuration of your nodes, cache configurations, what API you use to query and update the data.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email.
and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."


"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."