ClusterLocalNodeMetricsMXBean Stops Reporting Data

classic Classic list List threaded Threaded
4 messages Options
torjt torjt
Reply | Threaded
Open this post in threaded view
|

ClusterLocalNodeMetricsMXBean Stops Reporting Data

This post was updated on .
Production Setup:
Ignite Version: 2.7.0
JVM Info: Java(TM) SE Runtime Environment (build 1.8.0_77-b31);
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b31, mixed mode)
OS: Centos Linux 7 (3.10 Kernel)
3 server nodes in Ignite cluster
3 client nodes

Issue Observed:
Monitoring of ClusterLocalNodeMetricsMXBean properties getReceivedMessagesCount and
getSentMessagesCount no longer reports data.

Problem Overview:
* 2 of the 3 nodes stop returning data for the 2 metrics
* The nodes appear healthy and are processing as expected.  Specifically,
network traffic remains consistent
* Zabbix is used to monitor these metrics

Question:
If a server node stops reporting data, is it assumed to be unhealthy?  Is
this a known issue?  I've searched but couldn't find any related issues.

Thanks for your help.
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: ClusterMetricsMXBean Stops Reporting Data

Hello!

What do you mean by "not reporting data"? Do you get zeros/nulls? Does the operation hang? Maybe there are exceptions?

I remember there was the following issue about metrics: https://issues.apache.org/jira/browse/IGNITE-11352
But it's hard to trigger. Maybe, if you're using Zk discovery.

Regards,
--
Ilya Kasnacheev


ср, 5 июн. 2019 г. в 23:32, Jeremy Torres <[hidden email]>:
Production Setup:
Ignite Version: 2.7.0
JVM Info: Java(TM) SE Runtime Environment (build 1.8.0_77-b31);
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b31, mixed mode)
OS: Centos Linux 7 (3.10 Kernel)
3 servers nodes in Ignite cluster
3 client nodes

Issue Observed:
Monitoring of ClusterMetricsMXBean properties getReceivedMessagesCount and getSentMessagesCount no longer reports data.

Problem Overview:
* 2 of the 3 nodes stop returning data for the 2 metrics
* The nodes appear healthy and are processing as expected.  Specifically, network traffic remains consistent
* Zabbix is used to monitor these metrics

Question:
If a server node stops reporting data, is it assumed to be unhealthy?  Is this a known issue?  I've searched but couldn't find any related issues.

Thanks for your help.
torjt torjt
Reply | Threaded
Open this post in threaded view
|

Re: ClusterMetricsMXBean Stops Reporting Data

Hello.  Thanks for the response.  The operation returns with 0 values.  There
is no Zookeeper being used in this setup.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: ClusterMetricsMXBean Stops Reporting Data

Hello!

This is unexpected. Is it possible to create a reproducer project to reproduce this behavior? Can you share it?

Regards,
--
Ilya Kasnacheev


пт, 7 июн. 2019 г. в 18:37, torjt <[hidden email]>:
Hello.  Thanks for the response.  The operation returns with 0 values.  There
is no Zookeeper being used in this setup.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/