recoveryBallotBoxes in MvccProcessorImpl memory leak?

classic Classic list List threaded Threaded
26 messages Options
12
mvkarp mvkarp
Reply | Threaded
Open this post in threaded view
|

Re: recoveryBallotBoxes in MvccProcessorImpl memory leak?

Hi, after investigating I have few questions regarding this issue.

1. Having lack of knowledge in what MVCC coordinator is used for, are you
able to shed some light on the role and purpose of the MVCC coordinator?
What does the MVCC coordinator do, why is one selected? Should an MVCC
coordinator be selected regardless of MVCC being disabled? (i.e. is it used
for any other base features and is it just the way Ignite is meant to work)

2. Following on from this, after looking at the code of the
MvccProcessorImpl.java class in Ignite 2.7.5 Github, it looks like an MVCC
coordinator is ALWAYS selected and assigns one of the server nodes as the
MVCC coordinator, regardless of having TRANSACTIONAL_SNAPSHOT cache or not
(mvccEnabled can be false but a MVCC coordinator is still be selected).

https://github.com/apache/ignite/blob/ignite-2.7.5/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/mvcc/MvccProcessorImpl.java

On Line 861, in assignMvccCoordinator method, it loops through all nodes in
the cluster with only these two conditions.

*if (!node.isClient() && supportsMvcc(node))*

It only checks if the node is not a client, and that is supportsMvcc (which
is true for all versions > 2.7). It does not check mvccEnabled at all.


Can you confirm the above is intentional/expected or if there is another
piece of code I am missing?


3. As extra information, the node that happens to be selected as MVCC
coordinator does not get the leak. But every other client/server gets the
leak.



Ivan Pavlukhin wrote

> Hi,
>
> I suspect a following here. Some node treats itself as a MVCC
> coordinator and creates a new RecoveryBallotBox when each client node
> leaves. Some (may be all) other nodes think that MVCC is disabled and
> do not send a vote (assumed for aforementioned ballot box) to MVCC
> coordinator. Consequently a memory leak.
>
> A following could be done:
> 1. Figure out why some node treats itself MVCC coordinator and others
> think that MVCC is disabled.
> 2. Try to introduce some defensive matters in Ignite code to protect
> from the leak in a long running cluster.
>
> As a last chance workaround I can suggest writing custom code, which
> cleans recoveryBallotBoxes map from time to time (most likely using
> reflection).
>
> пн, 11 нояб. 2019 г. в 08:53, mvkarp <

> liquid_ninja2k@

> >:
>>
>> We have frequently stopping and starting clients in short lived client
>> JVM
>> processes as required for our purposes, this seems to lead to a huge
>> bunch
>> of PME (but no rebalancing) and topology changes (topVer=300,000+)
>>
>> Still can not figure out why this map won't clear (there are no
>> exceptions
>> or err at all in the entire log)
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
> --
> Best regards,
> Ivan Pavlukhin





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ivan Pavlukhin Ivan Pavlukhin
Reply | Threaded
Open this post in threaded view
|

Re: recoveryBallotBoxes in MvccProcessorImpl memory leak?

1. MVCC coordinator should not do anything when there is no MVCC
caches, actually it should not be active in such case. Basically, MVCC
coordinator is needed to have a consistent order between transactions.
2. In 2.7.5 "assigned coordinator" is always selected but it does not
mean that it is active. MvccProcessorImpl.curCrd variable corresponds
to active MVCC coordinator.
3. If that statement is true, then it should be rather easy to
reproduce the problem by starting and stopping client nodes
frequently. recoveryBallotBoxes was not assumed to be populated on
nodes other than MVCC coordinator. If it happens than we found a bug.
Actually, the code in master is different and has an explicit check
that recoveryBallotBoxes are populated only on MVCC coordinator.

чт, 14 нояб. 2019 г. в 15:42, mvkarp <[hidden email]>:

>
> Hi, after investigating I have few questions regarding this issue.
>
> 1. Having lack of knowledge in what MVCC coordinator is used for, are you
> able to shed some light on the role and purpose of the MVCC coordinator?
> What does the MVCC coordinator do, why is one selected? Should an MVCC
> coordinator be selected regardless of MVCC being disabled? (i.e. is it used
> for any other base features and is it just the way Ignite is meant to work)
>
> 2. Following on from this, after looking at the code of the
> MvccProcessorImpl.java class in Ignite 2.7.5 Github, it looks like an MVCC
> coordinator is ALWAYS selected and assigns one of the server nodes as the
> MVCC coordinator, regardless of having TRANSACTIONAL_SNAPSHOT cache or not
> (mvccEnabled can be false but a MVCC coordinator is still be selected).
>
> https://github.com/apache/ignite/blob/ignite-2.7.5/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/mvcc/MvccProcessorImpl.java
>
> On Line 861, in assignMvccCoordinator method, it loops through all nodes in
> the cluster with only these two conditions.
>
> *if (!node.isClient() && supportsMvcc(node))*
>
> It only checks if the node is not a client, and that is supportsMvcc (which
> is true for all versions > 2.7). It does not check mvccEnabled at all.
>
>
> Can you confirm the above is intentional/expected or if there is another
> piece of code I am missing?
>
>
> 3. As extra information, the node that happens to be selected as MVCC
> coordinator does not get the leak. But every other client/server gets the
> leak.
>
>
>
> Ivan Pavlukhin wrote
> > Hi,
> >
> > I suspect a following here. Some node treats itself as a MVCC
> > coordinator and creates a new RecoveryBallotBox when each client node
> > leaves. Some (may be all) other nodes think that MVCC is disabled and
> > do not send a vote (assumed for aforementioned ballot box) to MVCC
> > coordinator. Consequently a memory leak.
> >
> > A following could be done:
> > 1. Figure out why some node treats itself MVCC coordinator and others
> > think that MVCC is disabled.
> > 2. Try to introduce some defensive matters in Ignite code to protect
> > from the leak in a long running cluster.
> >
> > As a last chance workaround I can suggest writing custom code, which
> > cleans recoveryBallotBoxes map from time to time (most likely using
> > reflection).
> >
> > пн, 11 нояб. 2019 г. в 08:53, mvkarp &lt;
>
> > liquid_ninja2k@
>
> > &gt;:
> >>
> >> We have frequently stopping and starting clients in short lived client
> >> JVM
> >> processes as required for our purposes, this seems to lead to a huge
> >> bunch
> >> of PME (but no rebalancing) and topology changes (topVer=300,000+)
> >>
> >> Still can not figure out why this map won't clear (there are no
> >> exceptions
> >> or err at all in the entire log)
> >>
> >>
> >>
> >> --
> >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/



--
Best regards,
Ivan Pavlukhin
mvkarp mvkarp
Reply | Threaded
Open this post in threaded view
|

Re: recoveryBallotBoxes in MvccProcessorImpl memory leak?

1 & 2. Actually, looking at latest Master on the release of 2.7.5 and the
current Master version, it is 'pickMvccCoordinator' function which returns
the coordinator (this is same function that selects node that is not Client
and Ignite version >= 2.7). curCrd is then assigned the return variable of
pickMvccCoordinator, which becomes the active Mvcc coordinator. So looks
like it does become active, but not sure the effect of that yet.

3. Assuming it is then active, looks like there are two entry points into
recoveryBallotBoxes. Through  onDiscovery() and via
processRecoveryFinishedMessage().

Is it possible that onDiscovery() does not populate recoveryBallotBoxes as
there is curCrd0.local() check - so processing will only be done if MVCC
coordinator is local - thus a node that is actually a MVCC coordinator will
clear out the recoveryBallotBoxes (which is the explicit check that you
mentioned).

But the population might be through processRecoveryFinishedMessage() - which
does not do any check for isLocal() and goes straight to processing message
since waitForCoordinatorInit() on a MvccRecoveryFinishedMessage always
returns false?


Ivan Pavlukhin wrote

> 1. MVCC coordinator should not do anything when there is no MVCC
> caches, actually it should not be active in such case. Basically, MVCC
> coordinator is needed to have a consistent order between transactions.
> 2. In 2.7.5 "assigned coordinator" is always selected but it does not
> mean that it is active. MvccProcessorImpl.curCrd variable corresponds
> to active MVCC coordinator.
> 3. If that statement is true, then it should be rather easy to
> reproduce the problem by starting and stopping client nodes
> frequently. recoveryBallotBoxes was not assumed to be populated on
> nodes other than MVCC coordinator. If it happens than we found a bug.
> Actually, the code in master is different and has an explicit check
> that recoveryBallotBoxes are populated only on MVCC coordinator.
>
> чт, 14 нояб. 2019 г. в 15:42, mvkarp &lt;

> liquid_ninja2k@

> &gt;:
>>
>> Hi, after investigating I have few questions regarding this issue.
>>
>> 1. Having lack of knowledge in what MVCC coordinator is used for, are you
>> able to shed some light on the role and purpose of the MVCC coordinator?
>> What does the MVCC coordinator do, why is one selected? Should an MVCC
>> coordinator be selected regardless of MVCC being disabled? (i.e. is it
>> used
>> for any other base features and is it just the way Ignite is meant to
>> work)
>>
>> 2. Following on from this, after looking at the code of the
>> MvccProcessorImpl.java class in Ignite 2.7.5 Github, it looks like an
>> MVCC
>> coordinator is ALWAYS selected and assigns one of the server nodes as the
>> MVCC coordinator, regardless of having TRANSACTIONAL_SNAPSHOT cache or
>> not
>> (mvccEnabled can be false but a MVCC coordinator is still be selected).
>>
>> https://github.com/apache/ignite/blob/ignite-2.7.5/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/mvcc/MvccProcessorImpl.java
>>
>> On Line 861, in assignMvccCoordinator method, it loops through all nodes
>> in
>> the cluster with only these two conditions.
>>
>> *if (!node.isClient() && supportsMvcc(node))*
>>
>> It only checks if the node is not a client, and that is supportsMvcc
>> (which
>> is true for all versions > 2.7). It does not check mvccEnabled at all.
>>
>>
>> Can you confirm the above is intentional/expected or if there is another
>> piece of code I am missing?
>>
>>
>> 3. As extra information, the node that happens to be selected as MVCC
>> coordinator does not get the leak. But every other client/server gets the
>> leak.
>>
>>
>>
>> Ivan Pavlukhin wrote
>> > Hi,
>> >
>> > I suspect a following here. Some node treats itself as a MVCC
>> > coordinator and creates a new RecoveryBallotBox when each client node
>> > leaves. Some (may be all) other nodes think that MVCC is disabled and
>> > do not send a vote (assumed for aforementioned ballot box) to MVCC
>> > coordinator. Consequently a memory leak.
>> >
>> > A following could be done:
>> > 1. Figure out why some node treats itself MVCC coordinator and others
>> > think that MVCC is disabled.
>> > 2. Try to introduce some defensive matters in Ignite code to protect
>> > from the leak in a long running cluster.
>> >
>> > As a last chance workaround I can suggest writing custom code, which
>> > cleans recoveryBallotBoxes map from time to time (most likely using
>> > reflection).
>> >
>> > пн, 11 нояб. 2019 г. в 08:53, mvkarp &lt;
>>
>> > liquid_ninja2k@
>>
>> > &gt;:
>> >>
>> >> We have frequently stopping and starting clients in short lived client
>> >> JVM
>> >> processes as required for our purposes, this seems to lead to a huge
>> >> bunch
>> >> of PME (but no rebalancing) and topology changes (topVer=300,000+)
>> >>
>> >> Still can not figure out why this map won't clear (there are no
>> >> exceptions
>> >> or err at all in the entire log)
>> >>
>> >>
>> >>
>> >> --
>> >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>> >
>> >
>> >
>> > --
>> > Best regards,
>> > Ivan Pavlukhin
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
> --
> Best regards,
> Ivan Pavlukhin





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ivan Pavlukhin Ivan Pavlukhin
Reply | Threaded
Open this post in threaded view
|

Re: recoveryBallotBoxes in MvccProcessorImpl memory leak?

> But the population might be through processRecoveryFinishedMessage() - which does not do any check for isLocal() and goes straight to processing message since waitForCoordinatorInit() on a MvccRecoveryFinishedMessage always returns false?

All nodes sends messages intended for handling in
processRecoveryFinishedMessage() to MVCC coordinator node. So, we
assume that only coordinator receives such messages. If you are
interested in, you can find other side code in
IgniteTxManager.NodeFailureTimeoutObject, "recovery finished" messages
are issued there.

But currently I suspect that you faced a leak in
MvccProcessorImpl.onDiscovery on non MVCC coordinator nodes. Do you
think that there is other reason in you case?

сб, 16 нояб. 2019 г. в 12:30, mvkarp <[hidden email]>:

>
> 1 & 2. Actually, looking at latest Master on the release of 2.7.5 and the
> current Master version, it is 'pickMvccCoordinator' function which returns
> the coordinator (this is same function that selects node that is not Client
> and Ignite version >= 2.7). curCrd is then assigned the return variable of
> pickMvccCoordinator, which becomes the active Mvcc coordinator. So looks
> like it does become active, but not sure the effect of that yet.
>
> 3. Assuming it is then active, looks like there are two entry points into
> recoveryBallotBoxes. Through  onDiscovery() and via
> processRecoveryFinishedMessage().
>
> Is it possible that onDiscovery() does not populate recoveryBallotBoxes as
> there is curCrd0.local() check - so processing will only be done if MVCC
> coordinator is local - thus a node that is actually a MVCC coordinator will
> clear out the recoveryBallotBoxes (which is the explicit check that you
> mentioned).
>
> But the population might be through processRecoveryFinishedMessage() - which
> does not do any check for isLocal() and goes straight to processing message
> since waitForCoordinatorInit() on a MvccRecoveryFinishedMessage always
> returns false?
>
>
> Ivan Pavlukhin wrote
> > 1. MVCC coordinator should not do anything when there is no MVCC
> > caches, actually it should not be active in such case. Basically, MVCC
> > coordinator is needed to have a consistent order between transactions.
> > 2. In 2.7.5 "assigned coordinator" is always selected but it does not
> > mean that it is active. MvccProcessorImpl.curCrd variable corresponds
> > to active MVCC coordinator.
> > 3. If that statement is true, then it should be rather easy to
> > reproduce the problem by starting and stopping client nodes
> > frequently. recoveryBallotBoxes was not assumed to be populated on
> > nodes other than MVCC coordinator. If it happens than we found a bug.
> > Actually, the code in master is different and has an explicit check
> > that recoveryBallotBoxes are populated only on MVCC coordinator.
> >
> > чт, 14 нояб. 2019 г. в 15:42, mvkarp &lt;
>
> > liquid_ninja2k@
>
> > &gt;:
> >>
> >> Hi, after investigating I have few questions regarding this issue.
> >>
> >> 1. Having lack of knowledge in what MVCC coordinator is used for, are you
> >> able to shed some light on the role and purpose of the MVCC coordinator?
> >> What does the MVCC coordinator do, why is one selected? Should an MVCC
> >> coordinator be selected regardless of MVCC being disabled? (i.e. is it
> >> used
> >> for any other base features and is it just the way Ignite is meant to
> >> work)
> >>
> >> 2. Following on from this, after looking at the code of the
> >> MvccProcessorImpl.java class in Ignite 2.7.5 Github, it looks like an
> >> MVCC
> >> coordinator is ALWAYS selected and assigns one of the server nodes as the
> >> MVCC coordinator, regardless of having TRANSACTIONAL_SNAPSHOT cache or
> >> not
> >> (mvccEnabled can be false but a MVCC coordinator is still be selected).
> >>
> >> https://github.com/apache/ignite/blob/ignite-2.7.5/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/mvcc/MvccProcessorImpl.java
> >>
> >> On Line 861, in assignMvccCoordinator method, it loops through all nodes
> >> in
> >> the cluster with only these two conditions.
> >>
> >> *if (!node.isClient() && supportsMvcc(node))*
> >>
> >> It only checks if the node is not a client, and that is supportsMvcc
> >> (which
> >> is true for all versions > 2.7). It does not check mvccEnabled at all.
> >>
> >>
> >> Can you confirm the above is intentional/expected or if there is another
> >> piece of code I am missing?
> >>
> >>
> >> 3. As extra information, the node that happens to be selected as MVCC
> >> coordinator does not get the leak. But every other client/server gets the
> >> leak.
> >>
> >>
> >>
> >> Ivan Pavlukhin wrote
> >> > Hi,
> >> >
> >> > I suspect a following here. Some node treats itself as a MVCC
> >> > coordinator and creates a new RecoveryBallotBox when each client node
> >> > leaves. Some (may be all) other nodes think that MVCC is disabled and
> >> > do not send a vote (assumed for aforementioned ballot box) to MVCC
> >> > coordinator. Consequently a memory leak.
> >> >
> >> > A following could be done:
> >> > 1. Figure out why some node treats itself MVCC coordinator and others
> >> > think that MVCC is disabled.
> >> > 2. Try to introduce some defensive matters in Ignite code to protect
> >> > from the leak in a long running cluster.
> >> >
> >> > As a last chance workaround I can suggest writing custom code, which
> >> > cleans recoveryBallotBoxes map from time to time (most likely using
> >> > reflection).
> >> >
> >> > пн, 11 нояб. 2019 г. в 08:53, mvkarp &lt;
> >>
> >> > liquid_ninja2k@
> >>
> >> > &gt;:
> >> >>
> >> >> We have frequently stopping and starting clients in short lived client
> >> >> JVM
> >> >> processes as required for our purposes, this seems to lead to a huge
> >> >> bunch
> >> >> of PME (but no rebalancing) and topology changes (topVer=300,000+)
> >> >>
> >> >> Still can not figure out why this map won't clear (there are no
> >> >> exceptions
> >> >> or err at all in the entire log)
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> > Ivan Pavlukhin
> >>
> >>
> >>
> >>
> >>
> >> --
> >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/



--
Best regards,
Ivan Pavlukhin
mvkarp mvkarp
Reply | Threaded
Open this post in threaded view
|

Re: recoveryBallotBoxes in MvccProcessorImpl memory leak?

Only other thing I can think of if it's through onDiscovery() is that
curCrd.local() somehow is returning true. However I am unable to find
exactly how local() is determined since there appears to be a big chain.

I know that the node uuid on the leaking server is on a different physical
node as well as has a completely different node ID
(bbbbb-bbbbb-bbbbb-bbbbb-bbbbb-bbbbb) to what the MVCC coordinator is
(mvccCrd=aaaaa-aaaa-aaaaa-aaaaa-aaaaa)

Is there any way that the curCrd.local() could be returning True on the
leaking server JVM? I am trying to investigate how local() is determined and
what could cause it to be true.


Ivan Pavlukhin wrote
> But currently I suspect that you faced a leak in
> MvccProcessorImpl.onDiscovery on non MVCC coordinator nodes. Do you
> think that there is other reason in you case?





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ivan Pavlukhin Ivan Pavlukhin
Reply | Threaded
Open this post in threaded view
|

Re: recoveryBallotBoxes in MvccProcessorImpl memory leak?

Hi,

Thank you for digging deeper! No good ideas about problems in curCrd.local().

Have you tried to reproduce a leak with starting/stopping huge amount
of client nodes?

вс, 17 нояб. 2019 г. в 13:16, mvkarp <[hidden email]>:

>
> Only other thing I can think of if it's through onDiscovery() is that
> curCrd.local() somehow is returning true. However I am unable to find
> exactly how local() is determined since there appears to be a big chain.
>
> I know that the node uuid on the leaking server is on a different physical
> node as well as has a completely different node ID
> (bbbbb-bbbbb-bbbbb-bbbbb-bbbbb-bbbbb) to what the MVCC coordinator is
> (mvccCrd=aaaaa-aaaa-aaaaa-aaaaa-aaaaa)
>
> Is there any way that the curCrd.local() could be returning True on the
> leaking server JVM? I am trying to investigate how local() is determined and
> what could cause it to be true.
>
>
> Ivan Pavlukhin wrote
> > But currently I suspect that you faced a leak in
> > MvccProcessorImpl.onDiscovery on non MVCC coordinator nodes. Do you
> > think that there is other reason in you case?
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/



--
Best regards,
Ivan Pavlukhin
12