Group eviction in progress for a very long time

classic Classic list List threaded Threaded
5 messages Options
18624049226 18624049226
Reply | Threaded
Open this post in threaded view
|

Group eviction in progress for a very long time

Hi community,

Two nodes are added to a running cluster with persistence enabled, after several days, it seems that the rebalancing is not over. The log output the following information about every 2 minutes:

why?How to end the rebalancing?

[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Eviction in progress [permits=0, threads=4, groups=4, remainingPartsToEvict=293]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812, remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728, remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO, grpId=64322847, remainingPartsToEvict=114, partsEvictInProgress=6, totalParts=544]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=CO_CUST, grpId=1684722246, remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Partitions have been scheduled for eviction: [grpId=64322847, grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601, 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
[2020-04-30T21:58:32,288][INFO ][grid-timeout-worker-#39][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9ebed61d, uptime=4 days, 19:34:36.448]
^-- H/N/C [hosts=5, nodes=5, CPUs=80]
^-- CPU [cur=18%, avg=18.24%, GC=0%]
^-- PageMemory [pages=36986314]
^-- Heap [used=12815MB, free=37.43%, comm=20480MB]
^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=146170MB, free=28.63%, comm=204800MB]
^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=134048MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=134047MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=16, idle=0, qSize=3]
^-- Striped thread pool [active=0, idle=16, qSize=0]
[2020-04-30T21:58:49,329][INFO ][db-checkpoint-thread-#409][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modified) [checkpointBeforeLockTime=13ms, checkpointLockWait=0ms, checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=7ms, reason='timeout']
[2020-04-30T21:59:32,297][INFO ][grid-timeout-worker-#39][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9ebed61d, uptime=4 days, 19:35:36.457]
^-- H/N/C [hosts=5, nodes=5, CPUs=80]
^-- CPU [cur=17.9%, avg=18.24%, GC=0%]
^-- PageMemory [pages=36986314]
^-- Heap [used=8335MB, free=59.3%, comm=20480MB]
^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=146170MB, free=28.63%, comm=204800MB]
^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=134048MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=134047MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=6, idle=10, qSize=0]
^-- Striped thread pool [active=0, idle=16, qSize=0]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Eviction in progress [permits=1, threads=4, groups=4, remainingPartsToEvict=291]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812, remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728, remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO, grpId=64322847, remainingPartsToEvict=112, partsEvictInProgress=6, totalParts=544]
[2020-04-30T21:59:49,899][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=CO_CUST, grpId=1684722246, remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
[2020-04-30T21:59:49,899][INFO ][sys-#28325][PartitionsEvictManager] Partitions have been scheduled for eviction: [grpId=64322847, grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601, 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
[2020-04-30T22:00:32,302][INFO ][grid-timeout-worker-#39][IgniteKernal]


ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Group eviction in progress for a very long time

Hello!

It's hard to say why rebalancing is not over yet, judging from this log portion alone, but "Group eviction in progress" would probably mean that some partitions were successfully rebalanced and they may now be removed from "old" backup node.

Are you getting the same #'s of partitions every time, or do they change? Any irregularities in your logs other than this one?

Regards,
--
Ilya Kasnacheev


вс, 3 мая 2020 г. в 04:49, 18624049226 <[hidden email]>:

Hi community,

Two nodes are added to a running cluster with persistence enabled, after several days, it seems that the rebalancing is not over. The log output the following information about every 2 minutes:

why?How to end the rebalancing?

[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Eviction in progress [permits=0, threads=4, groups=4, remainingPartsToEvict=293]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812, remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728, remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO, grpId=64322847, remainingPartsToEvict=114, partsEvictInProgress=6, totalParts=544]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=CO_CUST, grpId=1684722246, remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Partitions have been scheduled for eviction: [grpId=64322847, grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601, 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
[2020-04-30T21:58:32,288][INFO ][grid-timeout-worker-#39][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9ebed61d, uptime=4 days, 19:34:36.448]
^-- H/N/C [hosts=5, nodes=5, CPUs=80]
^-- CPU [cur=18%, avg=18.24%, GC=0%]
^-- PageMemory [pages=36986314]
^-- Heap [used=12815MB, free=37.43%, comm=20480MB]
^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=146170MB, free=28.63%, comm=204800MB]
^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=134048MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=134047MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=16, idle=0, qSize=3]
^-- Striped thread pool [active=0, idle=16, qSize=0]
[2020-04-30T21:58:49,329][INFO ][db-checkpoint-thread-#409][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modified) [checkpointBeforeLockTime=13ms, checkpointLockWait=0ms, checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=7ms, reason='timeout']
[2020-04-30T21:59:32,297][INFO ][grid-timeout-worker-#39][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9ebed61d, uptime=4 days, 19:35:36.457]
^-- H/N/C [hosts=5, nodes=5, CPUs=80]
^-- CPU [cur=17.9%, avg=18.24%, GC=0%]
^-- PageMemory [pages=36986314]
^-- Heap [used=8335MB, free=59.3%, comm=20480MB]
^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=146170MB, free=28.63%, comm=204800MB]
^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=134048MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=134047MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=6, idle=10, qSize=0]
^-- Striped thread pool [active=0, idle=16, qSize=0]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Eviction in progress [permits=1, threads=4, groups=4, remainingPartsToEvict=291]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812, remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728, remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO, grpId=64322847, remainingPartsToEvict=112, partsEvictInProgress=6, totalParts=544]
[2020-04-30T21:59:49,899][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=CO_CUST, grpId=1684722246, remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
[2020-04-30T21:59:49,899][INFO ][sys-#28325][PartitionsEvictManager] Partitions have been scheduled for eviction: [grpId=64322847, grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601, 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
[2020-04-30T22:00:32,302][INFO ][grid-timeout-worker-#39][IgniteKernal]


18624049226 18624049226
Reply | Threaded
Open this post in threaded view
|

Re: Group eviction in progress for a very long time

Hi Ilya,

It's hard to say why rebalancing is not over yet, judging from this log portion alone, but "Group eviction in progress" would probably mean that some partitions were successfully rebalanced and they may now be removed from "old" backup node.
That's how I understand it,According to the visor, there are more backups than the primary ones.

Are you getting the same #'s of partitions every time, or do they change? Any irregularities in your logs other than this one?

At present, the cluster is in the state of no access, so the current log scrolls out the following information about every 2 seconds, basically the same.

We have 5 nodes, 2 of which have the following logs.



Regards,
--
Ilya Kasnacheev


вс, 3 мая 2020 г. в 04:49, 18624049226 <[hidden email]>:

Hi community,

Two nodes are added to a running cluster with persistence enabled, after several days, it seems that the rebalancing is not over. The log output the following information about every 2 minutes:

why?How to end the rebalancing?

[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Eviction in progress [permits=0, threads=4, groups=4, remainingPartsToEvict=293]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812, remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728, remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO, grpId=64322847, remainingPartsToEvict=114, partsEvictInProgress=6, totalParts=544]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=CO_CUST, grpId=1684722246, remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Partitions have been scheduled for eviction: [grpId=64322847, grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601, 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
[2020-04-30T21:58:32,288][INFO ][grid-timeout-worker-#39][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9ebed61d, uptime=4 days, 19:34:36.448]
^-- H/N/C [hosts=5, nodes=5, CPUs=80]
^-- CPU [cur=18%, avg=18.24%, GC=0%]
^-- PageMemory [pages=36986314]
^-- Heap [used=12815MB, free=37.43%, comm=20480MB]
^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=146170MB, free=28.63%, comm=204800MB]
^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=134048MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=134047MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=16, idle=0, qSize=3]
^-- Striped thread pool [active=0, idle=16, qSize=0]
[2020-04-30T21:58:49,329][INFO ][db-checkpoint-thread-#409][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modified) [checkpointBeforeLockTime=13ms, checkpointLockWait=0ms, checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=7ms, reason='timeout']
[2020-04-30T21:59:32,297][INFO ][grid-timeout-worker-#39][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9ebed61d, uptime=4 days, 19:35:36.457]
^-- H/N/C [hosts=5, nodes=5, CPUs=80]
^-- CPU [cur=17.9%, avg=18.24%, GC=0%]
^-- PageMemory [pages=36986314]
^-- Heap [used=8335MB, free=59.3%, comm=20480MB]
^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=146170MB, free=28.63%, comm=204800MB]
^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=134048MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=134047MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=6, idle=10, qSize=0]
^-- Striped thread pool [active=0, idle=16, qSize=0]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Eviction in progress [permits=1, threads=4, groups=4, remainingPartsToEvict=291]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812, remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728, remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO, grpId=64322847, remainingPartsToEvict=112, partsEvictInProgress=6, totalParts=544]
[2020-04-30T21:59:49,899][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=CO_CUST, grpId=1684722246, remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
[2020-04-30T21:59:49,899][INFO ][sys-#28325][PartitionsEvictManager] Partitions have been scheduled for eviction: [grpId=64322847, grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601, 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
[2020-04-30T22:00:32,302][INFO ][grid-timeout-worker-#39][IgniteKernal]


ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Group eviction in progress for a very long time

Hello!

Can you search the previous logs for any exceptions coming from Ignite internals? Maybe something did indeed break down.

Regards,
--
Ilya Kasnacheev


пн, 4 мая 2020 г. в 16:22, 18624049226 <[hidden email]>:

Hi Ilya,

It's hard to say why rebalancing is not over yet, judging from this log portion alone, but "Group eviction in progress" would probably mean that some partitions were successfully rebalanced and they may now be removed from "old" backup node.
That's how I understand it,According to the visor, there are more backups than the primary ones.

Are you getting the same #'s of partitions every time, or do they change? Any irregularities in your logs other than this one?

At present, the cluster is in the state of no access, so the current log scrolls out the following information about every 2 seconds, basically the same.

We have 5 nodes, 2 of which have the following logs.



Regards,
--
Ilya Kasnacheev


вс, 3 мая 2020 г. в 04:49, 18624049226 <[hidden email]>:

Hi community,

Two nodes are added to a running cluster with persistence enabled, after several days, it seems that the rebalancing is not over. The log output the following information about every 2 minutes:

why?How to end the rebalancing?

[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Eviction in progress [permits=0, threads=4, groups=4, remainingPartsToEvict=293]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812, remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728, remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO, grpId=64322847, remainingPartsToEvict=114, partsEvictInProgress=6, totalParts=544]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Group eviction in progress [grpName=CO_CUST, grpId=1684722246, remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
[2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager] Partitions have been scheduled for eviction: [grpId=64322847, grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601, 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
[2020-04-30T21:58:32,288][INFO ][grid-timeout-worker-#39][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9ebed61d, uptime=4 days, 19:34:36.448]
^-- H/N/C [hosts=5, nodes=5, CPUs=80]
^-- CPU [cur=18%, avg=18.24%, GC=0%]
^-- PageMemory [pages=36986314]
^-- Heap [used=12815MB, free=37.43%, comm=20480MB]
^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=146170MB, free=28.63%, comm=204800MB]
^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=134048MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=134047MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=16, idle=0, qSize=3]
^-- Striped thread pool [active=0, idle=16, qSize=0]
[2020-04-30T21:58:49,329][INFO ][db-checkpoint-thread-#409][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modified) [checkpointBeforeLockTime=13ms, checkpointLockWait=0ms, checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=7ms, reason='timeout']
[2020-04-30T21:59:32,297][INFO ][grid-timeout-worker-#39][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9ebed61d, uptime=4 days, 19:35:36.457]
^-- H/N/C [hosts=5, nodes=5, CPUs=80]
^-- CPU [cur=17.9%, avg=18.24%, GC=0%]
^-- PageMemory [pages=36986314]
^-- Heap [used=8335MB, free=59.3%, comm=20480MB]
^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=146170MB, free=28.63%, comm=204800MB]
^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=134048MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=134047MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=6, idle=10, qSize=0]
^-- Striped thread pool [active=0, idle=16, qSize=0]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Eviction in progress [permits=1, threads=4, groups=4, remainingPartsToEvict=291]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812, remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728, remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
[2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=CO_CO, grpId=64322847, remainingPartsToEvict=112, partsEvictInProgress=6, totalParts=544]
[2020-04-30T21:59:49,899][INFO ][sys-#28325][PartitionsEvictManager] Group eviction in progress [grpName=CO_CUST, grpId=1684722246, remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
[2020-04-30T21:59:49,899][INFO ][sys-#28325][PartitionsEvictManager] Partitions have been scheduled for eviction: [grpId=64322847, grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601, 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
[2020-04-30T22:00:32,302][INFO ][grid-timeout-worker-#39][IgniteKernal]


henrique.arroyo henrique.arroyo
Reply | Threaded
Open this post in threaded view
|

Re: Group eviction in progress for a very long time

In reply to this post by 18624049226
This is related to the following bug
https://issues.apache.org/jira/browse/IGNITE-11974

We are currently running version 2.8.1 and it is affecting our servers as
well, the results are disastrous.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/