Quantcast

Node not starting and waiting for ever

classic Classic list List threaded Threaded
2 messages Options
Ranjit Sahu Ranjit Sahu
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Node not starting and waiting for ever

Hi,

We are trying to start the Ignite node on spark worker node. When i try to start 10 nodes, few starts and few not and its in hung state. 

The log shows non heap free -1% and the log is below. Any clue whats happening and how to fix this ?
17/02/16 13:04:54 INFO IgniteKernal%WCA: 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=a5f2ef72, name=WCA, uptime=00:12:00:068]
    ^-- H/N/C [hosts=8, nodes=8, CPUs=336]
    ^-- CPU [cur=0.03%, avg=0.1%, GC=0%]
    ^-- Heap [used=687MB, free=94.96%, comm=2609MB]
    ^-- Non heap [used=137MB, free=-1%, comm=139MB]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=0, qSize=0]
    ^-- Outbound messages queue [size=0]
17/02/16 13:05:54 INFO IgniteKernal%WCA: 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=a5f2ef72, name=WCA, uptime=00:13:00:077]
    ^-- H/N/C [hosts=8, nodes=8, CPUs=336]
    ^-- CPU [cur=0.03%, avg=0.1%, GC=0%]
    ^-- Heap [used=699MB, free=94.87%, comm=2609MB]
    ^-- Non heap [used=137MB, free=-1%, comm=139MB]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=0, qSize=0]
    ^-- Outbound messages queue [size=0]
17/02/16 13:06:54 INFO IgniteKernal%WCA: 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=a5f2ef72, name=WCA, uptime=00:14:00:085]
    ^-- H/N/C [hosts=8, nodes=8, CPUs=336]
    ^-- CPU [cur=0.03%, avg=0.09%, GC=0%]
    ^-- Heap [used=713MB, free=94.78%, comm=2609MB]
    ^-- Non heap [used=137MB, free=-1%, comm=139MB]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=0, qSize=0]
    ^-- Outbound messages queue [size=0]

Thanks,
Ranjit
Vladislav Pyatkov Vladislav Pyatkov
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Node not starting and waiting for ever

Hi,

The metrics are appear with specific frequency[1], and it does not mean the node a hung. This is normal behavior.

About the row:

^-- Non heap [used=137MB, free=-1%, comm=139MB]

If the JVM non-heap memory size is unlimited, the free size is showing "-1".

Why are you think this nodes did not started?


On Thu, Feb 16, 2017 at 10:17 PM, Ranjit Sahu <[hidden email]> wrote:
Hi,

We are trying to start the Ignite node on spark worker node. When i try to start 10 nodes, few starts and few not and its in hung state. 

The log shows non heap free -1% and the log is below. Any clue whats happening and how to fix this ?
17/02/16 13:04:54 INFO IgniteKernal%WCA: 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=a5f2ef72, name=WCA, uptime=00:12:00:068]
    ^-- H/N/C [hosts=8, nodes=8, CPUs=336]
    ^-- CPU [cur=0.03%, avg=0.1%, GC=0%]
    ^-- Heap [used=687MB, free=94.96%, comm=2609MB]
    ^-- Non heap [used=137MB, free=-1%, comm=139MB]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=0, qSize=0]
    ^-- Outbound messages queue [size=0]
17/02/16 13:05:54 INFO IgniteKernal%WCA: 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=a5f2ef72, name=WCA, uptime=00:13:00:077]
    ^-- H/N/C [hosts=8, nodes=8, CPUs=336]
    ^-- CPU [cur=0.03%, avg=0.1%, GC=0%]
    ^-- Heap [used=699MB, free=94.87%, comm=2609MB]
    ^-- Non heap [used=137MB, free=-1%, comm=139MB]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=0, qSize=0]
    ^-- Outbound messages queue [size=0]
17/02/16 13:06:54 INFO IgniteKernal%WCA: 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=a5f2ef72, name=WCA, uptime=00:14:00:085]
    ^-- H/N/C [hosts=8, nodes=8, CPUs=336]
    ^-- CPU [cur=0.03%, avg=0.09%, GC=0%]
    ^-- Heap [used=713MB, free=94.78%, comm=2609MB]
    ^-- Non heap [used=137MB, free=-1%, comm=139MB]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=0, qSize=0]
    ^-- Outbound messages queue [size=0]

Thanks,
Ranjit



--
Vladislav Pyatkov
Architect-Consultant "GridGain Rus" Llc.
+7 963 716 68 99
Loading...