Ignite on yarn doesn't started

classic Classic list List threaded Threaded
7 messages Options
v.shinkevich v.shinkevich
Reply | Threaded
Open this post in threaded view
|

Ignite on yarn doesn't started

Hi, All!

I tried to run Ignite on Yarn

My cluster properties:

#IGNITE_HOME = /tmp/ignite

# The HDFS path to the Apache Ignite config file.
IGNITE_XML_CONFIG =  /tmp/ignite/config/default-config.xml

# The directory which will be used for saving Apache Ignite distribution.
#IGNITE_WORKING_DIR = ./work

# The HDFS directory which will be used for saving Apache Ignite
distribution.
#IGNITE_RELEASES_DIR = /tmp/ignite/releases/

# The HDFS path to libs which will be added to classpath.
# IGNITE_USERS_LIBS = N/A  #/opt/libs/

# The number of megabytes of RAM for each Apache Ignite node.
# This is the size of the Java heap.
# This includes on-heap caching if it is used.
IGNITE_MEMORY_PER_NODE = 2048

# The amount of memory necessary for all data regions, with padding for JVM
native overhead, interned Strings, etc.
# This setting should always be adjusted for nodes that are used to store
data, not just for pure computations.
# Memory requested to YARN for containers running an Ignite node is the sum
of IGNITE_MEMORY_PER_NODE and IGNITE_MEMORY_OVERHEAD_PER_NODE.
IGNITE_MEMORY_OVERHEAD_PER_NODE = 16384
#IGNITE_MEMORY_PER_NODE * 0.10, with a minimum of 384

# The constraint on slave hosts.
# IGNITE_HOSTNAME_CONSTRAINT = N/A #192.168.0.[1-100]

# The number of nodes in the cluster.
IGNITE_NODE_COUNT = 16

# The number of CPU Cores for each Apache Ignite node.
IGNITE_RUN_CPU_PER_NODE = 2

# The version of Ignite which will be run on nodes.
IGNITE_VERSION = 2.7.6

# The HDFS path to the Apache Ignite build. This property can be useful when
the yarn
# is cluster running in net without internet access.
IGNITE_PATH = /tmp/ignite/apache-ignite-2.7.6-bin.zip

# Location where Ignite binary distribution is stored to be downloaded for
delivery. As per version 2.7, either IGNITE_PATH or IGNITE_URL is mandatory
in practice.
#IGNITE_URL =
http://ambari1.dmz.loc:8888/filebrowser/view=/tmp/ignite/apache-ignite-2.7.6-bin.zip

# Additional JVM options.
IGNITE_JVM_OPTS = -Djava.net.preferIPv4Stack=true

I also tried to add hdfs:// before paths, but result is the same.
I created dir /tmp/ignite on local node fs and copied it to hdfs.

apache-ignite-2.7.6-bin.zip
cluster.properties
commons-beanutils-1.9.3.jar
commons-codec-1.11.jar
commons-collections-3.2.2.jar
config
hadoop-common-2.7.3.jar
hadoop-yarn-client-2.7.3.jar
ignite-yarn-2.7.6.jar

Then run
yarn jar ignite-yarn-2.7.6.jar /tmp/ignite/ignite-yarn-2.7.6.jar
/tmp/ignite/cluster.properties

It prints in console: (I had to add some log.info to code)
 
Feb 10, 2020 10:33:35 PM org.apache.ignite.yarn.IgniteYarnClient main INFO:
ignite: hdfs:/tmp/ignite/apache-ignite-2.7.6-bin.zip
Feb 10, 2020 10:33:35 PM org.apache.ignite.yarn.IgniteYarnClient main INFO:
appJar: /ignite/workdir/ignite-yarn.jar
Feb 10, 2020 10:33:35 PM org.apache.ignite.yarn.IgniteYarnClient main INFO:
appMasterJar: scheme: "hdfs" host: "nmnode1.dmz.loc" port: 8020 file:
"/ignite/workdir/ignite-yarn.jar"
Feb 10, 2020 10:33:35 PM org.apache.ignite.yarn.IgniteYarnClient main INFO:
Submitted application. Application id: application_1581322307764_0057
Feb 10, 2020 10:33:39 PM org.apache.ignite.yarn.IgniteYarnClient main INFO:
Application application_1581322307764_0057 is RUNNING.

(/ignite/workdir/ was created on hdfs with ignite-yarn.jar)

But it's maximum that I got (Yarn application log) :

Log Type: stderr
Log Upload Time: Mon Feb 10 22:07:15 +0300 2020
Log Length: 7921
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/data/hdfs/v11/spill/usercache/root/appcache/application_1581322307764_0054/filecache/10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/02/10 22:06:38 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
20/02/10 22:06:38 INFO client.RMProxy: Connecting to ResourceManager at
ambari1.dmz.loc/10.254.62.127:8030
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Application master registered.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 18,432, cpu 2.
20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
dn00.dmz.loc:45454
20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
dn14.dmz.loc:45454
20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
dn01.dmz.loc:45454
20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
dn11.dmz.loc:45454
20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
dn04.dmz.loc:45454
20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
dn06.dmz.loc:45454
20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
dn15.dmz.loc:45454
Feb 10, 2020 10:06:39 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000003. State: COMPLETE.
Feb 10, 2020 10:06:39 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000004. State: COMPLETE.
Feb 10, 2020 10:06:39 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000002. State: COMPLETE.
20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
dn13.dmz.loc:45454
20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
dn16.dmz.loc:45454
20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
dn03.dmz.loc:45454
20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
dn07.dmz.loc:45454
20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
dn12.dmz.loc:45454
20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
dn10.dmz.loc:45454
20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
dn02.dmz.loc:45454
Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000006. State: COMPLETE.
Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000008. State: COMPLETE.
Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000007. State: COMPLETE.
Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000005. State: COMPLETE.
20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
dn17.dmz.loc:45454
20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
dn05.dmz.loc:45454
Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000012. State: COMPLETE.
Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000013. State: COMPLETE.
Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000015. State: COMPLETE.
Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000014. State: COMPLETE.
Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000011. State: COMPLETE.
Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000010. State: COMPLETE.
Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000009. State: COMPLETE.
Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000016. State: COMPLETE.
Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_e50_1581322307764_0054_01_000017. State: COMPLETE.


I didn't find any errors in hadoop-yarn logs.
I tried to complie for hadoop 2.7.3, but result is the same.

P.S. Ignite started locally on any node without problems.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite on yarn doesn't started

Hi,

Can you attach the Ignite node logs?

Also, I suggest to check:

1) check that Ignite libs (from ignite_binaries/libs) are available for
your YARN deployment.
2) check that path to the configuration file is reachable from every node

I see that you set absolute paths for to tmp folder. Probably you should
avoid using a temporary folder.

However, the real reason we can see only from Ignite logs.

BR,
Andrei

2/11/2020 8:49 AM, v.shinkevich пишет:

> Hi, All!
>
> I tried to run Ignite on Yarn
>
> My cluster properties:
>
> #IGNITE_HOME = /tmp/ignite
>
> # The HDFS path to the Apache Ignite config file.
> IGNITE_XML_CONFIG =  /tmp/ignite/config/default-config.xml
>
> # The directory which will be used for saving Apache Ignite distribution.
> #IGNITE_WORKING_DIR = ./work
>
> # The HDFS directory which will be used for saving Apache Ignite
> distribution.
> #IGNITE_RELEASES_DIR = /tmp/ignite/releases/
>
> # The HDFS path to libs which will be added to classpath.
> # IGNITE_USERS_LIBS = N/A  #/opt/libs/
>
> # The number of megabytes of RAM for each Apache Ignite node.
> # This is the size of the Java heap.
> # This includes on-heap caching if it is used.
> IGNITE_MEMORY_PER_NODE = 2048
>
> # The amount of memory necessary for all data regions, with padding for JVM
> native overhead, interned Strings, etc.
> # This setting should always be adjusted for nodes that are used to store
> data, not just for pure computations.
> # Memory requested to YARN for containers running an Ignite node is the sum
> of IGNITE_MEMORY_PER_NODE and IGNITE_MEMORY_OVERHEAD_PER_NODE.
> IGNITE_MEMORY_OVERHEAD_PER_NODE = 16384
> #IGNITE_MEMORY_PER_NODE * 0.10, with a minimum of 384
>
> # The constraint on slave hosts.
> # IGNITE_HOSTNAME_CONSTRAINT = N/A #192.168.0.[1-100]
>
> # The number of nodes in the cluster.
> IGNITE_NODE_COUNT = 16
>
> # The number of CPU Cores for each Apache Ignite node.
> IGNITE_RUN_CPU_PER_NODE = 2
>
> # The version of Ignite which will be run on nodes.
> IGNITE_VERSION = 2.7.6
>
> # The HDFS path to the Apache Ignite build. This property can be useful when
> the yarn
> # is cluster running in net without internet access.
> IGNITE_PATH = /tmp/ignite/apache-ignite-2.7.6-bin.zip
>
> # Location where Ignite binary distribution is stored to be downloaded for
> delivery. As per version 2.7, either IGNITE_PATH or IGNITE_URL is mandatory
> in practice.
> #IGNITE_URL =
> http://ambari1.dmz.loc:8888/filebrowser/view=/tmp/ignite/apache-ignite-2.7.6-bin.zip
>
> # Additional JVM options.
> IGNITE_JVM_OPTS = -Djava.net.preferIPv4Stack=true
>
> I also tried to add hdfs:// before paths, but result is the same.
> I created dir /tmp/ignite on local node fs and copied it to hdfs.
>
> apache-ignite-2.7.6-bin.zip
> cluster.properties
> commons-beanutils-1.9.3.jar
> commons-codec-1.11.jar
> commons-collections-3.2.2.jar
> config
> hadoop-common-2.7.3.jar
> hadoop-yarn-client-2.7.3.jar
> ignite-yarn-2.7.6.jar
>
> Then run
> yarn jar ignite-yarn-2.7.6.jar /tmp/ignite/ignite-yarn-2.7.6.jar
> /tmp/ignite/cluster.properties
>
> It prints in console: (I had to add some log.info to code)
>    
> Feb 10, 2020 10:33:35 PM org.apache.ignite.yarn.IgniteYarnClient main INFO:
> ignite: hdfs:/tmp/ignite/apache-ignite-2.7.6-bin.zip
> Feb 10, 2020 10:33:35 PM org.apache.ignite.yarn.IgniteYarnClient main INFO:
> appJar: /ignite/workdir/ignite-yarn.jar
> Feb 10, 2020 10:33:35 PM org.apache.ignite.yarn.IgniteYarnClient main INFO:
> appMasterJar: scheme: "hdfs" host: "nmnode1.dmz.loc" port: 8020 file:
> "/ignite/workdir/ignite-yarn.jar"
> Feb 10, 2020 10:33:35 PM org.apache.ignite.yarn.IgniteYarnClient main INFO:
> Submitted application. Application id: application_1581322307764_0057
> Feb 10, 2020 10:33:39 PM org.apache.ignite.yarn.IgniteYarnClient main INFO:
> Application application_1581322307764_0057 is RUNNING.
>
> (/ignite/workdir/ was created on hdfs with ignite-yarn.jar)
>
> But it's maximum that I got (Yarn application log) :
>
> Log Type: stderr
> Log Upload Time: Mon Feb 10 22:07:15 +0300 2020
> Log Length: 7921
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/data/hdfs/v11/spill/usercache/root/appcache/application_1581322307764_0054/filecache/10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 20/02/10 22:06:38 INFO impl.ContainerManagementProtocolProxy:
> yarn.client.max-cached-nodemanagers-proxies : 0
> 20/02/10 22:06:38 INFO client.RMProxy: Connecting to ResourceManager at
> ambari1.dmz.loc/10.254.62.127:8030
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Application master registered.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> Feb 10, 2020 10:06:38 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 18,432, cpu 2.
> 20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
> dn00.dmz.loc:45454
> 20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
> dn14.dmz.loc:45454
> 20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
> dn01.dmz.loc:45454
> 20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
> dn11.dmz.loc:45454
> 20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
> dn04.dmz.loc:45454
> 20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
> dn06.dmz.loc:45454
> 20/02/10 22:06:39 INFO impl.AMRMClientImpl: Received new token for :
> dn15.dmz.loc:45454
> Feb 10, 2020 10:06:39 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000003. State: COMPLETE.
> Feb 10, 2020 10:06:39 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000004. State: COMPLETE.
> Feb 10, 2020 10:06:39 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000002. State: COMPLETE.
> 20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
> dn13.dmz.loc:45454
> 20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
> dn16.dmz.loc:45454
> 20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
> dn03.dmz.loc:45454
> 20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
> dn07.dmz.loc:45454
> 20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
> dn12.dmz.loc:45454
> 20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
> dn10.dmz.loc:45454
> 20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
> dn02.dmz.loc:45454
> Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000006. State: COMPLETE.
> Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000008. State: COMPLETE.
> Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000007. State: COMPLETE.
> Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000005. State: COMPLETE.
> 20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
> dn17.dmz.loc:45454
> 20/02/10 22:06:40 INFO impl.AMRMClientImpl: Received new token for :
> dn05.dmz.loc:45454
> Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000012. State: COMPLETE.
> Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000013. State: COMPLETE.
> Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000015. State: COMPLETE.
> Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000014. State: COMPLETE.
> Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000011. State: COMPLETE.
> Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000010. State: COMPLETE.
> Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000009. State: COMPLETE.
> Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000016. State: COMPLETE.
> Feb 10, 2020 10:06:40 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_e50_1581322307764_0054_01_000017. State: COMPLETE.
>
>
> I didn't find any errors in hadoop-yarn logs.
> I tried to complie for hadoop 2.7.3, but result is the same.
>
> P.S. Ignite started locally on any node without problems.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
v.shinkevich v.shinkevich
Reply | Threaded
Open this post in threaded view
|

Re: Ignite on yarn doesn't started

aealexsandrov wrote
> 1) check that Ignite libs (from ignite_binaries/libs) are available for
> your YARN deployment.
> 2) check that path to the configuration file is reachable from every node

1) I don't understand what I need to check. Where should these libs be ?  Do
I need to unpack the distribution? To a local folder or to HDFS?

My /tmp/ignite folder (on HDFS, on local the same content + unpacked distro
for local run check)
<http://apache-ignite-users.70518.x6.nabble.com/file/t2775/200211-211426.png>
On HDFS I don't have any logs. Only one jar in workdir.
<http://apache-ignite-users.70518.x6.nabble.com/file/t2775/200211-210235.png>

Log of local run:
[root@dn07 /tmp/ignite/apache-ignite-2.7.6-bin/bin]# ./ignite.sh

[20:54:37]    __________  ________________
[20:54:37]   /  _/ ___/ |/ /  _/_  __/ __/
[20:54:37]  _/ // (7 7    // /  / / / _/  
[20:54:37] /___/\___/_/|_/___/ /_/ /___/  
[20:54:37]
[20:54:37] ver. 2.7.6#20190911-sha1:21f7ca41
[20:54:37] 2019 Copyright(C) Apache Software Foundation
[20:54:37]
[20:54:37] Ignite documentation: http://ignite.apache.org
[20:54:37]
[20:54:37] Quiet mode.
[20:54:37]   ^-- Logging to file
'/tmp/ignite/apache-ignite-2.7.6-bin/work/log/ignite-e2eeb3da.0.log'
[20:54:37]   ^-- Logging by 'JavaLogger [quiet=true, config=null]'
[20:54:37]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[20:54:37]
[20:54:37] OS: Linux 3.10.0-693.el7.x86_64 amd64
[20:54:37] VM information: Java(TM) SE Runtime Environment 1.8.0_141-b15
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.141-b15
[20:54:38] Please set system property '-Djava.net.preferIPv4Stack=true' to
avoid possible problems in mixed environments.
[20:54:38] Configured plugins:
[20:54:38]   ^-- None
[20:54:38]
[20:54:38] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0, super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
Java HotSpot(TM) 64-Bit Server VM warning: sched_getaffinity failed (Invalid
argument)- using online processor count (192) which may exceed available
processors
[20:54:38] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
[20:54:39] Security status [authentication=off, tls/ssl=off]
[20:54:44] Performance suggestions for grid  (fix if possible)
[20:54:44] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[20:54:44]   ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM
options)
[20:54:44]   ^-- Specify JVM heap max size (add '-Xmx<size>[g|G|m|M|k|K]' to
JVM options)
[20:54:44]   ^-- Set max direct memory size if getting 'OOME: Direct buffer
memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
[20:54:44]   ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
[20:54:44]   ^-- Speed up flushing of dirty pages by OS (alter
vm.dirty_expire_centisecs parameter by setting to 500)
[20:54:44] Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[20:54:44]
[20:54:44] To start Console Management & Monitoring run
ignitevisorcmd.{sh|bat}
[20:54:44] Data Regions Configured:
[20:54:44]   ^-- default [initSize=256.0 MiB, maxSize=403.0 GiB,
persistence=false]
[20:54:44]
[20:54:44] Ignite node started OK (id=e2eeb3da)
^C
[20:55:19] Ignite node stopped OK [uptime=00:00:35.686]




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite on yarn doesn't started

Hi,

I asked you to check it because I see the next option:

IGNITE_PATH = /tmp/ignite/apache-ignite-2.7.6-bin.zip

This option should be a path to unzipped Ignite distribution instead of the URL (you set zip)

Also I see commented IGNITE_URL option:

#IGNITE_URL =
http://ambari1.dmz.loc:8888/filebrowser/view=/tmp/ignite/apache-ignite-2.7.6-bin.zip

So it looks like you don't provide the Ignite binaries to your YARN deployment.

BR,
Andrei
2/11/2020 9:19 PM, v.shinkevich пишет:
aealexsandrov wrote
1) check that Ignite libs (from ignite_binaries/libs) are available for 
your YARN deployment.
2) check that path to the configuration file is reachable from every node
1) I don't understand what I need to check. Where should these libs be ?  Do
I need to unpack the distribution? To a local folder or to HDFS?

My /tmp/ignite folder (on HDFS, on local the same content + unpacked distro
for local run check)
<http://apache-ignite-users.70518.x6.nabble.com/file/t2775/200211-211426.png> 
On HDFS I don't have any logs. Only one jar in workdir.
<http://apache-ignite-users.70518.x6.nabble.com/file/t2775/200211-210235.png> 

Log of local run:
[root@dn07 /tmp/ignite/apache-ignite-2.7.6-bin/bin]# ./ignite.sh

[20:54:37]    __________  ________________ 
[20:54:37]   /  _/ ___/ |/ /  _/_  __/ __/ 
[20:54:37]  _/ // (7 7    // /  / / / _/   
[20:54:37] /___/\___/_/|_/___/ /_/ /___/  
[20:54:37] 
[20:54:37] ver. 2.7.6#20190911-sha1:21f7ca41
[20:54:37] 2019 Copyright(C) Apache Software Foundation
[20:54:37] 
[20:54:37] Ignite documentation: http://ignite.apache.org
[20:54:37] 
[20:54:37] Quiet mode.
[20:54:37]   ^-- Logging to file
'/tmp/ignite/apache-ignite-2.7.6-bin/work/log/ignite-e2eeb3da.0.log'
[20:54:37]   ^-- Logging by 'JavaLogger [quiet=true, config=null]'
[20:54:37]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[20:54:37] 
[20:54:37] OS: Linux 3.10.0-693.el7.x86_64 amd64
[20:54:37] VM information: Java(TM) SE Runtime Environment 1.8.0_141-b15
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.141-b15
[20:54:38] Please set system property '-Djava.net.preferIPv4Stack=true' to
avoid possible problems in mixed environments.
[20:54:38] Configured plugins:
[20:54:38]   ^-- None
[20:54:38] 
[20:54:38] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0, super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
Java HotSpot(TM) 64-Bit Server VM warning: sched_getaffinity failed (Invalid
argument)- using online processor count (192) which may exceed available
processors
[20:54:38] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
[20:54:39] Security status [authentication=off, tls/ssl=off]
[20:54:44] Performance suggestions for grid  (fix if possible)
[20:54:44] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[20:54:44]   ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM
options)
[20:54:44]   ^-- Specify JVM heap max size (add '-Xmx<size>[g|G|m|M|k|K]' to
JVM options)
[20:54:44]   ^-- Set max direct memory size if getting 'OOME: Direct buffer
memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
[20:54:44]   ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
[20:54:44]   ^-- Speed up flushing of dirty pages by OS (alter
vm.dirty_expire_centisecs parameter by setting to 500)
[20:54:44] Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[20:54:44] 
[20:54:44] To start Console Management & Monitoring run
ignitevisorcmd.{sh|bat}
[20:54:44] Data Regions Configured:
[20:54:44]   ^-- default [initSize=256.0 MiB, maxSize=403.0 GiB,
persistence=false]
[20:54:44] 
[20:54:44] Ignite node started OK (id=e2eeb3da)
^C
[20:55:19] Ignite node stopped OK [uptime=00:00:35.686]




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
v.shinkevich v.shinkevich
Reply | Threaded
Open this post in threaded view
|

Re: Ignite on yarn doesn't started

Hi,

I found problem, but it is correct :

# The HDFS path to the Apache Ignite build. This property can be useful when
the yarn
# is cluster running in net without internet access.
IGNITE_PATH = hdfs:///ignite/apache-ignite-2.7.6-bin.zip

The problem was in the Yarn configuration or other cluster settings...

Yarn didn't give containers with more than one processor.

Containers were checked and rejected. But this reason was not present in
logs.

I added PR to help to solve the problem.

https://github.com/apache/ignite/pull/7437

Thanks.






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite on yarn doesn't started

Hello!

Can you please also file a ticket against Apache Ignite JIRA?


I think it would not be a problem to merge it.

Regards,
--
Ilya Kasnacheev


вт, 18 февр. 2020 г. в 11:04, v.shinkevich <[hidden email]>:
Hi,

I found problem, but it is correct :

# The HDFS path to the Apache Ignite build. This property can be useful when
the yarn
# is cluster running in net without internet access.
IGNITE_PATH = hdfs:///ignite/apache-ignite-2.7.6-bin.zip

The problem was in the Yarn configuration or other cluster settings...

Yarn didn't give containers with more than one processor.

Containers were checked and rejected. But this reason was not present in
logs.

I added PR to help to solve the problem.

https://github.com/apache/ignite/pull/7437

Thanks.






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
v.shinkevich v.shinkevich
Reply | Threaded
Open this post in threaded view
|

Re: Ignite on yarn doesn't started