Ignite for Spark on YARN Deployment

classic Classic list List threaded Threaded
21 messages Options
12
Hongmei Zong Hongmei Zong
Reply | Threaded
Open this post in threaded view
|

Ignite for Spark on YARN Deployment

This post was updated on .
Hi there,

I would like to use "Ignite for Spark" to save the states of Spark jobs in memory and those states can be used for later jobs. For Shared Deployment, the document only offer two ways to deploy Ignite cluster. First is the standalone deployment, second is MESOS deployment. But Our Spark clusters are running on YARN. My question is: is it possible to run Ignite for Spark on YARN deployment???

I downloaded and installed Ignite on my machine. Next, I referenced the link below for YARN Deployment. http://apacheignite.gridgain.org/docs/yarn-deployment

I created the cluster.properties file and ran the application using the command:    hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

From the YARN console, The YARN ignite application works ok. It shows running, and 16 containers are allocated for the ignite application.

When I tried to "test ignite with Spark-shell" by running
/usr/bin/spark-shell --jars /u/hongmei/apache-ignite/libs/ignite-core-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-spark/ignite-spark-1.6.0.jar,/u/hongmei/apache-ignite/libs/cache-api-1.0.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/ignite-log4j-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/log4j-1.2.17.jar --packages org.apache.ignite:ignite spark:1.6.0,org.apache.ignite:ignite-spring:1.6.0

I get the following error:
ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.138.10.30:56747, retrying ...
ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.138.10.30:56747, retrying ...

My goal is using ignite YARN deployment with Spark YARN. what should I do ?

Many Thanks!!!

Mei
Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

Hi,

I’m not an expert in this area however have you tried to specify a Spark master like the following documentation says?
https://apacheignite-fs.readme.io/docs/testing-integration-with-spark-shell#working-with-spark-shell

If you did try please share the full logs, someone from the community will respond.


Denis

> On Jun 6, 2016, at 5:58 PM, Hongmei Zong <[hidden email]> wrote:
>
> Hi there,
>
> I would like to use "Ignite for Spark" to save the states of Spark jobs in
> memory and those states can be used for later jobs. For Shared Deployment,
> the document only offer two ways to deploy Ignite cluster. First is the
> standalone deployment, second is MESOS deployment. But Our Spark clusters
> are running on YARN. My question is: is it possible to run Ignite for Spark
> on YARN deployment???
>
> I downloaded and installed Ignite on my machine. Next, I referenced the link
> below for YARN Deployment.
> http://apacheignite.gridgain.org/docs/yarn-deployment
>
> I created the cluster.properties file and ran the application using the
> command:    hadoop jar
> /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar
> /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar
> /u/hongmei/apache-ignite/config/cluster.properties
>
> Form the YARN console, The YARN ignite application works ok. It shows
> running, and 16 containers are allocated for the ignite application.
>
> After this step, what should I do in order to run Spark with Ignite on YARN
> deployment??
>
> Many Thanks!!!
>
> Mei
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp5465.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Hongmei Zong Hongmei Zong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

Hi Denis,

Thank you very much for your good suggestions! If my Spark running on YARN and there are three master_hosts running, should I pick up one randomly? I will give a try to specify the master_host and master_port.

Thank you!
Mei

On Tue, Jun 7, 2016 at 10:26 AM, Denis Magda <[hidden email]> wrote:
Hi,

I’m not an expert in this area however have you tried to specify a Spark master like the following documentation says?
https://apacheignite-fs.readme.io/docs/testing-integration-with-spark-shell#working-with-spark-shell

If you did try please share the full logs, someone from the community will respond.


Denis

> On Jun 6, 2016, at 5:58 PM, Hongmei Zong <[hidden email]> wrote:
>
> Hi there,
>
> I would like to use "Ignite for Spark" to save the states of Spark jobs in
> memory and those states can be used for later jobs. For Shared Deployment,
> the document only offer two ways to deploy Ignite cluster. First is the
> standalone deployment, second is MESOS deployment. But Our Spark clusters
> are running on YARN. My question is: is it possible to run Ignite for Spark
> on YARN deployment???
>
> I downloaded and installed Ignite on my machine. Next, I referenced the link
> below for YARN Deployment.
> http://apacheignite.gridgain.org/docs/yarn-deployment
>
> I created the cluster.properties file and ran the application using the
> command:    hadoop jar
> /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar
> /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar
> /u/hongmei/apache-ignite/config/cluster.properties
>
> Form the YARN console, The YARN ignite application works ok. It shows
> running, and 16 containers are allocated for the ignite application.
>
> After this step, what should I do in order to run Spark with Ignite on YARN
> deployment??
>
> Many Thanks!!!
>
> Mei
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp5465.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.


Hongmei Zong Hongmei Zong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

In reply to this post by Denis Magda
Hi Denis,

I tried testing Ignite as the following steps:

Background information: 
1. Our Spark is running on YARN deployment; There are three Master hosts and many Worker nodes and three client nodes in the Spark clusters.
2. I installed Ignite on one of the client node and can launch Ignite-shell locally on this client node. Since for testing purpose, I did not install Ignite on any master nodes or worker nodes.
3. I logged into the client node which Ignite was installed and launched Ignite YARN application using the following command:

hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

I open the UI console for the Ignite YARN application, and it shows that Ignite YARN is running, Containers, CPU Cores, Memory are all allocated by YARN. screenshot is as the following 

application_1464374946035_26956hongmeiignitionYARNdefaultWed Jun 8 10:21:04 -0400 2016N/ARUNNINGUNDEFINED1212348169.63.8
ApplicationMaster

4. I open an other terminal log into the same client node and run the Ignite for Spark Shell command as the following:
The Spark shell started

/usr/bin/spark-shell --jars /u/hongmei/apache-ignite/libs/ignite-core-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-spark/ignite-spark-1.6.0.jar,/u/hongmei/apache-ignite/libs/cache-api-1.0.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/ignite-log4j-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/log4j-1.2.17.jar --packages org.apache.ignite:ignite-spark:1.6.0,org.apache.ignite:ignite-spring:1.6.0

The Spark shell launched successfully and I use these two commands to import the ignite spark files:
import org.apache.ignite.spark._
import org.apache.ignite.configuration._

Next I create an instance of Ignite context as the following syntax:
val ic = new IgniteContext[Integer, Integer](sc, () => new IgniteConfiguration())

I got the following message: So I stuck at this point.

16/06/08 10:28:27 WARN TcpDiscoverySpi: IP finder returned empty addresses list. Please check IP finder configuration and make sure multicast works on your network. Will retry every 2 secs.

So I stuck at this point.

The other scenario when I run command listed as Step 4. The Spark shell can not be launched successfully with the status of “Accept”, but never get chance to run.

Any good suggestions?????? Is there anything wrong with my test procedure??
Thanks in advance! 

Mei


On Jun 7, 2016, at 10:26 AM, Denis Magda <[hidden email]> wrote:

Hi,

I’m not an expert in this area however have you tried to specify a Spark master like the following documentation says?
https://apacheignite-fs.readme.io/docs/testing-integration-with-spark-shell#working-with-spark-shell

If you did try please share the full logs, someone from the community will respond.


Denis

On Jun 6, 2016, at 5:58 PM, Hongmei Zong <[hidden email]> wrote:

Hi there,

I would like to use "Ignite for Spark" to save the states of Spark jobs in
memory and those states can be used for later jobs. For Shared Deployment,
the document only offer two ways to deploy Ignite cluster. First is the
standalone deployment, second is MESOS deployment. But Our Spark clusters
are running on YARN. My question is: is it possible to run Ignite for Spark
on YARN deployment???

I downloaded and installed Ignite on my machine. Next, I referenced the link
below for YARN Deployment.
http://apacheignite.gridgain.org/docs/yarn-deployment

I created the cluster.properties file and ran the application using the
command:    hadoop jar
/u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar
/u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar
/u/hongmei/apache-ignite/config/cluster.properties

Form the YARN console, The YARN ignite application works ok. It shows
running, and 16 containers are allocated for the ignite application.

After this step, what should I do in order to run Spark with Ignite on YARN
deployment??

Many Thanks!!!

Mei



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp5465.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


Nikolai Tikhonov-2 Nikolai Tikhonov-2
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

Hi Hongmei Zong!

Client node which started from IgniteContext can't to find server nodes. By default ignite integration with YARN uses TcpDiscoveryVmIpFinder (if you don't use another ip finder in your configuration).
In the case you should set the ip finder in IgniteContext. The following code snippet shows how do it on Java:

IgniteConfiguration cfg = new IgniteConfiguration();

TcpDiscoverySpi spi = new TcpDiscoverySpi();

TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("your_address1:47500..47510", "your_address2:47500..47510",
"your_address3:47500..47510"));

spi.setIpFinder(ipFinder);

cfg.setDiscoverySpi(spi);

On Wed, Jun 8, 2016 at 6:31 PM, Hongmei Zong <[hidden email]> wrote:
Hi Denis,

I tried testing Ignite as the following steps:

Background information: 
1. Our Spark is running on YARN deployment; There are three Master hosts and many Worker nodes and three client nodes in the Spark clusters.
2. I installed Ignite on one of the client node and can launch Ignite-shell locally on this client node. Since for testing purpose, I did not install Ignite on any master nodes or worker nodes.
3. I logged into the client node which Ignite was installed and launched Ignite YARN application using the following command:

hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

I open the UI console for the Ignite YARN application, and it shows that Ignite YARN is running, Containers, CPU Cores, Memory are all allocated by YARN. screenshot is as the following 

application_1464374946035_26956hongmeiignitionYARNdefaultWed Jun 8 10:21:04 -0400 2016N/ARUNNINGUNDEFINED1212348169.63.8
ApplicationMaster

4. I open an other terminal log into the same client node and run the Ignite for Spark Shell command as the following:
The Spark shell started

/usr/bin/spark-shell --jars /u/hongmei/apache-ignite/libs/ignite-core-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-spark/ignite-spark-1.6.0.jar,/u/hongmei/apache-ignite/libs/cache-api-1.0.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/ignite-log4j-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/log4j-1.2.17.jar --packages org.apache.ignite:ignite-spark:1.6.0,org.apache.ignite:ignite-spring:1.6.0

The Spark shell launched successfully and I use these two commands to import the ignite spark files:
import org.apache.ignite.spark._
import org.apache.ignite.configuration._

Next I create an instance of Ignite context as the following syntax:
val ic = new IgniteContext[Integer, Integer](sc, () => new IgniteConfiguration())

I got the following message: So I stuck at this point.

16/06/08 10:28:27 WARN TcpDiscoverySpi: IP finder returned empty addresses list. Please check IP finder configuration and make sure multicast works on your network. Will retry every 2 secs.

So I stuck at this point.

The other scenario when I run command listed as Step 4. The Spark shell can not be launched successfully with the status of “Accept”, but never get chance to run.

Any good suggestions?????? Is there anything wrong with my test procedure??
Thanks in advance! 

Mei


On Jun 7, 2016, at 10:26 AM, Denis Magda <[hidden email]> wrote:

Hi,

I’m not an expert in this area however have you tried to specify a Spark master like the following documentation says?
https://apacheignite-fs.readme.io/docs/testing-integration-with-spark-shell#working-with-spark-shell

If you did try please share the full logs, someone from the community will respond.


Denis

On Jun 6, 2016, at 5:58 PM, Hongmei Zong <[hidden email]> wrote:

Hi there,

I would like to use "Ignite for Spark" to save the states of Spark jobs in
memory and those states can be used for later jobs. For Shared Deployment,
the document only offer two ways to deploy Ignite cluster. First is the
standalone deployment, second is MESOS deployment. But Our Spark clusters
are running on YARN. My question is: is it possible to run Ignite for Spark
on YARN deployment???

I downloaded and installed Ignite on my machine. Next, I referenced the link
below for YARN Deployment.
http://apacheignite.gridgain.org/docs/yarn-deployment

I created the cluster.properties file and ran the application using the
command:    hadoop jar
/u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar
/u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar
/u/hongmei/apache-ignite/config/cluster.properties

Form the YARN console, The YARN ignite application works ok. It shows
running, and 16 containers are allocated for the ignite application.

After this step, what should I do in order to run Spark with Ignite on YARN
deployment??

Many Thanks!!!

Mei



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp5465.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



Hongmei Zong Hongmei Zong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

Hi Nikolai,

Thank you very much for your solution! I would like to double check with you about my address1…...

ipFinder.setAddresses(Arrays.asList("your_address1:47500..47510", "your_address2:47500..47510",
    "your_address3:47500..47510"));
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

Is that necessary to install Ignite on these three YARN master_host? I guess the answer is “NO”.
Is the the master_port should be 47500 or 47510?

Sorry for these questions, I am new to Ignite and YARN framework.
Thanks again!!!

Hongmei


On Jun 8, 2016, at 12:30 PM, Nikolai Tikhonov <[hidden email]> wrote:

Hi Hongmei Zong!

Client node which started from IgniteContext can't to find server nodes. By default ignite integration with YARN uses TcpDiscoveryVmIpFinder (if you don't use another ip finder in your configuration).
In the case you should set the ip finder in IgniteContext. The following code snippet shows how do it on Java:

IgniteConfiguration cfg = new IgniteConfiguration();

TcpDiscoverySpi spi = new TcpDiscoverySpi();

TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("your_address1:47500..47510", "your_address2:47500..47510",
"your_address3:47500..47510"));

spi.setIpFinder(ipFinder);

cfg.setDiscoverySpi(spi);

On Wed, Jun 8, 2016 at 6:31 PM, Hongmei Zong <[hidden email]> wrote:
Hi Denis,

I tried testing Ignite as the following steps:

Background information: 
1. Our Spark is running on YARN deployment; There are three Master hosts and many Worker nodes and three client nodes in the Spark clusters.
2. I installed Ignite on one of the client node and can launch Ignite-shell locally on this client node. Since for testing purpose, I did not install Ignite on any master nodes or worker nodes.
3. I logged into the client node which Ignite was installed and launched Ignite YARN application using the following command:

hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

I open the UI console for the Ignite YARN application, and it shows that Ignite YARN is running, Containers, CPU Cores, Memory are all allocated by YARN. screenshot is as the following 

application_1464374946035_26956hongmeiignitionYARNdefaultWed Jun 8 10:21:04 -0400 2016N/ARUNNINGUNDEFINED1212348169.63.8
ApplicationMaster

4. I open an other terminal log into the same client node and run the Ignite for Spark Shell command as the following:
The Spark shell started

/usr/bin/spark-shell --jars /u/hongmei/apache-ignite/libs/ignite-core-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-spark/ignite-spark-1.6.0.jar,/u/hongmei/apache-ignite/libs/cache-api-1.0.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/ignite-log4j-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/log4j-1.2.17.jar --packages org.apache.ignite:ignite-spark:1.6.0,org.apache.ignite:ignite-spring:1.6.0

The Spark shell launched successfully and I use these two commands to import the ignite spark files:
import org.apache.ignite.spark._
import org.apache.ignite.configuration._

Next I create an instance of Ignite context as the following syntax:
val ic = new IgniteContext[Integer, Integer](sc, () => new IgniteConfiguration())

I got the following message: So I stuck at this point.

16/06/08 10:28:27 WARN TcpDiscoverySpi: IP finder returned empty addresses list. Please check IP finder configuration and make sure multicast works on your network. Will retry every 2 secs.

So I stuck at this point.

The other scenario when I run command listed as Step 4. The Spark shell can not be launched successfully with the status of “Accept”, but never get chance to run.

Any good suggestions?????? Is there anything wrong with my test procedure??
Thanks in advance! 

Mei


On Jun 7, 2016, at 10:26 AM, Denis Magda <[hidden email]> wrote:

Hi,

I’m not an expert in this area however have you tried to specify a Spark master like the following documentation says?
https://apacheignite-fs.readme.io/docs/testing-integration-with-spark-shell#working-with-spark-shell

If you did try please share the full logs, someone from the community will respond.


Denis

On Jun 6, 2016, at 5:58 PM, Hongmei Zong <[hidden email]> wrote:

Hi there,

I would like to use "Ignite for Spark" to save the states of Spark jobs in
memory and those states can be used for later jobs. For Shared Deployment,
the document only offer two ways to deploy Ignite cluster. First is the
standalone deployment, second is MESOS deployment. But Our Spark clusters
are running on YARN. My question is: is it possible to run Ignite for Spark
on YARN deployment???

I downloaded and installed Ignite on my machine. Next, I referenced the link
below for YARN Deployment.
http://apacheignite.gridgain.org/docs/yarn-deployment

I created the cluster.properties file and ran the application using the
command:    hadoop jar
/u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar
/u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar
/u/hongmei/apache-ignite/config/cluster.properties

Form the YARN console, The YARN ignite application works ok. It shows
running, and 16 containers are allocated for the ignite application.

After this step, what should I do in order to run Spark with Ignite on YARN
deployment??

Many Thanks!!!

Mei



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp5465.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.




Nikolai Tikhonov-2 Nikolai Tikhonov-2
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));

Hongmei Zong Hongmei Zong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

Thank you very much Nikolai !

I found another issue regarding my Ignite YARN Integration:

I  run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following:
Then I log onto one of the container node and the log is as the following:

[hongmei@c5hdpe001 apache-ignite]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

WARNING: Use "yarn jar" to launch YARN applications.

16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/

16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2

16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29511

Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Submitted application. Application id: application_1464374946035_29511

Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Application application_1464374946035_29511 is RUNNING. 


Then I log onto one of the container node and the log is as the following: "It failed to start grid". 
Any good suggestion? Thank you very much!!!


class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
        at org.apache.ignite.Ignition.start(Ignition.java:350)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
        at org.apache.ignite.Ignition.start(Ignition.java:347)
        ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
        ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
        ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Hongmei


On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[hidden email]> wrote:
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));


Nikolai Tikhonov-2 Nikolai Tikhonov-2
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

It seems that your ignite configuration invalid. Could you share ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties?

On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <[hidden email]> wrote:
Thank you very much Nikolai !

I found another issue regarding my Ignite YARN Integration:

I  run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following:
Then I log onto one of the container node and the log is as the following:

[hongmei@c5hdpe001 apache-ignite]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

WARNING: Use "yarn jar" to launch YARN applications.

16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/

16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2

16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29511

Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Submitted application. Application id: application_1464374946035_29511

Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Application application_1464374946035_29511 is RUNNING. 


Then I log onto one of the container node and the log is as the following: "It failed to start grid". 
Any good suggestion? Thank you very much!!!


class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
        at org.apache.ignite.Ignition.start(Ignition.java:350)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
        at org.apache.ignite.Ignition.start(Ignition.java:347)
        ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
        ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
        ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Hongmei


On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[hidden email]> wrote:
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));



Hongmei Zong Hongmei Zong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

Hi nikolai,

Thank you very much for prompt reply! 

I did not find the ignite-config.xml file under my ignite home directory(  /u/hongmei/apache-ignite/  ).

I find a "default-config.xml" at the path:  /u/hongmei/apache-ignite/config/default-config.xml

<?xml version="1.0" encoding="UTF-8"?>


<!--

  Licensed to the Apache Software Foundation (ASF) under one or more

  contributor license agreements.  See the NOTICE file distributed with

  this work for additional information regarding copyright ownership.

  The ASF licenses this file to You under the Apache License, Version 2.0

  (the "License"); you may not use this file except in compliance with

  the License.  You may obtain a copy of the License at


       http://www.apache.org/licenses/LICENSE-2.0


  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License.

-->


<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xsi:schemaLocation="

       http://www.springframework.org/schema/beans

       http://www.springframework.org/schema/beans/spring-beans.xsd">

    <!--

        Alter configuration below as needed.

    -->

        <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

                <property name="discoverySpi">

                        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">

                                <property name="ipFinder">

                                        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">

                                                <property name="addresses">

                                                        <list>

                                                                <!--

                                                                 Explicitly specifying address of a local node to let it start

                                                                and operate normally even if there is no more nodes in the cluster.

                                                                You can also optionally specify an individual port or port range.

                                                                -->

                                                                <!--

                                                                <value>1.2.3.4</value>

                                                                -->

                                                                <!-- 

                                                                        IP Address and optional port range of a remote node.

                                                                        You can also optionally specify an individual port and don't set

                                                                        the port range at all.                                  

                                                                -->

                                                                <value>c5hdpe001.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe002.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe003.c5.runwaynine.com:47500..47509</value>


                                                        </list>

                                                </property>

                                        </bean>

                                </property>

                        </bean>

                </property>

        </bean>

</beans>



The "cluster.properties" at the path:  /u/hongmei/apache-ignite/config/cluster.properties

# The number of nodes in the cluster.

IGNITE_NODE_COUNT=2


# The number of CPU Cores for each Apache Ignite node.

IGNITE_RUN_CPU_PER_NODE=1


# The number of Megabytes of RAM for each Apache Ignite node.

IGNITE_MEMORY_PER_NODE=2048


# The version ignite which will be run on nodes.

IGNITE_VERSION=1.6.0


# The hdfs directory which will be used for saving Apache Ignite disbributives.

IGNITE_RELEASES_DIR=/user/hongmei/ignite-yarn


#The directory which will be used for saving Apache Ignite distributives(copy .jar file to it).

IGNITE_WORKING_DIR=/user/hongmei/ignite/workdir


#The hdfs path to Apache Ignite config file.

#IGNITE_XML_CONFIG=/user/yarn/ignite/

IGNITE_XML_CONFIG=/user/hongmei/ignite/


#The hdfs path to libs which will be added to classpath.

IGNITE_USERS_LIBS=/user/hongmei/ignite/libs/


#The constraint on slave hosts.

#IGNITE_HOSTENAME_CONSTRAINT=


IGNITE_LOCAL_WORK_DIR=/u/hongmei/apache-ignite



Thank you very much!!!!

Hongmei





On Thu, Jun 9, 2016 at 9:43 AM, Nikolai Tikhonov <[hidden email]> wrote:
It seems that your ignite configuration invalid. Could you share ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties?

On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <[hidden email]> wrote:
Thank you very much Nikolai !

I found another issue regarding my Ignite YARN Integration:

I  run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following:
Then I log onto one of the container node and the log is as the following:

[hongmei@c5hdpe001 apache-ignite]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

WARNING: Use "yarn jar" to launch YARN applications.

16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/

16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2

16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29511

Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Submitted application. Application id: application_1464374946035_29511

Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Application application_1464374946035_29511 is RUNNING. 


Then I log onto one of the container node and the log is as the following: "It failed to start grid". 
Any good suggestion? Thank you very much!!!


class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
        at org.apache.ignite.Ignition.start(Ignition.java:350)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
        at org.apache.ignite.Ignition.start(Ignition.java:347)
        ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
        ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
        ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Hongmei


On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[hidden email]> wrote:
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));




Nikolai Tikhonov Nikolai Tikhonov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

You set wrong value to IGNITE_XML_CONFIG property. The property should contains path to ignite configuration file. For example 

IGNITE_XML_CONFIG=/u/hongmei/apache-ignite/config/default-config.xml

I think you can comment this line in property file and ignite will start with default configuration.

On Thu, Jun 9, 2016 at 5:08 PM, Hongmei Zong <[hidden email]> wrote:
Hi nikolai,

Thank you very much for prompt reply! 

I did not find the ignite-config.xml file under my ignite home directory(  /u/hongmei/apache-ignite/  ).

I find a "default-config.xml" at the path:  /u/hongmei/apache-ignite/config/default-config.xml

<?xml version="1.0" encoding="UTF-8"?>


<!--

  Licensed to the Apache Software Foundation (ASF) under one or more

  contributor license agreements.  See the NOTICE file distributed with

  this work for additional information regarding copyright ownership.

  The ASF licenses this file to You under the Apache License, Version 2.0

  (the "License"); you may not use this file except in compliance with

  the License.  You may obtain a copy of the License at


       http://www.apache.org/licenses/LICENSE-2.0


  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License.

-->


<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xsi:schemaLocation="

       http://www.springframework.org/schema/beans

       http://www.springframework.org/schema/beans/spring-beans.xsd">

    <!--

        Alter configuration below as needed.

    -->

        <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

                <property name="discoverySpi">

                        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">

                                <property name="ipFinder">

                                        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">

                                                <property name="addresses">

                                                        <list>

                                                                <!--

                                                                 Explicitly specifying address of a local node to let it start

                                                                and operate normally even if there is no more nodes in the cluster.

                                                                You can also optionally specify an individual port or port range.

                                                                -->

                                                                <!--

                                                                <value>1.2.3.4</value>

                                                                -->

                                                                <!-- 

                                                                        IP Address and optional port range of a remote node.

                                                                        You can also optionally specify an individual port and don't set

                                                                        the port range at all.                                  

                                                                -->

                                                                <value>c5hdpe001.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe002.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe003.c5.runwaynine.com:47500..47509</value>


                                                        </list>

                                                </property>

                                        </bean>

                                </property>

                        </bean>

                </property>

        </bean>

</beans>



The "cluster.properties" at the path:  /u/hongmei/apache-ignite/config/cluster.properties

# The number of nodes in the cluster.

IGNITE_NODE_COUNT=2


# The number of CPU Cores for each Apache Ignite node.

IGNITE_RUN_CPU_PER_NODE=1


# The number of Megabytes of RAM for each Apache Ignite node.

IGNITE_MEMORY_PER_NODE=2048


# The version ignite which will be run on nodes.

IGNITE_VERSION=1.6.0


# The hdfs directory which will be used for saving Apache Ignite disbributives.

IGNITE_RELEASES_DIR=/user/hongmei/ignite-yarn


#The directory which will be used for saving Apache Ignite distributives(copy .jar file to it).

IGNITE_WORKING_DIR=/user/hongmei/ignite/workdir


#The hdfs path to Apache Ignite config file.

#IGNITE_XML_CONFIG=/user/yarn/ignite/

IGNITE_XML_CONFIG=/user/hongmei/ignite/


#The hdfs path to libs which will be added to classpath.

IGNITE_USERS_LIBS=/user/hongmei/ignite/libs/


#The constraint on slave hosts.

#IGNITE_HOSTENAME_CONSTRAINT=


IGNITE_LOCAL_WORK_DIR=/u/hongmei/apache-ignite



Thank you very much!!!!

Hongmei





On Thu, Jun 9, 2016 at 9:43 AM, Nikolai Tikhonov <[hidden email]> wrote:
It seems that your ignite configuration invalid. Could you share ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties?

On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <[hidden email]> wrote:
Thank you very much Nikolai !

I found another issue regarding my Ignite YARN Integration:

I  run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following:
Then I log onto one of the container node and the log is as the following:

[hongmei@c5hdpe001 apache-ignite]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

WARNING: Use "yarn jar" to launch YARN applications.

16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/

16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2

16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29511

Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Submitted application. Application id: application_1464374946035_29511

Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Application application_1464374946035_29511 is RUNNING. 


Then I log onto one of the container node and the log is as the following: "It failed to start grid". 
Any good suggestion? Thank you very much!!!


class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
        at org.apache.ignite.Ignition.start(Ignition.java:350)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
        at org.apache.ignite.Ignition.start(Ignition.java:347)
        ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
        ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
        ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Hongmei


On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[hidden email]> wrote:
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));





Hongmei Zong Hongmei Zong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

Thank you very much Nikolay, I will give it a try! :)

Hongmei

On Thu, Jun 9, 2016 at 10:21 AM, Nikolay Tikhonov <[hidden email]> wrote:
You set wrong value to IGNITE_XML_CONFIG property. The property should contains path to ignite configuration file. For example 

IGNITE_XML_CONFIG=/u/hongmei/apache-ignite/config/default-config.xml

I think you can comment this line in property file and ignite will start with default configuration.

On Thu, Jun 9, 2016 at 5:08 PM, Hongmei Zong <[hidden email]> wrote:
Hi nikolai,

Thank you very much for prompt reply! 

I did not find the ignite-config.xml file under my ignite home directory(  /u/hongmei/apache-ignite/  ).

I find a "default-config.xml" at the path:  /u/hongmei/apache-ignite/config/default-config.xml

<?xml version="1.0" encoding="UTF-8"?>


<!--

  Licensed to the Apache Software Foundation (ASF) under one or more

  contributor license agreements.  See the NOTICE file distributed with

  this work for additional information regarding copyright ownership.

  The ASF licenses this file to You under the Apache License, Version 2.0

  (the "License"); you may not use this file except in compliance with

  the License.  You may obtain a copy of the License at


       http://www.apache.org/licenses/LICENSE-2.0


  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License.

-->


<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xsi:schemaLocation="

       http://www.springframework.org/schema/beans

       http://www.springframework.org/schema/beans/spring-beans.xsd">

    <!--

        Alter configuration below as needed.

    -->

        <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

                <property name="discoverySpi">

                        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">

                                <property name="ipFinder">

                                        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">

                                                <property name="addresses">

                                                        <list>

                                                                <!--

                                                                 Explicitly specifying address of a local node to let it start

                                                                and operate normally even if there is no more nodes in the cluster.

                                                                You can also optionally specify an individual port or port range.

                                                                -->

                                                                <!--

                                                                <value>1.2.3.4</value>

                                                                -->

                                                                <!-- 

                                                                        IP Address and optional port range of a remote node.

                                                                        You can also optionally specify an individual port and don't set

                                                                        the port range at all.                                  

                                                                -->

                                                                <value>c5hdpe001.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe002.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe003.c5.runwaynine.com:47500..47509</value>


                                                        </list>

                                                </property>

                                        </bean>

                                </property>

                        </bean>

                </property>

        </bean>

</beans>



The "cluster.properties" at the path:  /u/hongmei/apache-ignite/config/cluster.properties

# The number of nodes in the cluster.

IGNITE_NODE_COUNT=2


# The number of CPU Cores for each Apache Ignite node.

IGNITE_RUN_CPU_PER_NODE=1


# The number of Megabytes of RAM for each Apache Ignite node.

IGNITE_MEMORY_PER_NODE=2048


# The version ignite which will be run on nodes.

IGNITE_VERSION=1.6.0


# The hdfs directory which will be used for saving Apache Ignite disbributives.

IGNITE_RELEASES_DIR=/user/hongmei/ignite-yarn


#The directory which will be used for saving Apache Ignite distributives(copy .jar file to it).

IGNITE_WORKING_DIR=/user/hongmei/ignite/workdir


#The hdfs path to Apache Ignite config file.

#IGNITE_XML_CONFIG=/user/yarn/ignite/

IGNITE_XML_CONFIG=/user/hongmei/ignite/


#The hdfs path to libs which will be added to classpath.

IGNITE_USERS_LIBS=/user/hongmei/ignite/libs/


#The constraint on slave hosts.

#IGNITE_HOSTENAME_CONSTRAINT=


IGNITE_LOCAL_WORK_DIR=/u/hongmei/apache-ignite



Thank you very much!!!!

Hongmei





On Thu, Jun 9, 2016 at 9:43 AM, Nikolai Tikhonov <[hidden email]> wrote:
It seems that your ignite configuration invalid. Could you share ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties?

On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <[hidden email]> wrote:
Thank you very much Nikolai !

I found another issue regarding my Ignite YARN Integration:

I  run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following:
Then I log onto one of the container node and the log is as the following:

[hongmei@c5hdpe001 apache-ignite]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

WARNING: Use "yarn jar" to launch YARN applications.

16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/

16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2

16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29511

Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Submitted application. Application id: application_1464374946035_29511

Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Application application_1464374946035_29511 is RUNNING. 


Then I log onto one of the container node and the log is as the following: "It failed to start grid". 
Any good suggestion? Thank you very much!!!


class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
        at org.apache.ignite.Ignition.start(Ignition.java:350)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
        at org.apache.ignite.Ignition.start(Ignition.java:347)
        ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
        ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
        ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Hongmei


On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[hidden email]> wrote:
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));






Hongmei Zong Hongmei Zong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

Hi Nikolay,

I tried to comment the line IGNITE_XML_CONFIG= ,but the Ignite YARN failed.

[hongmei@c5hdpe001 config]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

WARNING: Use "yarn jar" to launch YARN applications.

16/06/09 10:31:43 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/

16/06/09 10:31:47 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2

16/06/09 10:31:49 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29710

Jun 09, 2016 10:31:49 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Submitted application. Application id: application_1464374946035_29710

Jun 09, 2016 10:31:59 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Application application_1464374946035_29710 is FAILED.


On Thu, Jun 9, 2016 at 10:23 AM, Hongmei Zong <[hidden email]> wrote:
Thank you very much Nikolay, I will give it a try! :)

Hongmei

On Thu, Jun 9, 2016 at 10:21 AM, Nikolay Tikhonov <[hidden email]> wrote:
You set wrong value to IGNITE_XML_CONFIG property. The property should contains path to ignite configuration file. For example 

IGNITE_XML_CONFIG=/u/hongmei/apache-ignite/config/default-config.xml

I think you can comment this line in property file and ignite will start with default configuration.

On Thu, Jun 9, 2016 at 5:08 PM, Hongmei Zong <[hidden email]> wrote:
Hi nikolai,

Thank you very much for prompt reply! 

I did not find the ignite-config.xml file under my ignite home directory(  /u/hongmei/apache-ignite/  ).

I find a "default-config.xml" at the path:  /u/hongmei/apache-ignite/config/default-config.xml

<?xml version="1.0" encoding="UTF-8"?>


<!--

  Licensed to the Apache Software Foundation (ASF) under one or more

  contributor license agreements.  See the NOTICE file distributed with

  this work for additional information regarding copyright ownership.

  The ASF licenses this file to You under the Apache License, Version 2.0

  (the "License"); you may not use this file except in compliance with

  the License.  You may obtain a copy of the License at


       http://www.apache.org/licenses/LICENSE-2.0


  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License.

-->


<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xsi:schemaLocation="

       http://www.springframework.org/schema/beans

       http://www.springframework.org/schema/beans/spring-beans.xsd">

    <!--

        Alter configuration below as needed.

    -->

        <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

                <property name="discoverySpi">

                        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">

                                <property name="ipFinder">

                                        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">

                                                <property name="addresses">

                                                        <list>

                                                                <!--

                                                                 Explicitly specifying address of a local node to let it start

                                                                and operate normally even if there is no more nodes in the cluster.

                                                                You can also optionally specify an individual port or port range.

                                                                -->

                                                                <!--

                                                                <value>1.2.3.4</value>

                                                                -->

                                                                <!-- 

                                                                        IP Address and optional port range of a remote node.

                                                                        You can also optionally specify an individual port and don't set

                                                                        the port range at all.                                  

                                                                -->

                                                                <value>c5hdpe001.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe002.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe003.c5.runwaynine.com:47500..47509</value>


                                                        </list>

                                                </property>

                                        </bean>

                                </property>

                        </bean>

                </property>

        </bean>

</beans>



The "cluster.properties" at the path:  /u/hongmei/apache-ignite/config/cluster.properties

# The number of nodes in the cluster.

IGNITE_NODE_COUNT=2


# The number of CPU Cores for each Apache Ignite node.

IGNITE_RUN_CPU_PER_NODE=1


# The number of Megabytes of RAM for each Apache Ignite node.

IGNITE_MEMORY_PER_NODE=2048


# The version ignite which will be run on nodes.

IGNITE_VERSION=1.6.0


# The hdfs directory which will be used for saving Apache Ignite disbributives.

IGNITE_RELEASES_DIR=/user/hongmei/ignite-yarn


#The directory which will be used for saving Apache Ignite distributives(copy .jar file to it).

IGNITE_WORKING_DIR=/user/hongmei/ignite/workdir


#The hdfs path to Apache Ignite config file.

#IGNITE_XML_CONFIG=/user/yarn/ignite/

IGNITE_XML_CONFIG=/user/hongmei/ignite/


#The hdfs path to libs which will be added to classpath.

IGNITE_USERS_LIBS=/user/hongmei/ignite/libs/


#The constraint on slave hosts.

#IGNITE_HOSTENAME_CONSTRAINT=


IGNITE_LOCAL_WORK_DIR=/u/hongmei/apache-ignite



Thank you very much!!!!

Hongmei





On Thu, Jun 9, 2016 at 9:43 AM, Nikolai Tikhonov <[hidden email]> wrote:
It seems that your ignite configuration invalid. Could you share ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties?

On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <[hidden email]> wrote:
Thank you very much Nikolai !

I found another issue regarding my Ignite YARN Integration:

I  run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following:
Then I log onto one of the container node and the log is as the following:

[hongmei@c5hdpe001 apache-ignite]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

WARNING: Use "yarn jar" to launch YARN applications.

16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/

16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2

16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29511

Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Submitted application. Application id: application_1464374946035_29511

Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Application application_1464374946035_29511 is RUNNING. 


Then I log onto one of the container node and the log is as the following: "It failed to start grid". 
Any good suggestion? Thank you very much!!!


class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
        at org.apache.ignite.Ignition.start(Ignition.java:350)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
        at org.apache.ignite.Ignition.start(Ignition.java:347)
        ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
        ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
        ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Hongmei


On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[hidden email]> wrote:
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));







Hongmei Zong Hongmei Zong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

In reply to this post by Nikolai Tikhonov
Hi Nikolay,

I tried to comment the line IGNITE_XML_CONFIG= ,but the Ignite YARN failed.

[hongmei@c5hdpe001 config]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

WARNING: Use "yarn jar" to launch YARN applications.

16/06/09 10:31:43 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/

16/06/09 10:31:47 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2

16/06/09 10:31:49 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29710

Jun 09, 2016 10:31:49 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Submitted application. Application id: application_1464374946035_29710

Jun 09, 2016 10:31:59 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Application application_1464374946035_29710 is FAILED.


On Thu, Jun 9, 2016 at 10:21 AM, Nikolay Tikhonov <[hidden email]> wrote:
You set wrong value to IGNITE_XML_CONFIG property. The property should contains path to ignite configuration file. For example 

IGNITE_XML_CONFIG=/u/hongmei/apache-ignite/config/default-config.xml

I think you can comment this line in property file and ignite will start with default configuration.

On Thu, Jun 9, 2016 at 5:08 PM, Hongmei Zong <[hidden email]> wrote:
Hi nikolai,

Thank you very much for prompt reply! 

I did not find the ignite-config.xml file under my ignite home directory(  /u/hongmei/apache-ignite/  ).

I find a "default-config.xml" at the path:  /u/hongmei/apache-ignite/config/default-config.xml

<?xml version="1.0" encoding="UTF-8"?>


<!--

  Licensed to the Apache Software Foundation (ASF) under one or more

  contributor license agreements.  See the NOTICE file distributed with

  this work for additional information regarding copyright ownership.

  The ASF licenses this file to You under the Apache License, Version 2.0

  (the "License"); you may not use this file except in compliance with

  the License.  You may obtain a copy of the License at


       http://www.apache.org/licenses/LICENSE-2.0


  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License.

-->


<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xsi:schemaLocation="

       http://www.springframework.org/schema/beans

       http://www.springframework.org/schema/beans/spring-beans.xsd">

    <!--

        Alter configuration below as needed.

    -->

        <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

                <property name="discoverySpi">

                        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">

                                <property name="ipFinder">

                                        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">

                                                <property name="addresses">

                                                        <list>

                                                                <!--

                                                                 Explicitly specifying address of a local node to let it start

                                                                and operate normally even if there is no more nodes in the cluster.

                                                                You can also optionally specify an individual port or port range.

                                                                -->

                                                                <!--

                                                                <value>1.2.3.4</value>

                                                                -->

                                                                <!-- 

                                                                        IP Address and optional port range of a remote node.

                                                                        You can also optionally specify an individual port and don't set

                                                                        the port range at all.                                  

                                                                -->

                                                                <value>c5hdpe001.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe002.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe003.c5.runwaynine.com:47500..47509</value>


                                                        </list>

                                                </property>

                                        </bean>

                                </property>

                        </bean>

                </property>

        </bean>

</beans>



The "cluster.properties" at the path:  /u/hongmei/apache-ignite/config/cluster.properties

# The number of nodes in the cluster.

IGNITE_NODE_COUNT=2


# The number of CPU Cores for each Apache Ignite node.

IGNITE_RUN_CPU_PER_NODE=1


# The number of Megabytes of RAM for each Apache Ignite node.

IGNITE_MEMORY_PER_NODE=2048


# The version ignite which will be run on nodes.

IGNITE_VERSION=1.6.0


# The hdfs directory which will be used for saving Apache Ignite disbributives.

IGNITE_RELEASES_DIR=/user/hongmei/ignite-yarn


#The directory which will be used for saving Apache Ignite distributives(copy .jar file to it).

IGNITE_WORKING_DIR=/user/hongmei/ignite/workdir


#The hdfs path to Apache Ignite config file.

#IGNITE_XML_CONFIG=/user/yarn/ignite/

IGNITE_XML_CONFIG=/user/hongmei/ignite/


#The hdfs path to libs which will be added to classpath.

IGNITE_USERS_LIBS=/user/hongmei/ignite/libs/


#The constraint on slave hosts.

#IGNITE_HOSTENAME_CONSTRAINT=


IGNITE_LOCAL_WORK_DIR=/u/hongmei/apache-ignite



Thank you very much!!!!

Hongmei





On Thu, Jun 9, 2016 at 9:43 AM, Nikolai Tikhonov <[hidden email]> wrote:
It seems that your ignite configuration invalid. Could you share ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties?

On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <[hidden email]> wrote:
Thank you very much Nikolai !

I found another issue regarding my Ignite YARN Integration:

I  run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following:
Then I log onto one of the container node and the log is as the following:

[hongmei@c5hdpe001 apache-ignite]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

WARNING: Use "yarn jar" to launch YARN applications.

16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/

16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2

16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29511

Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Submitted application. Application id: application_1464374946035_29511

Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Application application_1464374946035_29511 is RUNNING. 


Then I log onto one of the container node and the log is as the following: "It failed to start grid". 
Any good suggestion? Thank you very much!!!


class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
        at org.apache.ignite.Ignition.start(Ignition.java:350)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
        at org.apache.ignite.Ignition.start(Ignition.java:347)
        ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
        ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
        ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Hongmei


On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[hidden email]> wrote:
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));






Hongmei Zong Hongmei Zong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

In reply to this post by Nikolai Tikhonov
Hi Nikolay,

After I changed the value of IGNITE_XML_CONFIG=/user/hongmei/ignite/config/  (a HDFS path). Ignite RARN is running now. I use the Hadoop UI console to check the log of the application, the attached is the stderr log information about containers:

It looks like that the containers are allocated and then completed! The stderr log is very long and the container ID from XXXXX01_0000001 to XXXXXX01_013582.  Finally all these containers are completed.

I have no idea, is there anything not right?

There is no information in stdout log.

Thank you!

Hongmei

Logs for container_e24_1464374946035_29722_01_000001


SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.4.0-3485/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_29722/filecache/10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/06/09 10:48:16 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
16/06/09 10:48:16 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Application master registered.
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000002.
16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000003.
16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000002. State: COMPLETE.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000003. State: COMPLETE.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : c5hdp109.c5.runwaynine.com:45454
16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : c5hdp105.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000004.
16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000005.
16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000006. State: COMPLETE.
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000007. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000004. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000005. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000008.
16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : c5hdp112.c5.runwaynine.com:45454
16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : c5hdp114.c5.runwaynine.com:45454
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000009.
16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000012. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000010. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000011. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000013. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000008. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000009. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000014.
16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp107.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp110.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp101.c5.runwaynine.com:45454
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000015.
16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000020. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000019. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000018. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000017. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000016. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000021. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000014. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000015. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:34 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated

On Thu, Jun 9, 2016 at 10:21 AM, Nikolay Tikhonov <[hidden email]> wrote:
You set wrong value to IGNITE_XML_CONFIG property. The property should contains path to ignite configuration file. For example 

IGNITE_XML_CONFIG=/u/hongmei/apache-ignite/config/default-config.xml

I think you can comment this line in property file and ignite will start with default configuration.

On Thu, Jun 9, 2016 at 5:08 PM, Hongmei Zong <[hidden email]> wrote:
Hi nikolai,

Thank you very much for prompt reply! 

I did not find the ignite-config.xml file under my ignite home directory(  /u/hongmei/apache-ignite/  ).

I find a "default-config.xml" at the path:  /u/hongmei/apache-ignite/config/default-config.xml

<?xml version="1.0" encoding="UTF-8"?>


<!--

  Licensed to the Apache Software Foundation (ASF) under one or more

  contributor license agreements.  See the NOTICE file distributed with

  this work for additional information regarding copyright ownership.

  The ASF licenses this file to You under the Apache License, Version 2.0

  (the "License"); you may not use this file except in compliance with

  the License.  You may obtain a copy of the License at


       http://www.apache.org/licenses/LICENSE-2.0


  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License.

-->


<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xsi:schemaLocation="

       http://www.springframework.org/schema/beans

       http://www.springframework.org/schema/beans/spring-beans.xsd">

    <!--

        Alter configuration below as needed.

    -->

        <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

                <property name="discoverySpi">

                        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">

                                <property name="ipFinder">

                                        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">

                                                <property name="addresses">

                                                        <list>

                                                                <!--

                                                                 Explicitly specifying address of a local node to let it start

                                                                and operate normally even if there is no more nodes in the cluster.

                                                                You can also optionally specify an individual port or port range.

                                                                -->

                                                                <!--

                                                                <value>1.2.3.4</value>

                                                                -->

                                                                <!-- 

                                                                        IP Address and optional port range of a remote node.

                                                                        You can also optionally specify an individual port and don't set

                                                                        the port range at all.                                  

                                                                -->

                                                                <value>c5hdpe001.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe002.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe003.c5.runwaynine.com:47500..47509</value>


                                                        </list>

                                                </property>

                                        </bean>

                                </property>

                        </bean>

                </property>

        </bean>

</beans>



The "cluster.properties" at the path:  /u/hongmei/apache-ignite/config/cluster.properties

# The number of nodes in the cluster.

IGNITE_NODE_COUNT=2


# The number of CPU Cores for each Apache Ignite node.

IGNITE_RUN_CPU_PER_NODE=1


# The number of Megabytes of RAM for each Apache Ignite node.

IGNITE_MEMORY_PER_NODE=2048


# The version ignite which will be run on nodes.

IGNITE_VERSION=1.6.0


# The hdfs directory which will be used for saving Apache Ignite disbributives.

IGNITE_RELEASES_DIR=/user/hongmei/ignite-yarn


#The directory which will be used for saving Apache Ignite distributives(copy .jar file to it).

IGNITE_WORKING_DIR=/user/hongmei/ignite/workdir


#The hdfs path to Apache Ignite config file.

#IGNITE_XML_CONFIG=/user/yarn/ignite/

IGNITE_XML_CONFIG=/user/hongmei/ignite/


#The hdfs path to libs which will be added to classpath.

IGNITE_USERS_LIBS=/user/hongmei/ignite/libs/


#The constraint on slave hosts.

#IGNITE_HOSTENAME_CONSTRAINT=


IGNITE_LOCAL_WORK_DIR=/u/hongmei/apache-ignite



Thank you very much!!!!

Hongmei





On Thu, Jun 9, 2016 at 9:43 AM, Nikolai Tikhonov <[hidden email]> wrote:
It seems that your ignite configuration invalid. Could you share ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties?

On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <[hidden email]> wrote:
Thank you very much Nikolai !

I found another issue regarding my Ignite YARN Integration:

I  run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following:
Then I log onto one of the container node and the log is as the following:

[hongmei@c5hdpe001 apache-ignite]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

WARNING: Use "yarn jar" to launch YARN applications.

16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/

16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2

16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29511

Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Submitted application. Application id: application_1464374946035_29511

Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Application application_1464374946035_29511 is RUNNING. 


Then I log onto one of the container node and the log is as the following: "It failed to start grid". 
Any good suggestion? Thank you very much!!!


class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
        at org.apache.ignite.Ignition.start(Ignition.java:350)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
        at org.apache.ignite.Ignition.start(Ignition.java:347)
        ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
        ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
        ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Hongmei


On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[hidden email]> wrote:
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));






Nikolai Tikhonov-2 Nikolai Tikhonov-2
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

Hi Hongmei Zong!

Could you show logs from other containers (container_e24_1464374946035_29722_01_000015) which was completed?

On Thu, Jun 9, 2016 at 6:29 PM, Hongmei Zong <[hidden email]> wrote:
Hi Nikolay,

After I changed the value of IGNITE_XML_CONFIG=/user/hongmei/ignite/config/  (a HDFS path). Ignite RARN is running now. I use the Hadoop UI console to check the log of the application, the attached is the stderr log information about containers:

It looks like that the containers are allocated and then completed! The stderr log is very long and the container ID from XXXXX01_0000001 to XXXXXX01_013582.  Finally all these containers are completed.

I have no idea, is there anything not right?

There is no information in stdout log.

Thank you!

Hongmei

Logs for container_e24_1464374946035_29722_01_000001


SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.4.0-3485/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_29722/filecache/10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/06/09 10:48:16 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
16/06/09 10:48:16 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Application master registered.
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000002.
16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000003.
16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000002. State: COMPLETE.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000003. State: COMPLETE.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : c5hdp109.c5.runwaynine.com:45454
16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : c5hdp105.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000004.
16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000005.
16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000006. State: COMPLETE.
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000007. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000004. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000005. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000008.
16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : c5hdp112.c5.runwaynine.com:45454
16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : c5hdp114.c5.runwaynine.com:45454
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000009.
16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000012. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000010. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000011. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000013. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000008. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000009. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000014.
16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp107.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp110.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp101.c5.runwaynine.com:45454
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000015.
16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000020. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000019. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000018. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000017. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000016. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000021. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000014. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000015. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:34 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated

On Thu, Jun 9, 2016 at 10:21 AM, Nikolay Tikhonov <[hidden email]> wrote:
You set wrong value to IGNITE_XML_CONFIG property. The property should contains path to ignite configuration file. For example 

IGNITE_XML_CONFIG=/u/hongmei/apache-ignite/config/default-config.xml

I think you can comment this line in property file and ignite will start with default configuration.

On Thu, Jun 9, 2016 at 5:08 PM, Hongmei Zong <[hidden email]> wrote:
Hi nikolai,

Thank you very much for prompt reply! 

I did not find the ignite-config.xml file under my ignite home directory(  /u/hongmei/apache-ignite/  ).

I find a "default-config.xml" at the path:  /u/hongmei/apache-ignite/config/default-config.xml

<?xml version="1.0" encoding="UTF-8"?>


<!--

  Licensed to the Apache Software Foundation (ASF) under one or more

  contributor license agreements.  See the NOTICE file distributed with

  this work for additional information regarding copyright ownership.

  The ASF licenses this file to You under the Apache License, Version 2.0

  (the "License"); you may not use this file except in compliance with

  the License.  You may obtain a copy of the License at


       http://www.apache.org/licenses/LICENSE-2.0


  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License.

-->


<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xsi:schemaLocation="

       http://www.springframework.org/schema/beans

       http://www.springframework.org/schema/beans/spring-beans.xsd">

    <!--

        Alter configuration below as needed.

    -->

        <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

                <property name="discoverySpi">

                        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">

                                <property name="ipFinder">

                                        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">

                                                <property name="addresses">

                                                        <list>

                                                                <!--

                                                                 Explicitly specifying address of a local node to let it start

                                                                and operate normally even if there is no more nodes in the cluster.

                                                                You can also optionally specify an individual port or port range.

                                                                -->

                                                                <!--

                                                                <value>1.2.3.4</value>

                                                                -->

                                                                <!-- 

                                                                        IP Address and optional port range of a remote node.

                                                                        You can also optionally specify an individual port and don't set

                                                                        the port range at all.                                  

                                                                -->

                                                                <value>c5hdpe001.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe002.c5.runwaynine.com:47500..47509</value>

                                                                <value>c5hdpe003.c5.runwaynine.com:47500..47509</value>


                                                        </list>

                                                </property>

                                        </bean>

                                </property>

                        </bean>

                </property>

        </bean>

</beans>



The "cluster.properties" at the path:  /u/hongmei/apache-ignite/config/cluster.properties

# The number of nodes in the cluster.

IGNITE_NODE_COUNT=2


# The number of CPU Cores for each Apache Ignite node.

IGNITE_RUN_CPU_PER_NODE=1


# The number of Megabytes of RAM for each Apache Ignite node.

IGNITE_MEMORY_PER_NODE=2048


# The version ignite which will be run on nodes.

IGNITE_VERSION=1.6.0


# The hdfs directory which will be used for saving Apache Ignite disbributives.

IGNITE_RELEASES_DIR=/user/hongmei/ignite-yarn


#The directory which will be used for saving Apache Ignite distributives(copy .jar file to it).

IGNITE_WORKING_DIR=/user/hongmei/ignite/workdir


#The hdfs path to Apache Ignite config file.

#IGNITE_XML_CONFIG=/user/yarn/ignite/

IGNITE_XML_CONFIG=/user/hongmei/ignite/


#The hdfs path to libs which will be added to classpath.

IGNITE_USERS_LIBS=/user/hongmei/ignite/libs/


#The constraint on slave hosts.

#IGNITE_HOSTENAME_CONSTRAINT=


IGNITE_LOCAL_WORK_DIR=/u/hongmei/apache-ignite



Thank you very much!!!!

Hongmei





On Thu, Jun 9, 2016 at 9:43 AM, Nikolai Tikhonov <[hidden email]> wrote:
It seems that your ignite configuration invalid. Could you share ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties?

On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <[hidden email]> wrote:
Thank you very much Nikolai !

I found another issue regarding my Ignite YARN Integration:

I  run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following:
Then I log onto one of the container node and the log is as the following:

[hongmei@c5hdpe001 apache-ignite]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties

WARNING: Use "yarn jar" to launch YARN applications.

16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/

16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2

16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29511

Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Submitted application. Application id: application_1464374946035_29511

Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main

INFO: Application application_1464374946035_29511 is RUNNING. 


Then I log onto one of the container node and the log is as the following: "It failed to start grid". 
Any good suggestion? Thank you very much!!!


class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
        at org.apache.ignite.Ignition.start(Ignition.java:350)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
        at org.apache.ignite.Ignition.start(Ignition.java:347)
        ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
        ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
        ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Hongmei


On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[hidden email]> wrote:
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));







Hongmei Zong Hongmei Zong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

Hi Nikolai,

Here is the log of one of the container:

Logs for container_e24_1464374946035_32114_01_000014



Showing 4096 bytes. Click here for full log
SpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
	at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
	at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
	at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
	at org.apache.ignite.Ignition.start(Ignition.java:347)
	... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
	at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
	... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
	at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
	at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
	at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
	at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
	... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Thank you very much!
Hongmei

On Jun 10, 2016, at 5:56 AM, Nikolai Tikhonov <[hidden email]> wrote:

Hi Hongmei Zong!

Could you show logs from other containers (container_e24_1464374946035_29722_01_000015) which was completed?

On Thu, Jun 9, 2016 at 6:29 PM, Hongmei Zong <[hidden email]> wrote:
Hi Nikolay,

After I changed the value of IGNITE_XML_CONFIG=/user/hongmei/ignite/config/  (a HDFS path). Ignite RARN is running now. I use the Hadoop UI console to check the log of the application, the attached is the stderr log information about containers:

It looks like that the containers are allocated and then completed! The stderr log is very long and the container ID from XXXXX01_0000001 to XXXXXX01_013582.  Finally all these containers are completed.

I have no idea, is there anything not right?

There is no information in stdout log.

Thank you!

Hongmei

Logs for container_e24_1464374946035_29722_01_000001


SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.4.0-3485/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_29722/filecache/10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/06/09 10:48:16 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
16/06/09 10:48:16 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Application master registered.
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000002.
16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000003.
16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000002. State: COMPLETE.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000003. State: COMPLETE.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : c5hdp109.c5.runwaynine.com:45454
16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : c5hdp105.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000004.
16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000005.
16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000006. State: COMPLETE.
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000007. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000004. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000005. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000008.
16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : c5hdp112.c5.runwaynine.com:45454
16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : c5hdp114.c5.runwaynine.com:45454
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000009.
16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000012. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000010. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000011. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000013. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000008. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000009. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000014.
16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp107.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp110.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp101.c5.runwaynine.com:45454
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000015.
16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000020. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000019. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000018. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000017. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000016. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000021. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000014. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000015. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:34 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated

On Thu, Jun 9, 2016 at 10:21 AM, Nikolay Tikhonov <[hidden email]> wrote:
You set wrong value to IGNITE_XML_CONFIG property. The property should contains path to ignite configuration file. For example 

IGNITE_XML_CONFIG=/u/hongmei/apache-ignite/config/default-config.xml

I think you can comment this line in property file and ignite will start with default configuration.

On Thu, Jun 9, 2016 at 5:08 PM, Hongmei Zong <[hidden email]> wrote:
Hi nikolai,

Thank you very much for prompt reply! 

I did not find the ignite-config.xml file under my ignite home directory(  /u/hongmei/apache-ignite/  ).

I find a "default-config.xml" at the path:  /u/hongmei/apache-ignite/config/default-config.xml

<?xml version="1.0" encoding="UTF-8"?>

<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at


  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

       xsi:schemaLocation="
    <!--
        Alter configuration below as needed.
    -->
        <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
                <property name="discoverySpi">
                        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                                <property name="ipFinder">
                                        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                                                <property name="addresses">
                                                        <list>
                                                                <!--
                                                                 Explicitly specifying address of a local node to let it start
                                                                and operate normally even if there is no more nodes in the cluster.
                                                                You can also optionally specify an individual port or port range.
                                                                -->
                                                                <!--
                                                                <value>1.2.3.4</value>
                                                                -->
                                                                <!-- 
                                                                        IP Address and optional port range of a remote node.
                                                                        You can also optionally specify an individual port and don't set
                                                                        the port range at all.                                  
                                                                -->
                                                                <value>c5hdpe001.c5.runwaynine.com:47500..47509</value>
                                                                <value>c5hdpe002.c5.runwaynine.com:47500..47509</value>
                                                                <value>c5hdpe003.c5.runwaynine.com:47500..47509</value>

                                                        </list>
                                                </property>
                                        </bean>
                                </property>
                        </bean>
                </property>
        </bean>

</beans>


The "cluster.properties" at the path:  /u/hongmei/apache-ignite/config/cluster.properties

# The number of nodes in the cluster.
IGNITE_NODE_COUNT=2

# The number of CPU Cores for each Apache Ignite node.
IGNITE_RUN_CPU_PER_NODE=1

# The number of Megabytes of RAM for each Apache Ignite node.
IGNITE_MEMORY_PER_NODE=2048

# The version ignite which will be run on nodes.
IGNITE_VERSION=1.6.0

# The hdfs directory which will be used for saving Apache Ignite disbributives.
IGNITE_RELEASES_DIR=/user/hongmei/ignite-yarn

#The directory which will be used for saving Apache Ignite distributives(copy .jar file to it).
IGNITE_WORKING_DIR=/user/hongmei/ignite/workdir

#The hdfs path to Apache Ignite config file.
#IGNITE_XML_CONFIG=/user/yarn/ignite/
IGNITE_XML_CONFIG=/user/hongmei/ignite/

#The hdfs path to libs which will be added to classpath.
IGNITE_USERS_LIBS=/user/hongmei/ignite/libs/

#The constraint on slave hosts.
#IGNITE_HOSTENAME_CONSTRAINT=

IGNITE_LOCAL_WORK_DIR=/u/hongmei/apache-ignite


Thank you very much!!!!
Hongmei




On Thu, Jun 9, 2016 at 9:43 AM, Nikolai Tikhonov <[hidden email]> wrote:
It seems that your ignite configuration invalid. Could you share ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties?

On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <[hidden email]> wrote:
Thank you very much Nikolai !

I found another issue regarding my Ignite YARN Integration:

I  run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following:
Then I log onto one of the container node and the log is as the following:

[hongmei@c5hdpe001 apache-ignite]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties
WARNING: Use "yarn jar" to launch YARN applications.
16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/
16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29511
Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main
INFO: Submitted application. Application id: application_1464374946035_29511
Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main
INFO: Application application_1464374946035_29511 is RUNNING. 


Then I log onto one of the container node and the log is as the following: "It failed to start grid". 
Any good suggestion? Thank you very much!!!


class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
        at org.apache.ignite.Ignition.start(Ignition.java:350)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
        at org.apache.ignite.Ignition.start(Ignition.java:347)
        ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
        ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
        ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Hongmei


On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[hidden email]> wrote:
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));








Nikolai Tikhonov Nikolai Tikhonov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

Could you download a configuration file (https://gist.github.com/ntikhonov/d2a2ede2faca7b533dc643e0da475959) and put it to
/user/hongmei/ignite/config/ignite-default-config.xml? Also change IGNITE_XML_CONFIG from /user/hongmei/ignite/config/ to /user/hongmei/ignite/config/ignite-default-config.xml?

IGNITE_XML_CONFIG=/user/hongmei/ignite/config/ignite-default-config.xml

On Fri, Jun 10, 2016 at 2:58 PM, Hongmei Zong <[hidden email]> wrote:
Hi Nikolai,

Here is the log of one of the container:

Logs for container_e24_1464374946035_32114_01_000014



Showing 4096 bytes. Click here for full log
SpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
	at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
	at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
	at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
	at org.apache.ignite.Ignition.start(Ignition.java:347)
	... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
	at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
	... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
	at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
	at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
	at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
	at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
	... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Thank you very much!
Hongmei

On Jun 10, 2016, at 5:56 AM, Nikolai Tikhonov <[hidden email]> wrote:

Hi Hongmei Zong!

Could you show logs from other containers (container_e24_1464374946035_29722_01_000015) which was completed?

On Thu, Jun 9, 2016 at 6:29 PM, Hongmei Zong <[hidden email]> wrote:
Hi Nikolay,

After I changed the value of IGNITE_XML_CONFIG=/user/hongmei/ignite/config/  (a HDFS path). Ignite RARN is running now. I use the Hadoop UI console to check the log of the application, the attached is the stderr log information about containers:

It looks like that the containers are allocated and then completed! The stderr log is very long and the container ID from XXXXX01_0000001 to XXXXXX01_013582.  Finally all these containers are completed.

I have no idea, is there anything not right?

There is no information in stdout log.

Thank you!

Hongmei

Logs for container_e24_1464374946035_29722_01_000001


SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.4.0-3485/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_29722/filecache/10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/06/09 10:48:16 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
16/06/09 10:48:16 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Application master registered.
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000002.
16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000003.
16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000002. State: COMPLETE.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000003. State: COMPLETE.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : c5hdp109.c5.runwaynine.com:45454
16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : c5hdp105.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000004.
16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000005.
16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000006. State: COMPLETE.
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000007. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000004. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000005. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000008.
16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : c5hdp112.c5.runwaynine.com:45454
16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : c5hdp114.c5.runwaynine.com:45454
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000009.
16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000012. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000010. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000011. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000013. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000008. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000009. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000014.
16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp107.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp110.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp101.c5.runwaynine.com:45454
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000015.
16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000020. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000019. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000018. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000017. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000016. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000021. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000014. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000015. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:34 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated

On Thu, Jun 9, 2016 at 10:21 AM, Nikolay Tikhonov <[hidden email]> wrote:
You set wrong value to IGNITE_XML_CONFIG property. The property should contains path to ignite configuration file. For example 

IGNITE_XML_CONFIG=/u/hongmei/apache-ignite/config/default-config.xml

I think you can comment this line in property file and ignite will start with default configuration.

On Thu, Jun 9, 2016 at 5:08 PM, Hongmei Zong <[hidden email]> wrote:
Hi nikolai,

Thank you very much for prompt reply! 

I did not find the ignite-config.xml file under my ignite home directory(  /u/hongmei/apache-ignite/  ).

I find a "default-config.xml" at the path:  /u/hongmei/apache-ignite/config/default-config.xml

<?xml version="1.0" encoding="UTF-8"?>

<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at


  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

       xsi:schemaLocation="
    <!--
        Alter configuration below as needed.
    -->
        <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
                <property name="discoverySpi">
                        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                                <property name="ipFinder">
                                        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                                                <property name="addresses">
                                                        <list>
                                                                <!--
                                                                 Explicitly specifying address of a local node to let it start
                                                                and operate normally even if there is no more nodes in the cluster.
                                                                You can also optionally specify an individual port or port range.
                                                                -->
                                                                <!--
                                                                <value>1.2.3.4</value>
                                                                -->
                                                                <!-- 
                                                                        IP Address and optional port range of a remote node.
                                                                        You can also optionally specify an individual port and don't set
                                                                        the port range at all.                                  
                                                                -->
                                                                <value>c5hdpe001.c5.runwaynine.com:47500..47509</value>
                                                                <value>c5hdpe002.c5.runwaynine.com:47500..47509</value>
                                                                <value>c5hdpe003.c5.runwaynine.com:47500..47509</value>

                                                        </list>
                                                </property>
                                        </bean>
                                </property>
                        </bean>
                </property>
        </bean>

</beans>


The "cluster.properties" at the path:  /u/hongmei/apache-ignite/config/cluster.properties

# The number of nodes in the cluster.
IGNITE_NODE_COUNT=2

# The number of CPU Cores for each Apache Ignite node.
IGNITE_RUN_CPU_PER_NODE=1

# The number of Megabytes of RAM for each Apache Ignite node.
IGNITE_MEMORY_PER_NODE=2048

# The version ignite which will be run on nodes.
IGNITE_VERSION=1.6.0

# The hdfs directory which will be used for saving Apache Ignite disbributives.
IGNITE_RELEASES_DIR=/user/hongmei/ignite-yarn

#The directory which will be used for saving Apache Ignite distributives(copy .jar file to it).
IGNITE_WORKING_DIR=/user/hongmei/ignite/workdir

#The hdfs path to Apache Ignite config file.
#IGNITE_XML_CONFIG=/user/yarn/ignite/
IGNITE_XML_CONFIG=/user/hongmei/ignite/

#The hdfs path to libs which will be added to classpath.
IGNITE_USERS_LIBS=/user/hongmei/ignite/libs/

#The constraint on slave hosts.
#IGNITE_HOSTENAME_CONSTRAINT=

IGNITE_LOCAL_WORK_DIR=/u/hongmei/apache-ignite


Thank you very much!!!!
Hongmei




On Thu, Jun 9, 2016 at 9:43 AM, Nikolai Tikhonov <[hidden email]> wrote:
It seems that your ignite configuration invalid. Could you share ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties?

On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <[hidden email]> wrote:
Thank you very much Nikolai !

I found another issue regarding my Ignite YARN Integration:

I  run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following:
Then I log onto one of the container node and the log is as the following:

[hongmei@c5hdpe001 apache-ignite]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties
WARNING: Use "yarn jar" to launch YARN applications.
16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/
16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29511
Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main
INFO: Submitted application. Application id: application_1464374946035_29511
Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main
INFO: Application application_1464374946035_29511 is RUNNING. 


Then I log onto one of the container node and the log is as the following: "It failed to start grid". 
Any good suggestion? Thank you very much!!!


class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
        at org.apache.ignite.Ignition.start(Ignition.java:350)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
        at org.apache.ignite.Ignition.start(Ignition.java:347)
        ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
        ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
        ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Hongmei


On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[hidden email]> wrote:
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));









Hongmei Zong Hongmei Zong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

Hi Nikolay,


Thank you very much for your solution. I think the ignite started successfully with your updated version of configuration file!!! :)
I think so far ignite running ok on YARN, right?  Next I will try to Test ignite from Spark-shell. Should I follow the link below:


In Starting up the cluster, step 1-4, Since our spark clusters are alway running on YARN, and I do not think I need to start spark, right?
In Starting up the cluster, step 5 , do I need to start Ignite node using the following command? Since I have started a ignite YARN application and currently running.
 
  1. On each of the worker nodes cd to $IGNITE_HOME and start an Ignite node by running the following command:bin/ignite.sh

In Working with Spark shell section, I run the following command and Spark shell launched successfully.
/usr/bin/spark-shell --jars /u/hongmei/apache-ignite/libs/ignite-core-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-spark/ignite-spark-1.6.0.jar,/u/hongmei/apache-ignite/libs/cache-api-1.0.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/ignite-log4j-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/log4j-1.2.17.jar


Then I ran these three command: The Warning message as below: I think I need to put the ip addresses of ignite server node into the ignite-default-config.xml, am I right?
import org.apache.ignite.spark._
import org.apache.ignite.configuration._
val ic = new IgniteContext[Integer, Integer](sc, () => new IgniteConfiguration())

16/06/10 09:18:13 WARN TcpDiscoverySpi: IP finder returned empty addresses list. Please check IP finder configuration and make sure multicast works on your network. Will retry every 2 secs.

Thank you very much for all your help!!!!

Hongmei

Starting up the cluster


 

Here is the log from my console:

Logs for container_e24_1464374946035_32224_01_000001




SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.4.0-3485/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/disk/09/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32224/filecache/10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/06/10 08:43:48 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
16/06/10 08:43:48 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
Jun 10, 2016 8:43:48 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Application master registered.
Jun 10, 2016 8:43:48 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 10, 2016 8:43:48 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/06/10 08:43:49 INFO impl.AMRMClientImpl: Received new token for : c5hdp105.c5.runwaynine.com:45454
16/06/10 08:43:49 INFO impl.AMRMClientImpl: Received new token for : c5hdp112.c5.runwaynine.com:45454
Jun 10, 2016 8:43:49 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 10, 2016 8:43:49 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 10, 2016 8:43:49 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_32224_01_000002.
16/06/10 08:43:49 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp105.c5.runwaynine.com:45454
Jun 10, 2016 8:43:49 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_32224_01_000003.
16/06/10 08:43:49 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp112.c5.runwaynine.com:45454
16/06/10 08:43:50 INFO impl.AMRMClientImpl: Received new token for : c5hdp114.c5.runwaynine.com:45454
16/06/10 08:43:50 INFO impl.AMRMClientImpl: Received new token for : c5hdp115.c5.runwaynine.com:45454
Jun 10, 2016 8:43:50 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_32224_01_000006. State: COMPLETE.
Jun 10, 2016 8:43:50 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_32224_01_000007. State: COMPLETE.
Jun 10, 2016 8:43:50 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_32224_01_000004. State: COMPLETE.
Jun 10, 2016 8:43:50 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_32224_01_000005. State: COMPLETE.

Logs for container_e24_1464374946035_32224_01_000002


[08:43:59]    __________  ________________ 
[08:43:59]   /  _/ ___/ |/ /  _/_  __/ __/ 
[08:43:59]  _/ // (7 7    // /  / / / _/   
[08:43:59] /___/\___/_/|_/___/ /_/ /___/  
[08:43:59] 
[08:43:59] ver. 1.5.22#20160519-sha1:f242f351
[08:43:59] 2016 Copyright(C) Apache Software Foundation
[08:43:59] 
[08:43:59] Ignite documentation: http://ignite.apache.org
[08:43:59] 
[08:43:59] Quiet mode.
[08:43:59]   ^-- Logging to file '/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32224/container_e24_1464374946035_32224_01_000002/ignite/gridgain-professional-fabric-1.5.22/work/log/ignite-8bf39a16.0.log'
[08:43:59]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
[08:43:59] 
[08:43:59] OS: Linux 2.6.32-573.18.1.el6.x86_64 amd64
[08:43:59] VM information: Java(TM) SE Runtime Environment 1.7.0_45-b18 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 24.45-b08
[08:44:00] Configured plugins:
[08:44:00]   ^-- None
[08:44:00] 
[08:44:00] Security status [authentication=off, tls/ssl=off]
[08:44:01] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[08:44:01] 
[08:44:01] Ignite node started OK (id=8bf39a16)
[08:44:01] Topology snapshot [ver=1, servers=1, clients=0, CPUs=24, heap=2.0GB]
[08:44:03] Topology snapshot [ver=2, servers=2, clients=0, CPUs=48, heap=4.0GB]


On Jun 10, 2016, at 8:20 AM, Nikolay Tikhonov <[hidden email]> wrote:

Could you download a configuration file (https://gist.github.com/ntikhonov/d2a2ede2faca7b533dc643e0da475959) and put it to
/user/hongmei/ignite/config/ignite-default-config.xml? Also change IGNITE_XML_CONFIG from /user/hongmei/ignite/config/ to /user/hongmei/ignite/config/ignite-default-config.xml?

IGNITE_XML_CONFIG=/user/hongmei/ignite/config/ignite-default-config.xml

On Fri, Jun 10, 2016 at 2:58 PM, Hongmei Zong <[hidden email]> wrote:
Hi Nikolai,

Here is the log of one of the container:

Logs for container_e24_1464374946035_32114_01_000014



Showing 4096 bytes. Click here for full log
SpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
	at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
	at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
	at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
	at org.apache.ignite.Ignition.start(Ignition.java:347)
	... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
	at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
	... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
	at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
	at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
	at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
	at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
	... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Thank you very much!
Hongmei

On Jun 10, 2016, at 5:56 AM, Nikolai Tikhonov <[hidden email]> wrote:

Hi Hongmei Zong!

Could you show logs from other containers (container_e24_1464374946035_29722_01_000015) which was completed?

On Thu, Jun 9, 2016 at 6:29 PM, Hongmei Zong <[hidden email]> wrote:
Hi Nikolay,

After I changed the value of IGNITE_XML_CONFIG=/user/hongmei/ignite/config/  (a HDFS path). Ignite RARN is running now. I use the Hadoop UI console to check the log of the application, the attached is the stderr log information about containers:

It looks like that the containers are allocated and then completed! The stderr log is very long and the container ID from XXXXX01_0000001 to XXXXXX01_013582.  Finally all these containers are completed.

I have no idea, is there anything not right?

There is no information in stdout log.

Thank you!

Hongmei

Logs for container_e24_1464374946035_29722_01_000001


SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.4.0-3485/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_29722/filecache/10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/06/09 10:48:16 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
16/06/09 10:48:16 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Application master registered.
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000002.
16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000003.
16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000002. State: COMPLETE.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000003. State: COMPLETE.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : c5hdp109.c5.runwaynine.com:45454
16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : c5hdp105.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000004.
16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000005.
16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000006. State: COMPLETE.
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000007. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000004. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000005. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000008.
16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : c5hdp112.c5.runwaynine.com:45454
16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : c5hdp114.c5.runwaynine.com:45454
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000009.
16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000012. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000010. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000011. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000013. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000008. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000009. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000014.
16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp107.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp110.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp101.c5.runwaynine.com:45454
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000015.
16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000020. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000019. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000018. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000017. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000016. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000021. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000014. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000015. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:34 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated

On Thu, Jun 9, 2016 at 10:21 AM, Nikolay Tikhonov <[hidden email]> wrote:
You set wrong value to IGNITE_XML_CONFIG property. The property should contains path to ignite configuration file. For example 

IGNITE_XML_CONFIG=/u/hongmei/apache-ignite/config/default-config.xml

I think you can comment this line in property file and ignite will start with default configuration.

On Thu, Jun 9, 2016 at 5:08 PM, Hongmei Zong <[hidden email]> wrote:
Hi nikolai,

Thank you very much for prompt reply! 

I did not find the ignite-config.xml file under my ignite home directory(  /u/hongmei/apache-ignite/  ).

I find a "default-config.xml" at the path:  /u/hongmei/apache-ignite/config/default-config.xml

<?xml version="1.0" encoding="UTF-8"?>

<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at


  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

       xsi:schemaLocation="
    <!--
        Alter configuration below as needed.
    -->
        <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
                <property name="discoverySpi">
                        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                                <property name="ipFinder">
                                        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                                                <property name="addresses">
                                                        <list>
                                                                <!--
                                                                 Explicitly specifying address of a local node to let it start
                                                                and operate normally even if there is no more nodes in the cluster.
                                                                You can also optionally specify an individual port or port range.
                                                                -->
                                                                <!--
                                                                <value>1.2.3.4</value>
                                                                -->
                                                                <!-- 
                                                                        IP Address and optional port range of a remote node.
                                                                        You can also optionally specify an individual port and don't set
                                                                        the port range at all.                                  
                                                                -->
                                                                <value>c5hdpe001.c5.runwaynine.com:47500..47509</value>
                                                                <value>c5hdpe002.c5.runwaynine.com:47500..47509</value>
                                                                <value>c5hdpe003.c5.runwaynine.com:47500..47509</value>

                                                        </list>
                                                </property>
                                        </bean>
                                </property>
                        </bean>
                </property>
        </bean>

</beans>


The "cluster.properties" at the path:  /u/hongmei/apache-ignite/config/cluster.properties

# The number of nodes in the cluster.
IGNITE_NODE_COUNT=2

# The number of CPU Cores for each Apache Ignite node.
IGNITE_RUN_CPU_PER_NODE=1

# The number of Megabytes of RAM for each Apache Ignite node.
IGNITE_MEMORY_PER_NODE=2048

# The version ignite which will be run on nodes.
IGNITE_VERSION=1.6.0

# The hdfs directory which will be used for saving Apache Ignite disbributives.
IGNITE_RELEASES_DIR=/user/hongmei/ignite-yarn

#The directory which will be used for saving Apache Ignite distributives(copy .jar file to it).
IGNITE_WORKING_DIR=/user/hongmei/ignite/workdir

#The hdfs path to Apache Ignite config file.
#IGNITE_XML_CONFIG=/user/yarn/ignite/
IGNITE_XML_CONFIG=/user/hongmei/ignite/

#The hdfs path to libs which will be added to classpath.
IGNITE_USERS_LIBS=/user/hongmei/ignite/libs/

#The constraint on slave hosts.
#IGNITE_HOSTENAME_CONSTRAINT=

IGNITE_LOCAL_WORK_DIR=/u/hongmei/apache-ignite


Thank you very much!!!!
Hongmei




On Thu, Jun 9, 2016 at 9:43 AM, Nikolai Tikhonov <[hidden email]> wrote:
It seems that your ignite configuration invalid. Could you share ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties?

On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <[hidden email]> wrote:
Thank you very much Nikolai !

I found another issue regarding my Ignite YARN Integration:

I  run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following:
Then I log onto one of the container node and the log is as the following:

[hongmei@c5hdpe001 apache-ignite]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties
WARNING: Use "yarn jar" to launch YARN applications.
16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/
16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29511
Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main
INFO: Submitted application. Application id: application_1464374946035_29511
Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main
INFO: Application application_1464374946035_29511 is RUNNING. 


Then I log onto one of the container node and the log is as the following: "It failed to start grid". 
Any good suggestion? Thank you very much!!!


class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
        at org.apache.ignite.Ignition.start(Ignition.java:350)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
        at org.apache.ignite.Ignition.start(Ignition.java:347)
        ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
        ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
        ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Hongmei


On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[hidden email]> wrote:
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));










Hongmei Zong Hongmei Zong
Reply | Threaded
Open this post in threaded view
|

Re: Ignite for Spark on YARN Deployment

In reply to this post by Nikolai Tikhonov
Hi Nikolai,

I ran the command to start the Spark-shell and Spark started successfully Then I import two classes and create a new IgniteContext, I got the error as posted below.  Any good suggestions??? Thank you very much!

Hongmei

First run:

/usr/bin/spark-shell --jars /u/hongmei/apache-ignite/libs/ignite-core-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-spark/ignite-spark-1.6.0.jar,/u/hongmei/apache-ignite/libs/cache-api-1.0.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/ignite-log4j-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/log4j-1.2.17.jar --packages org.apache.ignite:ignite-spark:1.6.0,org.apache.ignite:ignite-spring:1.6.0

Then import and new a ic:

scala> import org.apache.ignite.spark._
import org.apache.ignite.spark._

scala> import org.apache.ignite.configuration._
import org.apache.ignite.configuration._

scala> val ic = new IgniteContext[Integer, Integer](sc, "config/ignite-default-config.xml")
16/06/10 10:32:17 INFO XmlBeanDefinitionReader: Loading XML bean definitions from URL [file:/u/hongmei/apache-ignite/config/ignite-default-config.xml]
16/06/10 10:32:18 INFO GenericApplicationContext: Refreshing org.springframework.context.support.GenericApplicationContext@3cfbcf6a: startup date [Fri Jun 10 10:32:18 EDT 2016]; root of context hierarchy
16/06/10 10:32:19 INFO IgniteKernal: 

>>>    __________  ________________  
>>>   /  _/ ___/ |/ /  _/_  __/ __/  
>>>  _/ // (7 7    // /  / / / _/    
>>> /___/\___/_/|_/___/ /_/ /___/   
>>> 
>>> ver. 1.6.0#20160518-sha1:0b22c45b
>>> 2016 Copyright(C) Apache Software Foundation
>>> 
>>> Ignite documentation: http://ignite.apache.org

16/06/10 10:32:19 INFO IgniteKernal: Config URL: n/a
16/06/10 10:32:19 INFO IgniteKernal: Daemon mode: off
16/06/10 10:32:19 INFO IgniteKernal: OS: Linux 2.6.32-573.26.1.el6.x86_64 amd64
16/06/10 10:32:19 INFO IgniteKernal: OS user: hongmei
16/06/10 10:32:19 INFO IgniteKernal: Language runtime: Scala ver. 2.10.4
16/06/10 10:32:19 INFO IgniteKernal: VM information: Java(TM) SE Runtime Environment 1.7.0_45-b18 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 24.45-b08
16/06/10 10:32:19 INFO IgniteKernal: VM total memory: 4.2GB
16/06/10 10:32:19 INFO IgniteKernal: Remote Management [restart: off, REST: on, JMX (remote: off)]
16/06/10 10:32:19 INFO IgniteKernal: IGNITE_HOME=/u/hongmei/apache-ignite
16/06/10 10:32:19 INFO IgniteKernal: VM arguments: [-Dhdp.version=2.3.4.0-3485, -Dscala.usejavacp=true, -Xms4500m, -Xmx4500m, -XX:MaxPermSize=1024m, -XX:PermSize=256m]
16/06/10 10:32:19 INFO IgniteKernal: Configured caches ['ignite-marshaller-sys-cache', 'ignite-sys-cache', 'ignite-atomics-sys-cache']

…….

Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1536)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:897)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1736)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1589)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1042)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:569)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:530)
at org.apache.ignite.Ignition.getOrStart(Ignition.java:414)
... 53 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:258)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:677)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1531)
... 60 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Join process timed out, did not receive response for join request (consider increasing 'joinTimeout' configuration property) [joinTimeout=60000, sock=Socket[addr=<a href="http://c5hdp112.c5.runwaynine.com/10.138.10.47,port=47500,localport=59822]]" class="">c5hdp112.c5.runwaynine.com/10.138.10.47,port=47500,localport=59822]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1335)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)


scala>



On Jun 10, 2016, at 8:20 AM, Nikolay Tikhonov <[hidden email]> wrote:

Could you download a configuration file (https://gist.github.com/ntikhonov/d2a2ede2faca7b533dc643e0da475959) and put it to
/user/hongmei/ignite/config/ignite-default-config.xml? Also change IGNITE_XML_CONFIG from /user/hongmei/ignite/config/ to /user/hongmei/ignite/config/ignite-default-config.xml?

IGNITE_XML_CONFIG=/user/hongmei/ignite/config/ignite-default-config.xml

On Fri, Jun 10, 2016 at 2:58 PM, Hongmei Zong <[hidden email]> wrote:
Hi Nikolai,

Here is the log of one of the container:

Logs for container_e24_1464374946035_32114_01_000014



Showing 4096 bytes. Click here for full log
SpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
	at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
	at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
	at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
	at org.apache.ignite.Ignition.start(Ignition.java:347)
	... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
	at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
	... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
	at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
	at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
	at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
	at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
	... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Thank you very much!
Hongmei

On Jun 10, 2016, at 5:56 AM, Nikolai Tikhonov <[hidden email]> wrote:

Hi Hongmei Zong!

Could you show logs from other containers (container_e24_1464374946035_29722_01_000015) which was completed?

On Thu, Jun 9, 2016 at 6:29 PM, Hongmei Zong <[hidden email]> wrote:
Hi Nikolay,

After I changed the value of IGNITE_XML_CONFIG=/user/hongmei/ignite/config/  (a HDFS path). Ignite RARN is running now. I use the Hadoop UI console to check the log of the application, the attached is the stderr log information about containers:

It looks like that the containers are allocated and then completed! The stderr log is very long and the container ID from XXXXX01_0000001 to XXXXXX01_013582.  Finally all these containers are completed.

I have no idea, is there anything not right?

There is no information in stdout log.

Thank you!

Hongmei

Logs for container_e24_1464374946035_29722_01_000001


SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.4.0-3485/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_29722/filecache/10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/06/09 10:48:16 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
16/06/09 10:48:16 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Application master registered.
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000002.
16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000003.
16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000002. State: COMPLETE.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000003. State: COMPLETE.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : c5hdp109.c5.runwaynine.com:45454
16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : c5hdp105.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000004.
16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000005.
16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000006. State: COMPLETE.
Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000007. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000004. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000005. State: COMPLETE.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000008.
16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : c5hdp112.c5.runwaynine.com:45454
16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : c5hdp114.c5.runwaynine.com:45454
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000009.
16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000012. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000010. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000011. State: COMPLETE.
Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000013. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000008. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000009. State: COMPLETE.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000014.
16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp108.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp107.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp110.c5.runwaynine.com:45454
16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : c5hdp101.c5.runwaynine.com:45454
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated
INFO: Launching container: container_e24_1464374946035_29722_01_000015.
16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp111.c5.runwaynine.com:45454
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000020. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000019. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000018. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000017. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000016. State: COMPLETE.
Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000021. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000014. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted
INFO: Container completed. Container id: container_e24_1464374946035_29722_01_000015. State: COMPLETE.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Jun 09, 2016 10:48:34 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated

On Thu, Jun 9, 2016 at 10:21 AM, Nikolay Tikhonov <[hidden email]> wrote:
You set wrong value to IGNITE_XML_CONFIG property. The property should contains path to ignite configuration file. For example 

IGNITE_XML_CONFIG=/u/hongmei/apache-ignite/config/default-config.xml

I think you can comment this line in property file and ignite will start with default configuration.

On Thu, Jun 9, 2016 at 5:08 PM, Hongmei Zong <[hidden email]> wrote:
Hi nikolai,

Thank you very much for prompt reply! 

I did not find the ignite-config.xml file under my ignite home directory(  /u/hongmei/apache-ignite/  ).

I find a "default-config.xml" at the path:  /u/hongmei/apache-ignite/config/default-config.xml

<?xml version="1.0" encoding="UTF-8"?>

<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at


  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

       xsi:schemaLocation="
    <!--
        Alter configuration below as needed.
    -->
        <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
                <property name="discoverySpi">
                        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                                <property name="ipFinder">
                                        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                                                <property name="addresses">
                                                        <list>
                                                                <!--
                                                                 Explicitly specifying address of a local node to let it start
                                                                and operate normally even if there is no more nodes in the cluster.
                                                                You can also optionally specify an individual port or port range.
                                                                -->
                                                                <!--
                                                                <value>1.2.3.4</value>
                                                                -->
                                                                <!-- 
                                                                        IP Address and optional port range of a remote node.
                                                                        You can also optionally specify an individual port and don't set
                                                                        the port range at all.                                  
                                                                -->
                                                                <value>c5hdpe001.c5.runwaynine.com:47500..47509</value>
                                                                <value>c5hdpe002.c5.runwaynine.com:47500..47509</value>
                                                                <value>c5hdpe003.c5.runwaynine.com:47500..47509</value>

                                                        </list>
                                                </property>
                                        </bean>
                                </property>
                        </bean>
                </property>
        </bean>

</beans>


The "cluster.properties" at the path:  /u/hongmei/apache-ignite/config/cluster.properties

# The number of nodes in the cluster.
IGNITE_NODE_COUNT=2

# The number of CPU Cores for each Apache Ignite node.
IGNITE_RUN_CPU_PER_NODE=1

# The number of Megabytes of RAM for each Apache Ignite node.
IGNITE_MEMORY_PER_NODE=2048

# The version ignite which will be run on nodes.
IGNITE_VERSION=1.6.0

# The hdfs directory which will be used for saving Apache Ignite disbributives.
IGNITE_RELEASES_DIR=/user/hongmei/ignite-yarn

#The directory which will be used for saving Apache Ignite distributives(copy .jar file to it).
IGNITE_WORKING_DIR=/user/hongmei/ignite/workdir

#The hdfs path to Apache Ignite config file.
#IGNITE_XML_CONFIG=/user/yarn/ignite/
IGNITE_XML_CONFIG=/user/hongmei/ignite/

#The hdfs path to libs which will be added to classpath.
IGNITE_USERS_LIBS=/user/hongmei/ignite/libs/

#The constraint on slave hosts.
#IGNITE_HOSTENAME_CONSTRAINT=

IGNITE_LOCAL_WORK_DIR=/u/hongmei/apache-ignite


Thank you very much!!!!
Hongmei




On Thu, Jun 9, 2016 at 9:43 AM, Nikolai Tikhonov <[hidden email]> wrote:
It seems that your ignite configuration invalid. Could you share ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties?

On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <[hidden email]> wrote:
Thank you very much Nikolai !

I found another issue regarding my Ignite YARN Integration:

I  run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following:
Then I log onto one of the container node and the log is as the following:

[hongmei@c5hdpe001 apache-ignite]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/config/cluster.properties
WARNING: Use "yarn jar" to launch YARN applications.
16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/
16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application application_1464374946035_29511
Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main
INFO: Submitted application. Application id: application_1464374946035_29511
Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main
INFO: Application application_1464374946035_29511 is RUNNING. 


Then I log onto one of the container node and the log is as the following: "It failed to start grid". 
Any good suggestion? Thank you very much!!!


class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
        at org.apache.ignite.Ignition.start(Ignition.java:350)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
        at org.apache.ignite.Ignition.start(Ignition.java:347)
        ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
        ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
        ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, err=Line 1 in XML document from URL [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.]


Hongmei


On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[hidden email]> wrote:
your_address1:47500..47510,your_address2:47500..47510 and your_address3:47500..47510 are the YARN master_host address, right?

No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will have the following configuration:

ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510""10.0.0.2:47500..47510"));










12