Apache Ignite - YARN integration

classic Classic list List threaded Threaded
4 messages Options
vijayendra bhati vijayendra bhati
Reply | Threaded
Open this post in threaded view
|

Apache Ignite - YARN integration

Hi,
I am trying to run Ignite nodes over YARN cluster by following the documentation given on YARN Deployment · Apache Ignite

I am using Cloudera supplied VM for initial installation purpose and have downloaded Apache Ignite version 1.5.0.

But I am getting error when I am trying to run below command -

[cloudera@quickstart ignite-yarn]$ hadoop jar /home/cloudera/vij/ignite_config/apache-ignite-fabric-1.5.0.final-bin/libs/optional/ignite-yarn/ignite-yarn-1.5.0.final.jar ./home/cloudera/vij/ignite_config/apache-ignite-fabric-1.5.0.final-bin/libs/optional/ignite-yarn/ignite-yarn-1.5.0.final.jar /home/cloudera/vij/ignite_config/cluster.properties
16/04/17 05:29:16 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
Exception in thread "main" java.lang.RuntimeException: Got unexpected response code. Response code: 404
    at org.apache.ignite.yarn.IgniteProvider.updateIgnite(IgniteProvider.java:240)
    at org.apache.ignite.yarn.IgniteProvider.getIgnite(IgniteProvider.java:93)
    at org.apache.ignite.yarn.IgniteYarnClient.getIgnite(IgniteYarnClient.java:169)
    at org.apache.ignite.yarn.IgniteYarnClient.main(IgniteYarnClient.java:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Can somebody help me what could be the issue ?Also I want to understand whats the benefit of running Ignite cluster over YARN.We could run Ignite nodes separately as well.One reason I could think is management of cluster become easy, as you dont need to manually start Ignite nodes on each node.

Regards,
Vijayendra Bhati
vijayendra bhati vijayendra bhati
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite - YARN integration

Still I have not been able to figure out how to resolve this issue but I think I can change my approach.
I can start Ignite node on the same node of AWS cluster on which YARN (Node manager) and HDFS (Data node) would be configured but I want to understand is it necessary to run Ignite Node as YARN job to achieve data locality while accessing Ignite data using Spark job ?
 
Regards,
Vij


On Sunday, April 17, 2016 6:11 PM, vijayendra bhati <[hidden email]> wrote:


Hi,
I am trying to run Ignite nodes over YARN cluster by following the documentation given on YARN Deployment · Apache Ignite

I am using Cloudera supplied VM for initial installation purpose and have downloaded Apache Ignite version 1.5.0.

But I am getting error when I am trying to run below command -

[cloudera@quickstart ignite-yarn]$ hadoop jar /home/cloudera/vij/ignite_config/apache-ignite-fabric-1.5.0.final-bin/libs/optional/ignite-yarn/ignite-yarn-1.5.0.final.jar ./home/cloudera/vij/ignite_config/apache-ignite-fabric-1.5.0.final-bin/libs/optional/ignite-yarn/ignite-yarn-1.5.0.final.jar /home/cloudera/vij/ignite_config/cluster.properties
16/04/17 05:29:16 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
Exception in thread "main" java.lang.RuntimeException: Got unexpected response code. Response code: 404
    at org.apache.ignite.yarn.IgniteProvider.updateIgnite(IgniteProvider.java:240)
    at org.apache.ignite.yarn.IgniteProvider.getIgnite(IgniteProvider.java:93)
    at org.apache.ignite.yarn.IgniteYarnClient.getIgnite(IgniteYarnClient.java:169)
    at org.apache.ignite.yarn.IgniteYarnClient.main(IgniteYarnClient.java:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Can somebody help me what could be the issue ?Also I want to understand whats the benefit of running Ignite cluster over YARN.We could run Ignite nodes separately as well.One reason I could think is management of cluster become easy, as you dont need to manually start Ignite nodes on each node.

Regards,
Vijayendra Bhati


Vladimir Ozerov Vladimir Ozerov
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite - YARN integration

Hi Vij,

I am looking at the problem with "404" response at the moment. Will come back to you as soon as I have any additional information. 
As per Spark design, I hope other community members who a more familiar with it will answer you soon.

Vladimir.

On Tue, Apr 19, 2016 at 10:49 AM, vijayendra bhati <[hidden email]> wrote:
Still I have not been able to figure out how to resolve this issue but I think I can change my approach.
I can start Ignite node on the same node of AWS cluster on which YARN (Node manager) and HDFS (Data node) would be configured but I want to understand is it necessary to run Ignite Node as YARN job to achieve data locality while accessing Ignite data using Spark job ?
 
Regards,
Vij


On Sunday, April 17, 2016 6:11 PM, vijayendra bhati <[hidden email]> wrote:


Hi,
I am trying to run Ignite nodes over YARN cluster by following the documentation given on YARN Deployment · Apache Ignite
 
 
image
 
 
 
 
 
YARN Deployment · Apache Ignite
Deploy Ignite in YARN cluster.
Preview by Yahoo
 

I am using Cloudera supplied VM for initial installation purpose and have downloaded Apache Ignite version 1.5.0.

But I am getting error when I am trying to run below command -

[cloudera@quickstart ignite-yarn]$ hadoop jar /home/cloudera/vij/ignite_config/apache-ignite-fabric-1.5.0.final-bin/libs/optional/ignite-yarn/ignite-yarn-1.5.0.final.jar ./home/cloudera/vij/ignite_config/apache-ignite-fabric-1.5.0.final-bin/libs/optional/ignite-yarn/ignite-yarn-1.5.0.final.jar /home/cloudera/vij/ignite_config/cluster.properties
16/04/17 05:29:16 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
Exception in thread "main" java.lang.RuntimeException: Got unexpected response code. Response code: 404
    at org.apache.ignite.yarn.IgniteProvider.updateIgnite(IgniteProvider.java:240)
    at org.apache.ignite.yarn.IgniteProvider.getIgnite(IgniteProvider.java:93)
    at org.apache.ignite.yarn.IgniteYarnClient.getIgnite(IgniteYarnClient.java:169)
    at org.apache.ignite.yarn.IgniteYarnClient.main(IgniteYarnClient.java:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Can somebody help me what could be the issue ?Also I want to understand whats the benefit of running Ignite cluster over YARN.We could run Ignite nodes separately as well.One reason I could think is management of cluster become easy, as you dont need to manually start Ignite nodes on each node.

Regards,
Vijayendra Bhati



Vladimir Ozerov Vladimir Ozerov
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite - YARN integration

Vij,

Please try again. Problem should be resolved at the moment.

Vladimir.

On Tue, Apr 19, 2016 at 11:11 AM, Vladimir Ozerov <[hidden email]> wrote:
Hi Vij,

I am looking at the problem with "404" response at the moment. Will come back to you as soon as I have any additional information. 
As per Spark design, I hope other community members who a more familiar with it will answer you soon.

Vladimir.

On Tue, Apr 19, 2016 at 10:49 AM, vijayendra bhati <[hidden email]> wrote:
Still I have not been able to figure out how to resolve this issue but I think I can change my approach.
I can start Ignite node on the same node of AWS cluster on which YARN (Node manager) and HDFS (Data node) would be configured but I want to understand is it necessary to run Ignite Node as YARN job to achieve data locality while accessing Ignite data using Spark job ?
 
Regards,
Vij


On Sunday, April 17, 2016 6:11 PM, vijayendra bhati <[hidden email]> wrote:


Hi,
I am trying to run Ignite nodes over YARN cluster by following the documentation given on YARN Deployment · Apache Ignite
 
 
image
 
 
 
 
 
YARN Deployment · Apache Ignite
Deploy Ignite in YARN cluster.
Preview by Yahoo
 

I am using Cloudera supplied VM for initial installation purpose and have downloaded Apache Ignite version 1.5.0.

But I am getting error when I am trying to run below command -

[cloudera@quickstart ignite-yarn]$ hadoop jar /home/cloudera/vij/ignite_config/apache-ignite-fabric-1.5.0.final-bin/libs/optional/ignite-yarn/ignite-yarn-1.5.0.final.jar ./home/cloudera/vij/ignite_config/apache-ignite-fabric-1.5.0.final-bin/libs/optional/ignite-yarn/ignite-yarn-1.5.0.final.jar /home/cloudera/vij/ignite_config/cluster.properties
16/04/17 05:29:16 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
Exception in thread "main" java.lang.RuntimeException: Got unexpected response code. Response code: 404
    at org.apache.ignite.yarn.IgniteProvider.updateIgnite(IgniteProvider.java:240)
    at org.apache.ignite.yarn.IgniteProvider.getIgnite(IgniteProvider.java:93)
    at org.apache.ignite.yarn.IgniteYarnClient.getIgnite(IgniteYarnClient.java:169)
    at org.apache.ignite.yarn.IgniteYarnClient.main(IgniteYarnClient.java:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Can somebody help me what could be the issue ?Also I want to understand whats the benefit of running Ignite cluster over YARN.We could run Ignite nodes separately as well.One reason I could think is management of cluster become easy, as you dont need to manually start Ignite nodes on each node.

Regards,
Vijayendra Bhati