not working in Spark submit yarn mode.

classic Classic list List threaded Threaded
2 messages Options
java148 java148
Reply | Threaded
Open this post in threaded view
|

not working in Spark submit yarn mode.

Hi, I am trying to load Hdfs data into cache from Spark. it is working
in local mode, but failed in spark-submit yarn mode. it tried to find
Ignite Home path in the cluster. yes, it is true that Ignite is not
installed in cluster. but why needed ? Ignite instance is created inside
my java code and all of ignite jars are packaged. why need to find
ignite info outside of my jar.

My ignite version is 2.3.

Anyone has a working example of Spark submit + Yarn mode ?  any web link
is welcome.

Thanks. Edward

dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: not working in Spark submit yarn mode.

It's possible to do that but not sure the community has any ready-to-be-used code sample to share.

As for already available solutions, GridGain provides so-called Hadoop Connector that solves data loading task:

-
Denis


On Tue, Jun 4, 2019 at 1:49 PM Edward Chen <[hidden email]> wrote:
Hi, I am trying to load Hdfs data into cache from Spark. it is working
in local mode, but failed in spark-submit yarn mode. it tried to find
Ignite Home path in the cluster. yes, it is true that Ignite is not
installed in cluster. but why needed ? Ignite instance is created inside
my java code and all of ignite jars are packaged. why need to find
ignite info outside of my jar.

My ignite version is 2.3.

Anyone has a working example of Spark submit + Yarn mode ?  any web link
is welcome.

Thanks. Edward