Use apache Ignite outside spark-shell

classic Classic list List threaded Threaded
4 messages Options
luismattor luismattor
Reply | Threaded
Open this post in threaded view
|

Use apache Ignite outside spark-shell

Hi,

I use spark-submit script to launch my spark application. There is a problem with IgniteContext when starting ignite server nodes on each worker. It seems IgniteContext is trying to launch more than one ignite node instance per worker. Does Ignite works outiside spark-shell?
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Use apache Ignite outside spark-shell

Hi,

Why do you think this is happening? Do you have any exception? Generally, IgniteContext always reuses the Ignite client if it already exists in the current process.

Note that you can still have several clients in several processes. Usually you will have a client on the driver and a client per each executor (not worker!).

-Val
luismattor luismattor
Reply | Threaded
Open this post in threaded view
|

Re: Use apache Ignite outside spark-shell

Hi,

The problem is in IgniteContext.scala:

// Start ignite server node on each worker in server mode.
sparkContext.parallelize(1 to workers, workers).foreachPartition(it ignite())

As far as I understand, this line works as far as each executor holds a single partition. That is, if executor owns more than one partition IgniteContext will try to launch several Ignite nodes (in the same process); if there are executors with no partitions, no Ignite instance will be launched.

For example, Image a two worker cluster (node1, node2), each worker having two cores. The idea is to launch an Ignite node on each executor. So we call:
sparkContext.parallelize(1 to 2, 2).foreachPartition(it ignite())

It might be the chance the two partitions are handled by node1. Or it might be the case node1 handle one partition, and node2 the other. (this is what we want). So the question is how to make sure the latter scenario happens?

Regards
Luis




On 26 May 2016 at 05:19, vkulichenko [via Apache Ignite Users] <[hidden email]> wrote:
Hi,

Why do you think this is happening? Do you have any exception? Generally, IgniteContext always reuses the Ignite client if it already exists in the current process.

Note that you can still have several clients in several processes. Usually you will have a client on the driver and a client per each executor (not worker!).

-Val


If you reply to this email, your message will be added to the discussion below:
http://apache-ignite-users.70518.x6.nabble.com/Use-apache-Ignite-outside-spark-shell-tp5208p5228.html
To unsubscribe from Use apache Ignite outside spark-shell, click here.
NAML

vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Use apache Ignite outside spark-shell

Hi Luis,

IgniteContext.ignite() method handles this situation and reuses the Ignite client if it's already started. So if there are two partitions on the same worker, there still will be only one Ignite started per process. Is it working in a different way for you?

-Val