Cluster Setup Example

classic Classic list List threaded Threaded
3 messages Options
Kris Jirapinyo Kris Jirapinyo
Reply | Threaded
Open this post in threaded view
|

Cluster Setup Example

Is there any tutorial outlining how to set up multi node ignite clusters and best practices?  Kind of like this blog post for hadoop: http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/

I tried looking at the documentation on the apache ignite website regarding clusters, but they just contain code snippets that don't really take you from beginning to finish outlining all necessary steps and configurations in one coherent post.

My simple case is just trying to set up apache ignite on two remote servers that will be continuously running as a data grid for caching, with my laptop connecting to it as a client.

-- Kris.
Ognen Duzlevski Ognen Duzlevski
Reply | Threaded
Open this post in threaded view
|

Re: Cluster Setup Example

Hey Kris,

I have a 5 machine experimental setup on Amazon. If you are not coming from the Java world, Ignite can be confusing. The documentation is pretty decent but it does lack the basics that you mention since it kind of assumes you are coming from Java background. For example, I had never heard of Spring and beans before ;). In any case, you can start with an example config file like ignite/examples/config/example-ignite.xml, you can modify that to include the IPs of the two machines that can see each other. Both machines need to allow traffic on certain ports (see in the documentation online for ignite, these are mentioned in a table somewhere). You should probably replicate the ignite setup on both machines to make it easier for scripting - I unpack the latest zip for ignite binary and symlink ignite to it so all references to ignite/bin/ignite.sh ignite/examples/config/example-ignite.xml work on both machines. Best to do this as non-root user. Once you have the replicated stuff going, you can use something like supervisord to run ignite.sh with the config file passed to it or you can just run by hand - if you have set up your iptables properly, the two invocations of ignite on the two machines should see each other and connect.

I can dig up more details if the above is not enough, let me know...
Ognen

On Mon, Aug 10, 2015 at 3:37 PM, Kris Jirapinyo <[hidden email]> wrote:
Is there any tutorial outlining how to set up multi node ignite clusters and best practices?  Kind of like this blog post for hadoop: http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/

I tried looking at the documentation on the apache ignite website regarding clusters, but they just contain code snippets that don't really take you from beginning to finish outlining all necessary steps and configurations in one coherent post.

My simple case is just trying to set up apache ignite on two remote servers that will be continuously running as a data grid for caching, with my laptop connecting to it as a client.

-- Kris.

Kris Jirapinyo Kris Jirapinyo
Reply | Threaded
Open this post in threaded view
|

Re: Cluster Setup Example

Thanks Ognen. Yeah I was able to set up my example, but it felt very hack-ish since I'm just doing trial and error until it works.  Wish there was some kind of "official" post on how to do things properly.

Do I always need to copy over my config files to all the servers I want to deploy? I thought there was a feature where you don't need to do that, but I haven't gotten it working yet.

-- Kris.

On Mon, Aug 10, 2015 at 1:49 PM, Ognen Duzlevski <[hidden email]> wrote:
Hey Kris,

I have a 5 machine experimental setup on Amazon. If you are not coming from the Java world, Ignite can be confusing. The documentation is pretty decent but it does lack the basics that you mention since it kind of assumes you are coming from Java background. For example, I had never heard of Spring and beans before ;). In any case, you can start with an example config file like ignite/examples/config/example-ignite.xml, you can modify that to include the IPs of the two machines that can see each other. Both machines need to allow traffic on certain ports (see in the documentation online for ignite, these are mentioned in a table somewhere). You should probably replicate the ignite setup on both machines to make it easier for scripting - I unpack the latest zip for ignite binary and symlink ignite to it so all references to ignite/bin/ignite.sh ignite/examples/config/example-ignite.xml work on both machines. Best to do this as non-root user. Once you have the replicated stuff going, you can use something like supervisord to run ignite.sh with the config file passed to it or you can just run by hand - if you have set up your iptables properly, the two invocations of ignite on the two machines should see each other and connect.

I can dig up more details if the above is not enough, let me know...
Ognen

On Mon, Aug 10, 2015 at 3:37 PM, Kris Jirapinyo <[hidden email]> wrote:
Is there any tutorial outlining how to set up multi node ignite clusters and best practices?  Kind of like this blog post for hadoop: http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/

I tried looking at the documentation on the apache ignite website regarding clusters, but they just contain code snippets that don't really take you from beginning to finish outlining all necessary steps and configurations in one coherent post.

My simple case is just trying to set up apache ignite on two remote servers that will be continuously running as a data grid for caching, with my laptop connecting to it as a client.

-- Kris.