Apache Ignite Cloud Formation Template

classic Classic list List threaded Threaded
6 messages Options
sri hari kali charan Tummala sri hari kali charan Tummala
Reply | Threaded
Open this post in threaded view
|

Apache Ignite Cloud Formation Template

Hi All, 

did anyone have an cloud formation example for deploying Apache Ignite on AWS or steps to deploy or some successful deployment of Ignite onto AWS.

this below blog has missing artifacts so not working.


--
Thanks & Regards
Sri Tummala

sri hari kali charan Tummala sri hari kali charan Tummala
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite Cloud Formation Template

Hi Community, 

this doesn't launch a multinode ignite cluster am I right? it just launches a single node ignite cluster on one ec2 instance.


Thanks
Sri 

On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <[hidden email]> wrote:
Hi All, 

did anyone have an cloud formation example for deploying Apache Ignite on AWS or steps to deploy or some successful deployment of Ignite onto AWS.

this below blog has missing artifacts so not working.


--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala

dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite Cloud Formation Template

Hello Sri,

You’re right, you need to start several aws instances to create a distributed cluster if you follow the guide below. Ignite configuration for all the nodes/instances has to have a properly configured IPFinder. You can use a static IP finder if IPs are know in advance or AWS IPfinder if IPs are assigned dynamically.

Let us know if it helps and we’ll update the docs.

Denis

On Thursday, September 26, 2019, sri hari kali charan Tummala <[hidden email]> wrote:
Hi Community, 

this doesn't launch a multinode ignite cluster am I right? it just launches a single node ignite cluster on one ec2 instance.


Thanks
Sri 

On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <[hidden email]> wrote:
Hi All, 

did anyone have an cloud formation example for deploying Apache Ignite on AWS or steps to deploy or some successful deployment of Ignite onto AWS.

this below blog has missing artifacts so not working.


--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala



--
-
Denis

sri hari kali charan Tummala sri hari kali charan Tummala
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite Cloud Formation Template

looks complex and hard to get it right, it would be nice if Ignite community builds one working cloud formation template to launch ignite cluster and in the cloud formation template output a JDBC connection for connecting ignite cluster which would be awesome.

Many would leverage this stack to build Ignite cluster on AWS or how about ignite on EMR , while launching emr add ingite bootstrap script which launches ignite in emr which would be lot easier also JDBC connection for connecting ignite, this would change things for Ignite world.

Thanks
Sri 



On Thu, Sep 26, 2019 at 3:51 PM Denis Magda <[hidden email]> wrote:
Hello Sri,

You’re right, you need to start several aws instances to create a distributed cluster if you follow the guide below. Ignite configuration for all the nodes/instances has to have a properly configured IPFinder. You can use a static IP finder if IPs are know in advance or AWS IPfinder if IPs are assigned dynamically.

Let us know if it helps and we’ll update the docs.

Denis

On Thursday, September 26, 2019, sri hari kali charan Tummala <[hidden email]> wrote:
Hi Community, 

this doesn't launch a multinode ignite cluster am I right? it just launches a single node ignite cluster on one ec2 instance.


Thanks
Sri 

On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <[hidden email]> wrote:
Hi All, 

did anyone have an cloud formation example for deploying Apache Ignite on AWS or steps to deploy or some successful deployment of Ignite onto AWS.

this below blog has missing artifacts so not working.


--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala



--
-
Denis



--
Thanks & Regards
Sri Tummala

sri hari kali charan Tummala sri hari kali charan Tummala
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite Cloud Formation Template

I found one example of cloud formation after launching the template and started to execute ignite.sh it is failing with class not found exception AWSCredentialsProviderWrapper.

I sued:-
Ignitefabric.1.5.0
aws java sdk 1.3.21.1



ConfireIgnite.sh:-
#!/bin/bash

#
# This is a modified version of the file stored at s3://publicbucketbabupe/ignitelibrary/configureIgnite.sh
# which changes the config to use the instance provided credentials rather than requiring access/secret to be passed in
#
# Parameters are 
# 1 - Cache Name
# 2 - Number of replicas
# 3 - S3 Bucket Name
#

echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>

<!--
            Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the \"License\"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an \"AS IS\" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

<beans xmlns=\"<a href="http://www.springframework.org/schema/beans\">http://www.springframework.org/schema/beans\"
       xmlns:util=\"<a href="http://www.springframework.org/schema/util\">http://www.springframework.org/schema/util\"
       xmlns:xsi=\"<a href="http://www.w3.org/2001/XMLSchema-instance\">http://www.w3.org/2001/XMLSchema-instance\"
       xsi:schemaLocation=\"
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/util
       <a href="http://www.springframework.org/schema/util/spring-util.xsd\">http://www.springframework.org/schema/util/spring-util.xsd\">
    <!--
                          Alter configuration below as needed.
    -->
  <bean id=\"grid.cfg\" class=\"org.apache.ignite.configuration.IgniteConfiguration\">
    <property name=\"cacheConfiguration\">
      <list>
        <bean class=\"org.apache.ignite.configuration.CacheConfiguration\">
          <property name=\"name\" value=\"$1\"/>" > /tmp/igniteconfig.xml

echo "	  <property name=\"cacheMode\" value=\"PARTITIONED\"/>
          <property name=\"atomicWriteOrderMode\" value=\"PRIMARY\"/>
          <property name=\"writeSynchronizationMode\" value=\"PRIMARY_SYNC\"/>" >> /tmp/igniteconfig.xml

availfreeMemory=$(cat /proc/meminfo|grep MemTotal|awk '{print $2}')
memoryOverhead=$((availfreeMemory/1024/1024/10))
availfreeMemoryinGB=$((availfreeMemory/1024/1024 - memoryOverhead))
if [[ $availfreeMemoryinGB -gt 8 ]]; then
	offheapmemoryinGB=$((availfreeMemoryinGB-8))
	echo "	  <property name=\"memoryMode\" value=\"ONHEAP_TIERED\" />
	  <property name=\"offHeapMaxMemory\" value=\"#{$offheapmemoryinGB * 1024L * 1024L * 1024L}\" />" >> /tmp/igniteconfig.xml
	echo "8g" > /tmp/heapsize.log
else
	echo "${availfreeMemoryinGB}g" > /tmp/heapsize.log
fi
echo "    <property name=\"evictionPolicy\">
	  	<bean class=\"org.apache.ignite.cache.eviction.lru.LruEvictionPolicy\">
			<property name=\"maxSize\" value=\"100000000\"/>
		</bean>
	  </property>" >> /tmp/igniteconfig.xml
echo "	  <property name=\"swapEnabled\" value=\"false\"/>" >> /tmp/igniteconfig.xml
echo "    <property name=\"atomicityMode\" value=\"ATOMIC\" />" >> /tmp/igniteconfig.xml
echo "    <property name=\"backups\" value=\"$2\" />" >> /tmp/igniteconfig.xml
echo "	</bean>" >> /tmp/igniteconfig.xml
echo " </list>" >> /tmp/igniteconfig.xml
echo "</property>" >> /tmp/igniteconfig.xml
echo "<property name=\"discoverySpi\">
        <bean class=\"org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi\">
          <property name=\"ipFinder\">
            <bean class=\"org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder\">
              <property name=\"awsCredentials\" ref=\"aws.creds\"/>
              <property name=\"bucketName\" value=\"$3\"/>
             </bean>
          </property>
        </bean>
    </property>
    <property name=\"communicationSpi\">
    <bean class=\"org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi\">
      <property name=\"slowClientQueueLimit\" value=\"1000\"/>
    </bean>
  </property>
  </bean>
  <!-- AWS credentials. Use the default chain from the environment/instance. -->
  <bean id=\"aws.creds.factory\" class=\"com.amazonaws.auth.DefaultAWSCredentialsProviderChain\"/>
  <bean id=\"aws.creds\" class=\"com.amazon.connections.data.auth.AWSCredentialsProviderWrapper\">
      <constructor-arg ref=\"aws.creds.factory\"/>
  </bean>

</beans>" >> /tmp/igniteconfig.xml

On Thu, Sep 26, 2019 at 5:34 PM sri hari kali charan Tummala <[hidden email]> wrote:
looks complex and hard to get it right, it would be nice if Ignite community builds one working cloud formation template to launch ignite cluster and in the cloud formation template output a JDBC connection for connecting ignite cluster which would be awesome.

Many would leverage this stack to build Ignite cluster on AWS or how about ignite on EMR , while launching emr add ingite bootstrap script which launches ignite in emr which would be lot easier also JDBC connection for connecting ignite, this would change things for Ignite world.

Thanks
Sri 



On Thu, Sep 26, 2019 at 3:51 PM Denis Magda <[hidden email]> wrote:
Hello Sri,

You’re right, you need to start several aws instances to create a distributed cluster if you follow the guide below. Ignite configuration for all the nodes/instances has to have a properly configured IPFinder. You can use a static IP finder if IPs are know in advance or AWS IPfinder if IPs are assigned dynamically.

Let us know if it helps and we’ll update the docs.

Denis

On Thursday, September 26, 2019, sri hari kali charan Tummala <[hidden email]> wrote:
Hi Community, 

this doesn't launch a multinode ignite cluster am I right? it just launches a single node ignite cluster on one ec2 instance.


Thanks
Sri 

On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <[hidden email]> wrote:
Hi All, 

did anyone have an cloud formation example for deploying Apache Ignite on AWS or steps to deploy or some successful deployment of Ignite onto AWS.

this below blog has missing artifacts so not working.


--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala



--
-
Denis



--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Apache Ignite Cloud Formation Template

Hello!

If that is really Apache Ignite 1.5, please don't use it. Please only use the latest 2.7.6 release.

Regards,
--
Ilya Kasnacheev


пт, 27 сент. 2019 г. в 18:56, sri hari kali charan Tummala <[hidden email]>:
I found one example of cloud formation after launching the template and started to execute ignite.sh it is failing with class not found exception AWSCredentialsProviderWrapper.

I sued:-
Ignitefabric.1.5.0
aws java sdk 1.3.21.1



ConfireIgnite.sh:-
#!/bin/bash

#
# This is a modified version of the file stored at s3://publicbucketbabupe/ignitelibrary/configureIgnite.sh
# which changes the config to use the instance provided credentials rather than requiring access/secret to be passed in
#
# Parameters are 
# 1 - Cache Name
# 2 - Number of replicas
# 3 - S3 Bucket Name
#

echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>

<!--
            Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the \"License\"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an \"AS IS\" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

<beans xmlns=\"http://www.springframework.org/schema/beans\"
       xmlns:util=\"http://www.springframework.org/schema/util\"
       xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"
       xsi:schemaLocation=\"
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/util
       http://www.springframework.org/schema/util/spring-util.xsd\">
    <!--
                          Alter configuration below as needed.
    -->
  <bean id=\"grid.cfg\" class=\"org.apache.ignite.configuration.IgniteConfiguration\">
    <property name=\"cacheConfiguration\">
      <list>
        <bean class=\"org.apache.ignite.configuration.CacheConfiguration\">
          <property name=\"name\" value=\"$1\"/>" > /tmp/igniteconfig.xml

echo "	  <property name=\"cacheMode\" value=\"PARTITIONED\"/>
          <property name=\"atomicWriteOrderMode\" value=\"PRIMARY\"/>
          <property name=\"writeSynchronizationMode\" value=\"PRIMARY_SYNC\"/>" >> /tmp/igniteconfig.xml

availfreeMemory=$(cat /proc/meminfo|grep MemTotal|awk '{print $2}')
memoryOverhead=$((availfreeMemory/1024/1024/10))
availfreeMemoryinGB=$((availfreeMemory/1024/1024 - memoryOverhead))
if [[ $availfreeMemoryinGB -gt 8 ]]; then
	offheapmemoryinGB=$((availfreeMemoryinGB-8))
	echo "	  <property name=\"memoryMode\" value=\"ONHEAP_TIERED\" />
	  <property name=\"offHeapMaxMemory\" value=\"#{$offheapmemoryinGB * 1024L * 1024L * 1024L}\" />" >> /tmp/igniteconfig.xml
	echo "8g" > /tmp/heapsize.log
else
	echo "${availfreeMemoryinGB}g" > /tmp/heapsize.log
fi
echo "    <property name=\"evictionPolicy\">
	  	<bean class=\"org.apache.ignite.cache.eviction.lru.LruEvictionPolicy\">
			<property name=\"maxSize\" value=\"100000000\"/>
		</bean>
	  </property>" >> /tmp/igniteconfig.xml
echo "	  <property name=\"swapEnabled\" value=\"false\"/>" >> /tmp/igniteconfig.xml
echo "    <property name=\"atomicityMode\" value=\"ATOMIC\" />" >> /tmp/igniteconfig.xml
echo "    <property name=\"backups\" value=\"$2\" />" >> /tmp/igniteconfig.xml
echo "	</bean>" >> /tmp/igniteconfig.xml
echo " </list>" >> /tmp/igniteconfig.xml
echo "</property>" >> /tmp/igniteconfig.xml
echo "<property name=\"discoverySpi\">
        <bean class=\"org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi\">
          <property name=\"ipFinder\">
            <bean class=\"org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder\">
              <property name=\"awsCredentials\" ref=\"aws.creds\"/>
              <property name=\"bucketName\" value=\"$3\"/>
             </bean>
          </property>
        </bean>
    </property>
    <property name=\"communicationSpi\">
    <bean class=\"org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi\">
      <property name=\"slowClientQueueLimit\" value=\"1000\"/>
    </bean>
  </property>
  </bean>
  <!-- AWS credentials. Use the default chain from the environment/instance. -->
  <bean id=\"aws.creds.factory\" class=\"com.amazonaws.auth.DefaultAWSCredentialsProviderChain\"/>
  <bean id=\"aws.creds\" class=\"com.amazon.connections.data.auth.AWSCredentialsProviderWrapper\">
      <constructor-arg ref=\"aws.creds.factory\"/>
  </bean>

</beans>" >> /tmp/igniteconfig.xml

On Thu, Sep 26, 2019 at 5:34 PM sri hari kali charan Tummala <[hidden email]> wrote:
looks complex and hard to get it right, it would be nice if Ignite community builds one working cloud formation template to launch ignite cluster and in the cloud formation template output a JDBC connection for connecting ignite cluster which would be awesome.

Many would leverage this stack to build Ignite cluster on AWS or how about ignite on EMR , while launching emr add ingite bootstrap script which launches ignite in emr which would be lot easier also JDBC connection for connecting ignite, this would change things for Ignite world.

Thanks
Sri 



On Thu, Sep 26, 2019 at 3:51 PM Denis Magda <[hidden email]> wrote:
Hello Sri,

You’re right, you need to start several aws instances to create a distributed cluster if you follow the guide below. Ignite configuration for all the nodes/instances has to have a properly configured IPFinder. You can use a static IP finder if IPs are know in advance or AWS IPfinder if IPs are assigned dynamically.

Let us know if it helps and we’ll update the docs.

Denis

On Thursday, September 26, 2019, sri hari kali charan Tummala <[hidden email]> wrote:
Hi Community, 

this doesn't launch a multinode ignite cluster am I right? it just launches a single node ignite cluster on one ec2 instance.


Thanks
Sri 

On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <[hidden email]> wrote:
Hi All, 

did anyone have an cloud formation example for deploying Apache Ignite on AWS or steps to deploy or some successful deployment of Ignite onto AWS.

this below blog has missing artifacts so not working.


--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala



--
-
Denis



--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala