Amazon S3 Based Discovery NOT USING BasicAWSCredentials

classic Classic list List threaded Threaded
5 messages Options
Max Barrios Max Barrios
Reply | Threaded
Open this post in threaded view
|

Amazon S3 Based Discovery NOT USING BasicAWSCredentials

I am running Apache Ignite 2.6.0 in AWS and am using S3 Based Discovery,

However, I DO NOT want to embed AWS Access or Secret Keys in my ignite.xml

I have AWS EC2 Instance Metadata Service for my instances so that the creds can be loaded from there. 

However, there's no guidance or documentation on how to do this. Is this even supported?

For example, I want to do this:
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
          <property name="awsCredentialsProvider" ref="aws.creds"/>
          <property name="bucketName" value="YOUR_BUCKET_NAME"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>

<!-- AWS credentials. Provide your access key ID and secret access key. -->
<bean id="aws.creds" class="com.amazonaws.auth.InstanceProfileCredentialsProvider">
</bean>

But I get this exception when I try the above:

Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'awsCredentialsProvider' of bean class [org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder]: Bean property 'awsCredentialsProvider' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?

If using an AWS Credentials Provider *is* supported, where are the bean properties for documented, so I can see what I may be doing wrong? Are there any working examples for anything other than BasicAWSCCredentials?

Please help. 

Max
Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

RE: Amazon S3 Based Discovery NOT USING BasicAWSCredentials

Hi,

 

Were you able to solve this?

 

It seems that your config is actually fine… The feature was added by https://issues.apache.org/jira/browse/IGNITE-4530.

 

Does it work if you replace `ref` with just a value?

Like

<property name="awsCredentialsProvider">

    <bean class="com.amazonaws.auth.InstanceProfileCredentialsProvider"/>

</property>

 

Stan

 

From: [hidden email]
Sent: 12 декабря 2018 г. 23:51
To: [hidden email]
Subject: Amazon S3 Based Discovery NOT USING BasicAWSCredentials

 

I am running Apache Ignite 2.6.0 in AWS and am using S3 Based Discovery,

 

However, I DO NOT want to embed AWS Access or Secret Keys in my ignite.xml

 

I have AWS EC2 Instance Metadata Service for my instances so that the creds can be loaded from there. 

 

However, there's no guidance or documentation on how to do this. Is this even supported?

 

For example, I want to do this:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
          <property name="awsCredentialsProvider" ref="aws.creds"/>
          <property name="bucketName" value="YOUR_BUCKET_NAME"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>
 
<!-- AWS credentials. Provide your access key ID and secret access key. -->
<bean id="aws.creds" class="com.amazonaws.auth.InstanceProfileCredentialsProvider">
</bean>

 

But I get this exception when I try the above:

 

Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'awsCredentialsProvider' of bean class [org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder]: Bean property 'awsCredentialsProvider' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?

 

If using an AWS Credentials Provider *is* supported, where are the bean properties for documented, so I can see what I may be doing wrong? Are there any working examples for anything other than BasicAWSCCredentials?

 

Please help. 

 

Max

 

Max Barrios Max Barrios
Reply | Threaded
Open this post in threaded view
|

Re: Amazon S3 Based Discovery NOT USING BasicAWSCredentials

I’m still seeing this problem

I’ll try the suggested approach 

Sent from my iPhone

On Jan 10, 2019, at 08:25, Stanislav Lukyanov <[hidden email]> wrote:

Hi,

 

Were you able to solve this?

 

It seems that your config is actually fine… The feature was added by https://issues.apache.org/jira/browse/IGNITE-4530.

 

Does it work if you replace `ref` with just a value?

Like

<property name="awsCredentialsProvider">

    <bean class="com.amazonaws.auth.InstanceProfileCredentialsProvider"/>

</property>

 

Stan

 

From: [hidden email]
Sent: 12 декабря 2018 г. 23:51
To: [hidden email]
Subject: Amazon S3 Based Discovery NOT USING BasicAWSCredentials

 

I am running Apache Ignite 2.6.0 in AWS and am using S3 Based Discovery,

 

However, I DO NOT want to embed AWS Access or Secret Keys in my ignite.xml

 

I have AWS EC2 Instance Metadata Service for my instances so that the creds can be loaded from there. 

 

However, there's no guidance or documentation on how to do this. Is this even supported?

 

For example, I want to do this:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
          <property name="awsCredentialsProvider" ref="aws.creds"/>
          <property name="bucketName" value="YOUR_BUCKET_NAME"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>
 
<!-- AWS credentials. Provide your access key ID and secret access key. -->
<bean id="aws.creds" class="com.amazonaws.auth.InstanceProfileCredentialsProvider">
</bean>

 

But I get this exception when I try the above:

 

Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'awsCredentialsProvider' of bean class [org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder]: Bean property 'awsCredentialsProvider' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?

 

If using an AWS Credentials Provider *is* supported, where are the bean properties for documented, so I can see what I may be doing wrong? Are there any working examples for anything other than BasicAWSCCredentials?

 

Please help. 

 

Max

 

Max Barrios Max Barrios
Reply | Threaded
Open this post in threaded view
|

Re: Amazon S3 Based Discovery NOT USING BasicAWSCredentials

In reply to this post by Stanislav Lukyanov
I’m still seeing this error even when passing a value via ‘ref’. 

Looks like there’s some ignite lib not getting loaded that can resolve the awsCredentials property, when running in a spark cluster.

Is there any comprehensive guidance for running a spark app using ignite in a spark cluster? 

Sent from my iPhone

On Jan 10, 2019, at 08:25, Stanislav Lukyanov <[hidden email]> wrote:

Hi,

 

Were you able to solve this?

 

It seems that your config is actually fine… The feature was added by https://issues.apache.org/jira/browse/IGNITE-4530.

 

Does it work if you replace `ref` with just a value?

Like

<property name="awsCredentialsProvider">

    <bean class="com.amazonaws.auth.InstanceProfileCredentialsProvider"/>

</property>

 

Stan

 

From: [hidden email]
Sent: 12 декабря 2018 г. 23:51
To: [hidden email]
Subject: Amazon S3 Based Discovery NOT USING BasicAWSCredentials

 

I am running Apache Ignite 2.6.0 in AWS and am using S3 Based Discovery,

 

However, I DO NOT want to embed AWS Access or Secret Keys in my ignite.xml

 

I have AWS EC2 Instance Metadata Service for my instances so that the creds can be loaded from there. 

 

However, there's no guidance or documentation on how to do this. Is this even supported?

 

For example, I want to do this:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
          <property name="awsCredentialsProvider" ref="aws.creds"/>
          <property name="bucketName" value="YOUR_BUCKET_NAME"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>
 
<!-- AWS credentials. Provide your access key ID and secret access key. -->
<bean id="aws.creds" class="com.amazonaws.auth.InstanceProfileCredentialsProvider">
</bean>

 

But I get this exception when I try the above:

 

Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'awsCredentialsProvider' of bean class [org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder]: Bean property 'awsCredentialsProvider' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?

 

If using an AWS Credentials Provider *is* supported, where are the bean properties for documented, so I can see what I may be doing wrong? Are there any working examples for anything other than BasicAWSCCredentials?

 

Please help. 

 

Max

 

Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

RE: Amazon S3 Based Discovery NOT USING BasicAWSCredentials

awsCredentialsProvider is just a property of the  TcpDiscoveryS3IpFinder.

If the class is loaded then the property should also be there.

 

I’ll try this as well and let you know if that works for me.

 

Stan

 

From: [hidden email]
Sent: 11 января 2019 г. 4:47
To: [hidden email]
Subject: Re: Amazon S3 Based Discovery NOT USING BasicAWSCredentials

 

I’m still seeing this error even when passing a value via ‘ref’. 

 

Looks like there’s some ignite lib not getting loaded that can resolve the awsCredentials property, when running in a spark cluster.

 

Is there any comprehensive guidance for running a spark app using ignite in a spark cluster? 

Sent from my iPhone


On Jan 10, 2019, at 08:25, Stanislav Lukyanov <[hidden email]> wrote:

Hi,

 

Were you able to solve this?

 

It seems that your config is actually fine… The feature was added by https://issues.apache.org/jira/browse/IGNITE-4530.

 

Does it work if you replace `ref` with just a value?

Like

<property name="awsCredentialsProvider">

    <bean class="com.amazonaws.auth.InstanceProfileCredentialsProvider"/>

</property>

 

Stan

 

From: [hidden email]
Sent: 12 декабря 2018 г. 23:51
To: [hidden email]
Subject: Amazon S3 Based Discovery NOT USING BasicAWSCredentials

 

I am running Apache Ignite 2.6.0 in AWS and am using S3 Based Discovery,

 

However, I DO NOT want to embed AWS Access or Secret Keys in my ignite.xml

 

I have AWS EC2 Instance Metadata Service for my instances so that the creds can be loaded from there. 

 

However, there's no guidance or documentation on how to do this. Is this even supported?

 

For example, I want to do this:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
          <property name="awsCredentialsProvider" ref="aws.creds"/>
          <property name="bucketName" value="YOUR_BUCKET_NAME"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>
 
<!-- AWS credentials. Provide your access key ID and secret access key. -->
<bean id="aws.creds" class="com.amazonaws.auth.InstanceProfileCredentialsProvider">
</bean>

 

But I get this exception when I try the above:

 

Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'awsCredentialsProvider' of bean class [org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder]: Bean property 'awsCredentialsProvider' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?

 

If using an AWS Credentials Provider *is* supported, where are the bean properties for documented, so I can see what I may be doing wrong? Are there any working examples for anything other than BasicAWSCCredentials?

 

Please help. 

 

Max