Ignite on RBAC enabled K8s cluster

classic Classic list List threaded Threaded
5 messages Options
vbm vbm
Reply | Threaded
Open this post in threaded view
|

Ignite on RBAC enabled K8s cluster

Hi,

I am trying to bring up Ignite on RBAC enabled K8s cluster. As part of
discovery, when ignite nodes are trying to discover one another, I am seeing
below error:

[02:16:08,755][SEVERE][tcp-disco-ip-finder-cleaner-#4][TcpDiscoverySpi]
Failed to clean IP finder up.
class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite
pods IP addresses.
        at
org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172)
        at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1810)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.cleanIpFinder(ServerImpl.java:1921)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1896)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.io.IOException: Server returned HTTP response code: 403 for
URL:
https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/ignite
        at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
        at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
        at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
        at
org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:153)


I see there is a ticket on it, but it uses a cluster-admin role. Not sure
why we need cluster-admin role.
I tried the steps mentioned in the ticket, but still I am getting the same
error.

I was not able to find any documentation on the same. Has anyone tried
bringing up Ignite on RBAC enabled K8s cluster ?


Regards,
Vishwas



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Roman Guseinov Roman Guseinov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite on RBAC enabled K8s cluster

Hi Vishwas,

Usually, this issue happens because of:
1. Default service account doesn't have permissions to use the endpoint to
get all related IP addresses (of Ignite nodes).
2. A cluster is deployed in not default namespace. For example, in Openshift
namespace is a project name.

To resolve that you need to create a service account and grant necessary
permissions (we don't need cluster-admin role):

*sa.yml*

*role.yml*

*rolebind.yml*


Do not forget to replace <namespace>, add `serviceAccountName` into
DeploymentConfig [1] and specify namespace property of
TcpDiscoveryKubernetesIpFinder [1].

Best Regards,
Roman

[1]
https://apacheignite.readme.io/docs/kubernetes-deployment#section-deploying-on-openshift



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
vbm vbm
Reply | Threaded
Open this post in threaded view
|

Re: Ignite on RBAC enabled K8s cluster

Hi Roman,

Thanks for the response. I followed the steps you mentioned and also the
below link
https://stackoverflow.com/questions/50817940/failed-to-retrieve-ignite-pods-ip-addresses/50818842#50818842

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ignite
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ignite-endpoint-access
  namespace: default
  labels:
    app: ignite
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    resourceNames: ["ignite"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ignite-role-binding
  namespace: default
  labels:
    app: ignite
subjects:
  - kind: ServiceAccount
    name: ignite
roleRef:
  kind: Role
  name: ignite-endpoint-access
  apiGroup: rbac.authorization.k8s.io

I have added,  serviceAccount: ignite in deployment spec.

But still I am facing issues, I have done the deployment with 2 pods. Both
of them are in running state, but they are not able to discover one another.
Below is the error trace:

16:37:06,185][SEVERE][tcp-disco-ip-finder-cleaner-#4][TcpDiscoverySpi]
Failed to clean IP finder up.
class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite
pods IP addresses.
        at
org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172)
        at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1810)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.cleanIpFinder(ServerImpl.java:1921)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1896)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.net.UnknownHostException:
kubernetes.default.svc.cluster.local
        at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
        at
sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)

The only difference is that the ignite server in the container is running
with non root user. I have created a non root user and started the process
with that user.
Not sure if that makes a difference ?

Regards,
Vishwas



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Roman Guseinov Roman Guseinov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite on RBAC enabled K8s cluster

Hi Vishwas,

Where your kubernetes cluster is located? Google Cloud, Openshift or
somewhere else?

I've checked your service account configs (with role and binding) on Google
Cloud. It works well. Maybe the problem in your deployment file or in
changes related to docker image.

Here are the steps to deploy Ignite 2.5 on Google Cloud:

1. Grant cluster-admin role to current google user (to allow create roles):

$ kubectl create clusterrolebinding myname-cluster-admin-binding \
  --clusterrole=cluster-admin \
  --user=

2. Create service account and grant permissions:

$ kubectl create -f sa.yaml
$ kubectl create -f role.yaml
$ kubectl create -f rolebind.yaml

3. Create a grid service:

$ kubectl create -f service.yaml

4. Deploy GridGain cluster:

$ kubectl create -f grid.yaml

YAML-files are attached. Please try to use them.

sa.yaml <http://apache-ignite-users.70518.x6.nabble.com/file/t1601/sa.yaml>  
role.yaml
<http://apache-ignite-users.70518.x6.nabble.com/file/t1601/role.yaml>  
rolebind.yaml
<http://apache-ignite-users.70518.x6.nabble.com/file/t1601/rolebind.yaml>  
service.yaml
<http://apache-ignite-users.70518.x6.nabble.com/file/t1601/service.yaml>  
grid.yaml
<http://apache-ignite-users.70518.x6.nabble.com/file/t1601/grid.yaml>  

I suggest firstly make it works and then make changes you need. Also, please
make sure that your namespace is really `default`.

Best Regards,
Roman



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
vbm vbm
Reply | Threaded
Open this post in threaded view
|

Re: Ignite on RBAC enabled K8s cluster

Hi Roman,

Thanks for the information. I was able to get ignite working with RBAC
enabled K8s cluster.
The issue was with K8s calico networking, I had set it up wrongly because of
which I was getting errors.


Regards,
Vishwas



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/