Kubernetes- Failed to retrieve Ignite pods IP addresses.

classic Classic list List threaded Threaded
3 messages Options
Gokulnath Chidambaram Gokulnath Chidambaram
Reply | Threaded
Open this post in threaded view
|

Kubernetes- Failed to retrieve Ignite pods IP addresses.

Hello,

I am trying to install apache ignite:2.7.6 in kubernetes cluster (deployed in aws). 
I created

1.service account
2. role access
3.rolebinding
4.deployment.
4. added namesapce (bean property)  in xml configuration file. 

I am getting the following error message.

11:18:39,724][INFO][main][PartitionsEvictManager] Evict partition permits=2
[11:18:44,719][INFO][main][ClientListenerProcessor] Client connector processor has started on TCP port 10800
[11:18:46,418][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
[11:18:48,617][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible too long JVM pause: 850 milliseconds.
[11:18:52,420][INFO][main][GridJettyRestProtocol] Command protocol successfully started [name=Jetty REST, host=/0.0.0.0, port=8080]
[11:18:53,721][INFO][main][IgniteKernal] Non-loopback local IPs: 10.42.5.132
[11:18:53,721][INFO][main][IgniteKernal] Enabled local MACs: 363E412C208F
[11:18:54,625][INFO][main][TcpDiscoverySpi] Connection check threshold is calculated: 10000
[11:18:55,024][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port [port=47500, localHost=0.0.0.0/0.0.0.0, locNodeId=6bc33d41-6dba-422b-832d-ed8cae326a00]
[11:19:03,423][SEVERE][main][TcpDiscoverySpi] Failed to get registered addresses from IP finder on start (retrying every 2000ms; change 'reconnectDelay' to configure the frequency of retries).
class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite pods IP addresses.
at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1848)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1049)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:910)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
at org.apache.ignite.Ignition.start(Ignition.java:348)
at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
Caused by: java.io.IOException: Server returned HTTP response code: 403 for URL: https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/dataobjns/endpoints/ignite
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:153)
... 20 more
[11:19:15,821][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible too long JVM pause: 500 milliseconds. 

any help is appreciated. 
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes- Failed to retrieve Ignite pods IP addresses.

Hi,

Using the search I found that current issue was resolved in next thread:

http://apache-ignite-users.70518.x6.nabble.com/Ignite-on-RBAC-enabled-K8s-cluster-td22165.html

I guess that you should also read next article:

https://apacheignite.readme.io/docs/rbac-authorization

BR,
Andrei

11/7/2019 2:27 PM, Gokulnath Chidambaram пишет:
Hello,

I am trying to install apache ignite:2.7.6 in kubernetes cluster (deployed in aws). 
I created

1.service account
2. role access
3.rolebinding
4.deployment.
4. added namesapce (bean property)  in xml configuration file. 

I am getting the following error message.

11:18:39,724][INFO][main][PartitionsEvictManager] Evict partition permits=2
[11:18:44,719][INFO][main][ClientListenerProcessor] Client connector processor has started on TCP port 10800
[11:18:46,418][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
[11:18:48,617][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible too long JVM pause: 850 milliseconds.
[11:18:52,420][INFO][main][GridJettyRestProtocol] Command protocol successfully started [name=Jetty REST, host=/0.0.0.0, port=8080]
[11:18:53,721][INFO][main][IgniteKernal] Non-loopback local IPs: 10.42.5.132
[11:18:53,721][INFO][main][IgniteKernal] Enabled local MACs: 363E412C208F
[11:18:54,625][INFO][main][TcpDiscoverySpi] Connection check threshold is calculated: 10000
[11:18:55,024][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port [port=47500, localHost=0.0.0.0/0.0.0.0, locNodeId=6bc33d41-6dba-422b-832d-ed8cae326a00]
[11:19:03,423][SEVERE][main][TcpDiscoverySpi] Failed to get registered addresses from IP finder on start (retrying every 2000ms; change 'reconnectDelay' to configure the frequency of retries).
class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite pods IP addresses.
at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1848)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1049)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:910)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
at org.apache.ignite.Ignition.start(Ignition.java:348)
at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
Caused by: java.io.IOException: Server returned HTTP response code: 403 for URL: https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/dataobjns/endpoints/ignite
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:153)
... 20 more
[11:19:15,821][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible too long JVM pause: 500 milliseconds. 

any help is appreciated. 
Gokulnath Chidambaram Gokulnath Chidambaram
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes- Failed to retrieve Ignite pods IP addresses.

Thanks Andrei. the solution described in  https://apacheignite.readme.io/docs/rbac-authorization works with minorchange in Role access. If we have restricted access, Role and Rolebinding works. 

Thanks again for your help,

-gokul

On Thu, Nov 7, 2019 at 5:09 PM Andrei Aleksandrov <[hidden email]> wrote:

Hi,

Using the search I found that current issue was resolved in next thread:

http://apache-ignite-users.70518.x6.nabble.com/Ignite-on-RBAC-enabled-K8s-cluster-td22165.html

I guess that you should also read next article:

https://apacheignite.readme.io/docs/rbac-authorization

BR,
Andrei

11/7/2019 2:27 PM, Gokulnath Chidambaram пишет:
Hello,

I am trying to install apache ignite:2.7.6 in kubernetes cluster (deployed in aws). 
I created

1.service account
2. role access
3.rolebinding
4.deployment.
4. added namesapce (bean property)  in xml configuration file. 

I am getting the following error message.

11:18:39,724][INFO][main][PartitionsEvictManager] Evict partition permits=2
[11:18:44,719][INFO][main][ClientListenerProcessor] Client connector processor has started on TCP port 10800
[11:18:46,418][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
[11:18:48,617][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible too long JVM pause: 850 milliseconds.
[11:18:52,420][INFO][main][GridJettyRestProtocol] Command protocol successfully started [name=Jetty REST, host=/0.0.0.0, port=8080]
[11:18:53,721][INFO][main][IgniteKernal] Non-loopback local IPs: 10.42.5.132
[11:18:53,721][INFO][main][IgniteKernal] Enabled local MACs: 363E412C208F
[11:18:54,625][INFO][main][TcpDiscoverySpi] Connection check threshold is calculated: 10000
[11:18:55,024][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port [port=47500, localHost=0.0.0.0/0.0.0.0, locNodeId=6bc33d41-6dba-422b-832d-ed8cae326a00]
[11:19:03,423][SEVERE][main][TcpDiscoverySpi] Failed to get registered addresses from IP finder on start (retrying every 2000ms; change 'reconnectDelay' to configure the frequency of retries).
class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite pods IP addresses.
at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1848)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1049)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:910)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
at org.apache.ignite.Ignition.start(Ignition.java:348)
at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
Caused by: java.io.IOException: Server returned HTTP response code: 403 for URL: https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/dataobjns/endpoints/ignite
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:153)
... 20 more
[11:19:15,821][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible too long JVM pause: 500 milliseconds. 

any help is appreciated.