Kubernetes on Azure: Connection timed out (Connection timed out)

classic Classic list List threaded Threaded
11 messages Options
steve.hostettler steve.hostettler
Reply | Threaded
Open this post in threaded view
|

Kubernetes on Azure: Connection timed out (Connection timed out)

This post was updated on .
Hello,

Since this morning I get a strange Timeout on the connection to the cluster
even with only one node. I should specify that this on Azure
It used to work just fine until yesterday evening. At that point I destroyed
the clustered and recreated it this morning.

I do so with a batch so it is supposed to be the same all the time. That's
what I do not understand, I cannot see what change to cause that problem.
Here is some logs

any idea??

deployment/processing-engine (master)
$ kubectl get  svc
NAME                             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
kubernetes                       ClusterIP   10.0.0.1      <none>        443/TCP          88m
processing-engine-pe-v1-ignite   ClusterIP   None          <none>        47500/TCP        7m12s
processing-engine-pe-v1-master   NodePort    10.0.35.252   <none>        8080:32566/TCP   7m12s


2020-07-28 09:55:44,105 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Node version to set: 2.7.5#20190603-sha1:be4f2a15
2020-07-28 09:55:44,117 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Using parameter [localHost=0.0.0.0]
2020-07-28 09:55:44,117 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Using parameter [localPort=47500]
2020-07-28 09:55:44,117 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Using parameter [localPortRange=0]
2020-07-28 09:55:44,118 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Using parameter [threadPri=10]
2020-07-28 09:55:44,118 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Using parameter [failureDetectionTimeout=10000]
2020-07-28 09:55:44,118 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Using parameter [ipFinder=TcpDiscoveryIpFinderAdapter
[shared=true]]
2020-07-28 09:55:44,118 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Using parameter [ipFinderCleanFreq=60000]
2020-07-28 09:55:44,118 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Using parameter [metricsUpdateFreq=2000]
2020-07-28 09:55:44,118 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Using parameter [statsPrintFreq=0]
2020-07-28 09:55:44,122 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Registered SPI MBean:
org.apache:clsLdr=407b70da,igniteInstanceName=com.wolterskluwer.processing,group=SPIs,name=TcpDiscoverySpi
2020-07-28 09:55:44,133 INFO
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Connection check threshold is calculated: 10000
2020-07-28 09:55:44,134 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Grid runnable
started: tcp-disco-msg-worker
2020-07-28 09:55:44,135 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Message worker
started [locNodeId=32c22aa5-430c-4352-a8d3-55e0a7b2671e]
2020-07-28 09:55:44,135 INFO
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Successfully bound to TCP port [port=47500,
localHost=0.0.0.0/0.0.0.0, locNodeId=32c22aa5-430c-4352-a8d3-55e0a7b2671e]
2020-07-28 09:55:44,137 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Local node initialized: TcpDiscoveryNode
[id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1],
sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500,
/127.0.0.1:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15,
isClient=false]
2020-07-28 09:55:44,138 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-srvr-#3%com.wolterskluwer.processing%) Grid runnable started:
tcp-disco-srvr
2020-07-28 09:55:44,149 DEBUG
[org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder]
(ServerService Thread Pool -- 11) Getting Apache Ignite endpoints from:
https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/processing-engine-pe-v1-ignite
2020-07-28 09:55:50,187 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Message has been
added to queue: TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode
[id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1],
sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500,
/127.0.0.1:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15,
isClient=false], failedNodeId=null, status=0,
super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=44e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null,
topVer=0, pendingIdx=0, failedNodes=null, isClient=false]]
2020-07-28 09:55:50,187 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Processing message
[cls=TcpDiscoveryStatusCheckMessage,
id=44e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e]
2020-07-28 09:55:50,187 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Ignore message,
local node order is not initialized [msg=TcpDiscoveryStatusCheckMessage
[creatorNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e,
addrs=[10.244.0.8, 127.0.0.1],
sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500,
/127.0.0.1:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15,
isClient=false], failedNodeId=null, status=0,
super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=44e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null,
topVer=0, pendingIdx=0, failedNodes=null, isClient=false]],
locNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e,
addrs=[10.244.0.8, 127.0.0.1],
sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500,
/127.0.0.1:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15,
isClient=false]]
2020-07-28 09:55:56,239 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Message has been
added to queue: TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode
[id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1],
sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500,
/127.0.0.1:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15,
isClient=false], failedNodeId=null, status=0,
super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=54e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null,
topVer=0, pendingIdx=0, failedNodes=null, isClient=false]]
2020-07-28 09:55:56,239 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Processing message
[cls=TcpDiscoveryStatusCheckMessage,
id=54e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e]
2020-07-28 09:55:56,240 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Ignore message,
local node order is not initialized [msg=TcpDiscoveryStatusCheckMessage
[creatorNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e,
addrs=[10.244.0.8, 127.0.0.1],
sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500,
/127.0.0.1:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15,
isClient=false], failedNodeId=null, status=0,
super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=54e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null,
topVer=0, pendingIdx=0, failedNodes=null, isClient=false]],
locNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e,
addrs=[10.244.0.8, 127.0.0.1],
sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500,
/127.0.0.1:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15,
isClient=false]]
2020-07-28 09:56:02,294 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Message has been
added to queue: TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode
[id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1],
sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500,
/127.0.0.1:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15,
isClient=false], failedNodeId=null, status=0,
super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=64e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null,
topVer=0, pendingIdx=0, failedNodes=null, isClient=false]]
2020-07-28 09:56:02,294 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Processing message
[cls=TcpDiscoveryStatusCheckMessage,
id=64e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e]
2020-07-28 09:56:02,295 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Ignore message,
local node order is not initialized [msg=TcpDiscoveryStatusCheckMessage
[creatorNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e,
addrs=[10.244.0.8, 127.0.0.1],
sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500,
/127.0.0.1:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15,
isClient=false], failedNodeId=null, status=0,
super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=64e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null,
topVer=0, pendingIdx=0, failedNodes=null, isClient=false]],
locNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e,
addrs=[10.244.0.8, 127.0.0.1],
sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500,
/127.0.0.1:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15,
isClient=false]]


and here the exception


2020-07-28 09:57:51,279 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Message has been
added to queue: TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode
[id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1],
sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500,
/127.0.0.1:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15,
isClient=false], failedNodeId=null, status=0,
super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=85e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null,
topVer=0, pendingIdx=0, failedNodes=null, isClient=false]]
2020-07-28 09:57:51,279 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Processing message
[cls=TcpDiscoveryStatusCheckMessage,
id=85e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e]
2020-07-28 09:57:51,280 DEBUG
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]
(tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Ignore message,
local node order is not initialized [msg=TcpDiscoveryStatusCheckMessage
[creatorNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e,
addrs=[10.244.0.8, 127.0.0.1],
sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500,
/127.0.0.1:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15,
isClient=false], failedNodeId=null, status=0,
super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=85e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null,
topVer=0, pendingIdx=0, failedNodes=null, isClient=false]],
locNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e,
addrs=[10.244.0.8, 127.0.0.1],
sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500,
/127.0.0.1:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15,
isClient=false]]
2020-07-28 09:57:53,999 ERROR
[org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread
Pool -- 11) Failed to get registered addresses from IP finder on start
(retrying every 2000ms; change 'reconnectDelay' to configure the frequency
of retries).: class org.apache.ignite.spi.IgniteSpiException: Failed to
retrieve Ignite pods IP addresses.
        at
org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172)
        at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)
        at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1848)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1049)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:910)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
Caused by: java.net.ConnectException: Connection timed out (Connection timed
out)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
jose farfan jose farfan
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes : Connection timed out (Connection timed out)

Hi

I have similar problem.

For some reason this host are not resolve "kubernetes.default.svc.cluster.local".
I am still investigating the reason.
But in my case "kubernetes.default" was resolved, then
I have added the masterUrl in the config file
(I added directly the ip:                         <property name="masterUrl" value="https://172.22.100.9:6443"/> " of Master API.

<!--
    Configuration example with Kubernetes IP finder enabled.
-->
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd">

    <bean class="org.apache.ignite.configuration.IgniteConfiguration">

        <!-- Enabling Apache Ignite Persistent Store. -->
        <property name="dataStorageConfiguration">
            <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
                <property name="defaultDataRegionConfiguration">
                    <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                        <property name="persistenceEnabled" value="true"/>
                    </bean>
                </property>
            </bean>
        </property>

        <!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <!--
                        Enables Kubernetes IP finder and setting custom namespace and service names.
                    -->
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder">
                        <property name="namespace" value="development"/>
                        <property name="serviceName" value="ignite"/>
                        <property name="masterUrl" value="https://172.22.100.9:6443"/>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</beans>


You need to find your master api address.


On Tue, 28 Jul 2020 at 13:19, steve.hostettler <[hidden email]> wrote:
Hello,

making some progress
I added the missing

hostNetwork: true

and now I get

Caused by: java.net.UnknownHostException:
kubernetes.default.svc.cluster.local



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
steve.hostettler steve.hostettler
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes : Connection timed out (Connection timed out)

Hello Jose,

thanks for the answer. Are you on azure as well by any chance?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
steve.hostettler steve.hostettler
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes : Connection timed out (Connection timed out)

So it appears that since this morning I am not able to access the cluster
from within a pod in Azure. Very very strange

/ # wget http://kubernetes.default.svc.cluster.local:443/api
Connecting to kubernetes.default.svc.cluster.local:443 (10.0.0.1)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
steve.hostettler steve.hostettler
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes : Connection timed out (Connection timed out)

Running

 curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H
"Authorization: Bearer $(cat
/var/run/secrets/kubernetes.io/serviceaccount/token)" -H "Acept:
application/json"
https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/processing-engine-pe-v1-ignite

On my local K8S works but not on Azure. On Azure I end up with a Timeout.

Any Idea?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
jose farfan jose farfan
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes : Connection timed out (Connection timed out)

Hi Steve

No, I am using a on-premise cluster.

can you access a pod using the shell, and try to ping kubernetes.default.svc.cluster.local.

BR
Jose



On Tue, 28 Jul 2020 at 16:05, steve.hostettler <[hidden email]> wrote:
Running

 curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H
"Authorization: Bearer $(cat
/var/run/secrets/kubernetes.io/serviceaccount/token)" -H "Acept:
application/json"
https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/processing-engine-pe-v1-ignite

On my local K8S works but not on Azure. On Azure I end up with a Timeout.

Any Idea?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
jose farfan jose farfan
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes : Connection timed out (Connection timed out)

Hi

Also I am using this file ignite-rbac.yaml. Maybe can help you,..
I am still learning how to deploy ignite in the cluster.

BR
Jose


On Tue, 28 Jul 2020 at 20:53, jose farfan <[hidden email]> wrote:
Hi Steve

No, I am using a on-premise cluster.

can you access a pod using the shell, and try to ping kubernetes.default.svc.cluster.local.

BR
Jose



On Tue, 28 Jul 2020 at 16:05, steve.hostettler <[hidden email]> wrote:
Running

 curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H
"Authorization: Bearer $(cat
/var/run/secrets/kubernetes.io/serviceaccount/token)" -H "Acept:
application/json"
https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/processing-engine-pe-v1-ignite

On my local K8S works but not on Azure. On Azure I end up with a Timeout.

Any Idea?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

ignite-rbac.yaml (1K) Download Attachment
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes : Connection timed out (Connection timed out)

Steve, Jose,

Try to follow this instruction created specifically for Azure Kubernetes Service (AKS):

Not, if the cluster (server nodes) are planned to be deployed as virtual machines (on-premises) while the thick clients will be running in AKS then keep this configuration option in mind (it's already supported by GridGain and will be available in Ignite 2.9): https://www.gridgain.com/docs/latest/developers-guide/clustering/running-client-nodes-behind-nat

-
Denis


On Tue, Jul 28, 2020 at 12:03 PM jose farfan <[hidden email]> wrote:
Hi

Also I am using this file ignite-rbac.yaml. Maybe can help you,..
I am still learning how to deploy ignite in the cluster.

BR
Jose


On Tue, 28 Jul 2020 at 20:53, jose farfan <[hidden email]> wrote:
Hi Steve

No, I am using a on-premise cluster.

can you access a pod using the shell, and try to ping kubernetes.default.svc.cluster.local.

BR
Jose



On Tue, 28 Jul 2020 at 16:05, steve.hostettler <[hidden email]> wrote:
Running

 curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H
"Authorization: Bearer $(cat
/var/run/secrets/kubernetes.io/serviceaccount/token)" -H "Acept:
application/json"
https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/processing-engine-pe-v1-ignite

On my local K8S works but not on Azure. On Azure I end up with a Timeout.

Any Idea?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
steve.hostettler steve.hostettler
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes : Connection timed out (Connection timed out)

Hello Denis,

same code, same config, today it works so it was an transient AKS thing. no
idea what

Best Regards



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
steve.hostettler steve.hostettler
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes : Connection timed out (Connection timed out)

In reply to this post by dmagda
Hello Dennis,

so it works again, it was apparently a transient Azure problem when creating
new clusters.
FYI the documentation and
https://github.com/apache/ignite/tree/master/modules/kubernetes/config/az
is not up to date and gave errors when executing them (from a K8S
perspective)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes : Connection timed out (Connection timed out)

Steve, 

Excellent, glad the problem is gone. The community is working on the new documentation and, as far as I know, the Kubernetes instructions will be brought up-to-date. Thanks for pointing that out.

-
Denis


On Wed, Jul 29, 2020 at 2:53 AM steve.hostettler <[hidden email]> wrote:
Hello Dennis,

so it works again, it was apparently a transient Azure problem when creating
new clusters.
FYI the documentation and
https://github.com/apache/ignite/tree/master/modules/kubernetes/config/az
is not up to date and gave errors when executing them (from a K8S
perspective)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/