Ignite JDBC connection pooling mechanism

classic Classic list List threaded Threaded
9 messages Options
Sanjaya Sanjaya
Reply | Threaded
Open this post in threaded view
|

Ignite JDBC connection pooling mechanism

Hi All,

we are trying to use HIkari connection pooling with ignite JdbcThinDriver.
we are facing issue as


Any idea what is the supported connection pooling mechanism work with
IgniteThinDriver


ERROR LOG
==================

WARN  com.zaxxer.hikari.pool.ProxyConnection.157 sm-event-consumer prod
sm-event-consumer-v1-55f4db767d-2kskt - HikariPool-1 - Connection
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection@68f0e2a1 marked as
broken because of SQLSTATE(08006), ErrorCode(0)

java.sql.SQLException: Failed to communicate with Ignite cluster.

                at
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:760)

                at
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.executeBatch(JdbcThinStatement.java:651)

                at
com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128)

                at
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)

                at
org.springframework.jdbc.core.JdbcTemplate.lambda$batchUpdate$2(JdbcTemplate.java:950)

                at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:617)

                at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:647)

                at
org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:936)

                at
org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.batchUpdate(NamedParameterJdbcTemplate.java:366)

                at
com.ecoenergy.cortix.sm.event.cache.SMIgniteCacheManager.updateObjectStates(SMIgniteCacheManager.java:118)

                at
com.ecoenergy.cortix.sm.event.notifcator.SMIgniteNotificator.notify(SMIgniteNotificator.java:69)

                at
com.ecoenergy.cortix.sm.event.eventhandler.ObjectEventHandler.notify(ObjectEventHandler.java:100)

                at
com.ecoenergy.cortix.sm.event.eventhandler.ObjectEventHandler.receiveEvents(ObjectEventHandler.java:86)

                at
com.ecoenergy.cortix.sm.event.consumer.ObjectEventConsumer.processObjectEvents(ObjectEventConsumer.java:60)





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Vladimir Pligin Vladimir Pligin
Reply | Threaded
Open this post in threaded view
|

Re: Ignite JDBC connection pooling mechanism

Hi,

What java version do you use? What about Hikari & Ignite versions? Do you
have SSL enabled?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Sanjaya Sanjaya
Reply | Threaded
Open this post in threaded view
|

Re: Ignite JDBC connection pooling mechanism

Please find details below

Java - 1.8
Hikari - 3.4.1
Ignite - 2.6.0
SSL Enabled - false 

Thanks,
Sanjaya





 

On Mon, Nov 2, 2020 at 8:11 PM Vladimir Pligin <[hidden email]> wrote:
Hi,

What java version do you use? What about Hikari & Ignite versions? Do you
have SSL enabled?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Vladimir Pligin Vladimir Pligin
Reply | Threaded
Open this post in threaded view
|

Re: Ignite JDBC connection pooling mechanism

I wasn't able to reproduce that hurriedly. What Hikari settings do you have?
Maybe you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite JDBC connection pooling mechanism

In reply to this post by Sanjaya
Hello!

Are you sure that the Ignite cluster is in fact up? :)

If it is, maybe your usage patterns of this pool somehow assign the connection to two different threads, which try to do queries in parallel. In theory, this is what connection pools are explicitly created to avoid, but maybe there's some knob you have to turn to actually make them thread-exclusive.

Also, does it happen every time? How soon would it happen?

Regards,
--
Ilya Kasnacheev


пн, 2 нояб. 2020 г. в 12:31, Sanjaya <[hidden email]>:
Hi All,

we are trying to use HIkari connection pooling with ignite JdbcThinDriver.
we are facing issue as


Any idea what is the supported connection pooling mechanism work with
IgniteThinDriver


ERROR LOG
==================

WARN  com.zaxxer.hikari.pool.ProxyConnection.157 sm-event-consumer prod
sm-event-consumer-v1-55f4db767d-2kskt - HikariPool-1 - Connection
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection@68f0e2a1 marked as
broken because of SQLSTATE(08006), ErrorCode(0)

java.sql.SQLException: Failed to communicate with Ignite cluster.

                at
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:760)

                at
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.executeBatch(JdbcThinStatement.java:651)

                at
com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128)

                at
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)

                at
org.springframework.jdbc.core.JdbcTemplate.lambda$batchUpdate$2(JdbcTemplate.java:950)

                at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:617)

                at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:647)

                at
org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:936)

                at
org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.batchUpdate(NamedParameterJdbcTemplate.java:366)

                at
com.ecoenergy.cortix.sm.event.cache.SMIgniteCacheManager.updateObjectStates(SMIgniteCacheManager.java:118)

                at
com.ecoenergy.cortix.sm.event.notifcator.SMIgniteNotificator.notify(SMIgniteNotificator.java:69)

                at
com.ecoenergy.cortix.sm.event.eventhandler.ObjectEventHandler.notify(ObjectEventHandler.java:100)

                at
com.ecoenergy.cortix.sm.event.eventhandler.ObjectEventHandler.receiveEvents(ObjectEventHandler.java:86)

                at
com.ecoenergy.cortix.sm.event.consumer.ObjectEventConsumer.processObjectEvents(ObjectEventConsumer.java:60)





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Sanjaya Sanjaya
Reply | Threaded
Open this post in threaded view
|

Re: Ignite JDBC connection pooling mechanism

Hi,

I truly appreciated the support we are getting from the community.

As of now we don't have a re-producer, The above issue basically comes once in a while. 

The server is up and running, Note: The ignite cluster has been installed in azure kubernetes cluster as statefulset pods.

We have other application pods they frequently talk to ignite.

While analyzing we understood that the application pod which creating problem, is running with ignite 2.6.0 where as the Ignite server is 2.8.1 for us

We followed below steps and deployed in production

1- Changed version of Hikari to 3.4.5
2- Ignite core changed to 2.8.1
3- Spring boot was auto configuring jdbc templates (with hikari), we disabled auto configuration and configured manually.

We deployed the application and we are monitoring, and will publish the result.


Thanks,
Sanjaya
    




On Tue, Nov 3, 2020 at 8:28 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Are you sure that the Ignite cluster is in fact up? :)

If it is, maybe your usage patterns of this pool somehow assign the connection to two different threads, which try to do queries in parallel. In theory, this is what connection pools are explicitly created to avoid, but maybe there's some knob you have to turn to actually make them thread-exclusive.

Also, does it happen every time? How soon would it happen?

Regards,
--
Ilya Kasnacheev


пн, 2 нояб. 2020 г. в 12:31, Sanjaya <[hidden email]>:
Hi All,

we are trying to use HIkari connection pooling with ignite JdbcThinDriver.
we are facing issue as


Any idea what is the supported connection pooling mechanism work with
IgniteThinDriver


ERROR LOG
==================

WARN  com.zaxxer.hikari.pool.ProxyConnection.157 sm-event-consumer prod
sm-event-consumer-v1-55f4db767d-2kskt - HikariPool-1 - Connection
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection@68f0e2a1 marked as
broken because of SQLSTATE(08006), ErrorCode(0)

java.sql.SQLException: Failed to communicate with Ignite cluster.

                at
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:760)

                at
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.executeBatch(JdbcThinStatement.java:651)

                at
com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128)

                at
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)

                at
org.springframework.jdbc.core.JdbcTemplate.lambda$batchUpdate$2(JdbcTemplate.java:950)

                at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:617)

                at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:647)

                at
org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:936)

                at
org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.batchUpdate(NamedParameterJdbcTemplate.java:366)

                at
com.ecoenergy.cortix.sm.event.cache.SMIgniteCacheManager.updateObjectStates(SMIgniteCacheManager.java:118)

                at
com.ecoenergy.cortix.sm.event.notifcator.SMIgniteNotificator.notify(SMIgniteNotificator.java:69)

                at
com.ecoenergy.cortix.sm.event.eventhandler.ObjectEventHandler.notify(ObjectEventHandler.java:100)

                at
com.ecoenergy.cortix.sm.event.eventhandler.ObjectEventHandler.receiveEvents(ObjectEventHandler.java:86)

                at
com.ecoenergy.cortix.sm.event.consumer.ObjectEventConsumer.processObjectEvents(ObjectEventConsumer.java:60)





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Sanjaya Sanjaya
Reply | Threaded
Open this post in threaded view
|

Re: Ignite JDBC connection pooling mechanism

The above idea did not work. The ignite setup is as follows/

Ignite 2.8.1
Hikari 3.4.5
Java 1.8
Spring JdbcTemplate
Apache ignite is in Azure K8S cluster, and the service is being exposed as a Azure internal load balancer.

The api works well for some time, (till  1 hour of restart), after that we are getting below error, if we restart then it works for another approximately 1 hour.

is it not advisable to use a connection pooling  mechanism with ignite, if yes, then what is the best way to serve concurrent requests ? is it kind creating connection per user request and close once job done ?

Request to help on this we are completely stuck on this use case in production.


LOGS IN IGNITE
=========================
    ^-- System thread pool [active=0, idle=6, qSize=0]
[06:51:33,191][SEVERE][grid-nio-worker-client-listener-2-#32][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=2, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-2, igniteInstanceName=null, finished=false, heartbeatTs=1604472690975, hashCode=1771197860, interrupted=false, runner=grid-nio-worker-client-listener-2-#32]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, closeSocket=true, outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl [locAddr=/10.188.0.115:10800, rmtAddr=/10.189.3.42:46262, createTime=1604464791433, closeTime=0, bytesSent=46, bytesRcvd=51, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1604464791514, lastSndTime=1604464791514, lastRcvTime=1604464791433, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]], accepted=true, markedForClose=false]]]
java.io.IOException: Operation timed out
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1162)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2449)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2216)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1857)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
[06:51:33,191][WARNING][grid-nio-worker-client-listener-2-#32][ClientListenerProcessor] Client disconnected abruptly due to network connection loss or because the connection was left open on application shutdown. [cls=class o.a.i.i.util.nio.GridNioException, msg=Operation timed out]
[06:52:25,552][INFO][db-checkpoint-thread-#68][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modified) [checkpointBeforeLockTime=17ms, checkpointLockWait=0ms, checkpointListenersExecuteTime=21ms, checkpointLockHoldTime=23ms, reason='timeout']
[06:52:25,716][INFO][grid-timeout-worker-#23][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=747a4939, uptime=41 days, 13:40:47.769]


LOGS IN APPLICATION SIDE
=======================
03-11-2020 23:00:44.027 [http-nio-8080-exec-4] WARN  com.zaxxer.hikari.pool.ProxyConnection.157 cache-query-service prod v1 cache-query-service-v1-5c5d8cd74d-jgnbb - HikariPool-1 - Connection org.apache.ignite.internal.jdbc.thin.JdbcThinConnection@62708a92 marked as broken because of SQLSTATE(08006), ErrorCode(0)
java.sql.SQLException: Failed to communicate with Ignite cluster.
at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:760)
at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:212)
at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.executeQuery(JdbcThinStatement.java:123)
at com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:111)
at com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:439)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:376)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:452)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:462)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:473)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:480)
at com.ecoenergy.cortix.cache.repository.IgniteRepositoryImpl.lambda$health$0(IgniteRepositoryImpl.java:26)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
at com.ecoenergy.cortix.cache.repository.IgniteRepositoryImpl.health(IgniteRepositoryImpl.java:24)
at com.ecoenergy.cortix.cache.handler.health.HealthCheckHandler.health(HealthCheckHandler.java:31)
at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:95)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:43)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:108)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateHealth(HealthEndpointSupport.java:119)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:105)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:83)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:70)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:81)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:70)
at jdk.internal.reflect.GeneratedMethodAccessor99.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:279)
at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:77)
at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:305)
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:388)
at jdk.internal.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.ecoenergy.cortix.cache.controller.RequestLoggingFilter.doFilterInternal(RequestLoggingFilter.java:38)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:209)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:108)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.microsoft.applicationinsights.web.internal.WebRequestTrackingFilter.doFilter(WebRequestTrackingFilter.java:129)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1579)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketException: Operation timed out (Read failed)
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at java.base/java.io.BufferedInputStream.fill(Unknown Source)
at java.base/java.io.BufferedInputStream.read1(Unknown Source)
at java.base/java.io.BufferedInputStream.read(Unknown Source)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.read(JdbcThinTcpIo.java:605)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.read(JdbcThinTcpIo.java:586)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.readResponse(JdbcThinTcpIo.java:525)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.sendRequest(JdbcThinTcpIo.java:510)
at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:747)
... 100 common frames omitted
03-11-2020 23:00:44.029 [http-nio-8080-exec-4] WARN  c.e.cortix.cache.repository.IgniteRepositoryImpl.29 cache-query-service prod v1 cache-query-service-v1-5c5d8cd74d-jgnbb - Ignite is not reachable
org.springframework.dao.DataAccessResourceFailureException: StatementCallback; SQL [SELECT 1]; Failed to communicate with Ignite cluster.; nested exception is java.sql.SQLException: Failed to communicate with Ignite cluster.
at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:107)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1443)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:388)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:452)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:462)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:473)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:480)
at com.ecoenergy.cortix.cache.repository.IgniteRepositoryImpl.lambda$health$0(IgniteRepositoryImpl.java:26)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
at com.ecoenergy.cortix.cache.repository.IgniteRepositoryImpl.health(IgniteRepositoryImpl.java:24)
at com.ecoenergy.cortix.cache.handler.health.HealthCheckHandler.health(HealthCheckHandler.java:31)
at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:95)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:43)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:108)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateHealth(HealthEndpointSupport.java:119)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:105)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:83)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:70)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:81)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:70)
at jdk.internal.reflect.GeneratedMethodAccessor99.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:279)
at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:77)
at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:305)
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:388)
at jdk.internal.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.ecoenergy.cortix.cache.controller.RequestLoggingFilter.doFilterInternal(RequestLoggingFilter.java:38)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:209)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:108)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.microsoft.applicationinsights.web.internal.WebRequestTrackingFilter.doFilter(WebRequestTrackingFilter.java:129)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1579)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.sql.SQLException: Failed to communicate with Ignite cluster.
at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:760)
at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:212)
at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.executeQuery(JdbcThinStatement.java:123)
at com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:111)
at com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:439)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:376)
... 94 common frames omitted
Caused by: java.net.SocketException: Operation timed out (Read failed)
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at java.base/java.io.BufferedInputStream.fill(Unknown Source)
at java.base/java.io.BufferedInputStream.read1(Unknown Source)
at java.base/java.io.BufferedInputStream.read(Unknown Source)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.read(JdbcThinTcpIo.java:605)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.read(JdbcThinTcpIo.java:586)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.readResponse(JdbcThinTcpIo.java:525)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.sendRequest(JdbcThinTcpIo.java:510)
at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:747)
... 100 common frames omitted
03-11-2020 23:00:44.032 [http-nio-8080-exec-4] INFO  c.e.cortix.cache.handler.health.HealthCheckHandler.38 cache-query-service prod v1 cache-query-service-v1-5c5d8cd74d-jgnbb - Health is DOWN
03-11-2020 23:00:52.114 [http-nio-8080-exec-6] INFO  c.e.cortix.cache.controller.RequestLoggingFilter.37 cache-query-service prod v1 cache-query-service-v1-5c5d8cd74d-jgnbb 



















On Wed, Nov 4, 2020 at 11:20 AM Sanjaya Kumar Sahoo <[hidden email]> wrote:
Hi,

I truly appreciated the support we are getting from the community.

As of now we don't have a re-producer, The above issue basically comes once in a while. 

The server is up and running, Note: The ignite cluster has been installed in azure kubernetes cluster as statefulset pods.

We have other application pods they frequently talk to ignite.

While analyzing we understood that the application pod which creating problem, is running with ignite 2.6.0 where as the Ignite server is 2.8.1 for us

We followed below steps and deployed in production

1- Changed version of Hikari to 3.4.5
2- Ignite core changed to 2.8.1
3- Spring boot was auto configuring jdbc templates (with hikari), we disabled auto configuration and configured manually.

We deployed the application and we are monitoring, and will publish the result.


Thanks,
Sanjaya
    




On Tue, Nov 3, 2020 at 8:28 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Are you sure that the Ignite cluster is in fact up? :)

If it is, maybe your usage patterns of this pool somehow assign the connection to two different threads, which try to do queries in parallel. In theory, this is what connection pools are explicitly created to avoid, but maybe there's some knob you have to turn to actually make them thread-exclusive.

Also, does it happen every time? How soon would it happen?

Regards,
--
Ilya Kasnacheev


пн, 2 нояб. 2020 г. в 12:31, Sanjaya <[hidden email]>:
Hi All,

we are trying to use HIkari connection pooling with ignite JdbcThinDriver.
we are facing issue as


Any idea what is the supported connection pooling mechanism work with
IgniteThinDriver


ERROR LOG
==================

WARN  com.zaxxer.hikari.pool.ProxyConnection.157 sm-event-consumer prod
sm-event-consumer-v1-55f4db767d-2kskt - HikariPool-1 - Connection
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection@68f0e2a1 marked as
broken because of SQLSTATE(08006), ErrorCode(0)

java.sql.SQLException: Failed to communicate with Ignite cluster.

                at
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:760)

                at
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.executeBatch(JdbcThinStatement.java:651)

                at
com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128)

                at
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)

                at
org.springframework.jdbc.core.JdbcTemplate.lambda$batchUpdate$2(JdbcTemplate.java:950)

                at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:617)

                at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:647)

                at
org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:936)

                at
org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.batchUpdate(NamedParameterJdbcTemplate.java:366)

                at
com.ecoenergy.cortix.sm.event.cache.SMIgniteCacheManager.updateObjectStates(SMIgniteCacheManager.java:118)

                at
com.ecoenergy.cortix.sm.event.notifcator.SMIgniteNotificator.notify(SMIgniteNotificator.java:69)

                at
com.ecoenergy.cortix.sm.event.eventhandler.ObjectEventHandler.notify(ObjectEventHandler.java:100)

                at
com.ecoenergy.cortix.sm.event.eventhandler.ObjectEventHandler.receiveEvents(ObjectEventHandler.java:86)

                at
com.ecoenergy.cortix.sm.event.consumer.ObjectEventConsumer.processObjectEvents(ObjectEventConsumer.java:60)





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Vladimir Pligin Vladimir Pligin
Reply | Threaded
Open this post in threaded view
|

Re: Ignite JDBC connection pooling mechanism

In general it should be ok to use connection pooling with Ignite. Is your
network ok? It look like a connection is being closed because of network
issues.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Sanjaya Sanjaya
Reply | Threaded
Open this post in threaded view
|

Re: Ignite JDBC connection pooling mechanism

We solved the problem by removing the complete hikari connection pooling mechanism.

Instead we use IgniteJdbcThinDataSource (https://apacheignite-sql.readme.io/docs/jdbc-driver) , 

After doing a few hit and trials, we concluded ignite does not require connection pooling in the client side (like we do in RDBMS) 
, instead let the Ignite server handle the SQL queries, by providing appropriate client connection details.  




On Fri, Nov 6, 2020 at 7:01 PM Vladimir Pligin <[hidden email]> wrote:
In general it should be ok to use connection pooling with Ignite. Is your
network ok? It look like a connection is being closed because of network
issues.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/