Connection problem between client and server

classic Classic list List threaded Threaded
11 messages Options
Jeff Jiao Jeff Jiao
Reply | Threaded
Open this post in threaded view
|

Connection problem between client and server

Hi Ignite community,

we are encountering a connection problem, in our production environment,
we have 3 Ignite servers with 500G+ of data each, and about 30 Ignite
clients running.
then we tried to start a simple Ignite client in different environment,
Citrix environment, it cannot connect to the existing servers. (firewalls
are open in both way for 47500-47509 and 47100-47109)


we did some investigating,
1. if we start only one Ignite server node, this Ignite client from Citrix
can connect to it very fast, about 2-4 secs. (Server=1, Client=1)
2. then we start all of the 3 Ignite servers, it still can connect to
servers but took longer, about 20 secs. (Server=3, Client=1)
3. then we start 2 of our services which contains an Ignite client node,
then start this Ignite client in Citrix, it took 27-30 secs. (Server=3,
Client=3)


it looks like this connection establishment process depends on the amount of
Ignite nodes.
then we did some testing in our development environment and QA environment,
we used a tool to catch the packages transport from Ignite server to Ignite
client through port 47500 when they connect,
in DEV env, we have 1 server and 4 clients running, then start a client
node, the size of packages from server to this client is about 1MB.
in QA env, we have 3 servers and 12 clients running, then start a client
node, the size of packages from servers to this client is about 4MB.
The result also can prove the point.


we guess that between our Production env and Citrix env, when client try to
connect, there are too much infos need to be transport and reach some limit
or timeout...

so are we in the correct direction for debugging? do you have any ideas or
suggestions for this situation? or does Ignite has some configurations which
can control the information transportation when connect?


Thanks a lot,
Jeff



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Connection problem between client and server

Hi Jeff!

1 MB sound like a lot of data. Especially for discovery protocol. I didn't check, what is the volume of discovery data normally, but I think, it shouldn't be measured in megabytes. If you don't create thousands of caches, of cause.
What work load do you perform on your cluster? How many caches do you create? Can you provide node configuration files?

Node startup can be slow, if you have long ping between server nodes. Make sure, that network between nodes is stable and has low latency.

Denis

ср, 3 янв. 2018 г. в 9:40, Jeff Jiao <[hidden email]>:
Hi Ignite community,

we are encountering a connection problem, in our production environment,
we have 3 Ignite servers with 500G+ of data each, and about 30 Ignite
clients running.
then we tried to start a simple Ignite client in different environment,
Citrix environment, it cannot connect to the existing servers. (firewalls
are open in both way for 47500-47509 and 47100-47109)


we did some investigating,
1. if we start only one Ignite server node, this Ignite client from Citrix
can connect to it very fast, about 2-4 secs. (Server=1, Client=1)
2. then we start all of the 3 Ignite servers, it still can connect to
servers but took longer, about 20 secs. (Server=3, Client=1)
3. then we start 2 of our services which contains an Ignite client node,
then start this Ignite client in Citrix, it took 27-30 secs. (Server=3,
Client=3)


it looks like this connection establishment process depends on the amount of
Ignite nodes.
then we did some testing in our development environment and QA environment,
we used a tool to catch the packages transport from Ignite server to Ignite
client through port 47500 when they connect,
in DEV env, we have 1 server and 4 clients running, then start a client
node, the size of packages from server to this client is about 1MB.
in QA env, we have 3 servers and 12 clients running, then start a client
node, the size of packages from servers to this client is about 4MB.
The result also can prove the point.


we guess that between our Production env and Citrix env, when client try to
connect, there are too much infos need to be transport and reach some limit
or timeout...

so are we in the correct direction for debugging? do you have any ideas or
suggestions for this situation? or does Ignite has some configurations which
can control the information transportation when connect?


Thanks a lot,
Jeff



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Jeff Jiao Jeff Jiao
Reply | Threaded
Open this post in threaded view
|

Re: Connection problem between client and server

dev2.cap <http://apache-ignite-users.70518.x6.nabble.com/file/t757/dev2.cap>  

Hi Denis,

the attachment is the packages that server transport to client when connect,
we catched it using tool (Microsoft network monitor), can you please take a
look and see what you can find?

we only configured 12 caches in our Ignite, most of them have 3 or 4
indexes, only few has 7 or 8 indexes configured...




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Connection problem between client and server

Jeff,

Your have indeed a lot of data sent over discovery. Almost all of it is taken by node attributes. Try reducing Java classpath and PATH environment variable. Also increasing of TCP window could help, because now a lot of small messages are sent instead of a few big ones.

Cross-sending it to dev list:
Guys, turns out, we are sending a lot of irrelevant information in discovery messages. Some messages contain TcpDiscoveryNode objects, which in turn have such attributes like PATH, java.class.path, sun.boot.class.path, java.library.path, org.apache.ignite.jvm.args, etc.
Some of these attributes may contain huge strings, that can sum up to megabytes of data.

I think, we should filter-out such attributes, because they are not used anywhere, but  make messages grow enormous and slow down discovery.

What do you think?

Denis

чт, 4 янв. 2018 г. в 10:42, Jeff Jiao <[hidden email]>:
dev2.cap <http://apache-ignite-users.70518.x6.nabble.com/file/t757/dev2.cap>

Hi Denis,

the attachment is the packages that server transport to client when connect,
we catched it using tool (Microsoft network monitor), can you please take a
look and see what you can find?

we only configured 12 caches in our Ignite, most of them have 3 or 4
indexes, only few has 7 or 8 indexes configured...




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Jeff Jiao Jeff Jiao
Reply | Threaded
Open this post in threaded view
|

Re: Connection problem between client and server

Hi Denis,

Thanks for the analysis.

Today we tried to reduce the Java class path(.../lib/*) when start a server
node, but it still sends all the jar files(absolute path) to client.

and we also find that, it looks like client node sends its info(path...) to
server node, and then server node sends the client's info back to client
again...

for the TCP window increasing, does Ignite has some attribute that we can
configure? or we may need to modify our production system.

btw, we are using Ignite 2.3.0.

Thanks,
Jeff



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Connection problem between client and server

Jeff,

All node attributes, including environment variables and classpath are transmitted in discovery messages. 
So try to create an environment, where your app won't have this irrelevant information.
To check, what is included into the node attributes, you can use the following code:
ClusterNode locNode = ignite.cluster().localNode();

for (Map.Entry<String, Object> e : locNode.attributes().entrySet())
System.out.println(e.getKey() + "=" + e.getValue());

for the TCP window increasing, does Ignite has some attribute that we can configure? or we may need to modify our production system.
Sorry, just ignore this recommendation. I figured out, that the size of packages, that I see in your dump, depends on MTU network parameter. It's better not to change it.

Denis

пт, 5 янв. 2018 г. в 6:18, Jeff Jiao <[hidden email]>:
Hi Denis,

Thanks for the analysis.

Today we tried to reduce the Java class path(.../lib/*) when start a server
node, but it still sends all the jar files(absolute path) to client.

and we also find that, it looks like client node sends its info(path...) to
server node, and then server node sends the client's info back to client
again...

for the TCP window increasing, does Ignite has some attribute that we can
configure? or we may need to modify our production system.

btw, we are using Ignite 2.3.0.

Thanks,
Jeff



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Jeff Jiao Jeff Jiao
Reply | Threaded
Open this post in threaded view
|

Re: Connection problem between client and server

Hi Denis,

We think the problem is when a client connect to an existing cluster, server
node will send the attributes of all the nodes in the cluster(Servers &
Clients) to that new client. it will be a lot of data here...
We may also have problems for our network configuration, we are still
investigating.

For our system, it needs a lot of jars, we cannot avoid these infos...

Does Ignite dev team give any feedback for this?



Thanks,
Jeff



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Jeff Jiao Jeff Jiao
Reply | Threaded
Open this post in threaded view
|

Re: Connection problem between client and server

In reply to this post by Denis Mekhanikov
Hi Denis,

Does Ignite dev team give any feedback for this?

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Connection problem between client and server

Hi Jeff.

Looks like my letter wasn't noticed by the developer community.


In the meanwhile make sure, that this is really the cause of the discovery process being slow. Try deploying nodes on the same environment, but without additional jar files on the classpath. Will it make discovery work faster?

Denis

ср, 10 янв. 2018 г. в 8:39, Jeff Jiao <[hidden email]>:
Hi Denis,

Does Ignite dev team give any feedback for this?

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Connection problem between client and server

Hi Jeff!

There is a workaround for the problem of huge discovery messages.
Try changing IgniteConfiguration.includeProperties parameter to an empty array. It will make Ignite filter out all environment variables. 
It should improve node connection time in your case.

Add the following lines to IgniteConfiguration in your XML:
<property name="includeProperties">
    <array></array>
</property>

Denis 

ср, 10 янв. 2018 г. в 11:37, Denis Mekhanikov <[hidden email]>:
Hi Jeff.

Looks like my letter wasn't noticed by the developer community.


In the meanwhile make sure, that this is really the cause of the discovery process being slow. Try deploying nodes on the same environment, but without additional jar files on the classpath. Will it make discovery work faster?

Denis

ср, 10 янв. 2018 г. в 8:39, Jeff Jiao <[hidden email]>:
Hi Denis,

Does Ignite dev team give any feedback for this?

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Jeff Jiao Jeff Jiao
Reply | Threaded
Open this post in threaded view
|

Re: Connection problem between client and server

Thank you very much Denis, we will give it a try and let you know, I think
this should solve our problem.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/