Toxiproxy

classic Classic list List threaded Threaded
4 messages Options
Delian Delian
Reply | Threaded
Open this post in threaded view
|

Toxiproxy

Is anyone aware whether Toxiproxy can be set up to sit between Ignite nodes
in order to look at how things behave under various network conditions ? I
am new to Ignite and am wondering whether my thinking is flawed.

I have a simple 3 node cluster using the static TCP IP finder - all fine.
I've now set up Toxiproxy with a proxy (per node) for discovery,
communication, shared mem and timeserver as the config file for each node
allows me to explicitly set ports for these.  Finally, the ip finders in the
node configs point to the cluster nodes going through ToxiProxy - not
direct.

Nodes fire up but don't cluster, I'm seeing a lot of activity in Toxiproxy
console where by nodes are sending requests on ports other than the above
(in some cases incrementing so I assume a range is being attempted). As I
have not explicitly set these up in Toxiproxy the requests seem to get
routed to the upstream node on 47500 (service disco) which is obviously
wrong in some cases. I see a number of open ports for the process - some of
which I have set but some not and they are not the same across the nodes.
       
1) Can I statically set all these ports (even if I knew what they were) so I
can create proxies for them with the hope that allows me to cluster up ?

2) I believe a ring topology is in play - are the hosts/ip's set up in the
service disco config always used, i.e. so everything goes through Toxiproxy
or is there the possibility they will connect direct and bypass ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Mikhail Mikhail
Reply | Threaded
Open this post in threaded view
|

Re: Toxiproxy

Hi Delian,

I used it to test timeout for client nodes, I attached the example, I think it can be adapted for your purposes.

>I've now set up Toxiproxy with a proxy (per node) for discovery,
>communication, shared mem and timeserver as the config file for each node
>allows me to explicitly set ports for these. 
Ignite uses only communication and discovery by default. shared mem can't be proxied and it isn't used by default, and timeserver - is an obsolete configuration that isn't used anymore.

So you need to proxy only Discovery and Communication. In the attached example I created a server node that listens on port 47600 for discovery and 47200 for communication.
Client node has in the discovery list the following address: "localhost:47500" so at port 47500 we have Toxiproxy. Also note, that the very first server node need to connect it self, so I left "localhost:47600" in the discovery list.
Okay, now client node will connect to the server node via Toxiproxy, but, we can't specify particular address/port for the client to communicate with the server, because ignite uses autodiscovery and all nodes send communication address on node join view discovery protocol, and there a trick:
to server node I added address resolver, it is used for heterogeneous networks, when clients can't directly connect to servers and need to use another set of addresses:
@NotNull private static AddressResolver getRslvr(String s) {
return new AddressResolver() {
@Override public Collection<InetSocketAddress> getExternalAddresses(
InetSocketAddress addr) throws IgniteCheckedException {
List<InetSocketAddress> res = Collections.singletonList(
new InetSocketAddress(addr.getHostName(),
addr.getPort() == 0 ? 0 : addr.getPort() - 100)
);

System.out.println(Thread.currentThread().getName() + " " + s + "resolve: " + addr + " ->" + res);
return res;
}
};
}
So what does this code mean? The server listens on port 47200, but before to send it to the client, it will ask AdressResolver to convert its address to something new, in this case, I just reducer port number by 100. so Client will get localhost:47100 address for communication, however, even with new address localhost:47100, Server node will send its original address to client too localhost:47200, to rid of this original address I added this lines:
Map<String, List<String>> userAttr = Collections.singletonMap("TcpCommunicationSpi.comm.tcp.addrs",
Collections.emptyList());

igniteCfg.setUserAttributes(userAttr);
so, now the client will get an empty list instead of original communication address and will have to use adress returned by AddressResolver which is localhost:47100 where we have Toxiproxy.

I think this example can be adapted to your case. 
if you have any further question feel free to mail me, also I will appreciate if you will share the result of your work.

Thanks,
Mike.

ср, 29 мая 2019 г. в 15:37, Delian <[hidden email]>:
Is anyone aware whether Toxiproxy can be set up to sit between Ignite nodes
in order to look at how things behave under various network conditions ? I
am new to Ignite and am wondering whether my thinking is flawed.

I have a simple 3 node cluster using the static TCP IP finder - all fine.
I've now set up Toxiproxy with a proxy (per node) for discovery,
communication, shared mem and timeserver as the config file for each node
allows me to explicitly set ports for these.  Finally, the ip finders in the
node configs point to the cluster nodes going through ToxiProxy - not
direct.

Nodes fire up but don't cluster, I'm seeing a lot of activity in Toxiproxy
console where by nodes are sending requests on ports other than the above
(in some cases incrementing so I assume a range is being attempted). As I
have not explicitly set these up in Toxiproxy the requests seem to get
routed to the upstream node on 47500 (service disco) which is obviously
wrong in some cases. I see a number of open ports for the process - some of
which I have set but some not and they are not the same across the nodes.

1) Can I statically set all these ports (even if I knew what they were) so I
can create proxies for them with the hope that allows me to cluster up ?

2) I believe a ring topology is in play - are the hosts/ip's set up in the
service disco config always used, i.e. so everything goes through Toxiproxy
or is there the possibility they will connect direct and bypass ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

ToxiproxyTest.java (10K) Download Attachment
Delian Delian
Reply | Threaded
Open this post in threaded view
|

Re: Toxiproxy

Thanks very much Mike.........will run through this and report back. Looks
promising.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Delian Delian
Reply | Threaded
Open this post in threaded view
|

Re: Toxiproxy

Hi Mike,

Update on this - in the main I seem to have got things working but a couple
of unknowns. I can cluster up using service disco via Toxiproxy. I am
actually connecting server nodes rather than client ones and have a
localhost entry in the ipfinder for each.

If no Toxiproxy is running the servers start up non-clustered, with
Toxiproxy running they start and  cluster up via the proxies. If I then kill
Toxiproxy, the servers start pinging in the logs and when I start Toxiproxy
again the pinging stops and servers appear to cluster.

However, if I kill Toxiproxy again there is no pinging in the server logs
again and they seem to remain clustered. If I then physically kill a server
you will see node left events in the other so even with my addressResolver
and the fact I am blanking out the comm.tcp.addrs the servers seem to be
able to find themselves direct at some point .... I'm not 100% sure where as
yet but I can see sockAddrs which aren't the Toxiproxy ones in the server
logs.

Will persevere ...  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/