Node joining didn't trigger Affinity recalculation problem

classic Classic list List threaded Threaded
8 messages Options
relax ken relax ken
Reply | Threaded
Open this post in threaded view
|

Node joining didn't trigger Affinity recalculation problem

Hi,

I wrote an aws ELB based tcp discovery ip finder. When I launched two ec2 nodes in the same target group, the ip finder found node IPs correctly. However, affinity recalculation wasn't triggered. Basically nothing happened. also no exception. If I ran two nodes locally with multicast, affinity recalculation worked properly. I just wonder what else I need to do to make the custom IP finder works?

Here is the overrided method in IPFinder. Basically it just is a slightly modified version of TcpDiscoveryElbIpFinder

@Override public Collection<InetSocketAddress> getRegisteredAddresses() throws IgniteSpiException {
initClients();

logger.debug("elb host discovery");
List<String> privateIPs = awsService.getHealthyPrivateIPsFromTarget(targetGroupArn);
List<InetSocketAddress> addrs = new ArrayList<>();

for (String privateIP : privateIPs) {
logger.debug("tcp discovery. found host "+privateIP);
addrs.add(new InetSocketAddress(privateIP, 0));
}

return addrs;
}
@Override public void registerAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
//No-op, ELB will take care of registration.
}

@Override public void unregisterAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
// No-op, ELB will take care of this process.
}
And here is the ignite config setting
TcpDiscoverySpi spi = new TcpDiscoverySpi();
spi.setIpFinder(getTcpELBIpFinder(elbDiscoveryConfig));
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setMetricsLogFrequency(0);
igniteConfig.setDiscoverySpi(spi);
DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(false);
igniteConfig.setDataStorageConfiguration(storageCfg);
Any ideas? 

Thanks

Ken
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Node joining didn't trigger Affinity recalculation problem

Hi,

Could you confirm that both nodes on AWS joined the cluster? Data will be rebalanced between the nodes once you see 2 server nodes in your cluster topology. Might be some networking issue. 

-
Denis


On Tue, Apr 16, 2019 at 9:53 AM relax ken <[hidden email]> wrote:
Hi,

I wrote an aws ELB based tcp discovery ip finder. When I launched two ec2 nodes in the same target group, the ip finder found node IPs correctly. However, affinity recalculation wasn't triggered. Basically nothing happened. also no exception. If I ran two nodes locally with multicast, affinity recalculation worked properly. I just wonder what else I need to do to make the custom IP finder works?

Here is the overrided method in IPFinder. Basically it just is a slightly modified version of TcpDiscoveryElbIpFinder

@Override public Collection<InetSocketAddress> getRegisteredAddresses() throws IgniteSpiException {
initClients();

logger.debug("elb host discovery");
List<String> privateIPs = awsService.getHealthyPrivateIPsFromTarget(targetGroupArn);
List<InetSocketAddress> addrs = new ArrayList<>();

for (String privateIP : privateIPs) {
logger.debug("tcp discovery. found host "+privateIP);
addrs.add(new InetSocketAddress(privateIP, 0));
}

return addrs;
}
@Override public void registerAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
//No-op, ELB will take care of registration.
}

@Override public void unregisterAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
// No-op, ELB will take care of this process.
}
And here is the ignite config setting
TcpDiscoverySpi spi = new TcpDiscoverySpi();
spi.setIpFinder(getTcpELBIpFinder(elbDiscoveryConfig));
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setMetricsLogFrequency(0);
igniteConfig.setDiscoverySpi(spi);
DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(false);
igniteConfig.setDataStorageConfiguration(storageCfg);
Any ideas? 

Thanks

Ken
relax ken relax ken
Reply | Threaded
Open this post in threaded view
|

Re: Node joining didn't trigger Affinity recalculation problem

I don't think both nodes joined the cluster. That is the problem. I don't know what it's wrong and why they are not in the cluster. They are in the same subnet,. They can talk to each other. They found each other's IP. 

What port does ignite use for cluster joining? It may be security group issue. Will ignite throw exception or log warning if they can't join the cluster?

Thanks

On Sat, 20 Apr 2019, 05:50 Denis Magda, <[hidden email]> wrote:
Hi,

Could you confirm that both nodes on AWS joined the cluster? Data will be rebalanced between the nodes once you see 2 server nodes in your cluster topology. Might be some networking issue. 

-
Denis


On Tue, Apr 16, 2019 at 9:53 AM relax ken <[hidden email]> wrote:
Hi,

I wrote an aws ELB based tcp discovery ip finder. When I launched two ec2 nodes in the same target group, the ip finder found node IPs correctly. However, affinity recalculation wasn't triggered. Basically nothing happened. also no exception. If I ran two nodes locally with multicast, affinity recalculation worked properly. I just wonder what else I need to do to make the custom IP finder works?

Here is the overrided method in IPFinder. Basically it just is a slightly modified version of TcpDiscoveryElbIpFinder

@Override public Collection<InetSocketAddress> getRegisteredAddresses() throws IgniteSpiException {
initClients();

logger.debug("elb host discovery");
List<String> privateIPs = awsService.getHealthyPrivateIPsFromTarget(targetGroupArn);
List<InetSocketAddress> addrs = new ArrayList<>();

for (String privateIP : privateIPs) {
logger.debug("tcp discovery. found host "+privateIP);
addrs.add(new InetSocketAddress(privateIP, 0));
}

return addrs;
}
@Override public void registerAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
//No-op, ELB will take care of registration.
}

@Override public void unregisterAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
// No-op, ELB will take care of this process.
}
And here is the ignite config setting
TcpDiscoverySpi spi = new TcpDiscoverySpi();
spi.setIpFinder(getTcpELBIpFinder(elbDiscoveryConfig));
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setMetricsLogFrequency(0);
igniteConfig.setDiscoverySpi(spi);
DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(false);
igniteConfig.setDataStorageConfiguration(storageCfg);
Any ideas? 

Thanks

Ken
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Node joining didn't trigger Affinity recalculation problem

By default, each node will be connecting to port 47500 and the next 100 ports if nobody responds on 45700, 457001, etc.

Please check that the port is opened. Alternatively, you can always manage the port with TcpDiscoverySpi.setLocalPort [1]. The ports range can be adjusted as well. [2]


On Fri, Apr 19, 2019 at 9:38 PM relax ken <[hidden email]> wrote:
I don't think both nodes joined the cluster. That is the problem. I don't know what it's wrong and why they are not in the cluster. They are in the same subnet,. They can talk to each other. They found each other's IP. 

What port does ignite use for cluster joining? It may be security group issue. Will ignite throw exception or log warning if they can't join the cluster?

Thanks

On Sat, 20 Apr 2019, 05:50 Denis Magda, <[hidden email]> wrote:
Hi,

Could you confirm that both nodes on AWS joined the cluster? Data will be rebalanced between the nodes once you see 2 server nodes in your cluster topology. Might be some networking issue. 

-
Denis


On Tue, Apr 16, 2019 at 9:53 AM relax ken <[hidden email]> wrote:
Hi,

I wrote an aws ELB based tcp discovery ip finder. When I launched two ec2 nodes in the same target group, the ip finder found node IPs correctly. However, affinity recalculation wasn't triggered. Basically nothing happened. also no exception. If I ran two nodes locally with multicast, affinity recalculation worked properly. I just wonder what else I need to do to make the custom IP finder works?

Here is the overrided method in IPFinder. Basically it just is a slightly modified version of TcpDiscoveryElbIpFinder

@Override public Collection<InetSocketAddress> getRegisteredAddresses() throws IgniteSpiException {
initClients();

logger.debug("elb host discovery");
List<String> privateIPs = awsService.getHealthyPrivateIPsFromTarget(targetGroupArn);
List<InetSocketAddress> addrs = new ArrayList<>();

for (String privateIP : privateIPs) {
logger.debug("tcp discovery. found host "+privateIP);
addrs.add(new InetSocketAddress(privateIP, 0));
}

return addrs;
}
@Override public void registerAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
//No-op, ELB will take care of registration.
}

@Override public void unregisterAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
// No-op, ELB will take care of this process.
}
And here is the ignite config setting
TcpDiscoverySpi spi = new TcpDiscoverySpi();
spi.setIpFinder(getTcpELBIpFinder(elbDiscoveryConfig));
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setMetricsLogFrequency(0);
igniteConfig.setDiscoverySpi(spi);
DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(false);
igniteConfig.setDataStorageConfiguration(storageCfg);
Any ideas? 

Thanks

Ken
relax ken relax ken
Reply | Threaded
Open this post in threaded view
|

Re: Node joining didn't trigger Affinity recalculation problem

Great! Thanks Denis! that's the issue. After I configured it in security group, it works

On Sat, Apr 20, 2019 at 6:03 AM Denis Magda <[hidden email]> wrote:
By default, each node will be connecting to port 47500 and the next 100 ports if nobody responds on 45700, 457001, etc.

Please check that the port is opened. Alternatively, you can always manage the port with TcpDiscoverySpi.setLocalPort [1]. The ports range can be adjusted as well. [2]


On Fri, Apr 19, 2019 at 9:38 PM relax ken <[hidden email]> wrote:
I don't think both nodes joined the cluster. That is the problem. I don't know what it's wrong and why they are not in the cluster. They are in the same subnet,. They can talk to each other. They found each other's IP. 

What port does ignite use for cluster joining? It may be security group issue. Will ignite throw exception or log warning if they can't join the cluster?

Thanks

On Sat, 20 Apr 2019, 05:50 Denis Magda, <[hidden email]> wrote:
Hi,

Could you confirm that both nodes on AWS joined the cluster? Data will be rebalanced between the nodes once you see 2 server nodes in your cluster topology. Might be some networking issue. 

-
Denis


On Tue, Apr 16, 2019 at 9:53 AM relax ken <[hidden email]> wrote:
Hi,

I wrote an aws ELB based tcp discovery ip finder. When I launched two ec2 nodes in the same target group, the ip finder found node IPs correctly. However, affinity recalculation wasn't triggered. Basically nothing happened. also no exception. If I ran two nodes locally with multicast, affinity recalculation worked properly. I just wonder what else I need to do to make the custom IP finder works?

Here is the overrided method in IPFinder. Basically it just is a slightly modified version of TcpDiscoveryElbIpFinder

@Override public Collection<InetSocketAddress> getRegisteredAddresses() throws IgniteSpiException {
initClients();

logger.debug("elb host discovery");
List<String> privateIPs = awsService.getHealthyPrivateIPsFromTarget(targetGroupArn);
List<InetSocketAddress> addrs = new ArrayList<>();

for (String privateIP : privateIPs) {
logger.debug("tcp discovery. found host "+privateIP);
addrs.add(new InetSocketAddress(privateIP, 0));
}

return addrs;
}
@Override public void registerAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
//No-op, ELB will take care of registration.
}

@Override public void unregisterAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
// No-op, ELB will take care of this process.
}
And here is the ignite config setting
TcpDiscoverySpi spi = new TcpDiscoverySpi();
spi.setIpFinder(getTcpELBIpFinder(elbDiscoveryConfig));
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setMetricsLogFrequency(0);
igniteConfig.setDiscoverySpi(spi);
DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(false);
igniteConfig.setDataStorageConfiguration(storageCfg);
Any ideas? 

Thanks

Ken
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Node joining didn't trigger Affinity recalculation problem

No problem, thanks for taking the time and sharing the issue. We'll update the docs. Good luck with your Ignite deployment!

-
Denis


On Fri, Apr 19, 2019 at 10:32 PM relax ken <[hidden email]> wrote:
Great! Thanks Denis! that's the issue. After I configured it in security group, it works

On Sat, Apr 20, 2019 at 6:03 AM Denis Magda <[hidden email]> wrote:
By default, each node will be connecting to port 47500 and the next 100 ports if nobody responds on 45700, 457001, etc.

Please check that the port is opened. Alternatively, you can always manage the port with TcpDiscoverySpi.setLocalPort [1]. The ports range can be adjusted as well. [2]


On Fri, Apr 19, 2019 at 9:38 PM relax ken <[hidden email]> wrote:
I don't think both nodes joined the cluster. That is the problem. I don't know what it's wrong and why they are not in the cluster. They are in the same subnet,. They can talk to each other. They found each other's IP. 

What port does ignite use for cluster joining? It may be security group issue. Will ignite throw exception or log warning if they can't join the cluster?

Thanks

On Sat, 20 Apr 2019, 05:50 Denis Magda, <[hidden email]> wrote:
Hi,

Could you confirm that both nodes on AWS joined the cluster? Data will be rebalanced between the nodes once you see 2 server nodes in your cluster topology. Might be some networking issue. 

-
Denis


On Tue, Apr 16, 2019 at 9:53 AM relax ken <[hidden email]> wrote:
Hi,

I wrote an aws ELB based tcp discovery ip finder. When I launched two ec2 nodes in the same target group, the ip finder found node IPs correctly. However, affinity recalculation wasn't triggered. Basically nothing happened. also no exception. If I ran two nodes locally with multicast, affinity recalculation worked properly. I just wonder what else I need to do to make the custom IP finder works?

Here is the overrided method in IPFinder. Basically it just is a slightly modified version of TcpDiscoveryElbIpFinder

@Override public Collection<InetSocketAddress> getRegisteredAddresses() throws IgniteSpiException {
initClients();

logger.debug("elb host discovery");
List<String> privateIPs = awsService.getHealthyPrivateIPsFromTarget(targetGroupArn);
List<InetSocketAddress> addrs = new ArrayList<>();

for (String privateIP : privateIPs) {
logger.debug("tcp discovery. found host "+privateIP);
addrs.add(new InetSocketAddress(privateIP, 0));
}

return addrs;
}
@Override public void registerAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
//No-op, ELB will take care of registration.
}

@Override public void unregisterAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
// No-op, ELB will take care of this process.
}
And here is the ignite config setting
TcpDiscoverySpi spi = new TcpDiscoverySpi();
spi.setIpFinder(getTcpELBIpFinder(elbDiscoveryConfig));
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setMetricsLogFrequency(0);
igniteConfig.setDiscoverySpi(spi);
DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(false);
igniteConfig.setDataStorageConfiguration(storageCfg);
Any ideas? 

Thanks

Ken
relax ken relax ken
Reply | Threaded
Open this post in threaded view
|

Re: Node joining didn't trigger Affinity recalculation problem

Hi Denis,

I just found an issue with the ec2 tcp discovery. If I have one node running in elb and launch a new node in elb, they will join the same cluster properly. However, if I launch two nodes separately first without joining elb and then add them into the same elb after they are fully running, they will find each other by ip finder but won't join the cluster. Any ideas?

Thanks

On Sat, Apr 20, 2019 at 7:01 AM Denis Magda <[hidden email]> wrote:
No problem, thanks for taking the time and sharing the issue. We'll update the docs. Good luck with your Ignite deployment!

-
Denis


On Fri, Apr 19, 2019 at 10:32 PM relax ken <[hidden email]> wrote:
Great! Thanks Denis! that's the issue. After I configured it in security group, it works

On Sat, Apr 20, 2019 at 6:03 AM Denis Magda <[hidden email]> wrote:
By default, each node will be connecting to port 47500 and the next 100 ports if nobody responds on 45700, 457001, etc.

Please check that the port is opened. Alternatively, you can always manage the port with TcpDiscoverySpi.setLocalPort [1]. The ports range can be adjusted as well. [2]


On Fri, Apr 19, 2019 at 9:38 PM relax ken <[hidden email]> wrote:
I don't think both nodes joined the cluster. That is the problem. I don't know what it's wrong and why they are not in the cluster. They are in the same subnet,. They can talk to each other. They found each other's IP. 

What port does ignite use for cluster joining? It may be security group issue. Will ignite throw exception or log warning if they can't join the cluster?

Thanks

On Sat, 20 Apr 2019, 05:50 Denis Magda, <[hidden email]> wrote:
Hi,

Could you confirm that both nodes on AWS joined the cluster? Data will be rebalanced between the nodes once you see 2 server nodes in your cluster topology. Might be some networking issue. 

-
Denis


On Tue, Apr 16, 2019 at 9:53 AM relax ken <[hidden email]> wrote:
Hi,

I wrote an aws ELB based tcp discovery ip finder. When I launched two ec2 nodes in the same target group, the ip finder found node IPs correctly. However, affinity recalculation wasn't triggered. Basically nothing happened. also no exception. If I ran two nodes locally with multicast, affinity recalculation worked properly. I just wonder what else I need to do to make the custom IP finder works?

Here is the overrided method in IPFinder. Basically it just is a slightly modified version of TcpDiscoveryElbIpFinder

@Override public Collection<InetSocketAddress> getRegisteredAddresses() throws IgniteSpiException {
initClients();

logger.debug("elb host discovery");
List<String> privateIPs = awsService.getHealthyPrivateIPsFromTarget(targetGroupArn);
List<InetSocketAddress> addrs = new ArrayList<>();

for (String privateIP : privateIPs) {
logger.debug("tcp discovery. found host "+privateIP);
addrs.add(new InetSocketAddress(privateIP, 0));
}

return addrs;
}
@Override public void registerAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
//No-op, ELB will take care of registration.
}

@Override public void unregisterAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
// No-op, ELB will take care of this process.
}
And here is the ignite config setting
TcpDiscoverySpi spi = new TcpDiscoverySpi();
spi.setIpFinder(getTcpELBIpFinder(elbDiscoveryConfig));
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setMetricsLogFrequency(0);
igniteConfig.setDiscoverySpi(spi);
DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(false);
igniteConfig.setDataStorageConfiguration(storageCfg);
Any ideas? 

Thanks

Ken
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Node joining didn't trigger Affinity recalculation problem

Hello!

I guess this is because you can't join two clusters together. Once each node has formed a cluster at start, neither one can just other cluster.

A cluster can accept nodes, new nodes can join to cluster, but two clusters can't merge. The only way here is to restart one of nodes.

Regards,
--
Ilya Kasnacheev


чт, 25 апр. 2019 г. в 16:17, relax ken <[hidden email]>:
Hi Denis,

I just found an issue with the ec2 tcp discovery. If I have one node running in elb and launch a new node in elb, they will join the same cluster properly. However, if I launch two nodes separately first without joining elb and then add them into the same elb after they are fully running, they will find each other by ip finder but won't join the cluster. Any ideas?

Thanks

On Sat, Apr 20, 2019 at 7:01 AM Denis Magda <[hidden email]> wrote:
No problem, thanks for taking the time and sharing the issue. We'll update the docs. Good luck with your Ignite deployment!

-
Denis


On Fri, Apr 19, 2019 at 10:32 PM relax ken <[hidden email]> wrote:
Great! Thanks Denis! that's the issue. After I configured it in security group, it works

On Sat, Apr 20, 2019 at 6:03 AM Denis Magda <[hidden email]> wrote:
By default, each node will be connecting to port 47500 and the next 100 ports if nobody responds on 45700, 457001, etc.

Please check that the port is opened. Alternatively, you can always manage the port with TcpDiscoverySpi.setLocalPort [1]. The ports range can be adjusted as well. [2]


On Fri, Apr 19, 2019 at 9:38 PM relax ken <[hidden email]> wrote:
I don't think both nodes joined the cluster. That is the problem. I don't know what it's wrong and why they are not in the cluster. They are in the same subnet,. They can talk to each other. They found each other's IP. 

What port does ignite use for cluster joining? It may be security group issue. Will ignite throw exception or log warning if they can't join the cluster?

Thanks

On Sat, 20 Apr 2019, 05:50 Denis Magda, <[hidden email]> wrote:
Hi,

Could you confirm that both nodes on AWS joined the cluster? Data will be rebalanced between the nodes once you see 2 server nodes in your cluster topology. Might be some networking issue. 

-
Denis


On Tue, Apr 16, 2019 at 9:53 AM relax ken <[hidden email]> wrote:
Hi,

I wrote an aws ELB based tcp discovery ip finder. When I launched two ec2 nodes in the same target group, the ip finder found node IPs correctly. However, affinity recalculation wasn't triggered. Basically nothing happened. also no exception. If I ran two nodes locally with multicast, affinity recalculation worked properly. I just wonder what else I need to do to make the custom IP finder works?

Here is the overrided method in IPFinder. Basically it just is a slightly modified version of TcpDiscoveryElbIpFinder

@Override public Collection<InetSocketAddress> getRegisteredAddresses() throws IgniteSpiException {
initClients();

logger.debug("elb host discovery");
List<String> privateIPs = awsService.getHealthyPrivateIPsFromTarget(targetGroupArn);
List<InetSocketAddress> addrs = new ArrayList<>();

for (String privateIP : privateIPs) {
logger.debug("tcp discovery. found host "+privateIP);
addrs.add(new InetSocketAddress(privateIP, 0));
}

return addrs;
}
@Override public void registerAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
//No-op, ELB will take care of registration.
}

@Override public void unregisterAddresses(Collection<InetSocketAddress> addrs) throws IgniteSpiException {
// No-op, ELB will take care of this process.
}
And here is the ignite config setting
TcpDiscoverySpi spi = new TcpDiscoverySpi();
spi.setIpFinder(getTcpELBIpFinder(elbDiscoveryConfig));
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setMetricsLogFrequency(0);
igniteConfig.setDiscoverySpi(spi);
DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(false);
igniteConfig.setDataStorageConfiguration(storageCfg);
Any ideas? 

Thanks

Ken