Re:Isolating IgniteCache instances across JVMs on same machine by id

classic Classic list List threaded Threaded
12 messages Options
Mitchell Rathbun (BLOOMBERG/ 731 LEX) Mitchell Rathbun (BLOOMBERG/ 731 LEX)
Reply | Threaded
Open this post in threaded view
|

Re:Isolating IgniteCache instances across JVMs on same machine by id

Any thoughts on this?

From: [hidden email] At: 12/18/19 19:55:47
To: [hidden email]
Cc: [hidden email], [hidden email]
Subject: Isolating IgniteCache instances across JVMs on same machine by id

We have multiple different database instances for which we are looking into using Ignite as a local caching layer. Each db has a unique id that we are using as part of the IgniteInstanceName and for the WorkDirectory path. We are running IgniteCache in Local mode with persistence enabled, as we effectively want to completely separate the Ignite caches for each db id. However, this is not working due to the IgniteCluster being shared and the BaselineTopology concept. So a couple of questions:

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
-If rebalancing does come into play even if all caches are local, then it seems like separate clusters would have to be specified when starting up.
https://apacheignite.readme.io/docs/tcpip-discovery seems to provide a way to do that by making sure that TcpDiscoverySpi and TcpCommunicationSpi have different port ranges for every instance of the cluster. Again, this seems kind of like overkill if we know in advance that all of the caches will be local and will not need any of the cluster functionality.

Are there any other simpler options? We are only interested in the IgniteCache functionality at a local per process level, at least for now. We might have 10-20 different caches on the same machine, and need these caches to be completely separated with no risk of any of the data being rebalanced/exposed to a different client instance. Ideally we would get persistence without having to activate IgniteCluster, but it does not seem like that is an option.

ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Isolating IgniteCache instances across JVMs on same machine by id

Hi,

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?
Cluster needs to make sure that all nodes in cluster know about this cache to avoid situations in future when other nodes will try to create a distributed cache with the same name, for example.

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
It will define a subset of the nodes, which will store data for this cache. So, you can set there only one node.
>-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
No, rebalancing won't happen, it will be skipped. However, before that, nodes should exchange an information regarding new topology ad partition distribution.

Baseline topology should contains all nodes that will store the data.

Evgenii


пн, 30 дек. 2019 г. в 10:59, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Any thoughts on this?

From: [hidden email] At: 12/18/19 19:55:47
To: [hidden email]
Cc: [hidden email], [hidden email]
Subject: Isolating IgniteCache instances across JVMs on same machine by id

We have multiple different database instances for which we are looking into using Ignite as a local caching layer. Each db has a unique id that we are using as part of the IgniteInstanceName and for the WorkDirectory path. We are running IgniteCache in Local mode with persistence enabled, as we effectively want to completely separate the Ignite caches for each db id. However, this is not working due to the IgniteCluster being shared and the BaselineTopology concept. So a couple of questions:

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
-If rebalancing does come into play even if all caches are local, then it seems like separate clusters would have to be specified when starting up.
https://apacheignite.readme.io/docs/tcpip-discovery seems to provide a way to do that by making sure that TcpDiscoverySpi and TcpCommunicationSpi have different port ranges for every instance of the cluster. Again, this seems kind of like overkill if we know in advance that all of the caches will be local and will not need any of the cluster functionality.

Are there any other simpler options? We are only interested in the IgniteCache functionality at a local per process level, at least for now. We might have 10-20 different caches on the same machine, and need these caches to be completely separated with no risk of any of the data being rebalanced/exposed to a different client instance. Ideally we would get persistence without having to activate IgniteCluster, but it does not seem like that is an option.

Mitchell Rathbun (BLOOMBERG/ 731 LEX) Mitchell Rathbun (BLOOMBERG/ 731 LEX)
Reply | Threaded
Open this post in threaded view
|

Re: Isolating IgniteCache instances across JVMs on same machine by id

In reply to this post by Mitchell Rathbun (BLOOMBERG/ 731 LEX)
Thank you for the response. A couple of things:

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?

-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.


From: [hidden email] At: 12/30/19 14:16:33
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Hi,

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?
Cluster needs to make sure that all nodes in cluster know about this cache to avoid situations in future when other nodes will try to create a distributed cache with the same name, for example.

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
It will define a subset of the nodes, which will store data for this cache. So, you can set there only one node.
>-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
No, rebalancing won't happen, it will be skipped. However, before that, nodes should exchange an information regarding new topology ad partition distribution.

Baseline topology should contains all nodes that will store the data.

Evgenii


пн, 30 дек. 2019 г. в 10:59, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Any thoughts on this?

From: [hidden email] At: 12/18/19 19:55:47
To: [hidden email]
Cc: [hidden email], [hidden email]
Subject: Isolating IgniteCache instances across JVMs on same machine by id

We have multiple different database instances for which we are looking into using Ignite as a local caching layer. Each db has a unique id that we are using as part of the IgniteInstanceName and for the WorkDirectory path. We are running IgniteCache in Local mode with persistence enabled, as we effectively want to completely separate the Ignite caches for each db id. However, this is not working due to the IgniteCluster being shared and the BaselineTopology concept. So a couple of questions:

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
-If rebalancing does come into play even if all caches are local, then it seems like separate clusters would have to be specified when starting up.
https://apacheignite.readme.io/docs/tcpip-discovery seems to provide a way to do that by making sure that TcpDiscoverySpi and TcpCommunicationSpi have different port ranges for every instance of the cluster. Again, this seems kind of like overkill if we know in advance that all of the caches will be local and will not need any of the cluster functionality.

Are there any other simpler options? We are only interested in the IgniteCache functionality at a local per process level, at least for now. We might have 10-20 different caches on the same machine, and need these caches to be completely separated with no risk of any of the data being rebalanced/exposed to a different client instance. Ideally we would get persistence without having to activate IgniteCluster, but it does not seem like that is an option.


ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Isolating IgniteCache instances across JVMs on same machine by id

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.
No, it will be handled automatically, you don't need to do anything.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?
It will tell cluster that only nodes, which returns true after applying a predicate will store data for this cache. Of course, data won't be rebalanced to the nodes, that shouldn't store data for this cache.

>-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.
Yes, it's equivalent to just adding the node, but using this method you can just use one method for several topology changes(for example, adding 2 nodes to the BT), so it will merge events on the cluster level.

пн, 30 дек. 2019 г. в 12:08, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Thank you for the response. A couple of things:

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?

-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.


From: [hidden email] At: 12/30/19 14:16:33
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Hi,

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?
Cluster needs to make sure that all nodes in cluster know about this cache to avoid situations in future when other nodes will try to create a distributed cache with the same name, for example.

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
It will define a subset of the nodes, which will store data for this cache. So, you can set there only one node.
>-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
No, rebalancing won't happen, it will be skipped. However, before that, nodes should exchange an information regarding new topology ad partition distribution.

Baseline topology should contains all nodes that will store the data.

Evgenii


пн, 30 дек. 2019 г. в 10:59, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Any thoughts on this?

From: [hidden email] At: 12/18/19 19:55:47
To: [hidden email]
Cc: [hidden email], [hidden email]
Subject: Isolating IgniteCache instances across JVMs on same machine by id

We have multiple different database instances for which we are looking into using Ignite as a local caching layer. Each db has a unique id that we are using as part of the IgniteInstanceName and for the WorkDirectory path. We are running IgniteCache in Local mode with persistence enabled, as we effectively want to completely separate the Ignite caches for each db id. However, this is not working due to the IgniteCluster being shared and the BaselineTopology concept. So a couple of questions:

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
-If rebalancing does come into play even if all caches are local, then it seems like separate clusters would have to be specified when starting up.
https://apacheignite.readme.io/docs/tcpip-discovery seems to provide a way to do that by making sure that TcpDiscoverySpi and TcpCommunicationSpi have different port ranges for every instance of the cluster. Again, this seems kind of like overkill if we know in advance that all of the caches will be local and will not need any of the cluster functionality.

Are there any other simpler options? We are only interested in the IgniteCache functionality at a local per process level, at least for now. We might have 10-20 different caches on the same machine, and need these caches to be completely separated with no risk of any of the data being rebalanced/exposed to a different client instance. Ideally we would get persistence without having to activate IgniteCluster, but it does not seem like that is an option.


Mitchell Rathbun (BLOOMBERG/ 731 LEX) Mitchell Rathbun (BLOOMBERG/ 731 LEX)
Reply | Threaded
Open this post in threaded view
|

Re: Isolating IgniteCache instances across JVMs on same machine by id

In reply to this post by Mitchell Rathbun (BLOOMBERG/ 731 LEX)
Makes sense. A couple of more related questions:

-For your first response, I just want to make sure I am clear how this works. Say I have two different server nodes with IgniteInstanceName A and B. If both nodes are part of the same cluster and both nodes write to a cache (in local mode) called "datacache", would this be treated as two separate caches or would there be an issue?

-Where is the shared information stored for an IgniteCluster? Looking at https://apacheignite.readme.io/v2.4/docs/cluster-config, it seems like different clusters can be configured by using different port ranges. But for each cluster, the nodes in the baseline topology must be stored. Where does this information get stored?

-If we need to specify different clusters on the same machine, do we need to specify a range of ports for both TcpDiscoverySpi and TcpCommunicationSpi? Or is one port for each fine?

From: [hidden email] At: 12/30/19 15:25:40
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.
No, it will be handled automatically, you don't need to do anything.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?
It will tell cluster that only nodes, which returns true after applying a predicate will store data for this cache. Of course, data won't be rebalanced to the nodes, that shouldn't store data for this cache.

>-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.
Yes, it's equivalent to just adding the node, but using this method you can just use one method for several topology changes(for example, adding 2 nodes to the BT), so it will merge events on the cluster level.

пн, 30 дек. 2019 г. в 12:08, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Thank you for the response. A couple of things:

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?

-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.


From: [hidden email] At: 12/30/19 14:16:33
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Hi,

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?
Cluster needs to make sure that all nodes in cluster know about this cache to avoid situations in future when other nodes will try to create a distributed cache with the same name, for example.

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
It will define a subset of the nodes, which will store data for this cache. So, you can set there only one node.
>-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
No, rebalancing won't happen, it will be skipped. However, before that, nodes should exchange an information regarding new topology ad partition distribution.

Baseline topology should contains all nodes that will store the data.

Evgenii


пн, 30 дек. 2019 г. в 10:59, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Any thoughts on this?

From: [hidden email] At: 12/18/19 19:55:47
To: [hidden email]
Cc: [hidden email], [hidden email]
Subject: Isolating IgniteCache instances across JVMs on same machine by id

We have multiple different database instances for which we are looking into using Ignite as a local caching layer. Each db has a unique id that we are using as part of the IgniteInstanceName and for the WorkDirectory path. We are running IgniteCache in Local mode with persistence enabled, as we effectively want to completely separate the Ignite caches for each db id. However, this is not working due to the IgniteCluster being shared and the BaselineTopology concept. So a couple of questions:

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
-If rebalancing does come into play even if all caches are local, then it seems like separate clusters would have to be specified when starting up.
https://apacheignite.readme.io/docs/tcpip-discovery seems to provide a way to do that by making sure that TcpDiscoverySpi and TcpCommunicationSpi have different port ranges for every instance of the cluster. Again, this seems kind of like overkill if we know in advance that all of the caches will be local and will not need any of the cluster functionality.

Are there any other simpler options? We are only interested in the IgniteCache functionality at a local per process level, at least for now. We might have 10-20 different caches on the same machine, and need these caches to be completely separated with no risk of any of the data being rebalanced/exposed to a different client instance. Ideally we would get persistence without having to activate IgniteCluster, but it does not seem like that is an option.



ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Isolating IgniteCache instances across JVMs on same machine by id

Mitchell,

-For your first response, I just want to make sure I am clear how this works. Say I have two different server nodes with IgniteInstanceName A and B. If both nodes are part of the same cluster and both nodes write to a cache (in local mode) called "datacache", would this be treated as two separate caches or would there be an issue?
It will be more like the same cache, but all data will be local on each node. It shouldn't be a problem. However, I would recommend using partitioned caches with a NodeFilter instead.

-Where is the shared information stored for an IgniteCluster? Looking at https://apacheignite.readme.io/v2.4/docs/cluster-config, it seems like different clusters can be configured by using different port ranges. But for each cluster, the nodes in the baseline topology must be stored. Where does this information get stored?
On a disk, it's stored in meta storage inside DB folder.

-If we need to specify different clusters on the same machine, do we need to specify a range of ports for both TcpDiscoverySpi and TcpCommunicationSpi? Or is one port for each fine?
Next port from port range is used only if a first port is busy. If you set only one port and it's busy at the moment, node won't be able to start. So, it's up to you if you want to set the port range or one port only.

Evgenii

пн, 30 дек. 2019 г. в 13:30, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Makes sense. A couple of more related questions:

-For your first response, I just want to make sure I am clear how this works. Say I have two different server nodes with IgniteInstanceName A and B. If both nodes are part of the same cluster and both nodes write to a cache (in local mode) called "datacache", would this be treated as two separate caches or would there be an issue?

-Where is the shared information stored for an IgniteCluster? Looking at https://apacheignite.readme.io/v2.4/docs/cluster-config, it seems like different clusters can be configured by using different port ranges. But for each cluster, the nodes in the baseline topology must be stored. Where does this information get stored?

-If we need to specify different clusters on the same machine, do we need to specify a range of ports for both TcpDiscoverySpi and TcpCommunicationSpi? Or is one port for each fine?

From: [hidden email] At: 12/30/19 15:25:40
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.
No, it will be handled automatically, you don't need to do anything.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?
It will tell cluster that only nodes, which returns true after applying a predicate will store data for this cache. Of course, data won't be rebalanced to the nodes, that shouldn't store data for this cache.

>-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.
Yes, it's equivalent to just adding the node, but using this method you can just use one method for several topology changes(for example, adding 2 nodes to the BT), so it will merge events on the cluster level.

пн, 30 дек. 2019 г. в 12:08, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Thank you for the response. A couple of things:

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?

-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.


From: [hidden email] At: 12/30/19 14:16:33
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Hi,

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?
Cluster needs to make sure that all nodes in cluster know about this cache to avoid situations in future when other nodes will try to create a distributed cache with the same name, for example.

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
It will define a subset of the nodes, which will store data for this cache. So, you can set there only one node.
>-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
No, rebalancing won't happen, it will be skipped. However, before that, nodes should exchange an information regarding new topology ad partition distribution.

Baseline topology should contains all nodes that will store the data.

Evgenii


пн, 30 дек. 2019 г. в 10:59, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Any thoughts on this?

From: [hidden email] At: 12/18/19 19:55:47
To: [hidden email]
Cc: [hidden email], [hidden email]
Subject: Isolating IgniteCache instances across JVMs on same machine by id

We have multiple different database instances for which we are looking into using Ignite as a local caching layer. Each db has a unique id that we are using as part of the IgniteInstanceName and for the WorkDirectory path. We are running IgniteCache in Local mode with persistence enabled, as we effectively want to completely separate the Ignite caches for each db id. However, this is not working due to the IgniteCluster being shared and the BaselineTopology concept. So a couple of questions:

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
-If rebalancing does come into play even if all caches are local, then it seems like separate clusters would have to be specified when starting up.
https://apacheignite.readme.io/docs/tcpip-discovery seems to provide a way to do that by making sure that TcpDiscoverySpi and TcpCommunicationSpi have different port ranges for every instance of the cluster. Again, this seems kind of like overkill if we know in advance that all of the caches will be local and will not need any of the cluster functionality.

Are there any other simpler options? We are only interested in the IgniteCache functionality at a local per process level, at least for now. We might have 10-20 different caches on the same machine, and need these caches to be completely separated with no risk of any of the data being rebalanced/exposed to a different client instance. Ideally we would get persistence without having to activate IgniteCluster, but it does not seem like that is an option.



Mitchell Rathbun (BLOOMBERG/ 731 LEX) Mitchell Rathbun (BLOOMBERG/ 731 LEX)
Reply | Threaded
Open this post in threaded view
|

Re: Isolating IgniteCache instances across JVMs on same machine by id

In reply to this post by Mitchell Rathbun (BLOOMBERG/ 731 LEX)
"On a disk, it's stored in meta storage inside DB folder."

We also set the IgniteWorkDirectory to be different for every server node id. However, we still will get "Node is not part of the baseline topology, not using persistence." How would this still occur if the there is no shared directory for the cluster?

From: [hidden email] At: 12/30/19 17:33:55
To: [hidden email]
Cc: [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Mitchell,

-For your first response, I just want to make sure I am clear how this works. Say I have two different server nodes with IgniteInstanceName A and B. If both nodes are part of the same cluster and both nodes write to a cache (in local mode) called "datacache", would this be treated as two separate caches or would there be an issue?
It will be more like the same cache, but all data will be local on each node. It shouldn't be a problem. However, I would recommend using partitioned caches with a NodeFilter instead.

-Where is the shared information stored for an IgniteCluster? Looking at https://apacheignite.readme.io/v2.4/docs/cluster-config, it seems like different clusters can be configured by using different port ranges. But for each cluster, the nodes in the baseline topology must be stored. Where does this information get stored?
On a disk, it's stored in meta storage inside DB folder.

-If we need to specify different clusters on the same machine, do we need to specify a range of ports for both TcpDiscoverySpi and TcpCommunicationSpi? Or is one port for each fine?
Next port from port range is used only if a first port is busy. If you set only one port and it's busy at the moment, node won't be able to start. So, it's up to you if you want to set the port range or one port only.

Evgenii

пн, 30 дек. 2019 г. в 13:30, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Makes sense. A couple of more related questions:

-For your first response, I just want to make sure I am clear how this works. Say I have two different server nodes with IgniteInstanceName A and B. If both nodes are part of the same cluster and both nodes write to a cache (in local mode) called "datacache", would this be treated as two separate caches or would there be an issue?

-Where is the shared information stored for an IgniteCluster? Looking at https://apacheignite.readme.io/v2.4/docs/cluster-config, it seems like different clusters can be configured by using different port ranges. But for each cluster, the nodes in the baseline topology must be stored. Where does this information get stored?

-If we need to specify different clusters on the same machine, do we need to specify a range of ports for both TcpDiscoverySpi and TcpCommunicationSpi? Or is one port for each fine?

From: [hidden email] At: 12/30/19 15:25:40
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.
No, it will be handled automatically, you don't need to do anything.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?
It will tell cluster that only nodes, which returns true after applying a predicate will store data for this cache. Of course, data won't be rebalanced to the nodes, that shouldn't store data for this cache.

>-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.
Yes, it's equivalent to just adding the node, but using this method you can just use one method for several topology changes(for example, adding 2 nodes to the BT), so it will merge events on the cluster level.

пн, 30 дек. 2019 г. в 12:08, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Thank you for the response. A couple of things:

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?

-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.


From: [hidden email] At: 12/30/19 14:16:33
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Hi,

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?
Cluster needs to make sure that all nodes in cluster know about this cache to avoid situations in future when other nodes will try to create a distributed cache with the same name, for example.

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
It will define a subset of the nodes, which will store data for this cache. So, you can set there only one node.
>-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
No, rebalancing won't happen, it will be skipped. However, before that, nodes should exchange an information regarding new topology ad partition distribution.

Baseline topology should contains all nodes that will store the data.

Evgenii


пн, 30 дек. 2019 г. в 10:59, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Any thoughts on this?

From: [hidden email] At: 12/18/19 19:55:47
To: [hidden email]
Cc: [hidden email], [hidden email]
Subject: Isolating IgniteCache instances across JVMs on same machine by id

We have multiple different database instances for which we are looking into using Ignite as a local caching layer. Each db has a unique id that we are using as part of the IgniteInstanceName and for the WorkDirectory path. We are running IgniteCache in Local mode with persistence enabled, as we effectively want to completely separate the Ignite caches for each db id. However, this is not working due to the IgniteCluster being shared and the BaselineTopology concept. So a couple of questions:

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
-If rebalancing does come into play even if all caches are local, then it seems like separate clusters would have to be specified when starting up.
https://apacheignite.readme.io/docs/tcpip-discovery seems to provide a way to do that by making sure that TcpDiscoverySpi and TcpCommunicationSpi have different port ranges for every instance of the cluster. Again, this seems kind of like overkill if we know in advance that all of the caches will be local and will not need any of the cluster functionality.

Are there any other simpler options? We are only interested in the IgniteCache functionality at a local per process level, at least for now. We might have 10-20 different caches on the same machine, and need these caches to be completely separated with no risk of any of the data being rebalanced/exposed to a different client instance. Ideally we would get persistence without having to activate IgniteCluster, but it does not seem like that is an option.




ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Isolating IgniteCache instances across JVMs on same machine by id

Mitchell,

Can you share logs from all nodes?

Evgenii

пн, 30 дек. 2019 г. в 14:42, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
"On a disk, it's stored in meta storage inside DB folder."

We also set the IgniteWorkDirectory to be different for every server node id. However, we still will get "Node is not part of the baseline topology, not using persistence." How would this still occur if the there is no shared directory for the cluster?

From: [hidden email] At: 12/30/19 17:33:55
To: [hidden email]
Cc: [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Mitchell,

-For your first response, I just want to make sure I am clear how this works. Say I have two different server nodes with IgniteInstanceName A and B. If both nodes are part of the same cluster and both nodes write to a cache (in local mode) called "datacache", would this be treated as two separate caches or would there be an issue?
It will be more like the same cache, but all data will be local on each node. It shouldn't be a problem. However, I would recommend using partitioned caches with a NodeFilter instead.

-Where is the shared information stored for an IgniteCluster? Looking at https://apacheignite.readme.io/v2.4/docs/cluster-config, it seems like different clusters can be configured by using different port ranges. But for each cluster, the nodes in the baseline topology must be stored. Where does this information get stored?
On a disk, it's stored in meta storage inside DB folder.

-If we need to specify different clusters on the same machine, do we need to specify a range of ports for both TcpDiscoverySpi and TcpCommunicationSpi? Or is one port for each fine?
Next port from port range is used only if a first port is busy. If you set only one port and it's busy at the moment, node won't be able to start. So, it's up to you if you want to set the port range or one port only.

Evgenii

пн, 30 дек. 2019 г. в 13:30, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Makes sense. A couple of more related questions:

-For your first response, I just want to make sure I am clear how this works. Say I have two different server nodes with IgniteInstanceName A and B. If both nodes are part of the same cluster and both nodes write to a cache (in local mode) called "datacache", would this be treated as two separate caches or would there be an issue?

-Where is the shared information stored for an IgniteCluster? Looking at https://apacheignite.readme.io/v2.4/docs/cluster-config, it seems like different clusters can be configured by using different port ranges. But for each cluster, the nodes in the baseline topology must be stored. Where does this information get stored?

-If we need to specify different clusters on the same machine, do we need to specify a range of ports for both TcpDiscoverySpi and TcpCommunicationSpi? Or is one port for each fine?

From: [hidden email] At: 12/30/19 15:25:40
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.
No, it will be handled automatically, you don't need to do anything.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?
It will tell cluster that only nodes, which returns true after applying a predicate will store data for this cache. Of course, data won't be rebalanced to the nodes, that shouldn't store data for this cache.

>-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.
Yes, it's equivalent to just adding the node, but using this method you can just use one method for several topology changes(for example, adding 2 nodes to the BT), so it will merge events on the cluster level.

пн, 30 дек. 2019 г. в 12:08, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Thank you for the response. A couple of things:

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?

-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.


From: [hidden email] At: 12/30/19 14:16:33
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Hi,

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?
Cluster needs to make sure that all nodes in cluster know about this cache to avoid situations in future when other nodes will try to create a distributed cache with the same name, for example.

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
It will define a subset of the nodes, which will store data for this cache. So, you can set there only one node.
>-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
No, rebalancing won't happen, it will be skipped. However, before that, nodes should exchange an information regarding new topology ad partition distribution.

Baseline topology should contains all nodes that will store the data.

Evgenii


пн, 30 дек. 2019 г. в 10:59, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Any thoughts on this?

From: [hidden email] At: 12/18/19 19:55:47
To: [hidden email]
Cc: [hidden email], [hidden email]
Subject: Isolating IgniteCache instances across JVMs on same machine by id

We have multiple different database instances for which we are looking into using Ignite as a local caching layer. Each db has a unique id that we are using as part of the IgniteInstanceName and for the WorkDirectory path. We are running IgniteCache in Local mode with persistence enabled, as we effectively want to completely separate the Ignite caches for each db id. However, this is not working due to the IgniteCluster being shared and the BaselineTopology concept. So a couple of questions:

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
-If rebalancing does come into play even if all caches are local, then it seems like separate clusters would have to be specified when starting up.
https://apacheignite.readme.io/docs/tcpip-discovery seems to provide a way to do that by making sure that TcpDiscoverySpi and TcpCommunicationSpi have different port ranges for every instance of the cluster. Again, this seems kind of like overkill if we know in advance that all of the caches will be local and will not need any of the cluster functionality.

Are there any other simpler options? We are only interested in the IgniteCache functionality at a local per process level, at least for now. We might have 10-20 different caches on the same machine, and need these caches to be completely separated with no risk of any of the data being rebalanced/exposed to a different client instance. Ideally we would get persistence without having to activate IgniteCluster, but it does not seem like that is an option.




Mitchell Rathbun (BLOOMBERG/ 731 LEX) Mitchell Rathbun (BLOOMBERG/ 731 LEX)
Reply | Threaded
Open this post in threaded view
|

Re: Isolating IgniteCache instances across JVMs on same machine by id

In reply to this post by Mitchell Rathbun (BLOOMBERG/ 731 LEX)
I will apply changes that you mentioned and see if this issue still occurs. If not, I will attach logs.

From: [hidden email] At: 12/30/19 17:58:04
To: [hidden email]
Cc: [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Mitchell,

Can you share logs from all nodes?

Evgenii

пн, 30 дек. 2019 г. в 14:42, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
"On a disk, it's stored in meta storage inside DB folder."

We also set the IgniteWorkDirectory to be different for every server node id. However, we still will get "Node is not part of the baseline topology, not using persistence." How would this still occur if the there is no shared directory for the cluster?

From: [hidden email] At: 12/30/19 17:33:55
To: [hidden email]
Cc: [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Mitchell,

-For your first response, I just want to make sure I am clear how this works. Say I have two different server nodes with IgniteInstanceName A and B. If both nodes are part of the same cluster and both nodes write to a cache (in local mode) called "datacache", would this be treated as two separate caches or would there be an issue?
It will be more like the same cache, but all data will be local on each node. It shouldn't be a problem. However, I would recommend using partitioned caches with a NodeFilter instead.

-Where is the shared information stored for an IgniteCluster? Looking at https://apacheignite.readme.io/v2.4/docs/cluster-config, it seems like different clusters can be configured by using different port ranges. But for each cluster, the nodes in the baseline topology must be stored. Where does this information get stored?
On a disk, it's stored in meta storage inside DB folder.

-If we need to specify different clusters on the same machine, do we need to specify a range of ports for both TcpDiscoverySpi and TcpCommunicationSpi? Or is one port for each fine?
Next port from port range is used only if a first port is busy. If you set only one port and it's busy at the moment, node won't be able to start. So, it's up to you if you want to set the port range or one port only.

Evgenii

пн, 30 дек. 2019 г. в 13:30, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Makes sense. A couple of more related questions:

-For your first response, I just want to make sure I am clear how this works. Say I have two different server nodes with IgniteInstanceName A and B. If both nodes are part of the same cluster and both nodes write to a cache (in local mode) called "datacache", would this be treated as two separate caches or would there be an issue?

-Where is the shared information stored for an IgniteCluster? Looking at https://apacheignite.readme.io/v2.4/docs/cluster-config, it seems like different clusters can be configured by using different port ranges. But for each cluster, the nodes in the baseline topology must be stored. Where does this information get stored?

-If we need to specify different clusters on the same machine, do we need to specify a range of ports for both TcpDiscoverySpi and TcpCommunicationSpi? Or is one port for each fine?

From: [hidden email] At: 12/30/19 15:25:40
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.
No, it will be handled automatically, you don't need to do anything.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?
It will tell cluster that only nodes, which returns true after applying a predicate will store data for this cache. Of course, data won't be rebalanced to the nodes, that shouldn't store data for this cache.

>-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.
Yes, it's equivalent to just adding the node, but using this method you can just use one method for several topology changes(for example, adding 2 nodes to the BT), so it will merge events on the cluster level.

пн, 30 дек. 2019 г. в 12:08, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Thank you for the response. A couple of things:

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?

-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.


From: [hidden email] At: 12/30/19 14:16:33
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Hi,

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?
Cluster needs to make sure that all nodes in cluster know about this cache to avoid situations in future when other nodes will try to create a distributed cache with the same name, for example.

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
It will define a subset of the nodes, which will store data for this cache. So, you can set there only one node.
>-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
No, rebalancing won't happen, it will be skipped. However, before that, nodes should exchange an information regarding new topology ad partition distribution.

Baseline topology should contains all nodes that will store the data.

Evgenii


пн, 30 дек. 2019 г. в 10:59, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Any thoughts on this?

From: [hidden email] At: 12/18/19 19:55:47
To: [hidden email]
Cc: [hidden email], [hidden email]
Subject: Isolating IgniteCache instances across JVMs on same machine by id

We have multiple different database instances for which we are looking into using Ignite as a local caching layer. Each db has a unique id that we are using as part of the IgniteInstanceName and for the WorkDirectory path. We are running IgniteCache in Local mode with persistence enabled, as we effectively want to completely separate the Ignite caches for each db id. However, this is not working due to the IgniteCluster being shared and the BaselineTopology concept. So a couple of questions:

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
-If rebalancing does come into play even if all caches are local, then it seems like separate clusters would have to be specified when starting up.
https://apacheignite.readme.io/docs/tcpip-discovery seems to provide a way to do that by making sure that TcpDiscoverySpi and TcpCommunicationSpi have different port ranges for every instance of the cluster. Again, this seems kind of like overkill if we know in advance that all of the caches will be local and will not need any of the cluster functionality.

Are there any other simpler options? We are only interested in the IgniteCache functionality at a local per process level, at least for now. We might have 10-20 different caches on the same machine, and need these caches to be completely separated with no risk of any of the data being rebalanced/exposed to a different client instance. Ideally we would get persistence without having to activate IgniteCluster, but it does not seem like that is an option.





Mitchell Rathbun (BLOOMBERG/ 731 LEX) Mitchell Rathbun (BLOOMBERG/ 731 LEX)
Reply | Threaded
Open this post in threaded view
|

Re: Isolating IgniteCache instances across JVMs on same machine by id

In reply to this post by Mitchell Rathbun (BLOOMBERG/ 731 LEX)
*If so

From: [hidden email] At: 12/30/19 18:35:35
To: [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

I will apply changes that you mentioned and see if this issue still occurs. If not, I will attach logs.

From: [hidden email] At: 12/30/19 17:58:04
To: Mitchell Rathbun (BLOOMBERG/ 731 LEX )
Cc: [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Mitchell,

Can you share logs from all nodes?

Evgenii

пн, 30 дек. 2019 г. в 14:42, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
"On a disk, it's stored in meta storage inside DB folder."

We also set the IgniteWorkDirectory to be different for every server node id. However, we still will get "Node is not part of the baseline topology, not using persistence." How would this still occur if the there is no shared directory for the cluster?

From: [hidden email] At: 12/30/19 17:33:55
To: [hidden email]
Cc: [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Mitchell,

-For your first response, I just want to make sure I am clear how this works. Say I have two different server nodes with IgniteInstanceName A and B. If both nodes are part of the same cluster and both nodes write to a cache (in local mode) called "datacache", would this be treated as two separate caches or would there be an issue?
It will be more like the same cache, but all data will be local on each node. It shouldn't be a problem. However, I would recommend using partitioned caches with a NodeFilter instead.

-Where is the shared information stored for an IgniteCluster? Looking at https://apacheignite.readme.io/v2.4/docs/cluster-config, it seems like different clusters can be configured by using different port ranges. But for each cluster, the nodes in the baseline topology must be stored. Where does this information get stored?
On a disk, it's stored in meta storage inside DB folder.

-If we need to specify different clusters on the same machine, do we need to specify a range of ports for both TcpDiscoverySpi and TcpCommunicationSpi? Or is one port for each fine?
Next port from port range is used only if a first port is busy. If you set only one port and it's busy at the moment, node won't be able to start. So, it's up to you if you want to set the port range or one port only.

Evgenii

пн, 30 дек. 2019 г. в 13:30, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Makes sense. A couple of more related questions:

-For your first response, I just want to make sure I am clear how this works. Say I have two different server nodes with IgniteInstanceName A and B. If both nodes are part of the same cluster and both nodes write to a cache (in local mode) called "datacache", would this be treated as two separate caches or would there be an issue?

-Where is the shared information stored for an IgniteCluster? Looking at https://apacheignite.readme.io/v2.4/docs/cluster-config, it seems like different clusters can be configured by using different port ranges. But for each cluster, the nodes in the baseline topology must be stored. Where does this information get stored?

-If we need to specify different clusters on the same machine, do we need to specify a range of ports for both TcpDiscoverySpi and TcpCommunicationSpi? Or is one port for each fine?

From: [hidden email] At: 12/30/19 15:25:40
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.
No, it will be handled automatically, you don't need to do anything.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?
It will tell cluster that only nodes, which returns true after applying a predicate will store data for this cache. Of course, data won't be rebalanced to the nodes, that shouldn't store data for this cache.

>-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.
Yes, it's equivalent to just adding the node, but using this method you can just use one method for several topology changes(for example, adding 2 nodes to the BT), so it will merge events on the cluster level.

пн, 30 дек. 2019 г. в 12:08, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Thank you for the response. A couple of things:

-From your first response, it seems like the db id should be included in the cache name. Currently we just included it in the Ignite instance, but not the cache name, which I believe led to some of the issues we have seen.

-What exactly would the node filter protect us from? Just the cache being explicitly restarted on a different node? Or any potential rebalancing as well?

-The cluster will be explicitly set to active the first time that it is created. All new nodes will then need to join the baseline topology to use persistence. It seems like the way to do this is to use IgniteCluster.setBaselineTopology. Is this method equivalent to adding the specified BaselineNode objects to the already existing BaselineTopology? Or does it completely reset the BaselineTopology with the specified BaselineNode objects? If the second, then to update the BaselineTopology, currentBaselineTopology would have to be called, the new node would have to be added, and then the collection would have to be passed back to setBaselineTopology. This would have a race condition if two nodes are trying to join the baseline topology at the same time.


From: [hidden email] At: 12/30/19 14:16:33
To: [hidden email], [hidden email]
Subject: Re: Isolating IgniteCache instances across JVMs on same machine by id

Hi,

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?
Cluster needs to make sure that all nodes in cluster know about this cache to avoid situations in future when other nodes will try to create a distributed cache with the same name, for example.

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
It will define a subset of the nodes, which will store data for this cache. So, you can set there only one node.
>-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
No, rebalancing won't happen, it will be skipped. However, before that, nodes should exchange an information regarding new topology ad partition distribution.

Baseline topology should contains all nodes that will store the data.

Evgenii


пн, 30 дек. 2019 г. в 10:59, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <[hidden email]>:
Any thoughts on this?

From: [hidden email] At: 12/18/19 19:55:47
To: [hidden email]
Cc: [hidden email], [hidden email]
Subject: Isolating IgniteCache instances across JVMs on same machine by id

We have multiple different database instances for which we are looking into using Ignite as a local caching layer. Each db has a unique id that we are using as part of the IgniteInstanceName and for the WorkDirectory path. We are running IgniteCache in Local mode with persistence enabled, as we effectively want to completely separate the Ignite caches for each db id. However, this is not working due to the IgniteCluster being shared and the BaselineTopology concept. So a couple of questions:

1. If the cache mode is local, why does IgniteCluster even come into play? All that we want to do is to read/write from the work directory corresponding to our db id. Is there a way to get persistence without activating the IgniteCluster?

2. How do we achieve the cache isolation we are looking for? Currently, if I have db A and B, this would result in IgniteA and IgniteB in separate JVM instances, each with their own directory. If I start IgniteA, and then IgniteB, IgniteB will complain about not being part of the BaselineTopology and will not use persistence. It seems like there are two options:
-Have every node join the baseline topology when it comes up (I think there is one per machine by default). This seems like it could work, but I am worried about how rebalancing would work. If all of the server nodes are using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, then this seems like it would be an easy enough solution. Also, where is the cluster state stored (Members of baseline topology, etc.)?
-If rebalancing does come into play even if all caches are local, then it seems like separate clusters would have to be specified when starting up.
https://apacheignite.readme.io/docs/tcpip-discovery seems to provide a way to do that by making sure that TcpDiscoverySpi and TcpCommunicationSpi have different port ranges for every instance of the cluster. Again, this seems kind of like overkill if we know in advance that all of the caches will be local and will not need any of the cluster functionality.

Are there any other simpler options? We are only interested in the IgniteCache functionality at a local per process level, at least for now. We might have 10-20 different caches on the same machine, and need these caches to be completely separated with no risk of any of the data being rebalanced/exposed to a different client instance. Ideally we would get persistence without having to activate IgniteCluster, but it does not seem like that is an option.






Nagarjun Ayyagari Nagarjun Ayyagari
Reply | Threaded
Open this post in threaded view
|

Re: Isolating IgniteCache instances across JVMs on same machine by id

Hi,
Is there a solution to this? We are experiencing a similar issue. We have
two separate clusters on the same physical server with 3 persistence enabled
nodes in each cluster. We are not able to restart nodes in one cluster
without restarting both the clusters.

Thanks
Nagarjun



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Isolating IgniteCache instances across JVMs on same machine by id

Hello!

Can you please elaborate what happens if you try to restart a node in just one cluster? How does it affect the other one?

Thanks,
--
Ilya Kasnacheev


вт, 25 авг. 2020 г. в 18:06, Nagarjun Ayyagari <[hidden email]>:
Hi,
Is there a solution to this? We are experiencing a similar issue. We have
two separate clusters on the same physical server with 3 persistence enabled
nodes in each cluster. We are not able to restart nodes in one cluster
without restarting both the clusters.

Thanks
Nagarjun



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/