Lot's of repeated error logs when a node is killed in cluster. 1GB+/minute
I have two Ignite nodes running in my cluster. Node1 is a discovery node, and Node2 is a regular node. During testing, we started a process on both nodes, then took down Node1 while the process was running (kill -9). The logs on Node2 immediately started growing at an immense rate, over 1GB of text per minute. Upon closer examination, they are all repeated errors that are the same.
java.lang.IllegalStateException: Queue has been removed from cache: GridCacheQueueAdapter [cap=2147483647, collocated=false, rmvd=true]
Seem's like ignite is trying to access a queue in the cluster cache, but the cluster cache should be available even if one node goes down right?
Re: Lot's of repeated error logs when a node is killed in cluster. 1GB+/minute
The queue gets removed because by default there are no backups and when you kill the node, you potentially lose part of your data. To fix this provide the CollectionConfiguration specifying the correct number of backups like this:
Having one backup guarantees that you won't lose data when one node fails. If you assume that there can be more failures at the same time, you will need to have more backups.
I'm not sure why the log is growing. Most likely your service continues to try polling from the queue even when it's not available anymore, so each iteration results in exception. So probably you should change your code to stop the service or recreate the queue after the first failure.