Embedded ignite and baseline upgrade questions

classic Classic list List threaded Threaded
6 messages Options
djm132 djm132
Reply | Threaded
Open this post in threaded view
|

Embedded ignite and baseline upgrade questions

Hi,

 

I am implementing distributed crawler engine using ignite in embedded mode. Each crawler node have persistence enabled and assigned predefined consistentId (dev-1, dev-2, dev-3, etc). Baseline topology (nodes which store data) can change and configured using config file like this:

 

# Cluster settings

cluster {

    # Baseline topology (list of node ids which store data)

    baseline = [

        "dev-1",

        "dev-2",

        "dev-3"

    ]

}

 

Each crawler node startup implemented like this:

 

        // Start ignite

        ignite = Ignition.start(igniteConfig)

 

        // Activate

        ignite.cluster().active(true)

 

        // Validate baseline topology

        val baseline = config.getStringAll(SpiderNodeOptions.CLUSTER_BASELINE).toSet()

        val curTopology = ignite.cluster().currentBaselineTopology()

        if (curTopology == null || baseline != curTopology.map { it.consistentId().toString() }.toSet()) {

            // Wait for all new topology members to join

            log.info("Cluster baseline topology changed - waiting data nodes to join...")

            while (true) {

                val nodes = mutableListOf<ClusterNode>()

                ignite.cluster().nodes().forEach {

                    if (baseline.contains(it.consistentId().toString()))

                        nodes.add(it)

                }

 

                if (nodes.size == baseline.size) {

                    log.info("Updated baseline topology of {} nodes", value("baseline_size", nodes.size))

                    ignite.cluster().setBaselineTopology(nodes)

                    break

                }

                else

                    Thread.sleep(1000)

            }

        }

 

We can’t set baseline topology before cluster activation so activate() called first. Is it possible to set baseline topology without activating cluster for first time ? Is it safe to activate node with the code above ? It seems that we have misconfigured state in clean cluster between activate() and waiting nodes to join when setBaseliteTopology not called yet. The steps for upgrading topology will be 1) modify and commit config files 2) shutdown all cluster members 3) update code 4) start each node one by one. Is this the right way of reconfiguration ?

 

Thanks,

Michael

djm132 djm132
Reply | Threaded
Open this post in threaded view
|

Re: Embedded ignite and baseline upgrade questions

Also when starting second node I am getting this:

2019/12/27 23:39:36.665 [disco-pool-#55] WARN BaselineTopology of joining
node (dev-1) is not compatible with BaselineTopology in the cluster.
Branching history of cluster BlT ([95475834]) doesn't contain branching
point hash of joining node BlT (95475833). Consider cleaning persistent
storage of the node and adding it to the cluster again.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Embedded ignite and baseline upgrade questions

Hello!

First of all, I really hope you're not using != to compare collections :)

Second, cluster will auto-activate when all baseline nodes have joined. For the first time you have to manually activate the cluster.

The algorithm looks OK and nodes should auto-activate once they are all up.

With regards to your first question - you have managed nodes from different baseline topologies, i.e., nodes from different clusters. Persistent node from the "wrong" cluster will not join.

Regards,
--
Ilya Kasnacheev


сб, 28 дек. 2019 г. в 00:41, djm132 <[hidden email]>:
Also when starting second node I am getting this:

2019/12/27 23:39:36.665 [disco-pool-#55] WARN BaselineTopology of joining
node (dev-1) is not compatible with BaselineTopology in the cluster.
Branching history of cluster BlT ([95475834]) doesn't contain branching
point hash of joining node BlT (95475833). Consider cleaning persistent
storage of the node and adding it to the cluster again.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
djm132 djm132
Reply | Threaded
Open this post in threaded view
|

Re: Embedded ignite and baseline upgrade questions

!= in kotlin uses equals() under the hood so it works here as expected.

I can't use control.sh to manage topology (becase ignite is embedded) and
trying to implement it in code. So the actual initialization sequence is:

1) Start all fresh nodes, wait for all specified persistence nodes to be
online.
2) Auto-activation sets baseline to all nodes online to the moment.
3) If it first launch - currentBaselineTopology() will be null (or probably
different set) and I will set it to defined list of nodes using
setBaselineTopology().
4) Later if extending baseline required - we wait for all of them and call
setBaselineTopology to new list.
5) Later if shrinking baseline required - we set it using the same
setBaselineTopology() again (excluded node need to be offline for sure), it
also triggers data rebalancing as expected.
6) If we need to include new node to baseline, its definitely need to be
cleaned from old stored baseline metadata, right?

So it seems that managing topology by specifying a list in config file is
possible with this code? Or may be I am missing something ?

Is there any global event for data rebalancing ? Or it should be tracked for
every cache with localListen ?

Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
akorensh akorensh
Reply | Threaded
Open this post in threaded view
|

Re: Embedded ignite and baseline upgrade questions

Hi,
  Your summary looks correct.
  It should be possible to manage your baseline topology using a config
file, provided you follow the steps outlined.

more info here:
https://apacheignite.readme.io/docs/baseline-topology#section-triggering-rebalancing-programmatically

  You can use
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/events/CacheRebalancingEvent.html
for each cache


You can also use JMX to monitor rebalance:
https://www.gridgain.com/docs/latest/administrators-guide/monitoring-metrics/metrics#monitoring-rebalancing

In general, it is advisable to clean old metadata when re-adding a baseline
node(especially if you made changes to you cache config), but be careful to
make sure that no data is lost inadvertently.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
djm132 djm132
Reply | Threaded
Open this post in threaded view
|

Re: Embedded ignite and baseline upgrade questions

Thanks, its definitely clear now that rebalancing should be triggered from
code if node removal detected. Assuming that number of backups > 0 and only
one node removed it looks like safe case. But what if backup count = 0 (bad
idea but the risk may be acceptable in some cases) and we need to shutdown
node to remove it from BL ? Is there any way to trigger exclusive
rebalancing (ie moving out all data from some node to other valid nodes) and
than exclude this node from data storage ?

Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/