Strict Mode Disabled
When this option is disabled, this strict synchronization is loosened, and other nodes are able to freely join the cluster without waiting for channel-specific tasks to finish. In addition, channel tasks will not block each other either. The startupDeploy and channelOperation cluster-wide locks are not acquired on startup in this case.
This means that it is possible that channels could get out-of-sync, at least temporarily. For example:
- Channel A is currently deployed and Started across the cluster.
- A new server joins the cluster and sees that Channel A is currently deployed and started across the cluster, so it prepares to deploy and start that channel as well.
- Meanwhile, a user stops Channel A.
- The newly joined server deploys and starts Channel A. Now Channel A is Started on one server but Stopped on another.
To address this, a separate channel synchronization process will be used to detect whether channels ever get out of sync and then deploy/undeploy/start/stop/etc them as necessary. Whenever a channel operation is taken (such as Deploy or Start), the "desired state" of the channel is written into the database. Each node will have its own synchronization process that will compare the current states of all channels against the "desired" state in the database. If they are different, the synchronization process will automatically adjust (deploy/start/etc) the channel/connector as needed.
Now, consider the example above again:
- Channel A is currently deployed and Started across the cluster.
- A new server joins the cluster and sees that Channel A is currently deployed and started across the cluster, so it prepares to deploy and start that channel as well.
- Meanwhile, a user stops Channel A.
- As part of the user Stop action, the channel desired state is updated to Stopped in the database.
- The newly joined server deploys and starts Channel A. Now Channel A is Started on one server but Stopped on another.
- At the next interval, the synchronization process on the newly joined server detects that Channel A is Started, but the desired state is Stopped.
- The synchronization process automatically stops Channel A on the newly joined server.
- Now the channel is back in sync across the cluster, Stopped on all servers.
The Channel Synchronization Interval option in the Clustering settings determines how often this process runs. By default it runs once every 30 seconds.
Example: Deploy Script Fails
Let's say you have a deploy script that does some non-trivial amount of work. Maybe it connects to a remote database and pulls some information into the global channel map. What happens if that deploy script fails?
The following deploy script simulates a channel failing to deploy half the time:
if(Math.random() > 0.5) a();