When this option is disabled, this strict synchronization is loosened, and other nodes are able to freely join the cluster without waiting for channel-specific tasks to finish. In addition, channel tasks will not block each other either. The startupDeploy and channelOperation cluster-wide locks are not acquired on startup in this case.
This means that it is possible that channels could get out-of-sync, at least temporarily. For example:
- Channel A is currently deployed and Started across the cluster.
- A new server joins the cluster and sees that Channel A is currently deployed and started across the cluster, so it prepares to deploy and start that channel as well.
- Meanwhile, a user stops Channel A.
- The newly joined server deploys and starts Channel A. Now Channel A is Started on one server but Stopped on another.
To address this, a separate channel synchronization process will be used to detect whether channels ever get out of sync and then deploy/undeploy/start/stop/etc them as necessary. Whenever a channel operation is taken (such as Deploy or Start), the "desired state" of the channel is written into the database. Each node will have its own synchronization process that will compare the current states of all channels against the "desired" state in the database. If they are different, the synchronization process will automatically adjust (deploy/start/etc) the channel/connector as needed.
Now, consider the example above again:
- Channel A is currently deployed and Started across the cluster.
- A new server joins the cluster and sees that Channel A is currently deployed and started across the cluster, so it prepares to deploy and start that channel as well.
- Meanwhile, a user stops Channel A.
- As part of the user Stop action, the channel desired state is updated to Stopped in the database.
- The newly joined server deploys and starts Channel A. Now Channel A is Started on one server but Stopped on another.
- At the next interval, the synchronization process on the newly joined server detects that Channel A is Started, but the desired state is Stopped.
- The synchronization process automatically stops Channel A on the newly joined server.
- Now the channel is back in sync across the cluster, Stopped on all servers.
The Channel Synchronization Interval option in the Clustering settings determines how often this process runs. By default it runs once every 30 seconds.
Note: To clarify again: This new channel synchronization process will only be active if you
disable the Strict Channel Synchronization option in the Clustering settings. By default it's enabled to preserve the same behavior as previous versions when you upgrade to 3.12.0.
Strict Channel Synchronization:
- Enabled: The old behavior. Any channel operations will block new servers from joining the cluster, and new servers joining the cluster will block any channel operations from being performed, and also any other servers from joining.
- Disabled: The new behavior. Servers can join the cluster and perform their initial deploy process without blocking or being blocked on any channel operations. If any channels get out-of-sync, a separate synchronization process will adjust them as needed to get them back in sync.
Example: Deploy Script Fails
Let's say you have a deploy script that does some non-trivial amount of work. Maybe it connects to a remote database and pulls some information into the global channel map. What happens if that deploy script fails?
The following deploy script simulates a channel failing to deploy half the time:
if(Math.random() > 0.5) a();
When you deploy the channel, it may succeed on one server, but not another. You can check on the status of your deploy task in the
Cluster Task History tab on the dashboard:
You can also double-click on that error to see what happened:
If you have Strict Channel Synchronization turned off, the synchronization process will detect that and automatically deploy the channel. When it takes action, it will notify you in the server log as well:
