Automate upgrades with Consul Enterprise

Introduction

Upgrades are always a delicate moment for any production environment and it is important to have to have best practices in place that simplify the process where possible.

Consul Enterprise provides automated upgrades with the autopilot feature that enable you to test the upgrade process before making changes to production. The feature allows you to start new Consul servers alongside the old ones and automatically switch to the new servers, as soon as they are able to reach quorum.

This process automates the leader election process and ensures that the new leader is elected among the new nodes so that removing the old nodes from the datacenter does not trigger a leader election.

Using this tutorial you will test the functionality and get examples on how to:

1. Upgrade Consul servers to a newer version
2. Patch configuration for server nodes without changing Consul version

This will help you create a standardized and repeatable internal process for server upgrades.

Prerequisites

To test the automated upgrades feature explained in this tutorial you will need:

• A Consul Enterprise datacenter with three servers running Consul Enterprise 1.7.0.
• Three extra nodes with Consul Enterprise 1.7.2 binary installed to be used as the new servers once the upgrade is concluded.

You will also need a text editor, the curl executable to test the API endpoints, and optionally the jq command to format the output for curl.

Upgrade to a new Consul version

The automated upgrades feature is enabled by default in Consul Enterprise. You can verify the configuration for your datacenter by using the consul operator autopilot command.

$ consul operator autopilot get-config

##...
DisableUpgradeMigration = false
UpgradeVersionTag = ""

Add new servers

Once you have verified that DisableUpgradeMigration is set to false, you can start the new servers and make sure they join the existing datacenter with the Consul 1.7.0 servers. In this example the old servers with Consul version 1.7.0 are identified as server-1, server-2, and server-3. While the new ones with Consul version 1.7.2 are identified as server-1a, server-2a, and server-3a.

When autopilot detects that a server with a newer version of Consul has joined the datacenter, it will wait to promote the new server as voter until enough newer-versioned servers have been added to the datacenter to reach quorum. When the count of new servers equals or exceeds that of the old servers, autopilot will begin promoting the new servers to voters and demoting the old servers. After this is finished, the old servers can be safely removed from the datacenter.

To check the Consul version on the old servers, you can either use the autopilot health endpoint or the consul members command.

$ consul members

Node      Address           Status  Type    Build      Protocol  DC   Segment
server-1  10.20.10.11:8301  alive   server  1.7.0+ent  2         dc1  <all>
server-2  10.20.10.12:8301  alive   server  1.7.0+ent  2         dc1  <all>
server-3  10.20.10.13:8301  alive   server  1.7.0+ent  2         dc1  <all>

$ curl http://127.0.0.1:8500/v1/operator/autopilot/health | jq .

{
  "Healthy": true,
  "FailureTolerance": 1,
  "Servers": [
    {
      # ...
      "Name": "server-1",
      "Address": "10.20.10.11:8300",
      "SerfStatus": "alive",
      "Version": "1.7.0",
      "Leader": true,
      # ...
      "Voter": true,
      # ...
    },
    {
      # ...
      "Name": "server-2",
      "Address": "10.20.10.12:8300",
      "SerfStatus": "alive",
      "Version": "1.7.0",
      "Leader": false,
      # ...
      "Voter": true,
      # ...
    },
    {
      # ...
      "Name": "server-3",
      "Address": "10.20.10.13:8300",
      "SerfStatus": "alive",
      "Version": "1.7.0",
      "Leader": false,
      # ...
      "Voter": false,
      # ...
    }
  ]
}

Now that you have checked the autopilot configuration and Consul version on your existing three servers, start two new servers with Consul version 1.7.2. Be sure to join them to the existing datacenter.

Verify the new servers are not yet part of the voters pool.

$ consul operator raft list-peers

Node       ID                                    Address           State     Voter  RaftProtocol
server-1   64858627-0565-0c81-234f-4d84b3455303  10.20.10.11:8300  leader    true   3
server-2   5b10789f-3fe6-2407-e6f2-9e85cf9cfeb3  10.20.10.12:8300  follower  true   3
server-3   b0ecba67-a422-e46d-ac53-8f48aa403944  10.20.10.13:8300  follower  true   3
server-1a  15a7b243-95ad-8c7a-4335-4c1690120bbd  10.20.10.21:8300  follower  false  3
server-2a  fd1e7110-cc90-cf0c-d1e1-a22190208b0a  10.20.10.22:8300  follower  false  3

Finally, start the third new server on Consul version 1.7.2. Once the third new server is started and autopilot detects the possibility to reach a quorum among the new servers only it promotes them as voters, triggers a new leader election, and demotes the old nodes as non-voters.

Verify the three new servers on Consul version 1.7.2 are now voters.

$ consul operator raft list-peers

Node       ID                                    Address           State     Voter  RaftProtocol
server-1   64858627-0565-0c81-234f-4d84b3455303  10.20.10.11:8300  follower  false  3
server-2   5b10789f-3fe6-2407-e6f2-9e85cf9cfeb3  10.20.10.12:8300  follower  false  3
server-3   b0ecba67-a422-e46d-ac53-8f48aa403944  10.20.10.13:8300  follower  false  3
server-1a  15a7b243-95ad-8c7a-4335-4c1690120bbd  10.20.10.21:8300  leader    true   3
server-2a  fd1e7110-cc90-cf0c-d1e1-a22190208b0a  10.20.10.22:8300  follower  true   3
server-3a  36d57c89-dc39-056d-f8eb-b1a63faa85b8  10.20.10.23:8300  follower  true   3

Stop old servers

Once the upgrade is rolled out you can proceed by removing the old servers from the datacenter using the consul leave command. This will leave only the new servers and will prevent a leader election during the process that you would have experienced without autopilot.

Migrations without a Consul version change

In some environments, Consul is not the only component that needs to be upgraded to ensure security or stability for the node. For example, you might need to roll out node updates to apply OS patches or new configuration settings.

To cover these cases, Consul autopilot offers the possibility to set the UpgradeVersionTag parameter to override the version information used during automated upgrades. The same logic can be used for updating the datacenter nodes even without the need to change Consul version.

If the UpgradeVersionTag is configured, Consul will use its value to look for a version in each server's specified -node-meta tag. For example, if UpgradeVersionTag is set to build, and -node-meta build:0.0.2 is used when starting a server, that server's version will be 0.0.2 when considered in a migration. The upgrade logic will follow semantic versioning and the version string must be in the form of either X, X.Y, or X.Y.Z.

Configure autopilot for existing servers

Before proceeding with the upgrade it is necessary to prepare the existing servers to be compared with the new ones. For all the existing servers, add the node_meta option to the configuration.

## ...
node_meta {
  build = "0.0.1"
}
## ...

## ...
"node_meta": {
    "build": "0.0.1"
  },
## ...

This change requires a reload.

$ consul reload

Configuration reload triggered

Once the configuration is updated, you should modify the Consul autopilot configuration to reflect the node_meta configuration.

$ consul operator autopilot set-config -upgrade-version-tag=build

Configuration updated!

Ensure the autopilot configuration has been successfully updated.

$ consul operator autopilot get-config

## ...
DisableUpgradeMigration = false
UpgradeVersionTag = "build"

The UpgradeVersionTag should be set to build.

Start new servers

For the new servers, you will define the UpgradeVersionTag directly in the configuration.

## ...
node_meta {
  build = "0.0.2"
}
autopilot {
  upgrade_version_tag = "build"
}
## ...

# ...
    "node_meta": {
        "build": "0.0.2"
    },
    "autopilot":{
        "upgrade_version_tag":"build"
    },
# ...

Once the third new server is started and autopilot detects the possibility to reach a quorum among the new servers only it promotes them as voters, triggers a new leader election, and demotes the old nodes as non-voters. This time the version comparison happened not using the Consul internal version but the value expressed in the build value.

$ consul operator raft list-peers

Node       ID                                    Address           State     Voter  RaftProtocol
server-1a   dcc542fc-ff83-2cab-5bbb-7615584be019  10.20.10.11:8300  follower  false  3
server-2a   f38a430b-dda0-ad3d-a394-08c59052723c  10.20.10.12:8300  follower  false  3
server-3a   647a98a1-d2ab-f5d3-e911-91c1c10a92f4  10.20.10.13:8300  follower  false  3
server-1b   7770d10c-fcca-f6f9-222a-e1777a7fd29e  10.20.10.21:8300  leader    true   3
server-2b   bace6b4d-5e2e-8942-d553-ca527cbc2386  10.20.10.22:8300  follower  true   3
server-3b   221de624-0f5c-29da-ffaf-39481adcd234  10.20.10.23:8300  follower  true   3

Stop old servers

Once the migration is rolled out you can proceed by removing the old servers from the datacenter using the consul leave command. This will leave only the new servers but will prevent a leader election during the process that you would have experienced without autopilot.

Next steps

In this tutorial you upgraded your Consul datacenter by using autopilot's automated upgrades functionality. The automated upgrade feature allows you to both upgrade the Consul version of your servers and to deploy updated version for your server instances without the need to change the Consul version to distribute patches. You now have an extra tool to use when planning upgrades of critical system and to reduce manual steps to a minimum.

Before planning any Consul upgrade make sure to review version specific instructions provided by:

• Upgrading Consul
• Upgrading specific Consul versions

You can learn more on other autopilot functionalities by checking our autopilot tutorial.