Canary upgrade

Use a canary upgrade model to upgrade Gloo Gateway, such as in production environments.

In the canary model, you run two deployments of gloo in your cluster: the existing deployment that runs your current version, and a canary deployment that runs the version you want to upgrade to. After you create the canary deployment, you verify that it handles traffic as you expect before upgrading to the new version. This approach helps you reduce potential downtime for production upgrades.

Step 1: Prepare to upgrade

Before you begin, follow the Prepare to upgrade guide to complete these preparatory steps:

• Review important changes made to Gloo Gateway in version 1.17, including CRD, Helm, CLI, and feature changes.
• Upgrade your current version to the latest patch.
• Upgrade any dependencies to the required supported versions.
• Consider other steps to prepare for upgrading.
• Review frequently-asked questions about the upgrade process.

Step 2: Upgrade glooctl

Follow the steps in Update glooctl CLI version to update glooctl to the version you want to upgrade to.

Step 3: Upgrade Gloo Gateway

Now you’re ready to upgrade. The steps vary depending on your Gloo Gateway installation.

• Open Source or Enterprise
• Federation

Upgrade Gloo Gateway Open Source or Enterprise

1. Set the version to upgrade Gloo Gateway to in an environment variable, such as the latest patch version for open source (1.17.15) or enterprise (1.17.3).

   export CANARY_VERSION=<version>
   

2. Update and pull the Gloo Gateway Helm chart for the canary version.

   • Open Source
   • Enterprise

   helm repo add gloo https://storage.googleapis.com/solo-public-helm
   helm repo update
   helm pull gloo/gloo --version $CANARY_VERSION --untar
   

   helm repo add glooe https://storage.googleapis.com/gloo-ee-helm
   helm repo update
   helm pull glooe/gloo-ee --version $CANARY_VERSION --untar
   

3. Apply the CRDs for the canary version to your cluster. The Gloo Gateway CRDs are designed to be backward compatible, so the new CRDs should not impact the performance of your existing installation. For more information, see the list of CRD changes for 1.17.

   • Open Source
   • Enterprise

   kubectl apply -f gloo/crds
   

   kubectl apply -f gloo-ee/charts/gloo/crds
   

4. Install the canary version of Gloo Gateway in your cluster in a new namespace, gloo-system-$CANARY_VERSION.

   • Open Source
   • Enterprise

   glooctl install gateway \
     --create-namespace \
     -n gloo-system-$CANARY_VERSION \
     --version $CANARY_VERSION
   

   Note that you must set your license key by using the --license-key $LICENSE_KEY flag or including the license_key: $LICENSE_KEY setting in your values file. If you do not have a license key, request a Gloo Gateway Enterprise trial.

   glooctl install gateway enterprise \
     --create-namespace \
     -n gloo-system-$CANARY_VERSION \
     --version $CANARY_VERSION \
     --license-key $LICENSE_KEY
   

5. Verify that your existing and canary versions of Gloo Gateway are running.

   kubectl get all -n gloo-system
   kubectl get all -n gloo-system-$CANARY_VERSION
   

6. Check the Feature changes to modify any custom resources for any changes or new capabilities in 1.17.

7. Test your routes and monitor the metrics of the canary version.

   glooctl check
   

8. Uninstall the previous version of Gloo Gateway so that your cluster uses the new version going forward.

   glooctl uninstall -n gloo-system
   

Upgrade Gloo Gateway Federation

In the canary upgrade model for Gloo Gateway Federation, you start with an existing version of Gloo Gateway Federation in your management cluster that manages one or more remote Gloo Gateway instances. To test a new version, you install the canary version of Gloo Gateway Federation in a new namespace on the management cluster. Then, you install one or more Gloo Gateway instances that run the canary version in remote test clusters, register the test clusters, and check your setup. Finally, you deregister the remote test clusters from the previous version of Gloo Gateway Federation, and uninstall the previous Gloo Gateway Federation version from the management cluster.

1. Set your Gloo Gateway license key in an environment variable. If you do not have a license key, request a Gloo Gateway Enterprise trial.

   export LICENSE_KEY=<version>
   

2. Set the version to upgrade Gloo Gateway to in an environment variable, such as the latest patch version (1.17.3).

   export CANARY_VERSION=<version>
   

3. Update and pull the Gloo Gateway Federation Helm chart for the canary version.

   helm repo add gloo-fed https://storage.googleapis.com/gloo-fed-helm
   helm repo update
   helm pull gloo-fed/gloo-fed --version $CANARY_VERSION --untar
   

4. Apply the CRDs for the canary version to your management cluster. The Gloo Gateway CRDs are designed to be backward compatible, so the new CRDs should not impact the performance of your existing installation. However, if after evaluating the canary installation you decide to continue to use the existing installation, you can easily remove any added CRDs by referring to the listed changes to CRD names and running kubectl delete crd <CRD>. Then, to re-apply previous versions of CRDs, you can run helm pull gloo/gloo --version <previous_version> --untar and kubectl apply -f gloo/crds.

   kubectl apply -f gloo-fed/crds
   

5. Install the canary version of Gloo Gateway Federation in a new namespace, gloo-system-$CANARY_VERSION, in your management cluster.

   helm install gloo-fed gloo-fed/gloo-fed \
     --create-namespace \
     -n gloo-system-$CANARY_VERSION \
     --version $CANARY_VERSION \
     --set-string license_key=$LICENSE_KEY
   

6. Verify that your existing and canary versions of Gloo Gateway Federation are running.

   kubectl get all -n gloo-system
   kubectl get all -n gloo-system-$CANARY_VERSION
   

7. Install and register Gloo Gateway Enterprise on one of your remote clusters. Make sure that the version of Gloo Gateway Enterprise matches the canary version of Gloo Gateway Federation that you installed.

8. Check the Feature changes for any changes or new capabilities in 1.17. In your management cluster, create or modify any federated custom resources with these changes to test your canary version of Gloo Gateway Federation. Because Gloo Gateway Federation scans all federated resources on the management cluster, you might want to reuse the existing federated resources in the gloo-system namespace.

   Expand for an example federated virtual service configuration

   This example configuration creates a federated virtual service with different features across versions 1.12 and 1.13.

   • Creates a shared gloo-fed namespace to show that both the existing and canary Gloo Gateway Federation instances can watch for changes to a resource in the same namespace.

   • Places the federated virtual service resource in both the remote1 existing version cluster (1.12) and the remote2 canary version cluster (1.13).

   • Configures settings such as numRetries: 10 that are shared across versions.

   • Adds the retryBackOff that is available in version 1.13 but not in 1.12. As such, the 1.12 federated resource ignores this new configuration.

     kubectl create ns gloo-fed
     kubectl apply -f - <<EOF
     apiVersion: fed.gateway.solo.io/v1
     kind: FederatedVirtualService
     metadata:
       name: my-fed-vs
       namespace: gloo-fed
     spec:
       placement:
         clusters:
           - remote1
           - remote2
         namespaces:
           - gloo-system
       template:
         metadata:
           name: my-vs
         spec:
           virtualHost:
             domains:
             - '*'
             routes:
             - matchers:
               - prefix: /
               routeAction:
                 single:
                   upstream:
                     name: default-petstore-8080
                     namespace: gloo-system
               options:
                 retries:
                   retryOn: 'connect-failure'
                   numRetries: 10
                   perTryTimeout: '5s'
                   retryBackOff:
                     baseInterval: 1s
                     maxInterval: 3s
     EOF
     

9. Check that the updated, federated custom resources from the management cluster are propagated to the remote test cluster.

   1. In the management cluster, verify that the federated resources are propagated to the remote clusters by reviewing the federated resource configuration. This example gets the configuration for the federated virtual service from the previous step.

      kubectl get federatedvirtualservice -n gloo-fed -o yaml
      

      In this example output, the PLACED state in the status section confirms that the resource is federated:

       
      ...
      status:
        namespacedPlacementStatuses:
          gloo-system-$CANARY_VERSION:
            clusters:
              remote2:
                namespaces:
                  gloo-system:
                    state: PLACED
          gloo-system:
            clusters:
              remote1:
                namespaces:
                  gloo-system:
                    state: PLACED
            

   2. In the remote clusters, confirm that the federated resources are created. The following commands are based on the previous step’s example.
      • In the existing version’s remote cluster, verify that the resource is federated with only the existing version’s configuration. In the example, the existing 1.12 version has an updated numRetries: 10, but no retryBackOff section, which is available only in version 1.13 or later.

        kubectl get virtualservices -n gloo-system --context remote1 -o yaml
        ...
        options:
          retries:
            numRetries: 10
            perTryTimeout: 5s
            retryOn: connect-failure
                

      • In the canary version’s remote cluster, verify that the resource is federated with only the canary version’s configuration. In the example, the canary 1.13 version has an updated numRetries: 10, as well as a retryBackOff section.

         
        kubectl get virtualservices -n gloo-system --context remote2 -o yaml
        ...
        options:
          retries:
            numRetries: 10
            perTryTimeout: 5s
            retryOn: connect-failure
            retryBackOff:
              baseInterval: 1s
              maxInterval: 3s
                

   3. Repeat the previous steps for each federated resource that you want to check.

10. Test your routes and monitor the metrics of the canary version in your remote test cluster.

    glooctl check
    

11. Shift traffic to the canary version of Gloo Gateway Federation by uninstalling the previous version.

    1. Deregister your other remote clusters that still use the previous version of Gloo Gateway Federation.
    2. Uninstall the previous version of Gloo Gateway Federation so that your management cluster uses only the new version.

       glooctl uninstall -n gloo-system
       

    3. Optionally delete the previous version namespace from each remote cluster. The deregister command does not clean up the namespace and custom resources in the previous version.

       kubectl delete ns gloo-system
       

Roll back a canary upgrade

As you test your environment with the canary version of Gloo Gateway, you might need to roll back to the previous version. You can follow a canary model for the rollback.

1. Set the previous version that you want to roll back to as an environment variable.

   export ROLLBACK_VERSION=<version>
   

2. If you already removed the previous version of Gloo Gateway from your cluster, re-install Gloo Gateway at the rollback version.

   1. Update and pull the Gloo Gateway Helm chart for the rollback version. The Helm charts vary depending on the Gloo Gateway installation option.
      • Open Source
      • Enterprise
      • Federation

      helm repo update
      helm pull gloo/gloo --version $ROLLBACK_VERSION --untar
      

      helm repo update
      helm pull glooe/gloo-ee --version $ROLLBACK_VERSION --untar
      

      helm repo update
      helm pull gloo-fed/gloo-fed --version $ROLLBACK_VERSION --untar
      

   2. Install the rollback version of Gloo Gateway in a new namespace in your cluster.
      • Open Source
      • Enterprise
      • Federation

      glooctl install gateway \
        --create-namespace \
        -n gloo-system-$ROLLBACK_VERSION \
        --version $ROLLBACK_VERSION
      

      glooctl install gateway enterprise \
        --create-namespace \
        -n gloo-system-$ROLLBACK_VERSION \
        --version $ROLLBACK_VERSION \
        --license-key $LICENSE_KEY
      

      helm install gloo-fed gloo-fed/gloo-fed \
        --create-namespace \
        -n gloo-system-$ROLLBACK_VERSION \
        --version $ROLLBACK_VERSION \
        --set-string license_key=$LICENSE_KEY
      

3. Revert any changes to custom resources that you made during the upgrade. For differences between versions, check the version change list and the changelogs.

4. Gloo Gateway Federation: Shift traffic from the canary version to the rollback version of Gloo Gateway Federation.

   1. Deregister your test remote clusters that use the canary version of Gloo Gateway Federation.
   2. Install and register Gloo Gateway Enterprise using the rollback version on each remote cluster.
   3. Optionally delete the canary version namespace. The deregister command does not clean up the namespace and custom resources in the previous version.

      kubectl delete ns gloo-system-$CANARY_VERSION
      

5. Uninstall the canary version of Gloo Gateway from your cluster.

   glooctl uninstall -n gloo-system-$CANARY_VERSION
   

6. Remove any new CRDs that were added in the canary version that are not present in the rollback version.

   1. Check the list of CRD changes in 1.17 for the names of any newly added CRDs.
   2. Delete any added CRDs.

      kubectl delete crd <CRD>
      

7. Apply the Gloo CRDs from the rollback version Helm chart to your cluster. In Gloo Gateway Federation, this cluster is the local management cluster.

   • Open Source
   • Enterprise
   • Federation

   kubectl apply -f gloo/crds
   

   kubectl apply -f gloo-ee/charts/gloo/crds
   

   kubectl apply -f gloo-fed/crds
   

8. Test your routes and monitor the metrics of the rollback version.

   glooctl check