Upgrade to Spinnaker (1.20.x+/2.20.x+) Causes Errors as Pipelines Deploy to Unavailable Namespace


Issue

After upgrading Spinnaker completed successfully from 1.19.x/2.19.x or below to 1.20.x/2.20.x or above, attempts to deploy pipelines causes errors.  The following is an example of the error

Deploy failed: Error from server (NotFound): error when creating "STDIN": namespaces "xxxxxxxxxxxxx" not found

The namespace is not a part of any declarations within your halconfig, etc.  
 

This error can also be applicable/appear where Spinnaker will attempt to deploy to the current namespace it occupies (basically what it defaults to) if the following is true:

  • kubeconfig doesn't define any namespace
  • Manifest doesn't define any namespace
  • Override namespace in stage definition is not enabled
  • Deployment targets a different cluster than the one where spinnaker lives

Cause

Spinnaker OSS change in 1.20.x (2.20.x Armory version):
https://github.com/spinnaker/spinnaker/issues/5731

A bugfix that shipped with Spinnaker 1.20 fixes an issue in versions 1.19 and earlier, where Spinnaker would attempt to read the default namespace in the context using a faulty JSON path.

As a result of this error, Spinnaker was always falling back to the default namespace. As a result of this change, Spinnaker is now correctly interpreting the default namespace declared in the kubeconfig file


Another possibility is that Spinnaker can also end up falling back in the namespace specified in:

/var/run/secrets/kubernetes.io/serviceaccount/namespace

in absence of any namespace specified in the kubeconfig file

Solution

There are a few solutions for resolving this issue

Setting the Namespace in your Kubeconfig

You can also set the namespace values within your kubeconfig.  Although this can require more changes, any references in your environment involving Kubernetes will directly refer to your kubeconfig file.

contexts:
- context:
    cluster: <someCluster>
    # Set namespace to default if you want to maintain the previous behavior.
    namespace: <targetNamespace>   
    user: <username>


Setting the Namespace in your Deployment Manifest

In your deployment manifest, set the namespace to the namespace that you want to be deploying to.  This defines it within Spinnaker only.

apiVersion: batch/v1
kind: Job
metadata:
    generateName: <someName> 
    # Set namespace to default if you want to maintain the previous behavior.
    namespace: <targetNamespace>


Change Kubectl Service Account

For full explanation: https://github.com/spinnaker/spinnaker/issues/5731#issuecomment-686838502

If you are unable to explicitly set a default namespace, you can attempt the following method to revert settings so that Kubernetes will no longer mount a service token in /var/run/secrets/kubernetes.io/serviceaccount

This will cause the logic to fall back to using the string default, so that the namespace default is once again used.  Since this is a reversion, it may be necessary to also process additional updates to different objects, artifacts and references. 

  1. Edit the service account YAML for the Spinnaker namespace to be reversed back to default

    # kubectl edit serviceaccount default -n spinnaker -o yaml

  2. Set the following config to false in the PodTemplate spec of the CloudDriver pod

    automountServiceAccountToken: false

  3. Redeploy the pods

As Tested On Version

2.20.x and above