Working with multiple clusters – A Murder of Crows

So for a while, I have had a very backward way of accessing multiple clusters: I would set the kubeconfig environment variable, or change the default file. If I had bothered to learn the first thing about contexts, I could have avoided the confusion of keeping track of multiple files.

When a cluster is created, we often get a basic config file to access the cluster. I had often looked at these as a black box of access. Here is an example below from my rancher cluster:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: https://rke1:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

Thanks to the official documentation (RTFM folks) I think it has finally clicked. We have lists of 3 different object types in the above config:
– Cluster: the connection to the cluster (contains a CA and endpoint)
– User: Identified with the client cert data and key data
– Context: Ties the above together (also namespaces if we want)

Contexts allow me to have multiple configurations and switch between them using the kubectl config use-context command. My goal is to have a connection to both my openshift cluster, and my rancher cluster. So I combined (and renamed some elements) the configuration:

apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://api.oc1.lab.local:6443
  name: api-oc1-lab-local:6443
- cluster:
    certificate-authority-data: REDACTED
    server: https://rke1:6443
  name: rancher
contexts:
- context:
    cluster: api-oc1-lab-local:6443
    namespace: default
    user: kube:admin/api-oc1-lab-local:6443
  name: default/api-oc1-lab-local:6443/kube:admin
- context:
    cluster: rancher
    user: rancherdefault
  name: rancher
current-context: rancher
kind: Config
preferences: {}
users:
- name: kube:admin/api-oc1-lab-local:6443
  user:
    token: REDACTED
- name: rancherdefault
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

If we understand a little YAML, we can easily combine the files. Now it is simple to switch between my clusters:

kubectl config get-contexts
CURRENT   NAME                                        CLUSTER                  AUTHINFO                            NAMESPACE
          default/api-oc1-lab-local:6443/kube:admin   api-oc1-lab-local:6443   kube:admin/api-oc1-lab-local:6443   default
*         rancher                                     rancher                  rancherdefault
kubectl config use-context default/api-oc1-lab-local:6443/kube:admin
Switched to context "default/api-oc1-lab-local:6443/kube:admin".