IKO Plus: Operator Works From Home - IrisCluster Provisioning Across Kubernetes Clusters

IKO Helm Status: WFH

Here is an option for your headspace if you are designing a multi-cluster architecture and the Operator is an FTE to the design. You can run the Operator from a central Kubernetes cluster (A), and point it to another Kubernetes cluster (B), so that when you apply an IrisCluster to B, the Operator works remotely on A and plans the cluster accordingly on B.

This design keeps resource heat off the actual workload cluster, spares us some ServiceAccounts/RBAC, and gives us only one operator deployment to worry about so we can concentrate on the IRIS workloads.

IKO woke up and decided against the commute for work. Using the saved windshield time, IKO upgraded its Helm values on a Kubernetes cluster at home, bounced itself, and went for a run. Once settled back in, inspecting its logs, it could see it had planned many IrisClusters on the Office Kubernetes cluster, all at the cost of its own internet and power.

---

The Setup

Clusters

Let’s provision two Kind clusters: ikohome and ikowork.

ikokind.sh

# Create ikohome cluster
cat <<EOF | kind create cluster --name ikohome --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
networking:
  disableDefaultCNI: true
EOF

# Create ikowork cluster
cat <<EOF | kind create cluster --name ikowork --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
networking:
  disableDefaultCNI: true
EOF

# Extract kubeconfigs
kind get kubeconfig --name ikohome > ikohome.kubeconfig
kind get kubeconfig --name ikowork > ikowork.kubeconfig

# Install Cilium CNI on both
KUBECONFIGS=("ikohome.kubeconfig" "ikowork.kubeconfig")
for cfg in "${KUBECONFIGS[@]}"; do
  echo ">>> Running against kubeconfig: $cfg"
  cilium install --version v1.18.0 --kubeconfig "$cfg"
  cilium status --wait --kubeconfig "$cfg"
  echo ">>> Finished $cfg"
done

---

Install IKO at HOME 🏠

First, we need to make the home cluster aware of the work cluster and load up its kubeconfig as a secret:

kubectl create secret generic work-kubeconfig \
  --from-file=config=ikowork.kubeconfig \
  --kubeconfig ikohome.kubeconfig

Now, we need to make two critical changes to the IKO chart to enable “Work From Home” mode:

1. Mount the kubeconfig Secret as a Volume in the Operator.
2. Args: Point the operator to the external kubeconfig in the container arguments (--kubeconfig).

1. The Mount

2. The Args

The arguments to the container are crucial. I tried this with the environment variable KUBECONFIG, but found there is a specific precedence in the controller code.

Full deployment.yaml (Highlights)

# ...
      containers:
        - name: operator
          image: {{ .Values.operator.registry }}/{{ .Values.operator.repository }}:{{ .Values.operator.tag }}
          args:
            - run
            - --v={{ .Values.logLevel }}
            - --kubeconfig=/airgap/.kube/config
# ...
          volumeMounts:
            - mountPath: /airgap/.kube
              name: kubeconfig
              readOnly: true
      volumes:
        - name: kubeconfig
          secret:
            secretName: work-kubeconfig
            items:
              - key: config
                path: config

Chart Values

In values.yaml, I also disabled the mutating and validating webhooks for this remote setup.

---

Install CRDs at WORK 🏢

The operator normally handles CRD installation. For this remote setup, the CRDs must exist in the Work cluster, even though the operator is not running there.

We can pull this maneuver:

# Export CRDs from home (where they are installed)
kubectl get crd irisclusters.intersystems.com --kubeconfig ikohome.kubeconfig -o yaml > ikocrds.yaml

# Create CRDs at work
kubectl create -f ikocrds.yaml --kubeconfig ikowork.kubeconfig

---

Final State 💥

• IKO is running at home, not at work.
• CRDs are loaded at work, but no operator is present.

When we apply IrisClusters at work, the operator at home will plan and schedule them remotely.

It’s alive! The operator has planned many IrisClusters on the Office cluster from its home base.

Related Reading:

• IKO Plus: Launching Airgapped IrisClusters with Hauler
• IKO Plus: Database Management, Migration, or Day 1 Seeding
• Controller Runtime Config Precedence