For one of our customers, we had a requirement to use ArgoCD to serve up applications inside a Big Bang-based Kubernetes Cluster deployed with Zarf. There is a lot to unpack there.

ArgoCD is a tool in the GitOps space that enables sort of an inversion of control pattern in a Kubernetes cluster (very similar to Flux or Fleet). Instead of pushing a Helm Chart and Images to a cluster, a Kubernetes cluster listens for changes in a Git repository and then takes appropriate reconciliatory actions. This enables rapid deployment at any stage of the lifecycle of an application. ArgoCD has the added benefit of having a very well-presented user interface to make it easy for non-technical users to debug what might be going on with their deployment.

Big Bang is an open-source and opinionated Helm chart with many well-thought-out capabilities, specifically designed for typical DoD scenarios (although other governmental organizations have adopted it). It also makes use of an image repository called Iron Bank, which is maintained through a set of standards to keep its security hardened.

Lastly, Zarf is another open-source tool that allows an operator to package up Kubernetes Manifests (and Helm Charts) and Images into a single well-formed package for deployment inside of an Airgapped Network.

Upon deployment, Zarf will package up the Git repositories and Images and then later “inject” them into the air-gapped cluster. Those URLs and Image references, however, would all be targeted toward a network location that is no longer accessible. Zarf utilizes Admission Controllers by way of a Mutating Webhook which analyzes those URLs and points them to the internal registry and git server inside that cluster. All of this was designed with the concept that one would be using Flux, not ArgoCD. Given that the Flux Mutating Webhook looks at a specific schema, this required enhancing the Zarf Agent to support ArgoCD.

Here are the steps we took in order to set up our development environment to add this capability:

• Set up a Kubernetes cluster. We used K3D in this situation.
• Initialize the cluster with Zarf.

zarf init -a amd64 --components=git-server --confirm --set K3S_ARGS="--disable=traefik --disable=metrics-server --disable=servicelb"

• Deploy ArgoCD. We did not use Zarf for this step, so you’ll have to temporarily disable the Zarf Agent from mutating these URLs).

kubectl create namespace argocd
kubectl label namespace argocd zarf.dev/agent=ignore --overwrite=true
helm upgrade --install --namespace argocd <argocd-helm-chart>

• After this is done deploying, you can remove the label from the ArgoCD namespace that tells the agent to ignore it.
• Now we can make changes to the agent and test them in the Zarf code base. Each time we must recompile the binary, create an image and then inject it into the cluster. It also requires removing the image from the cache. Here is a summary of that:

AGENT_IMAGE_TAG=local ARCH=amd64 make init-package-local-agent
docker save ghcr.io/defenseunicorns/zarf/agent:local > zarf-agent-local.tar
zarf tools crane push zarf-agent-local.tar 127.0.0.1:31999/defenseunicorns/zarf/agent:local
docker exec k3d-k3s-default-server-0 crictl rmi 127.0.0.1:31999/defenseunicorns/zarf/agent:local
kubectl rollout restart deployment agent-hook -n zarf

• This works very for rapidly iterating and deploying new changes.
• It also leverages a new feature coming to be able to push images directly into a running Zarf cluster without having to open a tunnel and authenticate.

Here is the pull request with the new feature that may appear in Zarf one day!

If you need any help with Zarf, Kubernetes, ArgoCD, reach out to us!