kubectl and helm diff challenges / Articles / dracoblue.net

When you are using helm and kubernetes to maintain your service workload, you might run into the question: what will be I execute kubectl apply, helm upgrade or helm install. There are multiple native ways, plugins and combined efforts to make this possible.

TLDR:

helm diff does only show changes between last helm revision and the new one
if you need to show diff between status quo in k8s and what helm would apply, combine helm template or helm upgrade --dry-run output with kubectl diff in --serverside=false mode. But in this case you have to ignore all fieldsType: FieldsV1 and related f: fields if you only want to see the real changes.

You can run through the entire blog post by using https://www.katacoda.com/dracoblue/scenarios/helm-and-kubectl-diff on katacoda. The blog post is based on k3s (kubernetes 1.20.12) and helm 3.6.3. And it requires helm plugin install https://github.com/databus23/helm-diff if you want to use helm diff plugin.

Setup a plain helm chart called foo:

$ helm create foo
$ cd foo

and install it:

$ helm upgrade --install foo .
Release "foo" does not exist. Installing it now.
NAME: foo
LAST DEPLOYED: Sun Nov 14 06:04:11 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=foo,app.kubernetes.io/instance=foo" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT

and to ensure that something is changed in k8s, which is not visible to helm without looking at the resources in k8s, change replica count manually by scaling:

kubectl scale deploy/foo --replicas=2

So now .Values.replicaCount is 1 in the chart and the deployment foo resource has replicas set to 2.

`helm diff` plugin does not show non-helm changes

Now let's see what helm diff says, if we set the image.tag to 1.14.0 but keeping replicaCount to 1 in the helm chart.

$ helm diff --install foo --set image.tag=1.14.0 .

You get:

            securityContext:
              {}
-           image: "nginx:1.16.0"
+           image: "nginx:1.14.0"
            imagePullPolicy: IfNotPresent

but the replicas are unaffected. So use this only if you do not care about the manual changes in the k8s (because you can ensure that they do not happen at all).

`helm template` + `kubectl diff` with `server-side=false` shows all changes, but also FieldsV1 updates

If you don't want (or have to) use the helm diff plugin, you can use helm template in combination with kubectl diff instead, which looks liks this:

$ helm template --is-upgrade --no-hooks --skip-crds foo --set image.tag=1.14.0 . | kubectl diff --server-side=false -f -

You will see something like this:

 spec:
   progressDeadlineSeconds: 600
-  replicas: 2
+  replicas: 1
   revisionHistoryLimit: 10
   selector:
     matchLabels:
@@ -154,7 +165,7 @@
         app.kubernetes.io/name: foo
     spec:
       containers:
-      - image: nginx:1.16.0
+      - image: nginx:1.14.0
         imagePullPolicy: IfNotPresent
         livenessProbe:
           failureThreshold: 3

and both parts (replicas and image tag) are included. Yay!

But it also includes lots of

   creationTimestamp: "2021-11-14T08:31:53Z"
-  generation: 2
+  generation: 3
   labels:
     app.kubernetes.io/instance: foo
     app.kubernetes.io/managed-by: Helm
@@ -31,7 +31,6 @@
           f:helm.sh/chart: {}
       f:spec:
         f:progressDeadlineSeconds: {}
-        f:replicas: {}
         f:revisionHistoryLimit: {}
         f:selector: {}
         f:strategy:
@@ -50,7 +49,6 @@
             f:containers:
               k:{"name":"foo"}:
                 .: {}
-                f:image: {}
                 f:imagePullPolicy: {}
                 f:livenessProbe:
                   .: {}
@@ -129,13 +127,26 @@
     manager: k3s
     operation: Update
     time: "2021-11-14T08:33:16Z"
+  - apiVersion: apps/v1
+    fieldsType: FieldsV1
+    fieldsV1:
+      f:spec:
+        f:replicas: {}
+        f:template:
+          f:spec:
+            f:containers:
+              k:{"name":"foo"}:
+                f:image: {}
+    manager: kubectl-client-side-apply
+    operation: Update
+    time: "2021-11-14T08:36:26Z"
   name: foo
   namespace: default

`helm template` + `kubectl diff` with `server-side=true` shows all changes, but also FieldsV1 updates

You will see something like this:

helm template --is-upgrade --no-hooks --skip-crds foo --set image.tag=1.14.0 . | kubectl diff --server-side=true -f -

and you will most probably get:

W1114 08:38:06.305048    5067 diff.go:544] Object (apps/v1, Kind=Deployment: foo) keeps changing, diffing without lock
Error from server (Conflict): Apply failed with 2 conflicts: conflicts with "helm" using apps/v1:
- .spec.replicas
- .spec.template.spec.containers[name="foo"].image

Now try with --force-conflicts=true so execute: helm template --is-upgrade --no-hooks --skip-crds foo --set image.tag=1.14.0 . | kubectl diff --server-side=true -f - --force-conflicts=true.

You will see the proper diff:

 spec:
   progressDeadlineSeconds: 600
-  replicas: 2
+  replicas: 1
   revisionHistoryLimit: 10
   selector:
     matchLabels:
@@ -154,7 +198,7 @@
         app.kubernetes.io/name: foo
     spec:
       containers:
-      - image: nginx:1.16.0
+      - image: nginx:1.14.0
         imagePullPolicy: IfNotPresent
         livenessProbe:
           failureThreshold: 3

but with the disadvantage that ALL resources (even unrelated ones like service accounts) are affected:

--- /tmp/LIVE-596917768/v1.ServiceAccount.default.foo
+++ /tmp/MERGED-086363591/v1.ServiceAccount.default.foo
@@ -16,6 +16,19 @@
     fieldsType: FieldsV1
     fieldsV1:
       f:metadata:
+        f:labels:
+          f:app.kubernetes.io/instance: {}
+          f:app.kubernetes.io/managed-by: {}
+          f:app.kubernetes.io/name: {}
+          f:app.kubernetes.io/version: {}
+          f:helm.sh/chart: {}
+    manager: kubectl
+    operation: Apply
+    time: "2021-11-14T08:44:12Z"
+  - apiVersion: v1
+    fieldsType: FieldsV1
+    fieldsV1:
+      f:metadata:
         f:annotations:
           .: {}
           f:meta.helm.sh/release-name: {}

Conclusion

|---------------|-------|----------|-------------|-------------|
|   command     | helm  |manual k8s|only affected|field changes|
|               |changes|  changes |  resources  |   hidden    |
|---------------|-------|----------|-------------|-------------|
| `helm diff`   |  YES  |   NO     |    YES      |    YES      |
|    plugin     |       |          |             |             |
|---------------|-------|----------|-------------|-------------|
| helm upgrade  |  YES  |  YES     |    YES      |    NO       |
|    --dry-run  |       |          |             |             |
| +kubectl diff |       |          |             |             |
| (client side) |       |          |             |             |
|---------------|-------|----------|-------------|-------------|
| helm template |  YES  |  YES     |    YES      |    NO       |
|    --dry-run  |       |          |             |             |
| +kubectl diff |       |          |             |             |
| (client side) |       |          |             |             |
|---------------|-------|----------|-------------|-------------|
| helm template |  YES  |  YES     |    NO       |    NO       |
|    --dry-run  |       |          |             |             |
| +kubectl diff |       |          |             |             |
| (server side) |       |          |             |             |
|---------------|-------|----------|-------------|-------------|

So if you use helm diff plugin, be aware that manual k8s changes won't be visible. If you have to check also for manual changes use either manual kubectl diff with --server-side=false on the output of helm template or helm upgrade --dry-run. But in this case you have to ignore all fieldsType: FieldsV1 and related f: fields if you only want to see the real changes.

dracoblue.net