How does Kubernetes handle multiple CustomResource versions? What happens when clients create objects in v1beta1 and you release v1beta2 with breaking changes?

Let’s explore this using a simple Widget CRD:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.example.com
spec:
  group: example.com
  names:
    kind: Widget
    plural: widgets
    singular: widget
  scope: Namespaced
  versions:
    - name: v1beta1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                name:
                  type: boolean
    - name: v1beta2
      served: true
      storage: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                name:
                  type: string

Different API Maturity Levels

Kubernetes defines three API maturity levels:

Read more on Kubernetes API versioning.

Multiple API Versions

CRDs can support multiple versions. In our example, we have v1beta1 and v1beta2:

versions:
  - name: v1beta1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              name:
                type: boolean
  - name: v1beta2
    served: true
    storage: false
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              name:
                type: string

Both versions are served (served: true), so you can create objects in either version:

v1beta1:

apiVersion: example.com/v1beta1
kind: Widget
metadata:
  name: widget-v1beta1
spec:
  name: true

v1beta2:

apiVersion: example.com/v1beta2
kind: Widget
metadata:
  name: widget-v1beta2
spec:
  name: "Piotr"

Applying both:

kubectl apply -f widget-v1beta1.yaml -f widget-v1beta2.yaml
widget.example.com/widget-v1beta1 created
widget.example.com/widget-v1beta2 created

However, when retrieving widget-v1beta1:

kubectl get widget.example.com/widget-v1beta1 -o yaml

You’ll notice:

apiVersion: example.com/v1beta2
kind: Widget
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"example.com/v1beta1","kind":"Widget","metadata":{"annotations":{},"name":"widget-v1beta1","namespace":"default"},"spec":{"name":true}}
  creationTimestamp: "2024-08-13T13:41:14Z"
  generation: 1
  name: widget-v1beta1
  namespace: default
  resourceVersion: "790652"
  uid: 47dcb472-7990-4ad8-8b64-45ec70c8ac2a
spec:
  name: true

The apiVersion returned is v1beta2, despite the object being created with v1beta1.

Prefered version

Kubernetes defaults to the newest available version when multiple versions are supported:

kubectl get --raw /apis/example.com | jq .
{
  "kind": "APIGroup",
  "apiVersion": "v1",
  "name": "example.com",
  "versions": [
    {
      "groupVersion": "example.com/v1beta2",
      "version": "v1beta2"
    },
    {
      "groupVersion": "example.com/v1beta1",
      "version": "v1beta1"
    }
  ],
  "preferredVersion": {
    "groupVersion": "example.com/v1beta2",
    "version": "v1beta2"
  }
}

“The newest available resource version is preferred, but strong consistency is not required.”
Learn more about resource versions.

Additionally, the version with the highest priority is used by kubectl as the default version to access objects:

“The version with the highest priority is used by kubectl as the default version to access objects.”
Learn more about version priority.

This behaviour means that even if an object was created with one version, Kubernetes may return it in the preferred(highest priority) version.

Storage field

The storage field dictates which version is stored in etcd:

versions:
  - name: v1beta1
    served: true
    storage: true

Creating a Widget using v1beta2:

apiVersion: example.com/v1beta2
kind: Widget
metadata:
  name: widget1
spec:
  name: "Piotr"

When checking etcd, the object is stored as v1beta1:

root@master:~/etcd-v3.4.12-linux-amd64# ./etcdctl --endpoints=https://localhost:2379   --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt   --key=/etc/kubernetes/pki/apiserver-etcd-client.key get /registry/example.com/widgets/default/widget1
/registry/example.com/widgets/default/widget1
{"apiVersion":"example.com/v1beta1","kind":"Widget","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"example.com/v1beta2\",\"kind\":\"Widget\",\"metadata\":{\"annotations\":{},\"name\":\"widget1\",\"namespace\":\"default\"},\"spec\":{\"name\":true}}\n"},"creationTimestamp":"2024-07-17T09:07:03Z","generation":2,"name":"widget1","namespace":"default","uid":"96d187bc-958d-4c95-8a59-8a2d33899d90"},"spec":{"name":true}}

Only one version is stored in etcd, and other versions are just different views of the same data.

Conversion Strategies

What happens if you create an object in one version but request it in another? Kubernetes tries to convert the object to match the requested version. This conversion can happen in one of two ways:

• None: Kubernetes simply updates the apiVersion field to the requested version without altering the object’s schema. This approach assumes the schemas are compatible, so only the version field changes, while the data remains the same.
• Webhook: If the schemas differ significantly, you can use a custom conversion webhook. The webhook handles the transformation, ensuring the object fits the requested version’s schema, maintaining data consistency across versions.

Learn more about conversion strategies.

Bonus

This Kubecon 2018 talk is the best one on the topic I could find.