Provisioning Admission Check Controller

The Provisioning AdmissionCheck Controller is an AdmissionCheck Controller designed to integrate Kueue with Kubernetes cluster-autoscaler. Its primary function is to create ProvisioningRequests for the workloads holding Quota Reservation and keeping the AdmissionCheckState in sync.

The controller is part of Kueue. You can enable it by setting the ProvisioningACC feature gate. Check the Installation guide for details on feature gate configuration.

The Provisioning Admission Check Controller is supported on Kubernetes cluster-autoscaler versions 1.29 and later. However, some cloud-providers may not have an implementation for it.

Usage

To use the Provisioning AdmissionCheck, create an AdmissionCheck with kueue.x-k8s.io/provisioning-request as a .spec.controllerName and create a ProvisioningRequest configuration usign a ProvisioningRequestConfig object. See an example below.

ProvisioningRequest configuration

There are two ways to configure ProvisioningRequest sent for your Jobs.

• ProvisioningRequestConfig: This configuration in the AdmissionCheck applies to all the jobs that go through this check. It enables you to set provisioningClassName, managedResources, and parameters
• Job annotation: This configuration enables you to set parameters to a specific job. If both the annotation and the ProvisioningRequestConfig refer to the same parameter, the annotation value takes precedence.

ProvisioningRequestConfig

A ProvisioningRequestConfig looks like the following:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ProvisioningRequestConfig
metadata:
  name: prov-test-config
spec:
  provisioningClassName: queued-provisioning.gke.io
  managedResources:
  - nvidia.com/gpu

Where:

• provisioningClassName - describes the different modes of provisioning the resources. Check autoscaling.x-k8s.io ProvisioningRequestSpec.provisioningClassName for details.
• managedResources - contains the list of resources managed by the autoscaling.

Check the API definition for more details.

Job annotation

Another way to pass ProvisioningRequest’s parameters is by using Job annotations. Every annotation with the provreq.kueue.x-k8s.io/ prefix will be directly passed to created ProvisioningRequest. E.g. provreq.kueue.x-k8s.io/ValidUntilSeconds: "60" will pass ValidUntilSeconds parameter with the value of 60. See more examples below.

Once Kueue creates a ProvisioningRequest for the job you submitted, modifying the value of annotations in the job will have no effect in the ProvisioningRequest.

Example

Setup

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: "default-flavor"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: "cluster-queue"
spec:
  namespaceSelector: {} # match all.
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 9
      - name: "memory"
        nominalQuota: 36Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 9
  admissionChecks:
  - sample-prov
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  namespace: "default"
  name: "user-queue"
spec:
  clusterQueue: "cluster-queue"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: AdmissionCheck
metadata:
  name: sample-prov
spec:
  controllerName: kueue.x-k8s.io/provisioning-request
  parameters:
    apiGroup: kueue.x-k8s.io
    kind: ProvisioningRequestConfig
    name: prov-test-config
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ProvisioningRequestConfig
metadata:
  name: prov-test-config
spec:
  provisioningClassName: queued-provisioning.gke.io
  managedResources:
  - nvidia.com/gpu

Job using a ProvisioningRequest

apiVersion: batch/v1
kind: Job
metadata:
  generateName: sample-job-
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: user-queue
  annotations:
    provreq.kueue.x-k8s.io/maxRunDurationSeconds: "600"
spec:
  parallelism: 3
  completions: 3
  suspend: true
  template:
    spec:
      tolerations:
      - key: "nvidia.com/gpu"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
      - name: dummy-job
        image: gcr.io/k8s-staging-perf-tests/sleep:v0.1.0
        args: ["120s"]
        resources:
          requests:
            cpu: "100m"
            memory: "100Mi"
            nvidia.com/gpu: 1
          limits:
            nvidia.com/gpu: 1
      restartPolicy: Never