Concepts on Kueue

Resource Flavor

Mon, 14 Mar 2022 00:00:00 +0000

Resources in a cluster are typically not homogeneous. Resources could differ in:

Pricing and availability (for example, spot versus on-demand VMs)
Architecture (for example, x86 versus ARM CPUs)
Brands and models (for example, Radeon 7000 versus Nvidia A100 versus T4 GPUs)

A ResourceFlavor is an object that represents these resource variations and allows you to associate them with cluster nodes through labels, taints and tolerations.

Note

If the resources in your cluster are homogeneous, you can use an empty ResourceFlavor instead of adding labels to custom ResourceFlavors.

ResourceFlavor tolerations for automatic scheduling

Requires Kubernetes 1.23 or newer

Cluster Queue

Tue, 14 Mar 2023 00:00:00 +0000

A ClusterQueue is a cluster-scoped object that governs a pool of resources such as pods, CPU, memory, and hardware accelerators. A ClusterQueue defines:

The quotas for the resource flavors that the ClusterQueue manages, with usage limits and order of consumption.
Fair Sharing rules across the multiple ClusterQueues in the cluster.

Only batch administrators should create ClusterQueue objects.

A sample ClusterQueue looks like the following:

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "cluster-queue"
spec:
  namespaceSelector: {} # match all.
  resourceGroups:
  - coveredResources: ["cpu", "memory", "pods"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 9
      - name: "memory"
        nominalQuota: 36Gi
      - name: "pods"
        nominalQuota: 5

This ClusterQueue admits Workloads if and only if:

Cohort

Wed, 16 Apr 2025 00:00:00 +0000

Hello, Cohorts

Cohorts give you the ability to organize your Quotas. ClusterQueues within the same Cohort (or same CohortTree for Hierarchical Cohorts) can share resources with each other. The simplest possible Cohort is the following:

apiVersion: kueue.x-k8s.io/v1beta2
kind: Cohort
metadata:
  name: "hello-cohort"

A ClusterQueue may join this Cohort by referencing it:

Elastic Workloads

Wed, 16 Apr 2025 00:00:00 +0000

Feature state alpha since Kueue v0.13

Elastic Workloads (Workload Slices)

Elastic Workloads extend the core Workload abstraction in Kueue to support dynamic scaling of admitted jobs, without requiring suspension or requeueing. This is achieved through the use of Workload Slices, which track partial allocations of a parent job’s scale-up and scale-down operations.

Local Queue

Mon, 14 Mar 2022 00:00:00 +0000

A LocalQueue is a namespaced object that groups closely related Workloads that belong to a single namespace. A namespace is typically assigned to a tenant (team or user) of the organization. A LocalQueue points to one ClusterQueue from which resources are allocated to run its Workloads.

A LocalQueue definition looks like the following:

apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: team-a 
  name: team-a-queue
spec:
  clusterQueue: cluster-queue

Users submit jobs to a LocalQueue, instead of to a ClusterQueue directly. Tenants can discover which queues they can submit jobs to by listing the local queues in their namespace. The command is similar to the following:

Workload

Mon, 14 Feb 2022 00:00:00 +0000

A workload is an application that will run to completion. It can be composed by one or multiple Pods that, loosely or tightly coupled, as a whole, complete a task. A workload is the unit of admission in Kueue.

The prototypical workload can be represented with a Kubernetes batch/v1.Job. For this reason, we sometimes use the word job to refer to any workload, and Job when we refer specifically to the Kubernetes API.

Topology

Thu, 29 Jan 2026 00:00:00 +0000

A Topology is a cluster-scoped object that defines the hierarchical structure of nodes in a data center. It enables Topology Aware Scheduling by providing a model for representing the hierarchy of organizational units (such as zones, blocks, and racks) using node labels.

The Topology object is referenced from a ResourceFlavor via the .spec.topologyName field to associate the flavor with a specific topology structure.

A Topology definition looks like the following:

apiVersion: kueue.x-k8s.io/v1beta2
kind: Topology
metadata:
  name: "default"
spec:
  levels:
  - nodeLabel: "topology.kubernetes.io/zone"
  - nodeLabel: "cloud.provider.com/topology-block"
  - nodeLabel: "cloud.provider.com/topology-rack"
  - nodeLabel: "kubernetes.io/hostname"

Topology levels

The .spec.levels field defines the hierarchy of topology levels, ordered from the highest (coarsest) level to the lowest (finest) level. Each level is identified by a node label that nodes in your cluster must have.

Admission Fair Sharing

Wed, 28 May 2025 00:00:00 +0000

Feature state beta since Kueue v0.15

Note

AdmissionFairSharing is currently an beta feature and is enabled by default.

You can disable it by editing the AdmissionFairSharing feature gate. Check the Installation guide for details on feature gate configuration.

Admission Fair Sharing helps distribute resources fairly between multiple LocalQueues targeting the same ClusterQueue. It orders workloads based on the historical resource usage of their source LocalQueues, giving preference to those that have consumed less resources over time.

Fair Sharing

Wed, 28 May 2025 00:00:00 +0000

A mechanism for ordering workloads based on the historical resource usage of their source LocalQueues, giving preference to those that have consumed fewer resources over time.

Proof that two workloads won’t preempt each other

Here we present a proof that FairSharing preemption strategies won’t result in a loop where two workloads in different ClusterQueues keep preempting each other.

Topology Aware Scheduling

Thu, 11 Apr 2024 00:00:00 +0000

Feature state beta since Kueue v0.14

Note

TopologyAwareScheduling is currently a beta feature and is enabled by default.

You can disable it by editing the TopologyAwareScheduling feature gate. Refer to the Installation guide for instructions on configuring feature gates.

It is common that AI/ML workloads require a significant amount of pod-to-pod communication. Therefore the network bandwidth between the running Pods translates into the workload execution time, and the cost of running such workloads. The available bandwidth between the Pods depends on the placement of the Nodes, running the Pods, in the data center.

Workload Priority Class

Mon, 02 Oct 2023 00:00:00 +0000

A WorkloadPriorityClass allows you to control the Workload’s priority without affecting the pod’s priority. This feature is useful for these cases:

want to prioritize workloads that remain inactive for a specific duration
want to set a lower priority for development workloads and higher priority for production workloads

A sample WorkloadPriorityClass looks like the following:

apiVersion: kueue.x-k8s.io/v1beta2
kind: WorkloadPriorityClass
metadata:
  name: sample-priority
value: 10000
description: "Sample priority"

WorkloadPriorityClass objects are cluster scoped, so they can be used by a job in any namespace.

Dynamic Resource Allocation

Sun, 22 Mar 2026 00:00:00 +0000

Dynamic Resource Allocation

Dynamic Resource Allocation (DRA) is a Kubernetes API for requesting and managing hardware devices such as GPUs, FPGAs, and network adapters. Kueue can account for DRA devices in quota management through two paths:

ResourceClaimTemplate path: Pods explicitly reference a ResourceClaimTemplate that specifies a device request. Kueue maps each DeviceClass referenced by the claim to a logical resource name using deviceClassMappings in the Kueue Configuration.

Preemption

Tue, 28 May 2024 00:00:00 +0000

In a preemption, the following terms are relevant:

Preemptees: The preempted Workloads.
Target ClusterQueues: The ClusterQueues to which the preemptees belong.
Preemptor: The Workload being accommodated.
Preempting ClusterQueue: The ClusterQueue to which the preemptor belongs.

Reasons for preemption

A Workload can preempt one or more Workloads if it is admitted in a ClusterQueue with preemption enabled and any of the following events happen:

MultiKueue

Mon, 11 Nov 2024 00:00:00 +0000

Feature state beta since Kueue v0.9

Note

MultiKueue is currently a beta feature and is enabled by default.

You can disable it by editing the MultiKueue feature gate. Refer to the Installation guide for instructions on configuring feature gates.

A MultiKueue setup is composed of a manager cluster and at least one worker cluster.

Cluster Roles

Manager Cluster

The manager cluster is responsible for:

Admission

Mon, 01 Jan 0001 00:00:00 +0000

Kueue’s admission process determines whether a Workload can begin execution. Admission may also be subject to the admission gate mechanism which is currently in alpha.

It involves verifying:

logical resource availability via quota reservation
physical resource availability via Topology-Aware Scheduling, when used,
optional AdmissionChecks for additional admission guards.

Kueue implements this through a two-phase admission cycle:

Quota Reservation: When a user submits a Workload, it enters a LocalQueue first. This LocalQueue points to a ClusterQueue which is responsible for managing the available resources. The Kueue checks if the targeted ClusterQueue’s available quota and resource flavors can accomodate requested resources (CPU, memory, GPUs, etc.). If the quota is available, the Kueue reserves resources for this Workload and prevents other Workloads from using the same resources. This phase also includes checking the availability of physical resources when Topology-Aware Scheduling is enabled.

Concepts on Kueue

Resource Flavor

Note

ResourceFlavor tolerations for automatic scheduling

Cluster Queue

Cohort

Hello, Cohorts

Elastic Workloads

Elastic Workloads (Workload Slices)

Local Queue

Workload

Topology

Topology levels

Admission Fair Sharing

Note

Admission Fair Sharing

Fair Sharing

Admission Fair Sharing

Preemption based Fair Sharing

Proof that two workloads won’t preempt each other

Topology Aware Scheduling

Note

Workload Priority Class

Dynamic Resource Allocation

Dynamic Resource Allocation

Preemption

Reasons for preemption

MultiKueue

Note

Cluster Roles

Manager Cluster

Admission