Kueue

Setup Prometheus

Wed, 04 Feb 2026 00:00:00 +0000

This page shows how to set up Prometheus to scrape Kueue metrics. For TLS-secured metrics endpoints, see Configure Prometheus with TLS.

The page is intended for a batch administrator.

Before you begin

Make sure the following conditions are met:

A Kubernetes cluster is running.
The kubectl command-line tool has communication with your cluster.
Kueue is installed.
Prometheus Operator is installed.

1. Setup

Choose the setup method that matches your Kueue installation.

Configure external cert-manager

Fri, 14 Mar 2025 00:00:00 +0000

This page shows how you can a third party certificate authority solution like Cert Manager.

The page is intended for a batch administrator.

Before you begin

Make sure you the following conditions are set:

A Kubernetes cluster is running.
The kubectl command-line tool has communication with your cluster.
Kueue is installed.
Cert Manager is installed

Kueue supports either Kustomize or installation via a Helm chart.

Run A Wrapped Custom Workload

Tue, 14 Jan 2025 00:00:00 +0000

This page shows how to use AppWrappers to make Kueue’s scheduling and resource management capabilities available to Workload types that do not have a dedicated Kueue integration. For Workloads that use PodSpecTemplates in their definition, this can provide a significantly easier approach than building a custom integration to enable the use of Kueue with a custom Workload type.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Troubleshooting Jobs

Thu, 21 Mar 2024 00:00:00 +0000

This doc is about troubleshooting pending Kubernetes Jobs, however, most of the ideas can be extrapolated to other supported job types.

See the Kueue overview to visualize the components that collaborate to run a Job.

In this document, assume that your Job is called my-job and it’s in the my-namespace namespace.

Identifying the Workload for your Job

For each Job, Kueue creates a Workload object to hold the details about the admission of the Job, whether it was admitted or not.

ProvisioningRequest

Mon, 23 Oct 2023 00:00:00 +0000

When AdmissionChecks or TopologyAwareScheduling were not configured, Admissions were mainly based on quota checks - if sufficient quota existed, Kueue admitted the Workload. While quota reservation confirmed logical resource availability, it didn’t guarantee that physical resources existed to schedule all Pods successfully. The ProvisioningRequest AdmissionCheck addresses this in cluster-autoscaler environments.

Kueue’s enhanced admission requires two sequential checks:

Quota Reservation: Kueue validates the resource requests against ClusterQueue’s available quota and resource flavors, reserves the required resources if available and locks the quota to prevent other Workloads from claiming it. This step verifies logical resource availability.

Setup RBAC

Mon, 14 Feb 2022 00:00:00 +0000

This page shows you how to setup role-based access control (RBAC) in your cluster to control the types of users that can view and create Kueue objects.

The page is intended for a batch administrator.

Before you begin

Make sure the following conditions are met:

Common Grafana Queries

Tue, 03 Feb 2026 00:00:00 +0000

This page shows you how to use common PromQL queries to monitor Kueue metrics in Grafana.

The intended audience for this page are batch administrators.

Before you begin

Make sure the following conditions are met:

A Kubernetes cluster is running.
The kubectl command-line tool has communication with your cluster.
Kueue is installed.
kube-prometheus is installed.
Kueue Prometheus metrics are enabled (see Setup Prometheus).

Quota utilization

Note

The queries in this section require metrics.enableClusterQueueResources: true in the Kueue configuration. See Installation for details.

To monitor the percentage of CPU quota being used in a ClusterQueue:

Configure Prometheus with TLS

Fri, 14 Mar 2025 00:00:00 +0000

Note

For basic Prometheus setup without TLS, see Setup Prometheus. This page covers advanced TLS configuration with cert-manager.

This page shows how to configure Kueue to use Prometheus metrics with TLS encryption.

The page is intended for a batch administrator.

Before you begin

Make sure the following conditions are met:

Run Deployment in Multi-Cluster

Mon, 17 Feb 2025 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.
Deployments receive live status updates through the status from the remote Pods created on the worker cluster.

Pod integration requirements

Since Kueue v0.15, you don’t need to explicitly enable "pod" integration to use the "deployment" integration.

Run Plain Pod in Multi-Cluster

Mon, 17 Feb 2025 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.
Follow steps in Run Plain Pods to learn how to enable and configure the pod integration.

Pods created on the manager cluster are automatically gated and receive live status updates from their remote counterparts

Run StatefulSet in Multi-Cluster

Wed, 15 Jan 2025 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

Pod integration requirements

Since Kueue v0.15, you don’t need to explicitly enable "pod" integration to use the "statefulset" integration.

For Kueue v0.14 and earlier, "pod" integration must be explicitly enabled.

Run Kubernetes Job in Multi-Cluster

Tue, 05 Nov 2024 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

The current status of a Job being executed by MultiKueue on a worker cluster is live-updated on the management cluster.

This gives the users and automation tools the ability to track the progress of Job status (.status) without lookup to the worker cluster, making MultiKueue transparent from that perspective.

Troubleshooting Queues

Thu, 21 Mar 2024 00:00:00 +0000

Why no workloads are admitted in the LocalQueue?

The status of the LocalQueue includes details of any configuration problems on the LocalQueue, as part of the Active condition.

Run the following command to see the status of the LocalQueue:

kubectl get localqueue -n my-namespace my-local-queue -o yaml

The status of the LocalQueue will be similar to the following:

Administer Cluster Quotas

Mon, 14 Mar 2022 00:00:00 +0000

This page shows you how to manage your cluster resource quotas and to establish Fair Sharing rules among the tenants.

The intended audience for this page are batch administrators.

Before you begin

Make sure the following conditions are met:

A Kubernetes cluster is running.
The kubectl command-line tool has communication with your cluster.
Kueue is installed.

Single ClusterQueue and single ResourceFlavor setup

In the following steps, you will create a queuing system with a single ClusterQueue and a single ResourceFlavor to govern the quota of your cluster.

Resource Flavor

Mon, 14 Mar 2022 00:00:00 +0000

Resources in a cluster are typically not homogeneous. Resources could differ in:

Pricing and availability (for example, spot versus on-demand VMs)
Architecture (for example, x86 versus ARM CPUs)
Brands and models (for example, Radeon 7000 versus Nvidia A100 versus T4 GPUs)

A ResourceFlavor is an object that represents these resource variations and allows you to associate them with cluster nodes through labels, taints and tolerations.

Note

If the resources in your cluster are homogeneous, you can use an empty ResourceFlavor instead of adding labels to custom ResourceFlavors.

ResourceFlavor tolerations for automatic scheduling

Requires Kubernetes 1.23 or newer

Run A Flux MiniCluster

Mon, 14 Feb 2022 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Flux Operator’s MiniClusters.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Check administer cluster quotas for details on the initial cluster setup.

Monitor actual resource usage

Tue, 24 Feb 2026 00:00:00 +0000

This page shows you how to use Kueue labels assigned to pods to monitor resource usage.

The intended audience for this page are batch administrators.

Before you begin

Make sure the following conditions are met:

A Kubernetes cluster is running.
The kubectl command-line tool has communication with your cluster.
Kueue is installed.
The cluster has at least one local queue and cluster queue. Presented commands assume your local queue is named “user-queue” and cluster queue is named “cluster-queue”, if this is not the case adjust them appropriately.

Note

The queries in this page require AssignQueueLabelsForPods Feature Gate, which is enabled by default. If it is not enabled, see Installation for details how to enable it.

`kubectl top` for command line resource usage debugging

Warning

As mentioned on Metrics Server repository site, Metrics Server and provided command line tool - kubectl top - is not meant as a monitoring solution. The tool is convenient for quick troubleshooting but is not a substitute for full-scale monitoring. See the Production resource monitoring section for a more robust setup.

To use kubectl top you need to install metrics-server for you cluster. Follow the Metrics Server Installation
Schedule a couple of jobs that have some actual cpu usage to your local queue:

for i in {1..3}; do
kubectl create -f - <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  generateName: sample-job-
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: "user-queue"
spec:
  parallelism: 1
  completions: 1
  template:
    spec:
      containers:
      - name: dummy-job
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        command: [ "/bin/sh" ]
        args: [ "-c", "end_time=$(($(date +%s) + 60)); while [ $(date +%s) -lt $end_time ]; do :; done"]
        resources:
          requests:
            cpu: "1"

      restartPolicy: Never
EOF
done

Monitor the usage of cpu and memory in nearly real time for this local queue with kubectl top:

watch -n 15  'kubectl top pod --sum -l kueue.x-k8s.io/local-queue-name=user-queue'

You will see a list of pods currently running on the local queue named “user-queue” with their current cpu and memory measurements and the sum of the usage at the bottom. The list will be automatically refreshed every 15 seconds. The outcome should look like this:

Share Quotas Across Resource Flavors

Wed, 28 Jan 2026 00:00:00 +0000

This page demonstrates how to use resource transformations to enforce a unified quota that applies across multiple resource flavors.

A common use case is when you have multiple hardware types (for example, different CPU architectures or GPU models) but want to limit the total usage per team regardless of which flavor they consume. By defining a virtual “credits” resource, you can assign a cost to each flavor and enforce a combined quota.

Run TrainJobs in Multi-Cluster

Thu, 06 Nov 2025 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

For the ease of setup and use we recommend using at least Kueue v0.11.0 and for Kubeflow Trainer at least v2.0.0.

See Trainer Installation for installation and configuration details of Kubeflow Trainer v2.

Run AppWrappers in Multi-Cluster

Mon, 23 Jun 2025 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

For the ease of setup and use we recommend using at least Kueue v0.11.0 and at least AppWrapper Operator v1.1.1.

See AppWrapper Quick-Start Guide for installation and configuration details of the AppWrapper Operator.

Run An Argo Workflow

Thu, 23 Jan 2025 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Argo Workflows.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Currently Kueue doesn’t support Argo Workflows Workflow resources directly, but you can take advantage of the ability for Kueue to manage plain pods to integrate them.

Before you begin

Learn how to install Kueue with a custom manager configuration.

Run Jobsets in Multi-Cluster

Wed, 25 Sep 2024 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

For the ease of setup and use we recommend using at least Kueue v0.8.1 and JobSet v0.6.0 (see JobSet Installation for more details).

Run Jobsets example

Once the setup is complete you can test it by running the jobset-sample.yaml Job.

Troubleshooting Provisioning Request in Kueue

Mon, 20 May 2024 00:00:00 +0000

This document helps you troubleshoot ProvisioningRequests, an API defined by ClusterAutoscaler.

Kueue creates ProvisioningRequests via the Provisioning Admission Check Controller, and treats them like an Admission Check. In order for Kueue to admit a Workload, the ProvisioningRequest created for it needs to succeed.

Before you begin

Before you begin troubleshooting, make sure your cluster meets the following requirements:

Enabling pprof endpoints

Fri, 21 Jul 2023 00:00:00 +0000

This page shows you how to enable pprof endpoints for Kueue controller manager.

The intended audience for this page are batch administrators.

Before you begin

Make sure the following conditions are met:

A Kubernetes cluster is running.
The kubectl command-line tool has communication with your cluster.
Kueue is installed.

Enabling pprof endpoints

Feature state stable since Kueue v0.5

To enable pprof endpoints, you need to set a pprofBindAddress is set in the manager’s configuration.

Cluster Queue

Tue, 14 Mar 2023 00:00:00 +0000

A ClusterQueue is a cluster-scoped object that governs a pool of resources such as pods, CPU, memory, and hardware accelerators. A ClusterQueue defines:

The quotas for the resource flavors that the ClusterQueue manages, with usage limits and order of consumption.
Fair Sharing rules across the multiple ClusterQueues in the cluster.

Only batch administrators should create ClusterQueue objects.

A sample ClusterQueue looks like the following:

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "cluster-queue"
spec:
  namespaceSelector: {} # match all.
  resourceGroups:
  - coveredResources: ["cpu", "memory", "pods"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 9
      - name: "memory"
        nominalQuota: 36Gi
      - name: "pods"
        nominalQuota: 5

This ClusterQueue admits Workloads if and only if:

Configure Custom Metric Labels

Tue, 10 Mar 2026 00:00:00 +0000

This page shows you how to configure custom metric labels that promote Kubernetes labels or annotations on ClusterQueues, LocalQueues, and Cohorts into additional Prometheus metric label dimensions.

The intended audience for this page are batch administrators.

Before you begin

Make sure the following conditions are met:

Cohort

Wed, 16 Apr 2025 00:00:00 +0000

Hello, Cohorts

Cohorts give you the ability to organize your Quotas. ClusterQueues within the same Cohort (or same CohortTree for Hierarchical Cohorts) can share resources with each other. The simplest possible Cohort is the following:

apiVersion: kueue.x-k8s.io/v1beta2
kind: Cohort
metadata:
  name: "hello-cohort"

A ClusterQueue may join this Cohort by referencing it:

Elastic Workloads

Wed, 16 Apr 2025 00:00:00 +0000

Feature state alpha since Kueue v0.13

Elastic Workloads (Workload Slices)

Elastic Workloads extend the core Workload abstraction in Kueue to support dynamic scaling of admitted jobs, without requiring suspension or requeueing. This is achieved through the use of Workload Slices, which track partial allocations of a parent job’s scale-up and scale-down operations.

Run KubeRay Jobs in Multi-Cluster

Wed, 19 Mar 2025 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

For the ease of setup and use we recommend using at least Kueue v0.11.0 and for KubeRay Operator at least v1.3.1.

See KubeRay Operator Installation for installation and configuration details of KubeRay Operator.

Run Kubeflow Jobs (v1) in Multi-Cluster

Wed, 25 Sep 2024 00:00:00 +0000

Note

This page covers legacy Kubeflow Trainer v1 jobs (PyTorchJob, TFJob, XGBoostJob, etc.).

For Kubeflow Trainer v2 TrainJob, see Run TrainJobs in Multi-Cluster.

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

For the ease of setup and use we recommend using at least Kueue v0.11.0 and for Kubeflow Trainer at least v1.9.0.

Troubleshooting Pods

Thu, 21 Mar 2024 00:00:00 +0000

This doc is about troubleshooting plain Pods when directly managed by Kueue, in other words, Pods that are not managed by kubernetes Jobs or supported CRDs.

Note

This doc focuses on the behavior of Kueue when managing Pods that is different from other job integrations. You can read Troubleshooting Jobs for more general troubleshooting steps.

Is my Pod managed directly by Kueue?

Kueue adds the label kueue.x-k8s.io/managed with value true to Pods that it manages. If the label is not present on a Pod, it means that Kueue is not going to admit or account for the resource usage of this Pod directly.

Run job with WorkloadPriority

Mon, 02 Oct 2023 00:00:00 +0000

Usually, in Kueue, workload’s priority is calculated using for pod’s priority for queuing and preemption.
By using a WorkloadPriorityClass, you can independently manage the priority of workloads for queuing and preemption, separate from pod’s priority.

This page contains instructions on how to run a job with workload priority.

Before you begin

Make sure the following conditions are met:

Local Queue

Mon, 14 Mar 2022 00:00:00 +0000

A LocalQueue is a namespaced object that groups closely related Workloads that belong to a single namespace. A namespace is typically assigned to a tenant (team or user) of the organization. A LocalQueue points to one ClusterQueue from which resources are allocated to run its Workloads.

A LocalQueue definition looks like the following:

apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: team-a 
  name: team-a-queue
spec:
  clusterQueue: cluster-queue

Users submit jobs to a LocalQueue, instead of to a ClusterQueue directly. Tenants can discover which queues they can submit jobs to by listing the local queues in their namespace. The command is similar to the following:

Run A Kubernetes Job

Mon, 14 Feb 2022 00:00:00 +0000

This page shows you how to run a Job in a Kubernetes cluster with Kueue enabled.

The intended audience for this page are batch users.

Before you begin

Make sure the following conditions are met:

A Kubernetes cluster is running.
The kubectl command-line tool has communication with your cluster.
Kueue is installed.
The cluster has quotas configured.

The following picture shows all the concepts you will interact with in this tutorial:

Setup Dev Monitoring

Wed, 04 Feb 2026 00:00:00 +0000

This page shows how to set up Prometheus for development, debugging, and testing Kueue metrics.

The page is intended for a platform developer.

Quick start with e2e infrastructure

The fastest way to get a Prometheus-enabled cluster is to use the e2e test infrastructure:

E2E_MODE=dev GINKGO_ARGS="--label-filter=feature:prometheus" make kind-image-build test-e2e

This provisions a Kind cluster with the Prometheus Operator, a Prometheus instance, Kueue, and all ServiceMonitors pre-configured.

Setup MultiKueue Development Environment

Tue, 13 Jan 2026 00:00:00 +0000

This tutorial explains how you can configure a manager cluster and worker clusters to run jobs with Topology-Aware Scheduling (TAS) in a MultiKueue environment. We also outline the automated steps using Kind for local testing.

Check the concepts section for a MultiKueue overview and Topology-Aware Scheduling overview.

Setup MultiKueue with E2E Test Cluster

The e2e test development mode can be used to maintain a MultiKueue cluster setup and run end-to-end tests against it without recreating and tearing it down each time.

Run External Framework Jobs in Multi-Cluster

Fri, 19 Sep 2025 00:00:00 +0000

Feature state alpha since Kueue v0.14

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.
Enable the MultiKueueAdaptersForCustomJobs feature gate. This feature is in Alpha and is disabled by default.

To enable the feature, add the --feature-gates=MultiKueueAdaptersForCustomJobs=true flag to the Kueue controller manager’s arguments. For example, you can apply it to your deployment with the following command:

Run MPIJobs in Multi-Cluster

Fri, 25 Oct 2024 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

For the proper setup and use it is required using at least Kueue v0.9.0 and for MPI Operator at least v0.6.0.

Installation on the Clusters

Note

While both MPI Operator and Trainer must be running on the same cluster, there are special steps that has to be applied to Trainer deployment. See Working alongside MPI Operator for more details.

See MPI Operator Installation for installation and configuration details of MPI Operator.

Troubleshooting delete ClusterQueue

Wed, 28 Aug 2024 00:00:00 +0000

Deleting a ClusterQueue might require additional steps when it has active Workloads.

This guide provides a clear approach to safely deleting a ClusterQueue, explaining the process and offering alternative methods to handle specific scenarios.

For this example, assume your ClusterQueue is named my-cq.

Understanding `kubectl delete clusterqueue`

When you run the following command:

Run A CronJob

Tue, 12 Dec 2023 00:00:00 +0000

This page shows you how to run a CronJob in a Kubernetes cluster with Kueue enabled.

The intended audience for this page are batch users.

Before you begin

Make sure the following conditions are met:

A Kubernetes cluster is running.
The kubectl command-line tool has communication with your cluster.
Kueue is installed.
The cluster has quotas configured.

0. Identify the queues available in your namespace

Run the following command to list the LocalQueues available in your namespace.

Setup All-or-nothing with ready Pods

Mon, 14 Mar 2022 00:00:00 +0000

Some jobs need all pods to be running at the same time to operate; for example, synchronized distributed training or MPI-based jobs which require pod-to-pod communication. On a default Kueue configuration, a pair of such jobs may deadlock if the physical availability of resources do not match the configured quotas in Kueue. The same pair of jobs could run to completion if their pods were scheduled sequentially.

To address this requirement, in version 0.3.0 we introduced an opt-in mechanism configured via the flag waitForPodsReady that provides a simple implementation of the all-or-nothing scheduling. When enabled, the workload is monitored by Kueue until all of its Pods are ready (meaning scheduled, running, and passing the optional readiness probe). If not all pods of the workload are ready within the configured timeout, then the workload is evicted and requeued.

Workload

Mon, 14 Feb 2022 00:00:00 +0000

A workload is an application that will run to completion. It can be composed by one or multiple Pods that, loosely or tightly coupled, as a whole, complete a task. A workload is the unit of admission in Kueue.

The prototypical workload can be represented with a Kubernetes batch/v1.Job. For this reason, we sometimes use the word job to refer to any workload, and Job when we refer specifically to the Kubernetes API.

Topology

Thu, 29 Jan 2026 00:00:00 +0000

A Topology is a cluster-scoped object that defines the hierarchical structure of nodes in a data center. It enables Topology Aware Scheduling by providing a model for representing the hierarchy of organizational units (such as zones, blocks, and racks) using node labels.

The Topology object is referenced from a ResourceFlavor via the .spec.topologyName field to associate the flavor with a specific topology structure.

A Topology definition looks like the following:

apiVersion: kueue.x-k8s.io/v1beta2
kind: Topology
metadata:
  name: "default"
spec:
  levels:
  - nodeLabel: "topology.kubernetes.io/zone"
  - nodeLabel: "cloud.provider.com/topology-block"
  - nodeLabel: "cloud.provider.com/topology-rack"
  - nodeLabel: "kubernetes.io/hostname"

Topology levels

The .spec.levels field defines the hierarchy of topology levels, ordered from the highest (coarsest) level to the lowest (finest) level. Each level is identified by a node label that nodes in your cluster must have.

Using HAMi

Mon, 15 Dec 2025 00:00:00 +0000

This page demonstrates how to use Kueue for running workloads with HAMi (Heterogeneous AI Computing Virtualization Middleware) for vGPU resource management.

When working with vGPU resources through HAMi, Pods request per-vGPU resources (nvidia.com/gpumem, nvidia.com/gpucores) along with the number of vGPUs (nvidia.com/gpu). For quota management, you need to track the total resource consumption across all vGPU instances.

Below we demonstrate how to support this with the resource transformation for quota management in Kueue.

Admission Fair Sharing

Wed, 28 May 2025 00:00:00 +0000

Feature state beta since Kueue v0.15

Note

AdmissionFairSharing is currently an beta feature and is enabled by default.

You can disable it by editing the AdmissionFairSharing feature gate. Check the Installation guide for details on feature gate configuration.

Admission Fair Sharing helps distribute resources fairly between multiple LocalQueues targeting the same ClusterQueue. It orders workloads based on the historical resource usage of their source LocalQueues, giving preference to those that have consumed less resources over time.

Fair Sharing

Wed, 28 May 2025 00:00:00 +0000

A mechanism for ordering workloads based on the historical resource usage of their source LocalQueues, giving preference to those that have consumed fewer resources over time.

Proof that two workloads won’t preempt each other

Here we present a proof that FairSharing preemption strategies won’t result in a loop where two workloads in different ClusterQueues keep preempting each other.

Run a JAXJob

Wed, 23 Apr 2025 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Trainer JAXJobs.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Check administer cluster quotas for details on the initial cluster setup.

Run LeaderWorkerSet

Mon, 17 Feb 2025 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running LeaderWorkerSet.

We demonstrate how to support scheduling LeaderWorkerSets where a group of Pods constitutes a unit of admission represented by a Workload. This allows to scale-up and down LeaderWorkerSets group by group.

This integration is based on the Plain Pod Group integration.

This guide is for serving users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Run An AppWrapper

Wed, 08 Jan 2025 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running AppWrappers.

AppWrappers provide a flexible and workload-agnostic mechanism for enabling Kueue to manage a group of Kubernetes resources as a single logical unit without requiring any Kueue-specific support by the controllers of those resources.

AppWrappers are designed to harden workloads by providing an additional level of automatic fault detection and recovery. The AppWrapper controller monitors the health of the workload and if corrective actions are not taken by the primary resource controllers within specified deadlines, the AppWrapper controller will orchestrate workload-level retries and resource deletion to ensure that either the workload returns to a healthy state or is cleanly removed from the cluster and its quota freed for use by other workloads.

Run A TrainJob

Tue, 05 Nov 2024 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Kubeflow Trainer TrainJobs.

Kubeflow Trainer v2 introduces the TrainJob API that works seamlessly with Kueue for batch scheduling and resource management. TrainJobs can be configured to use either:

ClusterTrainingRuntime: Cluster-scoped training runtimes that can be used across all namespaces
TrainingRuntime: Namespace-scoped training runtimes that are only available within a specific namespace

Kueue manages TrainJobs by scheduling their underlying jobs according to available quota and priority.

Run Deployment

Thu, 25 Jul 2024 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Deployments. Although Kueue does not yet support managing a Deployment as a single Workload, it’s still possible to leverage Kueue’s scheduling and resource management capabilities for the individual Pods of the Deployment.

We demonstrate how to support scheduling Deployments in Kueue based on the Plain Pod integration, where every Pod from a Deployment is represented as a single independent Plain Pod. This approach allows independent resource management for the Pods, and thus scale-out and scale-in of the Deployment.

Run StatefulSet

Thu, 25 Jul 2024 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running StatefulSets.

We demonstrate how to support scheduling StatefulSets in Kueue based on the Plain Pod Group integration, where StatefulSet is represented as a single independent workload.

This guide is for serving users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

The statefulset integration is enabled by default.

Topology Aware Scheduling

Thu, 11 Apr 2024 00:00:00 +0000

Feature state beta since Kueue v0.14

Note

TopologyAwareScheduling is currently a beta feature and is enabled by default.

You can disable it by editing the TopologyAwareScheduling feature gate. Refer to the Installation guide for instructions on configuring feature gates.

It is common that AI/ML workloads require a significant amount of pod-to-pod communication. Therefore the network bandwidth between the running Pods translates into the workload execution time, and the cost of running such workloads. The available bandwidth between the Pods depends on the placement of the Nodes, running the Pods, in the data center.

Workload Priority Class

Mon, 02 Oct 2023 00:00:00 +0000

A WorkloadPriorityClass allows you to control the Workload’s priority without affecting the pod’s priority. This feature is useful for these cases:

want to prioritize workloads that remain inactive for a specific duration
want to set a lower priority for development workloads and higher priority for production workloads

A sample WorkloadPriorityClass looks like the following:

apiVersion: kueue.x-k8s.io/v1beta2
kind: WorkloadPriorityClass
metadata:
  name: sample-priority
value: 10000
description: "Sample priority"

WorkloadPriorityClass objects are cluster scoped, so they can be used by a job in any namespace.

Run Plain Pods

Wed, 27 Sep 2023 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running plain Pods. Kueue supports management of both individual Pods, or Pod groups.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

The pod integration is enabled by default.

Run a PaddleJob

Fri, 22 Sep 2023 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Trainer PaddleJobs.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Check administer cluster quotas for details on the initial cluster setup.

Run a XGBoostJob

Thu, 14 Sep 2023 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Trainer XGBoostJobs.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Check administer cluster quotas for details on the initial cluster setup.

Run a TFJob

Wed, 23 Aug 2023 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Trainer TFJobs.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Check administer cluster quotas for details on the initial cluster setup.

Run a PyTorchJob

Wed, 09 Aug 2023 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Trainer PyTorchJobs.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Check administer cluster quotas for details on the initial cluster setup.

Run a MPIJob

Tue, 16 May 2023 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running MPI Operator MPIJobs.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Check administer cluster quotas for details on the initial cluster setup.

Run Workloads With DRA Devices

Sun, 22 Mar 2026 00:00:00 +0000

This page shows you how to run workloads that request hardware devices (such as GPUs) managed by Dynamic Resource Allocation (DRA) in a Kubernetes cluster with Kueue enabled. The examples use a batch Job, but the same approach works with any workload type that Kueue supports.

The intended audience for this page are batch users.

For conceptual details about how Kueue handles DRA resources, see Dynamic Resource Allocation concepts.

Before you begin

Make sure the following conditions are met:

Dynamic Resource Allocation

Sun, 22 Mar 2026 00:00:00 +0000

Dynamic Resource Allocation

Dynamic Resource Allocation (DRA) is a Kubernetes API for requesting and managing hardware devices such as GPUs, FPGAs, and network adapters. Kueue can account for DRA devices in quota management through two paths:

ResourceClaimTemplate path: Pods explicitly reference a ResourceClaimTemplate that specifies a device request. Kueue maps each DeviceClass referenced by the claim to a logical resource name using deviceClassMappings in the Kueue Configuration.

Set Up Dynamic Resource Allocation

Sun, 22 Mar 2026 00:00:00 +0000

This page shows you how to configure Kueue to account for DRA devices in quota management.

The intended audience for this page are batch administrators.

For conceptual details, see Dynamic Resource Allocation concepts. For instructions on submitting workloads with DRA devices, see Run Workloads With DRA Devices.

Before you begin

Make sure the following conditions are met:

Run a SparkApplication

Mon, 17 Nov 2025 00:00:00 +0000

Feature state alpha since Kueue v0.17

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Spark Operator SparkApplication.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Note

SparkApplicationIntegration is currently an alpha feature and is disabled by default.

To enable it, the SparkApplicationIntegration feature gate needs to be activated, and sparkoperator.k8s.io/sparkapplication must be added as an allowed workload.

Run A Tekton Pipeline

Sat, 01 Feb 2025 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Tekton pipelines.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

We demonstrate how to support scheduling Tekton Pipelines Tasks in Kueue based on the Plain Pod integration, where every Pod from a Pipeline is represented as a single independent Plain Pod.

Before you begin

Learn how to install Kueue with a custom manager configuration.

Preemption

Tue, 28 May 2024 00:00:00 +0000

In a preemption, the following terms are relevant:

Preemptees: The preempted Workloads.
Target ClusterQueues: The ClusterQueues to which the preemptees belong.
Preemptor: The Workload being accommodated.
Preempting ClusterQueue: The ClusterQueue to which the preemptor belongs.

Reasons for preemption

A Workload can preempt one or more Workloads if it is admitted in a ClusterQueue with preemption enabled and any of the following events happen:

Run Jobs Using Python

Wed, 05 Jul 2023 00:00:00 +0000

This guide is for batch users that have a basic understanding of interacting with Kubernetes from Python. For more information, see Kueue’s overview.

Before you begin

Check administer cluster quotas for details on the initial cluster setup. You’ll also need kubernetes python installed. We recommend a virtual environment.

Run A JobSet

Fri, 16 Jun 2023 00:00:00 +0000

This document explains how you can use Kueue’s scheduling and resource management functionality when running JobSet Operator JobSet.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Check Administer cluster quotas for details on the initial Kueue setup.

Run A Sandbox

Sun, 19 Apr 2026 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Sandboxes.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

The Sandbox operator provides isolated environments for running AI agent workloads. Kueue manages the Pods created by the Sandbox controller through the Plain Pod integration, where each Sandbox Pod is represented as a single independent Plain Pod.

Before you begin

Learn how to install Kueue with a custom manager configuration.

External Frameworks

Fri, 17 Jan 2025 00:00:00 +0000

See external frameworks for examples of using existing integrations to integrate external frameworks.

MultiKueue

Mon, 11 Nov 2024 00:00:00 +0000

Feature state beta since Kueue v0.9

Note

MultiKueue is currently a beta feature and is enabled by default.

You can disable it by editing the MultiKueue feature gate. Refer to the Installation guide for instructions on configuring feature gates.

A MultiKueue setup is composed of a manager cluster and at least one worker cluster.

Cluster Roles

Manager Cluster

The manager cluster is responsible for:

Integrate a custom Job with Kueue

Tue, 25 Jul 2023 00:00:00 +0000

Kueue has built-in integrations for several Job types, including Kubernetes batch Job, MPIJob, RayJob and JobSet.

There are three options for using Kueue to manage Job-like CRDs that lack built-in integrations.

Leverage the built-in AppWrapper integration by wrapping instances of the custom Job in an AppWrapper. See Running a Wrapped Custom Workload for details.
Build a new integration as part of the Kueue repository.
Build a new integration as an external controller.

This guide is for platform developers and describes how to build a new integration. Integrations should be built using the APIs provided by Kueue’s jobframework package. This will both simplify development and ensure that your controller will be properly structured to become a core built-in integration if your Job type is widely used by the community.

Develop an AdmissionCheck Controller

Tue, 03 Sep 2024 00:00:00 +0000

An Admission Check Controller, referred to as ACC in the rest of the doc, is a component that manages the AdmissionChecks associated with it (given by the corresponding value of spec.controllerName) and the workloads queued against ClusterQueues configured to use those AdmissionChecks.

Read Admission Check to learn more about the mechanism from a user perspective.

Subcomponents

The ACC can be built-in into Kueue or run in a different kubernetes controller manager and should implement reconcilers for AdmissionChecks and Workloads.

Setup a MultiKueue environment

Mon, 26 Feb 2024 00:00:00 +0000

This tutorial explains how you can configure a management cluster and one worker cluster to run JobSets and batch/Jobs in a MultiKueue environment.

Check the concepts section for a MultiKueue overview.

Let’s assume that your manager cluster is named manager-cluster and your worker cluster is named worker1-cluster. To follow this tutorial, ensure that the credentials for all these clusters are present in the kubeconfig in your local machine. Check the kubectl documentation to learn more about how to Configure Access to Multiple Clusters.

Run LeaderWorkerSet in Multi-Cluster

Fri, 16 Jan 2026 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

How it works

LeaderWorkerSet creates one Workload per replica, which requires special handling in MultiKueue.

Run A RayService

Mon, 30 Jun 2025 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running RayService.

Before v0.17.0, Kueue manages the RayService through the RayCluster created for it. Since v0.17.0, Kueue manages the RayService as top level job which is similar like how Kueue manages the RayJob.

This guide is for serving users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Make sure you are using Kueue v0.6.0 version or newer and KubeRay v1.3.0 or newer.

Setup default LocalQueue

Thu, 12 Dec 2024 00:00:00 +0000

Feature state stable since Kueue v0.17

This page describes how to setup default LocalQueue to ensure that all Workloads submitted to a specific namespace are managed by Kueue, even if the kueue.x-k8s.io/queue-name label is not specified explicitly.

Setup default LocalQueue

LocalQueueDefaulting is a feature that allows the use of a LocalQueue with name default as the default LocalQueue for workloads in the same namespace that do not have the kueue.x-k8s.io/queue-name label.

Pending Workloads on-demand

Mon, 30 Sep 2024 00:00:00 +0000

This page shows you how to monitor pending workloads with VisibilityOnDemand feature.

The intended audience for this page are batch administrators for Cluster Queue visibility section, and batch users for Local Queue Visibility section.

From version v0.6.0, Kueue provides the ability for a batch administrators to monitor the pipeline of pending jobs, and help users to estimate when their jobs will start.

Before you begin

Make sure the following conditions are met:

Run A RayCluster

Wed, 07 Aug 2024 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running RayCluster.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Make sure you are using Kueue v0.6.0 version or newer and KubeRay v1.1.0 or newer.

Run A RayJob

Wed, 07 Aug 2024 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running KubeRay’s RayJob.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Make sure you are using Kueue v0.6.0 version or newer and KubeRay v1.1.0 or newer.

Setup a Job Admission Policy

Fri, 28 Jun 2024 00:00:00 +0000

This page shows how you can set up for Kueue a Job Admission Policy using the Kubernetes Validating Admission Policy, based on the Common Expression Language (CEL).

Before you begin

Ensure the following conditions are met:

A Kubernetes cluster is running.
The ValidatingAdmissionPolicy feature gate is enabled. In Kubernetes 1.30 or newer, the feature gate is enabled by default.
The kubectl command-line tool can communicate with your cluster.
Kueue is installed.

Enforce Queue Name - Example

The example below shows you how to set up a Job Admission Policy to reject early all Job or JobSets without the queue-name if sent to a namespace labeled as a kueue-managed namespace.

Setup manageJobsWithoutQueueName

Fri, 28 Jun 2024 00:00:00 +0000

This page describes how to configure Kueue to ensure that all Workloads submitted in namespaces intended for use by batch users will be managed by Kueue even if they lack a kueue.x-k8s.io/queue-name label.

Before you begin

Learn how to install Kueue with a custom manager configuration.

Feature Gates (removed)

Mon, 01 Jan 0001 00:00:00 +0000

This page contains list of feature gates that have been removed. The information on this page is for reference. A removed feature gate is different from a GA’ed or deprecated one in that a removed one is no longer recognized as a valid feature gate. However, a GA’ed or a deprecated feature gate is still recognized by the corresponding Kueue components although they are unable to cause any behavior differences in a cluster.

Setup garbage-collection of workload

Fri, 16 May 2025 00:00:00 +0000

This guide shows you how to enable and configure optional object retention policies in Kueue to automatically delete finished or deactivated Workloads after a specified time. By default, Kueue leaves all Workload objects in cluster indefinitely; with object retention you can free up etcd storage and reduce Kueue’s memory footprint.

Prerequisites

A running Kueue installation at v0.12 or newer.

Set up a retention policy

Follow the instructions described here to install a release version by extending the configuration with the following fields:

Setup failure recovery

Thu, 04 Dec 2025 00:00:00 +0000

This describes how to configure the failure recovery mechanism that automatically transitions pods into the Failed phase when they are assigned to unreachable nodes and stuck terminating. This is especially relevant for Job-based workloads that specify podReplacementPolicy: Failed, preventing the Job from making progress if such scenario occurrs.

Without the failure recovery controller, a node failure (e.g. network partition or a malfunction of the kubelet) results in the pods running on that node becoming stuck. The failure recovery mechanism unblocks workloads relying on pods being in a terminated state by forcefully moving them to the Failed phase after a grace period of 60 seconds since they became stuck.

Enable KueueViz

Fri, 18 Jul 2025 00:00:00 +0000

Note

For a streamlined installation and simplified upgrades, we recommend deploying Kueue using Helm.
You can customize the deployment with a local values.yaml file to fit your environment.
See the Helm chart installation guide for full instructions.

KueueViz is a web-based visualization tool that provides real-time monitoring of Kueue workloads, queues, and resource allocation. It offers an intuitive dashboard for observing job queue status, resource utilization, and workload progression.

This page shows how to install and configure KueueViz in your cluster.

Opt-in Namespace Management

Tue, 25 Nov 2025 00:00:00 +0000

This page describes how to configure Kueue to enforce opt-in namespace management, ensuring that only workloads in explicitly labeled namespaces are reconciled by Kueue.

Before you begin

Learn how to install Kueue with a custom manager configuration.
Understand how to configure manageJobsWithoutQueueName.
Learn how to change the feature gates configuration.

Why Use Opt-in Namespace Management?

Opt-in namespace management allows cluster administrators to enforce strict quota controls by ensuring that Kueue only manages workloads in explicitly designated namespaces. This provides several key benefits:

Pending Workloads in Grafana

Thu, 12 Jun 2025 00:00:00 +0000

This guide explains how to monitor pending workloads in Grafana using the VisibilityOnDemand feature.

The intended audience for this page are batch administrators for ClusterQueue visibility, and batch users for LocalQueue visibility.

Before you begin

Make sure the following conditions are met:

A Kubernetes cluster is running.
Kueue v0.15.0 or later is installed.
Prometheus Operator is installed. See Setup Prometheus for instructions.
The VisibilityOnDemand feature is enabled.

Setting Up Grafana for Pending Workloads

Step 1: Configure Cluster Permissions

To enable visibility, create a ClusterRole and ClusterRoleBinding for either ClusterQueue or LocalQueue:

Installation

Thu, 09 May 2024 00:00:00 +0000

Installing via Krew

kubectl krew install kueue

Installing From Release Binaries

1. Download the latest release:

On Linux:

Running and debugging tests

Mon, 01 Jan 0001 00:00:00 +0000

Running presubmission verification tests

make verify

Running unit tests

To run all unit tests:

make test

To run unit tests for webhooks:

Adopters

Mon, 01 Jan 0001 00:00:00 +0000

Below is a list of organizations that use Kueue in production along with the integrations they use. If you are using Kueue, feel free to open a pull request to add your organization to the list.

Adopters

Organization	Type	Description	Integrations	Contact
CyberAgent, Inc.	End User	On-premise ML Platform	batch/job kubeflow.org/mpijob	@tenzen-y
DaoCloud, Inc.	End User	Part of the AI Platform for managing all kinds of Jobs.	batch/job RayJob …	@kerthcet
WattIQ, Inc.	End User	SaaS/IoT product	batch/job RayJob	@madsenwattiq
Horizon, Inc.	End User	AI training platform	batch/job …	@GhangZh
FAR AI	End User	AI alignment research nonprofit	batch/job	@rhaps0dy
Shopee, Inc.	End User	Training/batch inference/data processes in AI platform test env	Customized job RayJob …	@denkensk
Mondoo, Inc.	End User	Helps power Mondoo’s hosted security scanner	batch/job	@jaym
Google Cloud	Provider	Part of kit for training ML workloads on TPUs	JobSet	@mrozacki
Onna Technologies, Inc	End User	Unstructured Data Management Platform	batch/job	@gitcarbs
IBM Research	End User	Part of MLBatch stack for running AI/ML workloads	AppWrapper PyTorchJob RayJob	dgrove-oss
Innovatrics	End User	On-premise ML Platform	batch/job pod	@mmolisch
Red Hat, Inc.	End User	Cloud/On-premise ML Platform	Ray Cluster RayJob Pytorch Job TensorFlow Job …	@varshaprasad96
Octue	End User	Scientific data service framework	batch/job	@cortadocodes
LY Corporation	End User	On-premise AI Platform	batch/job TensorFlow Job Ray Cluster pod …	@everpeace
Preferred Networks, Inc.	End User	Cloud/On-premise ML Platform	batch/job pod …	@wafrelka
Wayve	End User	Part of the AI Platform powering Wayve AI Driver	RayJob pod	@mukund-wayve

Labels and Annotations

Thu, 19 Mar 2026 00:00:00 +0000

This page serves as a reference for all labels and annotations in Kueue.

kueue.x-k8s.io/admission-gated-by

Type: Annotation

Example: kueue.x-k8s.io/admission-gated-by: "example.com/mygate,example.com/mygate2"

Used on: Kueue-managed Jobs.

This annotation requires the AdmissionGatedBy feature that is disabled by default.

When it contains a value, the annotation prevents the Job from entering the Quota Reservation phase of Kueue’s Admission Flow. The annotation value is a comma-separated list of 1 or more gate names and can only be added during Job creation. After creation, the annotation may only be deleted or modified to remove 1 or more gates.

Prometheus Metrics

Mon, 14 Feb 2022 00:00:00 +0000

Note

To enable metrics scraping, see Setup Prometheus.

Kueue exposes prometheus metrics to monitor the health of the system and the status of ClusterQueues and LocalQueues.

Kueue health

Use the following metrics to monitor the health of the kueue controllers:

Admission

Mon, 01 Jan 0001 00:00:00 +0000

Kueue’s admission process determines whether a Workload can begin execution. Admission may also be subject to the admission gate mechanism which is currently in alpha.

It involves verifying:

logical resource availability via quota reservation
physical resource availability via Topology-Aware Scheduling, when used,
optional AdmissionChecks for additional admission guards.

Kueue implements this through a two-phase admission cycle:

Quota Reservation: When a user submits a Workload, it enters a LocalQueue first. This LocalQueue points to a ClusterQueue which is responsible for managing the available resources. The Kueue checks if the targeted ClusterQueue’s available quota and resource flavors can accomodate requested resources (CPU, memory, GPUs, etc.). If the quota is available, the Kueue reserves resources for this Workload and prevents other Workloads from using the same resources. This phase also includes checking the availability of physical resources when Topology-Aware Scheduling is enabled.

Kueue Configuration v1beta1 API

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

Configuration

`Configuration`

Configuration is the Schema for the kueueconfigurations API

Field Description

apiVersion
string config.kueue.x-k8s.io/v1beta1

kind
string Configuration

namespace [Required]
string

Namespace is the namespace in which kueue is deployed. It is used as part of DNSName of the webhook Service. If not set, the value is set from the file /var/run/secrets/kubernetes.io/serviceaccount/namespace If the file doesn't exist, default value is kueue-system.

Kueue Configuration v1beta2 API

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

Configuration

`Configuration`

Configuration is the Schema for the kueueconfigurations API

Field Description

apiVersion
string config.kueue.x-k8s.io/v1beta2

kind
string Configuration

namespace [Required]
string

Kueue v1beta1 API

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

`AdmissionCheck`

Appears in:

Kueue v1beta2 API

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

`AdmissionCheck`

Appears in:

kueuectl

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Controls Kueue queueing manager

Options

--as string

Username to impersonate for the operation. User could be a regular user or a service account in a namespace.

kueuectl create clusterqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Creates a ClusterQueue with the given name.

kueuectl create clusterqueue NAME [--cohort COHORT_NAME] [--queuing-strategy QUEUEING_STRATEGY] [--namespace-selector KEY=VALUE] [--reclaim-within-cohort PREEMPTION_POLICY] [--preemption-within-cluster-queue PREEMPTION_POLICY] [--nominal-quota RESOURCE_FLAVOR:RESOURCE=VALUE] [--borrowing-limit RESOURCE_FLAVOR:RESOURCE=VALUE] [--lending-limit RESOURCE_FLAVOR:RESOURCE=VALUE] [--dry-run STRATEGY]

Examples

  # Create a ClusterQueue
  kueuectl create clusterqueue my-cluster-queue
  
  # Create a ClusterQueue with cohort, namespace selector and other details
  kueuectl create clusterqueue my-cluster-queue \
  --cohort cohortname \
  --queuing-strategy StrictFIFO \
  --namespace-selector fooX=barX,fooY=barY \
  --reclaim-within-cohort Any \
  --preemption-within-cluster-queue LowerPriority
  
  # Create a ClusterQueue with nominal quota and one resource flavor named alpha
  kueuectl create clusterqueue my-cluster-queue --nominal-quota "alpha:cpu=9;memory=36Gi"
  
  # Create a ClusterQueue with multiple resource flavors named alpha and beta
  kueuectl create clusterqueue my-cluster-queue \
  --cohort cohortname \
  --nominal-quota "alpha:cpu=9;memory=36Gi;nvidia.com/gpu=10,beta:cpu=18;memory=72Gi;nvidia.com/gpu=20" \
  --borrowing-limit "alpha:cpu=1;memory=1Gi;nvidia.com/gpu=1,beta:cpu=2;memory=2Gi;nvidia.com/gpu=2" \
  --lending-limit "alpha:cpu=1;memory=1Gi;nvidia.com/gpu=1,beta:cpu=2;memory=2Gi;nvidia.com/gpu=2"

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kueuectl create localqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a local queue with the given name in the specified namespace.

kueuectl create localqueue NAME -c CLUSTER_QUEUE_NAME [--ignore-unknown-cq] [--dry-run STRATEGY]

Examples

  # Create a local queue
  kueuectl create localqueue my-local-queue -c my-cluster-queue
  
  # Create a local queue with unknown cluster queue
  kueuectl create localqueue my-local-queue -c my-cluster-queue -i

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kueuectl create resourceflavor

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a resource flavor with the given name.

kueuectl create resourceflavor NAME [--node-labels KEY=VALUE] [--node-taints KEY[=VALUE]:EFFECT] [--tolerations KEY[=VALUE][:EFFECT]]|:EFFECT [--dry-run STRATEGY]

Examples

  # Create a resource flavor
  kueuectl create resourceflavor my-resource-flavor
  
  # Create a resource flavor with labels
  kueuectl create resourceflavor my-resource-flavor \
  --node-labels kubernetes.io/arch=arm64,kubernetes.io/os=linux
  
  # Create a resource flavor with node taints
  kueuectl create resourceflavor my-resource-flavor \
  --node-taints key1=value:NoSchedule,key2:NoExecute
  
  # Create a resource flavor with tolerations
  kueuectl create resourceflavor my-resource-flavor \
  --tolerations key1=value:NoSchedule,key2:NoExecute,key3=value,key4,:PreferNoSchedule

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kueuectl delete clusterqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "delete clusterqueue" to kubectl

kueuectl delete clusterqueue [flags]

Options

-h, --help

help for clusterqueue

kueuectl delete localqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "delete localqueue" to kubectl

kueuectl delete localqueue [flags]

Options

-h, --help

help for localqueue

kueuectl delete resourceflavor

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "delete resourceflavor" to kubectl

kueuectl delete resourceflavor [flags]

Options

-h, --help

help for resourceflavor

kueuectl delete workload

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

If the Workload has associated Jobs, the command will prompt for deletion approval of the affected Jobs, and delete them. The Workload will then be asynchronously deleted by the garbage collector. If there are no associated Jobs, the command will directly delete the Workload.

kueuectl describe clusterqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "describe clusterqueue" to kubectl

kueuectl describe clusterqueue [flags]

Options

-h, --help

help for clusterqueue

kueuectl describe localqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "describe localqueue" to kubectl

kueuectl describe localqueue [flags]

Options

-h, --help

help for localqueue

kueuectl describe resourceflavor

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "describe resourceflavor" to kubectl

kueuectl describe resourceflavor [flags]

Options

-h, --help

help for resourceflavor

kueuectl describe workload

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "describe workload" to kubectl

kueuectl describe workload [flags]

Options

-h, --help

help for workload

kueuectl edit clusterqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "edit clusterqueue" to kubectl

kueuectl edit clusterqueue [flags]

Options

-h, --help

help for clusterqueue

kueuectl edit localqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "edit localqueue" to kubectl

kueuectl edit localqueue [flags]

Options

-h, --help

help for localqueue

kueuectl edit resourceflavor

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "edit resourceflavor" to kubectl

kueuectl edit resourceflavor [flags]

Options

-h, --help

help for resourceflavor

kueuectl edit workload

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "edit workload" to kubectl

kueuectl edit workload [flags]

Options

-h, --help

help for workload

kueuectl get clusterqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "get clusterqueue" to kubectl

kueuectl get clusterqueue [flags]

Options

-h, --help

help for clusterqueue

kueuectl get localqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "get localqueue" to kubectl

kueuectl get localqueue [flags]

Options

-h, --help

help for localqueue

kueuectl get resourceflavor

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "get resourceflavor" to kubectl

kueuectl get resourceflavor [flags]

Options

-h, --help

help for resourceflavor

kueuectl get workload

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "get workload" to kubectl

kueuectl get workload [flags]

Options

-h, --help

help for workload

kueuectl list clusterqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Lists all ClusterQueues, potentially limiting output to those that are active/inactive and matching the label selector.

kueuectl list clusterqueue [--selector key1=value1] [--field-selector key1=value1] [--active=true|false]

Examples

  # List ClusterQueue
  kueuectl list clusterqueue

Options

--active bools Default: []

Filter by active status. Valid values: 'true' and 'false'.

kueuectl list localqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Lists LocalQueues that match the given criteria: point to a specific CQ, being active/inactive, belonging to the specified namespace, matching the label selector or the field selector.

kueuectl list pods

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Lists all pods that matches the given criteria: should be part of the specified Job kind, belonging to the specified namespace, matching the label selector or the field selector.)

kueuectl list resourceflavor

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

List ResourceFlavor

kueuectl list resourceflavor [--selector KEY=VALUE] [--field-selector FIELD_NAME=VALUE]

Examples

  # List ResourceFlavor
  kueuectl list resourceflavor

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kueuectl list workload

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Lists Workloads that match the provided criteria.

kueuectl list workload [--clusterqueue CLUSTER_QUEUE_NAME] [--localqueue LOCAL_QUEUE_NAME] [--status STATUS] [--selector key1=value1] [--field-selector key1=value1] [--all-namespaces] [--for TYPE[.API-GROUP]/NAME]

Examples

  # List Workload
  kueuectl list kueueworkload

Options

-A, --all-namespaces

If present, list the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace.

kueuectl patch clusterqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "patch clusterqueue" to kubectl

kueuectl patch clusterqueue [flags]

Options

-h, --help

help for clusterqueue

kueuectl patch localqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "patch localqueue" to kubectl

kueuectl patch localqueue [flags]

Options

-h, --help

help for localqueue

kueuectl patch resourceflavor

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "patch resourceflavor" to kubectl

kueuectl patch resourceflavor [flags]

Options

-h, --help

help for resourceflavor

kueuectl patch workload

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pass-through "patch workload" to kubectl

kueuectl patch workload [flags]

Options

-h, --help

help for workload

kueuectl resume clusterqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Resumes the previously held ClusterQueue.

kueuectl resume clusterqueue NAME

Examples

  # Resume the clusterqueue
  kueuectl resume clusterqueue my-clusterqueue

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kueuectl resume localqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Resumes the previously held LocalQueue.

kueuectl resume localqueue NAME

Examples

  # Resume the localqueue
  kueuectl resume localqueue my-localqueue

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kueuectl resume workload

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Resumes the Workload, allowing its admission according to regular ClusterQueue rules.

kueuectl resume workload NAME [--namespace NAMESPACE] [--dry-run STRATEGY]

Examples

  # Resume the workload
  kueuectl resume kueueworkload my-workload

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kueuectl stop clusterqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Puts the given ClusterQueue on hold.

kueuectl stop clusterqueue NAME [--keep-already-running]

Examples

  # Stop the clusterqueue
  kueuectl stop clusterqueue my-clusterqueue

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kueuectl stop localqueue

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Puts the given LocalQueue on hold.

kueuectl stop localqueue NAME [--keep-already-running]

Examples

  # Stop the localqueue
  kueuectl stop localqueue my-localqueue

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kueuectl stop workload

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Puts the given Workload on hold. The Workload will not be admitted and if it is already admitted it will be put back to queue just as if it was preempted (using .spec.active field).

Multikueue

Mon, 01 Jan 0001 00:00:00 +0000

MultiKueue AdmissionCheck is the central component responsible for admission of workloads created on the management cluster, based on their of the workloads on the selected worker cluster.

Search Results

Mon, 01 Jan 0001 00:00:00 +0000

Kueue

Setup Prometheus

Before you begin

1. Setup

Configure external cert-manager

Before you begin

Run A Wrapped Custom Workload

Troubleshooting Jobs

Identifying the Workload for your Job

ProvisioningRequest

Setup RBAC

Before you begin

Common Grafana Queries

Before you begin

Quota utilization

Note

Configure Prometheus with TLS

Note

Before you begin

Run Deployment in Multi-Cluster

Before you begin

Pod integration requirements

Run Plain Pod in Multi-Cluster

Before you begin

Run StatefulSet in Multi-Cluster

Before you begin

Pod integration requirements

Run Kubernetes Job in Multi-Cluster

Before you begin

Troubleshooting Queues

Why no workloads are admitted in the LocalQueue?

Administer Cluster Quotas

Before you begin

Single ClusterQueue and single ResourceFlavor setup

Resource Flavor

Note

ResourceFlavor tolerations for automatic scheduling

Run A Flux MiniCluster

Before you begin

Monitor actual resource usage

Before you begin

Note

kubectl top for command line resource usage debugging

Warning

Share Quotas Across Resource Flavors

Run TrainJobs in Multi-Cluster

Before you begin

Run AppWrappers in Multi-Cluster

Before you begin

Run An Argo Workflow

Before you begin

Run Jobsets in Multi-Cluster

Before you begin

Run Jobsets example

Troubleshooting Provisioning Request in Kueue

Before you begin

Enabling pprof endpoints

Before you begin

Enabling pprof endpoints

Cluster Queue

Configure Custom Metric Labels

Before you begin

Cohort

Hello, Cohorts

Elastic Workloads

Elastic Workloads (Workload Slices)

Run KubeRay Jobs in Multi-Cluster

Before you begin

Run Kubeflow Jobs (v1) in Multi-Cluster

Note

Before you begin

Troubleshooting Pods

Note

Is my Pod managed directly by Kueue?

Run job with WorkloadPriority

Before you begin

Local Queue

Run A Kubernetes Job

Before you begin

Setup Dev Monitoring

`kubectl top` for command line resource usage debugging

Understanding `kubectl delete clusterqueue`