GitHub - logicalclocks/run-ai-helm: Helm chart that installs NVIDIA Run:ai (control plane + cluster) and KAI Scheduler

runai

Installs NVIDIA Run:ai (control plane and cluster) and the KAI Scheduler it schedules with

The chart wraps the published upstream charts and adds the glue between them: release profiles, validation, the Secrets the upstream charts expect to exist, and the cluster registration step.

Releases

The chart is installed once per namespace. Each release enables its components with a profile file:

The control plane and the cluster cannot share a release. The cluster's pre-install hooks call the control-plane API with a cluster UUID and a bootstrap token, and the control plane issues both only once it is running. scripts/register_cluster.sh registers the cluster and writes them to a values file for release 3.

KAI Scheduler comes with release 3: runai-cluster depends on kai-resource-management, which depends on kai-scheduler. The Run:ai operator writes KAI's Config and the kai-resource-management node-pool controller writes one SchedulingShard per node pool. The standalone kai-scheduler profile is for clusters that run KAI without Run:ai, and the chart refuses to render it together with the cluster profile.

Prerequisites

• Kubernetes >= 1.30, Helm 3.x or 4.x, kubectl, curl, jq.
• An ingress controller. The control plane creates an Ingress with class nginx by default (global.ingress.ingressClass, nginx or haproxy). OpenShift uses Routes.
• A DNS name for the control plane (global.domain) that resolves to the ingress controller, both from where scripts/install.sh runs and from inside the cluster.
• A TLS certificate for global.domain: PEM in controlplane.tls, a cert-manager issuer in controlplane.tls.certManager, or a kubernetes.io/tls Secret runai-backend-tls created in namespace runai-backend before the install.
• A default StorageClass for the PostgreSQL, Thanos (100Gi) and NATS volumes.
• A Prometheus Operator and an NVIDIA GPU stack (DCGM exporter). Release 1 installs both; skip it with --skip-prerequisites when the cluster already has them.

Install

From a git checkout:

./helm_dependencies.sh
scripts/install.sh --domain runai.example.com -f my-values.yaml

my-values.yaml is passed to every release. A minimal one with a certificate for the domain:

controlplane:
  tls:
    certificate: |
      -----BEGIN CERTIFICATE-----
      ...
    key: |
      -----BEGIN PRIVATE KEY-----
      ...

When the certificate is signed by a private CA, also set the CA so the cluster components trust it:

global:
  customCA:
    enabled: true
  _runai:
    customCA:
      certificate: |
        -----BEGIN CERTIFICATE-----
        ...

and pass --cacert ca.pem to scripts/install.sh for the registration calls.

scripts/install.sh --help lists the options: --skip-prerequisites, --skip-control-plane, --skip-cluster, --dev (fake GPUs), --cluster-name, --admin-user, --cacert, --insecure, --timeout.

The UI is at https://<global.domain>. The admin user is test@run.ai (controlplane.control-plane.tenantsManager.config.adminUsername) and its password is generated:

kubectl get secret runai-backend-gen-tenants-manager -n runai-backend \
  -o jsonpath='{.data.ADMIN_PASSWORD}' | base64 -d

Step by step

helm upgrade --install runai-prerequisites . -n runai-prerequisites --create-namespace \
  -f values.prerequisites.yaml --wait

helm upgrade --install runai-backend . -n runai-backend --create-namespace \
  -f values.control-plane.yaml -f my-values.yaml --set global.domain=runai.example.com \
  --wait --wait-for-jobs --timeout 20m

scripts/register_cluster.sh --domain runai.example.com --output cluster-registration.yaml

helm upgrade --install runai-cluster . -n runai --create-namespace \
  -f values.cluster.yaml -f my-values.yaml -f cluster-registration.yaml \
  --wait --wait-for-jobs --timeout 20m

cluster-registration.yaml holds a bootstrap token. The control plane mints a new one on every call, so regenerate the file before each install or upgrade of runai-cluster.

Test clusters without GPUs

scripts/install.sh --domain runai.example.com --dev -f my-values.yaml
kubectl label node <node> run.ai/simulated-gpu-node-pool=default

values.dev.yaml replaces the NVIDIA GPU Operator with the fake GPU operator, which reports 8 Tesla-A10 GPUs on each labelled node.

KAI Scheduler only

helm upgrade --install kai-scheduler . -n kai-scheduler --create-namespace \
  -f values.kai-scheduler.yaml --wait

KAI creates the queues default-parent-queue and default-queue on first install. Pods opt in with schedulerName: kai-scheduler and the label kai.scheduler/queue: default-queue.

OpenShift

Layer values.openshift.yaml on every release. Upstream detects OpenShift with lookup, which returns nothing under helm template, so the file states it: it sets global.config.kubernetesDistribution: openshift (Routes and SCCs in the control plane) and forces OpenShift mode in runai-cluster, kai-resource-management and kai-scheduler. Leave global._runai.monitoring off: the Run:ai operator uses openshift-monitoring.

GitOps (ArgoCD)

ArgoCD renders with helm template, so:

• Set global._runai.mode: install (or upgrade): lookup returns nothing, and the chart then renders the runai-reservation Namespace unconditionally.
• Create one Application per release, synced in the order of the table above.
• The cluster Application needs the registration values. Run scripts/register_cluster.sh and store its output in a Secret-backed values source, since it holds a bootstrap token.

Cluster ingress

The Run:ai cluster takes its routing settings through clusterConfig, a merge patch into its RunaiConfig. To use HAProxy instead of nginx:

cluster:
  runai-cluster:
    clusterConfig:
      global:
        ingress:
          ingressClass: haproxy

Commercial charts

NVIDIA also publishes the commercial Run:ai line (2.x) as control-plane and runai-cluster in https://helm.ngc.nvidia.com/nvidia/runai, which needs an NGC API key. Those charts take the same install values (controlPlane.url, controlPlane.clientSecret, cluster.uid, cluster.url, clusterConfig). Their runai-cluster 2.26.50 schedules with the proprietary runai-engine (v3.41.19, from nvcr.io/artifactory) instead of KAI Scheduler. This chart wraps the open-source 3.x line from nvidia/runai-oss.

Upstream globals

Each subchart passes its upstream chart's values one level down, for example controlplane.control-plane.keycloakx, cluster.runai-cluster.kai-resource-management or monitoring.kube-prometheus-stack.grafana. The upstream charts also read these globals:

Values