Editorial
Guide
Is AI Making Your Organization More Productive or More Dependent?
Key Takeaways
- Kubernetes’ CPU-based autoscaling can miss overloaded GPUs, causing latency spikes and SLA violations.
- Direct NVML-based scaling cuts GPU metric latency from 15-30 seconds to roughly 2-4 seconds.
- Workload-specific scaling profiles can improve performance while reducing the cost of idle GPU capacity.
Your vLLM pod shows 8% CPU usage. Kubernetes considers everything to be okay. But the A100 on your node works at 97% utilization, VRAM close to being exhausted and inference latency has exceeded the SLA limits.
This occurs in every Kubernetes cluster which does GPU inference tasks. HPA observes CPU and memory metrics. But it does not know anything about GPUs.
The Visibility Problem
Kubernetes was built for generic computation. The scheduler can schedule pods on GPUs thanks to the device plugin for NVIDIA, which exposes an extended resource. But that’s just allocation. It doesn’t tell you anything about how much load a GPU is under.
That leads to a major problem. Auto-scaling based on CPU metrics and GPU workload are worlds apart. You can fully utilize the GPU while putting no strain at all on the CPU. The HPA will see idle pods, but the GPU will be burning hot.
Usually, the visibility problem becomes apparent in the following way: inference performance drops, people start complaining, someone connects via SSH and runs nvidia-smi.
The Standard Fix and Its Problems
The traditional way involves a pipeline: NVIDIA DCGM Exporter gathers telemetry from GPUs, exports it to Prometheus, which is queried by either KEDA or custom HPA adapter.
It works fine, but it creates a set of issues that scale badly:
- Latency: There are latency intervals introduced by Prometheus scrapes, Prometheus recording rules and KEDA polls. As a result, the latency between GPU reaching 95% utilization and a new pod being spun up is between 15 and 30 seconds. For interactive chatbot inference, code autocomplete, real-time translation — this is an eternity.
- Complexity: You operate with five different components to collect one metric: DCGM Exporter, ServiceMonitor for DCGM Exporter, Prometheus, KEDA and ScaledObject. Every single component brings its unique set of failure modes and version compatibility matrix. Teams usually spend much more time on debugging the monitoring pipeline rather than the inference workload itself.
- Resource Overhead: Prometheus is a time-series database. Using Prometheus to gather GPU telemetry and pass it over to an autoscaler is a case of over-engineering. In a world where every megabyte of RAM counts, it becomes very significant.
Related Article: Why Enterprise AI Fails: The 4 Infrastructure Bottlenecks Nobody Wants to Talk About
Reading GPUs Directly
Our approach was quite different for the open-source keda-gpu-scaler. Rather than build another monitoring pipeline, we decided to go all the way down to the metal.
NVML stands for NVIDIA Management Library — it’s the C library that drives nvidia-smi behind the scenes. NVML gives direct access to all metrics available from the GPU: compute util, memory util, temperature, power draw, PCIe bandwidth, NVLink bandwidth. There is no exporter, there is no database and no scrape interval.
The tool runs inside Kubernetes DaemonSet running on GPU nodes and communicates directly to NVML through the bindings in Go. Metrics are exposed via gRPC ExternalScalerServer interface understood natively by KEDA. When KEDA asks what the GPU utilizations currently are, the response is taken directly from the hardware within a single second.
The result is two components rather than five, with metric latencies being just 2-4 seconds instead of 15-30. Scale-up, scale-down and scale-to-zero are performed by KEDA’s External Scaler protocol.
Why It Won’t Work Within KEDA
Though this functionality may appear to be a part of KEDA’s essence, there are architectural limitations preventing it.
First, KEDA uses CGO_ENABLED=0 and only pure Go without any C libraries. NVIDIA provides Go bindings using go-nvml, which use CGO due to its reliance on the NVML C library. Thus, adding such a CGO-dependent library will drastically change the deployment and building approach of KEDA.
The other limitation is that the KEDA operator operates as a single central pod. NVML needs access to GPU device files present on the host machine. The only reasonable deployment method is DaemonSet on each GPU nodes. You cannot check the GPU from a pod on another machine. See architectural reasoning in KEDA issue #7538.
Scaling Profiles for Real Workloads
Metrics alone are not sufficient. An endpoint that hosts a vLLM inference server and a distributed training job exhibit entirely distinct scaling properties. Equating them leads to inefficient use of the GPU or the model.
Pre-defined scalability profiles come out-of-the-box for specific workload types:
- LLM Inference: The workload scales on VRAM usage. The KV cache of the language model increases with the number of simultaneous requests. When memory usage reaches 80%, requests begin failing or get queued. This profile offers scale-to-zero functionality: If no requests come for a certain period of time, all replicas get scaled down to zero and the GPU is freed up.
- Triton Inference: The workload scales on GPU usage at a 75% threshold by default. Specifically tuned for NVIDIA Triton Inference Server multi-model serving platform workloads.
- Batch Inference: Utilizes a low memory threshold with aggressive scale-down. Batch jobs are tolerant of latency; it focuses on cost-efficiency and scales down rapidly once the batch runs dry.
- Training Workloads: Scales based on the compute resource usage and doesn’t have any scale-to-zero so that the pod isn’t killed mid-epoch and loses the progress in its checkpoints. It maintains a high threshold (90%).
- Distributed Training: Checks the inter-GPU bandwidth on NVLink clusters. Once the NVLink bandwidth is below the activation threshold, the GPUs are not communicating actively since the job isn’t busy. Replicas can be killed.
Cost and Operational Impact
The gap between 30-second scaling and 3-second scaling creates operational practices that were not previously possible.
With instant provisioning of GPU pods, running idle GPUs "just in case" simply does not make economic sense anymore. With a cost of around $2-$3/hr for an A100, 20 idle GPUs incur a $40-$60/hr cost. Instant scale to zero and quick reactions mean minimal waste of money at off-peak times.
Burst processing is another capability that comes into play. In cases where there is a 10x traffic spike to an LLM endpoint, the autoscaler quickly recognizes GPU saturation and launches more replicas to avoid any performance degradation.
How This Fills the Ecosystem Gaps
The presented solution does not take the place of NVIDIA’s DCGM Exporter nor Prometheus when it comes to long-term metrics, dashboards and alerts. If you use this stack and the performance is good enough, you have no need to make any changes.
keda-gpu-scaler allows for a direct connection from hardware to scaling actions. The tool is designed specifically for solving the task of autoscaling. Moreover, it works outside of Kubernetes, being capable of detecting SLURM and Flux job schedulers, providing the same output schemas in both cases.
Related Article: Taming GPU Burn: Cut GenAI Costs Without Slowing Delivery
The Bigger Problem
GPU aware autoscaling represents part of the greater problem of infrastructure readiness. Kubernetes is becoming the de facto system for AI workloads, yet its resource model has been optimized for CPUs.
HAMi is trying to solve the problem of GPU sharing, Volcano is implementing NUMA-aware GPU placement and the Kubernetes WG on Workload-aware Scheduling is implementing topology-aware scheduling as first class. They all represent the same trend of shifting to fundamentally different infrastructure needed by AI workloads.
From Automation to Intelligence: How Leading Teams Are Rethinking Operations
Register
Ai4 2026
Register
How to Win the War for Agentic Citations: The AEO Playbook You Need Now
Register
Content Leaders Collective: Proving Content's Business Impact Starts With the Right CCMS
Watch Now
GPU-aware autoscaling is going to be table stakes. The ecosystem is going to close this gap. The only question is how long teams keep paying for idle GPUs while it catches up.
Learn how you can join our contributor community.