Enhancing Kubernetes with NVIDIA’s NIM Microservices Autoscaling

Terrill Dicki
Jan 24, 2025 14:36

Explore NVIDIA’s approach to horizontal autoscaling of NIM microservices on Kubernetes, utilizing custom metrics for efficient resource management.

NVIDIA has introduced a comprehensive approach to horizontally autoscale its NIM microservices on Kubernetes, as detailed by Juana Nakfour on the NVIDIA Developer Blog. This method leverages Kubernetes Horizontal Pod Autoscaling (HPA) to dynamically adjust resources based on custom metrics, optimizing compute and memory usage.

Understanding NVIDIA NIM Microservices

NVIDIA NIM microservices serve as model inference containers deployable on Kubernetes, crucial for managing large-scale machine learning models. These microservices necessitate a clear understanding of their compute and memory profiles in a production environment to ensure efficient autoscaling.

Setting Up Autoscaling

The process begins with setting up a Kubernetes cluster equipped with essential components such as the Kubernetes Metrics Server, Prometheus, Prometheus Adapter, and Grafana. These tools are integral for scraping and displaying metrics required for the HPA service.

The Kubernetes Metrics Server collects resource metrics from Kubelets and exposes them via the Kubernetes API Server. Prometheus and Grafana are employed to scrape metrics from pods and create dashboards, while the Prometheus Adapter allows HPA to utilize custom metrics for scaling strategies.

Deploying NIM Microservices

NVIDIA provides a detailed guide for deploying NIM microservices, specifically using the NIM for LLMs model. This involves setting up the necessary infrastructure and ensuring the NIM for LLMs microservice is ready for scaling based on GPU cache usage metrics.

Grafana dashboards visualize these custom metrics, facilitating the monitoring and adjustment of resource allocation based on traffic and workload demands. The deployment process includes generating traffic with tools like genai-perf, which helps in assessing the impact of varying concurrency levels on resource utilization.

Implementing Horizontal Pod Autoscaling

To implement HPA, NVIDIA demonstrates creating an HPA resource focused on the gpu_cache_usage_perc metric. By running load tests at different concurrency levels, the HPA automatically adjusts the number of pods to maintain optimal performance, demonstrating its effectiveness in handling fluctuating workloads.

Future Prospects

NVIDIA’s approach opens avenues for further exploration, such as scaling based on multiple metrics like request latency or GPU compute utilization. Additionally, leveraging Prometheus Query Language (PromQL) to create new metrics can enhance the autoscaling capabilities.

For more detailed insights, visit the NVIDIA Developer Blog.

Image source: Shutterstock