Monitor LLM inference costs and GPU utilization in real-time with a self-hosted, OpenTelemetry-native metrics gateway.

What’s included:

• OpenTelemetry (OTel) native metric export for Prometheus and Grafana
• Real-time CUDA memory, VRAM allocation, and compute utilization tracking
• Per-token inference cost calculation and step-latency profiling
• Drop-in async middleware gateway for custom AI backends