OPENSHIFT AI · LIGHTNING TALK · 15 MIN

Serving LLMs on OpenShift AI

One gateway in front, KServe underneath: a request's journey from your app to a GPU and back — and where Red Hat is taking the gateway next.

Arsalan Naeem · OpenShift Solution Architect, Red Hat August 2026 current release: OpenShift AI 3.4 ⚠ roadmap ahead moves fast — may be outdated by the time you read it
0:00hello 1:00why not a Route? 2:30the two layers 4:30one request, animated 8:00the roadmap 9:30takeaways 10:00your questions
PART 1 — HELLO≈ 1 min

Hi, I'm Arsalan 👋

  • Arlington, Virginia → Manchester, England. Born in the first, raised in the second, and travelling ever since — 42 countries so far (spin the globe below).
  • Linux & Perl since age 10. My first production workloads were Quake 3 and Counter-Strike 1.6 servers — uptime mattered, the players made sure of it.
  • Biomedical engineer by education, specialized in hip replacements — then retrained as a container specialist. Same trade, honestly: replace the worn parts before they fail, with less paperwork.
  • The route here: a few years as a datacenter sysadmin — I've commissioned server rooms with hot/cold aisle containment — then DevOps: running Kubernetes for Harvard, shipping 5G cores on Kubernetes for major telcos at Rancher/SUSE, and since 2023, OpenShift Solution Architect at Red Hat.
  • I still run my own datacenter on OpenShift Virtualization: 3× R630, an R730xd, an R730 with 2× Tesla P40s, pfSense out front, and about 1 TB of DDR4 across the fleet. Old habits.
Arsalan's home datacenter: a rack of Dell PowerEdge servers and a switch in a white cabinet
The datacenter in question — it lives with me, not in a cloud region.

The travel log

Red pins are the origin story; every dot is a stamp in the passport.

Arlington, VA · born Manchester, UK · raised AustriaBahrainBelgium CanadaChinaColombia CroatiaDenmarkEgypt FranceGermanyGreece HungaryIcelandIndonesia IrelandItalyKazakhstan KuwaitLiechtensteinLuxembourg MalaysiaMaltaMexico MonacoMoroccoThe Netherlands OmanPakistanPanama PortugalQatarRomania Saudi ArabiaSloveniaSpain SwitzerlandThailandTurkey United Arab EmiratesUnited Kingdom United States
PART 2 — THE PROBLEM≈ 1.5 min

Why not just a Route and HAProxy?

What plain ingress sees
What LLM traffic actually is
A request: milliseconds, a few kilobytes, then done.
A response that streams for minutes — a default Route timeout of 30 s cuts your answer off mid-sentence.
Interchangeable backends — balance by connection count.
GPU pods where the right backend depends on model, version and load — and may not even be running yet.
Requests are free to serve.
Requests are metered by the token — every call is somebody's budget.
No concept of API keys, models, or quotas.
Per-team quotas, provider keys, fallbacks and canaries are the actual job.

Ingress moves bytes. LLM serving needs opinions — that's why there are two extra layers.

PART 3 — THE TWO LAYERS≈ 2 min

A gateway in front, a serving platform underneath

Consumers
the customers
chatbotsagents RAG pipelinesnotebooks they all speak the OpenAI API
▼ one URL, one token
Gateway layer — MLflow AI Gateway
the front desk
one API for every modelkey management rate limitsrouting & fallbackcost tracking
▼ routed to a model backend
Serving layer — OpenShift AI (KServe)
the engine room
InferenceService CRsvLLM runtime autoscaling → zerocanary rollouts
▼ scheduled onto
GPU nodes
the electricity bill
accelerated worker nodesonly running when needed

What each layer gives you, in current releases

MLflow AI Gatewayopen source · runs as a pod
  • One OpenAI-compatible API in front of every model — self-hosted on the cluster or SaaS providers (OpenAI, Anthropic, Bedrock, …).
  • Keys stay server-side. Apps get one URL and one token; provider credentials live only in the gateway.
  • Rate limits per endpoint — a team that burns its budget gets a polite 429, not a surprise invoice.
  • Routing is config, not code. Swap or fail over the backing model by editing the gateway, never the apps.
  • Usage & cost visibility — every request is logged and attributable.
Honesty note: a lightweight Python service you deploy yourself as a container in front of the platform — it is not part of the OpenShift AI product.
KServe on OpenShift AIGA · RHOAI 3.4
  • Model serving as YAML. One InferenceService per model — declarative, GitOps-friendly.
  • Autoscaling including scale-to-zero (Knative serverless mode): idle models cost $0 in GPU.
  • Canary rollouts — canaryTrafficPercent splits traffic between the last-good and the new revision.
  • Many models side by side, each its own service; vLLM is the LLM runtime of choice.
  • Model storage from S3, URI, PVC, or OCI images ("modelcars").
ModelMesh (the old multi-model platform) is deprecated — the path is one InferenceService per model. Distributed inference with llm-d: Tech Preview.

Each layer is one small YAML file

Gateway — config.yaml: a route, a key, a limit
endpoints:
  - name: chat
    endpoint_type: llm/v1/chat
    model:
      provider: openai        # any OpenAI-compatible backend…
      name: granite-8b
      config:
        openai_api_base: https://granite-8b.ai.svc/v1   # …like vLLM on KServe
        openai_api_key: $GRANITE_KEY    # stays server-side
    limit:
      renewal_period: minute
      calls: 100              # over budget → polite 429
KServe — InferenceService: serve, scale, canary
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: granite-8b
spec:
  predictor:
    minReplicas: 0            # scale to zero when idle
    canaryTrafficPercent: 10   # 10% to the new revision
    model:
      modelFormat:
        name: vLLM
      storageUri: oci://registry.example.com/granite-8b:1.3
      resources:
        limits:
          nvidia.com/gpu: "1"

That's the whole trick: the gateway's YAML decides who may call what; KServe's YAML decides how it runs.

PART 4 — ONE REQUEST, END TO END≈ 3.5 min

Press play: a chat request wakes up the whole stack

Ten steps, one request. Numbers are illustrative; the mechanics are what ships today.

Step 1 / 10
Your app chatbot · agent one URL, one token GATEWAY LAYER MLflow AI Gateway · runs as a pod AI Gateway endpoint POST /v1/chat/completions auth · quota · route · log quota: 0/100 req·min spend today: $0.00 every request metered Provider key vault OpenAI · Anthropic · local keys never reach apps Route table chat → granite-8b embed → bge-large fallback → mistral-7b swap models = edit config apps never change OPENSHIFT AI — KSERVE single-model serving · RHOAI 3.4 InferenceService granite-8b runtime: vLLM · GPU replicas: 0 traffic: v1 100% · v2 0% vllm-pod-1 vllm-pod-2 vllm-pod-3 v2-pod-1 canary revision v2 scaled to zero when idle $0 GPU while replicas: 0 Autoscaler Knative KPA target: 4 in-flight desired: 0 scale 0 ↔ N mistral-7b (fallback) second InferenceService same cluster · same API boring by design POST /v1/chat route: chat tokens (SSE) load 0→N on error

Inference traffic — requests & streamed tokens Control & autoscaling Metrics & policy
PART 5 — WHERE THIS IS GOING≈ 1.5 min

The gateway layer is consolidating: Praxis & Red Hat AI Gateway

Today the "AI gateway" story is several products standing in a trench coat. The plan is one front door. Dashed boxes are roadmap, not releases.

Today — Aug '26shipping
  • Connectivity Link (Kuadrant) — gateway policies: auth, rate limits (GA)
  • MaaS on OpenShift AI — model gateway: quotas, API keys (GA)
  • llm-d — distributed inference, cache-aware routing (Tech Preview)
  • MCP gateways — agent tool discovery (emerging)
Nov '26planned
  • Praxis replaces MaaS / IPP in Tech Preview, acting as backend proxy
  • Envoy + Kuadrant operate in front of Praxis
  • Connectivity Link is rebranded to Red Hat AI Gateway
  • Next target for Praxis: llm-d
Aug '27planned
  • Praxis goes GA on MaaS / IPP functionality
  • Praxis goes GA on llm-d — a supported llm-d gateway option
  • Praxis goes GA on MCP Gateway, incl. tool discovery
  • Praxis + Kuadrant at ingress — for all AI workloads
  • One gateway: RHCL · MaaS · llm-d · MCP

⚠ Objects in this roadmap are less stable than they appear. Verified August 2026 — names, dates and scope shift fast. Treat as direction, not commitment.

PART 6 — TAKEAWAYS≈ 30 sec

If you remember three things

1

KServe is the engine, and it's GA today. Autoscaling to zero, canary rollouts and vLLM runtimes — in the current OpenShift AI release, driven by YAML.

2

Never point apps straight at models. A gateway — MLflow AI Gateway is a lightweight way in — gives you one API, central keys, quotas and a cost story.

3

Watch Praxis. Red Hat's gateway pieces — Connectivity Link, MaaS, llm-d, MCP — are converging into one AI Gateway at ingress.

Questions? You have five minutes, and I have opinions.