LLM Inference on Autopilot
in Your AWS

LLM Inference
on Autopilot
in Your AWS

Own your inference. Skip the operational burden.
Define SLO targets per model. Rivvr handles everything below the line,
inside your AWS account.

Own your inference.
Skip the operational burden.
Define SLO targets per model. Rivvr handles everything below the line,
inside your AWS account.

Data stays in your cloud

Resilient SLOs under traffic bursts

Run any vLLM model

The SLO-cost tradeoff, eliminated.

With other providers

No guaranteed SLOs in a shared API

No control over prompts, weights, or logs leaving your environment

Limited model selection, fixed pre-configs

Unpredictable costs at scale

With Rivvr

Custom SLOs, held automatically, even under traffic bursts

Runs inside your own AWS account

Run any vLLM model, any quantization

Always predictable $/token economy

You set the targets.
Rivvr runs the cluster.

Define your model, not your infrastructure

Bring your own weights, or export from Hugging Face. Define model configs — data type, quantization, KV cache quantization, context length — or leave defaults. Set TTFT, TPOT, p50/p90, cost guardrails, and speed mode.

Define your model, not your infrastructure

Deployment Policy

llama-3-70b

Quantization

LoRa

TTFT p50

Output Token/s

TTFT p90

Cost guardrail

Speed mode

Loading...

Rivvr runs the cluster in real time

Rivvr finds the VMs and GPUs that fit your model's memory footprint. It continuously optimizes the mixed GPU fleet against your SLOs and cost guardrails — forecasting demand, shifting compute between models, maximizing KV cache reuse. Topology always matches real traffic, not your worst-case estimate.

Rivvr runs the cluster in real time

req / s

llama-3-70b

req / s

mistral-7b

GPU Cluster

A100

80GB

llama-3-70b

65%

H100

80GB

llama-3-70b

70%

L40S

48GB

llama-3-70b

58%

L4

24GB

mistral-7bllama-3-70b

62%

A100

40GB

mistral-7bllama-3-70b

55%

L4

24GB

mistral-7b

60%

You stay in control of the economics

Track live TTFT, TPOT, SLO compliance, cost per model, cost per token. Change your targets anytime — no redeploy, no downtime. Tighten an SLO, raise a cost guardrail, see the cluster adapt.

You stay in control of the economics

Track live TTFT, TPOT, SLO compliance, cost per model, cost per token. Change your targets anytime — no redeploy, no downtime. Tighten an SLO, raise a cost guardrail, see the cluster adapt.

Live Monitoring

llama-3-70b

TTFT

p50 0msp90 0ms

Tokens / s

p50 0p90 0

Prefill / M tokens

$0.00

Decode / M tokens

$0.00

SLO Compliance0.0%

Built for production
LLM workloads

Voice AI

Support chat agents

Coding assistants & agents

OCR & document intake

LATENCY-CRITICAL

The Challenge

•

Any TTFT variance is an audible pause; there’s no graceful degradation

•

Static warm pools protect latency, but bill you for capacity you don’t use

With Rivvr

•

Your pipeline stays under 400ms without a dedicated warm pool

•

GPU spend tracks actual call volume, not your peak estimate

Voice AI

Support chat agents

Coding assistants & agents

OCR & document intake

THROUGHPUT-BOUND

The Challenge

•

Batch jobs arrive unpredictably, and each one is GPU-heavy

•

No clear view of what each pipeline actually costs

With Rivvr

•

Each job gets the compute it needs, then releases it, no waste

•

Pipeline costs are visible and stable, not buried in shared cluster spend

Runs in your cloud.
Stays in your cloud.

VPC deployment

Rivvr runs inside your AWS account, in a network-isolated environment you control.

VPC deployment

Rivvr runs inside your AWS account, in a network-isolated environment you control.

Data privacy

Prompts, responses, logs, and model weights stay in your cloud.

Data privacy

Prompts, responses, logs, and model weights stay in your cloud.

RBAC and access control

Control who can deploy models, modify policies, and view cost and performance data.

RBAC and access control

Control who can deploy models, modify policies, and view cost and performance data.

Everything to run inference in production

SLO policy engine

TTFT, TPOT, p50/p90, speed modes, and cost guardrails enforced per model, automatically.

SLO policy engine

TTFT, TPOT, p50/p90, speed modes, and cost guardrails enforced per model, automatically.

Inference orchestration

Continuous GPU allocation, model placement, and routing optimization around live traffic.

Inference orchestration

Continuous GPU allocation, model placement, and routing optimization around live traffic.

Inference orchestration

Continuous GPU allocation, model placement, and routing optimization around live traffic.

Multi-model serving

Models share a GPU pool and swap compute in real time without idle resources.

Multi-model serving

Models share a GPU pool and swap compute in real time without idle resources.

Multi-model serving

Models share a GPU pool and swap compute in real time without idle resources.

Heterogeneous GPU support

Mixes AWS instance types and GPU sizes dynamically to maximize cost efficiency.

Heterogeneous GPU support

Mixes AWS instance types and GPU sizes dynamically to maximize cost efficiency.

Sub-10s cold starts

New capacity online in under 10 seconds, scale to bursts without padding headroom.

Sub-10s cold starts

New capacity online in under 10 seconds, scale to bursts without padding headroom.

Fleet management

Centralized model storage, deployment and rollout for your model fleet in one place.

Managed endpoints

Networking, DNS, API keys, and access permissions handled for you.

Managed endpoints

Networking, DNS, API keys, and access permissions handled for you.

Managed endpoints

Networking, DNS, API keys, and access permissions handled for you.

Cost and latency monitoring

TTFT, TPOT, SLO compliance, cluster cost, and cost per token, all in real time.

Cost and latency monitoring

TTFT, TPOT, SLO compliance, cluster cost, and cost per token, all in real time.

RBAC

Granular access control over deployments, policy changes, and performance data.

RBAC

Granular access control over deployments, policy changes, and performance data.

RBAC

Granular access control over deployments, policy changes, and performance data.

Multi-LoRa Serving

Run fine-tuned models alongside your base models, in the same cluster.

Multi-LoRa Serving

Run fine-tuned models alongside your base models, in the same cluster.

Batched Processing

Scales to zero when idle. Submit a job, poll for the result — capacity spins up only to run it.

Batched Processing

Scales to zero when idle. Submit a job, poll for the result — capacity spins up only to run it.

Batched Processing

Scales to zero when idle. Submit a job, poll for the result — capacity spins up only to run it.

Bring your own weights

From Hugging Face, S3 path, or upload directly. If it runs on vLLM, Rivvr can serve it.

Bring your own weights

From Hugging Face, S3 path, or upload directly. If it runs on vLLM, Rivvr can serve it.

Frequently Asked Questions

Frequently
Asked Questions

How is this different from managed platforms?

Rivvr gives you the managed service experience — no infra to run, no scaling rules to tune — but inside your own AWS account. You keep full control over security, SLOs, and cost, without the operational overhead.

How is this different from AIBrix or NVIDIA Dynamo?

Two things. First: you don't operate anything — Rivvr runs as a managed platform inside your AWS account, unlike the software your team deploys and maintains. Second: a different operational model. AIBrix and Dynamo ask you to configure autoscaling, profiling, and dozens of other parameters. Rivvr asks for two — your SLO and your cost guardrail. Set those, and you're running.

Where does Rivvr run?

Inside your AWS account, behind your VPC. Inference traffic, data, and model weights never leave your environment.

How is Rivvr priced?

Rivvr is priced as a management layer based on GPUs under orchestration. You pay AWS directly for infrastructure; Rivvr charges a separate management fee.

Do I need to change my integration code?

No. Rivvr uses an OpenAI-compatible API. Most teams point their existing client at a new endpoint, and nothing else changes.

Does Rivvr modify or compress my models?

No. Rivvr doesn't touch model weights. Optimization happens at the orchestration, placement, and infrastructure layers.

What models does Rivvr support?

Any model served by vLLM, up to 400B parameters. If you're running something larger or unusual, let's talk.

What inference engines are supported?

Rivvr uses a vLLM-compatible runtime.

What GPUs are supported?

NVIDIA GPUs with CUDA compute capability 7.0 or higher — including L4, L40S, A10G, T4, A100, H100, H200, B200, B300, V100.

See it running on your setup

Bring your current models and traffic. We'll show you what it looks like running on Rivvr — SLOs held automatically, your own AWS account, no operational burden.

Bring your current models and traffic. We'll show you what it looks like running on Rivvr — SLOs held automatically, your own AWS account,
no operational burden.

Bring your current models and traffic. We'll show you what it looks like running on Rivvr —
SLOs held automatically,
your own AWS account,
no operational burden.

LLM Inference on Autopilot in Your AWS