Managed LLM Inference
Inside Your AWS

Managed LLM Inference Inside Your AWS

Run vLLM instances at scale. Skip the operational burden.
Set SLO and cost targets. Rivvr handles infrastructure to meet them,
inside your AWS account.

Run vLLM instances at scale. Skip the operational burden. Set SLO and cost targets. Rivvr handles infrastructure to meet them, inside your AWS account.

Up to 70% cost savings

Zero inference ops

Any vLLM model or LoRA

VPC

Model + LoRAs

SLO + cost targets

Rivvr Orchestrator

Exact-target SLOs

Fleet optimization

Cost control

Observability

Mixed GPU Fleet

H100

Spot

vLLM · GPT-OSS 20B

L40S

Spot

vLLM · GPT-OSS 20B

A100

On-demand

vLLM · GPT-OSS 20B

L4

Spot

vLLM · GPT-OSS 20B

Live metrics

Stop configuring inference infrastructure

With SageMaker or DIY

Configure the infrastructure

Choose GPUs, capacity, and scaling

Monitor traffic

Retune manually

With Rivvr

Set latency and cost targets

Rivvr chooses the infrastructure

Adapts automatically as traffic changes

Continuously optimized in the background

With Rivvr

Set latency and cost targets

Rivvr chooses the infrastructure

Adapts automatically as traffic changes

Continuously optimized, in the background

You set the targets.
Rivvr runs the cluster.

Define your SLOs, not your infrastructure

Bring weights from Hugging Face, S3, or your local store. Optionally configure datatype, quantization, KV-cache quantization and context length. Then set TTFT, TPS, p50/p90, cost guardrails and SLO complience level.

Define your SLOs, not your infrastructure

Bring your own weights, or import from Hugging Face. Define model configs — data type, quantization, KV cache quantization, context length — or leave defaults. Set TTFT, TPS, p50/p90, cost guardrails, and speed mode.

Deployment Policy

llama-3-70b

Quantization

LoRa

TTFT p50

Output Token/s

TTFT p90

Cost guardrail

Speed mode

Loading...

Rivvr continuously operates the fleet

Rivvr continusly optimizes mixed GPU inference topology: GPU type, VM size, capacity type (reliable on-demand or cheap spot), vLLM workers, and replicas. Zero operations, up to 70% cost savings.

Rivvr continuously operates the fleet

Rivvr continusly optimizes mixed GPU inference topology: GPU type, VM size, capacity type (reliable on-demand or cheap spot), vLLM workers, and replicas. Zero operations, up to 70% cost savings.

req / s

llama-3-70b

req / s

mistral-7b

GPU Cluster

A100

80GB

on-demandspot

llama-3-70b

65%

H100

80GB

spot

llama-3-70b

70%

L40S

48GB

spoton-demand

llama-3-70b

58%

L4

24GB

spot

mistral-7bllama-3-70b

62%

A100

40GB

on-demandspot

mistral-7bllama-3-70b

55%

L4

24GB

spoton-demand

mistral-7b

60%

Monitor performance. Change objectives.

Track live TTFT, TPS, SLO compliance, cost per model, cost per token. Change your targets anytime,' no redeploy, no downtime. Rivvr re-adjusts the cluster automatically.

Monitor performance. Change objectives.

Track live TTFT, TPS, SLO compliance, cost per model, cost per token. Change your targets anytime — no redeploy, no downtime. Tighten an SLO, raise a cost guardrail, see the cluster adapt.

Live Monitoring

llama-3-70b

TTFT

p50 0msp90 0ms

Tokens / s

p50 0p90 0

Prefill / M tokens

$0.00

Decode / M tokens

$0.00

SLO Compliance0.0%

Built for production
AI agents

Voice AI

Support chat agents

Coding assistants & agents

OCR & document intake

LATENCY-CRITICAL

The Challenge

•

Any TTFT variance is an audible pause; there’s no graceful degradation

•

Static warm pools protect latency, but bill you for capacity you don’t use

With Rivvr

•

Your pipeline stays under 400ms without a dedicated warm pool

•

GPU spend tracks actual call volume, not your peak estimate

Voice AI

Support chat agents

Coding assistants & agents

OCR & document intake

LATENCY-CRITICAL

The Challenge

•

Any TTFT variance is an audible pause; there’s no graceful degradation

•

Static warm pools protect latency, but bill you for capacity you don’t use

With Rivvr

•

Your pipeline stays under 400ms without a dedicated warm pool

•

GPU spend tracks actual call volume, not your peak estimate

Runs in your cloud.
Stays in your cloud.

VPC deployment

Runs inside your AWS account, in a network-isolated environment you control.

VPC deployment

Runs inside your AWS account, in a network-isolated environment you control.

Data privacy

Prompts, responses, logs, and model weights stay in your cloud.

Data privacy

Prompts, responses, logs, and model weights stay in your cloud.

RBAC and access control

RBAC controls who can deploy models, change policies, and view cost or performance data.

RBAC and access control

RBAC controls who can deploy models, change policies, and view cost or performance data.

Flexible tenancy isolation

Use dedicated infrastructure where strict isolation is required or shared capacity where efficiency matters.

Flexible tenancy isolation

Use dedicated infrastructure where strict isolation is required or shared capacity where efficiency matters.

Everything to run inference in production

Bring your own weights

Deploy any vLLM-compatible model from Hugging Face, S3 or your model registry.

Managed endpoints

Every deployment gets a production-ready, OpenAI-compatible endpoint.

Multi-model & Multi-LoRA

Run independent models with separate configurations and SLOs, and share base-model capacity across LoRA adapters.

Batch processing

Scale capacity for asynchronous jobs only when needs.

Bring your own weights

Deploy any vLLM-compatible model from Hugging Face, S3 or your model registry.

Managed endpoints

Every deployment gets a production-ready, OpenAI-compatible endpoint.

Multi-model & Multi-LoRA

Run independent models with separate configurations and SLOs, and share base-model capacity across LoRA adapters.

Batch processing

Scale capacity for asynchronous jobs only when needs.

Frequently Asked Questions

Frequently
Asked Questions

How is this different from SageMaker?

SageMaker gives you infrastructure and autoscaling controls. Rivvr removes the need to operate them: set latency and cost targets, and Rivvr continuously runs the infrastructure for you.

How is this different from AIBrix or NVIDIA Dynamo?

AIBrix and Dynamo are infrastructure software your team deploys, configures and maintains. Rivvr is a managed platform inside your AWS account. Instead of configuring profiling, autoscaling and infrastructure parameters, you specify the outcome: SLO + cost target.

Isn't spot capacity risky for something with an SLO?

Spot capacity is interruptible. Rivvr spreads workloads across compatible GPU capacity pools and uses on-demand as fallback, allowing capacity to change without sacrificing the SLO.

Where does Rivvr run?

Inside your AWS account, behind your VPC. Inference traffic, data, and model weights never leave your environment.

How is Rivvr priced?

You pay AWS directly for infrastructure. Rivvr charges a separate management fee based on GPUs under orchestration.

Do I need to change my application?

Usually not. Rivvr exposes an OpenAI-compatible API, so most teams only change the endpoint.

Does Rivvr modify my model?

No. Model weights remain unchanged. Rivvr optimizes orchestration, placement and infrastructure.

What models are supported?

Any LLM supported by vLLM, currently up to 400B parameters.

What runtime does Rivvr use?

A vLLM-compatible runtime.

What GPUs are supported?

NVIDIA CUDA compute capability 7.0+, including L4, L40S, A10G, T4, A100, H100, H200, B200, B300 and V100.

Managed LLM Inference
Inside Your AWS