> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lazycloud.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenAI-compatible LLM service

> Serve a small instruction model from an authenticated vLLM pod on one GPU.

The model is `Qwen/Qwen2.5-1.5B-Instruct` on an NVIDIA L4, served by the
official vLLM OpenAI server. A volume keeps the downloaded model files, so a
replacement container reuses them instead of downloading again.

`app.py` in the downloaded project defines the pod and its resources.

The CLI installs once with `uv tool install lazycloud-client`. The commands
below download a standalone project and install its dependencies, with no
repository checkout. The rest of the guide runs from that directory through
`uv run`, which uses the project's pinned SDK.

```bash theme={null}
lazycloud example download openai-compatible-llm
cd openai-compatible-llm
uv sync
uv run lazycloud login
```

## Prerequisites

* Check that your account can run an L4 and has credit. The pod holds a GPU
  for as long as it runs. Stop it after your requests if you do not need a
  server.
* Create an access token in the dashboard to call the service, and export it:

```bash theme={null}
read -rsp "LazyCloud token: " LAZYCLOUD_TOKEN; echo
export LAZYCLOUD_TOKEN
```

The model is public, so you do not need a Hugging Face token.

## The pod

The pod runs the pinned `vllm/vllm-openai:v0.23.0` image. vLLM, PyTorch, and
CUDA stay in that image. Nothing installs on your machine.

```python theme={null}
from lazycloud import App, GpuType, Image, Volume

app = App("openai_compatible_llm")
model_cache = Volume("vllm-model-cache", "/root/.cache/huggingface")

server = app.pod(
    name="openai-server",
    image=Image.from_registry("vllm/vllm-openai:v0.23.0"),
    command=[
        "vllm", "serve",
        "--host", "0.0.0.0",
        "--port", "8000",
        "--model", "Qwen/Qwen2.5-1.5B-Instruct",
        "--served-model-name", "qwen-1.5b",
        "--download-dir", "/root/.cache/huggingface",
        "--max-model-len", "8192",
        "--gpu-memory-utilization", "0.90",
    ],
    ports={"http": 8000},
    env={"HF_HOME": "/root/.cache/huggingface"},
    cpu=4.0,
    memory="16Gi",
    gpu=GpuType.L4,
    gpu_count=1,
    volumes=[model_cache],
    authorized=True,
)
```

`authorized=True` makes LazyCloud require a bearer token before a request
reaches vLLM. The example sets no separate vLLM API key.

`volumes=[model_cache]` creates or reuses `vllm-model-cache` in your workspace
when the pod starts. The volume name and mount path live in this definition,
with no separate storage setup.

## Deploy

```bash theme={null}
uv run lazycloud deploy app:server
```

The first start takes longer because it downloads the model. Copy the pod URL
from the deploy output, without a trailing slash:

```bash theme={null}
export VLLM_URL="<pod-url>"
```

Check whether the server has loaded the model:

```bash theme={null}
curl --fail --show-error "$VLLM_URL/health" \
  -H "Authorization: Bearer $LAZYCLOUD_TOKEN"
```

LazyCloud rejects a request without the header before it reaches vLLM. If
health fails during startup, look for download or GPU errors in
`uv run lazycloud logs --deployment openai-server -n 50`, then check health
again.

## Use it

```bash theme={null}
curl --fail --show-error "$VLLM_URL/v1/models" \
  -H "Authorization: Bearer $LAZYCLOUD_TOKEN"
```

```bash theme={null}
curl --fail --show-error "$VLLM_URL/v1/chat/completions" \
  -H "Authorization: Bearer $LAZYCLOUD_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-1.5b",
    "messages": [
      {"role": "user", "content": "Explain durable model caches in one sentence."}
    ],
    "max_tokens": 80,
    "temperature": 0.2
  }'
```

Read the answer in `choices[0].message.content`. For an OpenAI-compatible
client, use `$VLLM_URL/v1` as the base URL, your LazyCloud token as the API
key, and `qwen-1.5b` as the model.

## Troubleshoot a request

```bash theme={null}
uv run lazycloud deployment list --app openai_compatible_llm
uv run lazycloud logs --deployment openai-server --follow --show-timestamp
```

Check these logs when startup or a request fails. They show model-loading and
GPU errors. A request without a bearer token gets an authorization error.

## Cost and licensing

The pod holds an L4 while it runs and bills GPU, CPU, and memory time until you
stop or delete the deployment. Check the Usage page before leaving it up. The
1.5B model fits on one L4, but concurrency and context length still affect GPU
memory.

* vLLM publishes the server image and its
  [Docker instructions](https://docs.vllm.ai/en/latest/deployment/docker/).
* The [Hugging Face model card](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct)
  lists `Qwen/Qwen2.5-1.5B-Instruct` under the Apache 2.0 license. Read the
  card before using the weights in a product.

## Clean up

To stop GPU charges while keeping the deployment and cache:

```bash theme={null}
uv run lazycloud deployment stop openai-server
```

Restart with `uv run lazycloud deployment start openai-server`. To remove it entirely,
delete the deployment first. Delete the cache only if this example created it
and you no longer need the model files:

```bash theme={null}
uv run lazycloud deployment delete openai-server
uv run lazycloud volume delete vllm-model-cache
unset VLLM_URL
```

Keep the volume if you want the next deployment to reuse the download.
LazyCloud bills the volume for as long as it exists.