OpenAI-compatible LLM service

The model is Qwen/Qwen2.5-1.5B-Instruct on an NVIDIA L4, served by the official vLLM OpenAI server. A volume keeps the downloaded model files, so a replacement container reuses them instead of downloading again. app.py in the downloaded project defines the pod and its resources. The CLI installs once with uv tool install lazycloud-client. The commands below download a standalone project and install its dependencies, with no repository checkout. The rest of the guide runs from that directory through uv run, which uses the project’s pinned SDK.

lazycloud example download openai-compatible-llm
cd openai-compatible-llm
uv sync
uv run lazycloud login

Prerequisites

Check that your account can run an L4 and has credit. The pod holds a GPU for as long as it runs. Stop it after your requests if you do not need a server.
Create an access token in the dashboard to call the service, and export it:

read -rsp "LazyCloud token: " LAZYCLOUD_TOKEN; echo
export LAZYCLOUD_TOKEN

The model is public, so you do not need a Hugging Face token.

The pod

The pod runs the pinned vllm/vllm-openai:v0.23.0 image. vLLM, PyTorch, and CUDA stay in that image. Nothing installs on your machine.

from lazycloud import App, GpuType, Image, Volume

app = App("openai_compatible_llm")
model_cache = Volume("vllm-model-cache", "/root/.cache/huggingface")

server = app.pod(
    name="openai-server",
    image=Image.from_registry("vllm/vllm-openai:v0.23.0"),
    command=[
        "vllm", "serve",
        "--host", "0.0.0.0",
        "--port", "8000",
        "--model", "Qwen/Qwen2.5-1.5B-Instruct",
        "--served-model-name", "qwen-1.5b",
        "--download-dir", "/root/.cache/huggingface",
        "--max-model-len", "8192",
        "--gpu-memory-utilization", "0.90",
    ],
    ports={"http": 8000},
    env={"HF_HOME": "/root/.cache/huggingface"},
    cpu=4.0,
    memory="16Gi",
    gpu=GpuType.L4,
    gpu_count=1,
    volumes=[model_cache],
    authorized=True,
)

authorized=True makes LazyCloud require a bearer token before a request reaches vLLM. The example sets no separate vLLM API key. volumes=[model_cache] creates or reuses vllm-model-cache in your workspace when the pod starts. The volume name and mount path live in this definition, with no separate storage setup.

Deploy

uv run lazycloud deploy app:server

The first start takes longer because it downloads the model. Copy the pod URL from the deploy output, without a trailing slash:

export VLLM_URL="<pod-url>"

Check whether the server has loaded the model:

curl --fail --show-error "$VLLM_URL/health" \
  -H "Authorization: Bearer $LAZYCLOUD_TOKEN"

LazyCloud rejects a request without the header before it reaches vLLM. If health fails during startup, look for download or GPU errors in uv run lazycloud logs --deployment openai-server -n 50, then check health again.

Use it

curl --fail --show-error "$VLLM_URL/v1/models" \
  -H "Authorization: Bearer $LAZYCLOUD_TOKEN"

curl --fail --show-error "$VLLM_URL/v1/chat/completions" \
  -H "Authorization: Bearer $LAZYCLOUD_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-1.5b",
    "messages": [
      {"role": "user", "content": "Explain durable model caches in one sentence."}
    ],
    "max_tokens": 80,
    "temperature": 0.2
  }'

Read the answer in choices[0].message.content. For an OpenAI-compatible client, use $VLLM_URL/v1 as the base URL, your LazyCloud token as the API key, and qwen-1.5b as the model.

Troubleshoot a request

uv run lazycloud deployment list --app openai_compatible_llm
uv run lazycloud logs --deployment openai-server --follow --show-timestamp

Check these logs when startup or a request fails. They show model-loading and GPU errors. A request without a bearer token gets an authorization error.

Cost and licensing

The pod holds an L4 while it runs and bills GPU, CPU, and memory time until you stop or delete the deployment. Check the Usage page before leaving it up. The 1.5B model fits on one L4, but concurrency and context length still affect GPU memory.

vLLM publishes the server image and its Docker instructions.
The Hugging Face model card lists Qwen/Qwen2.5-1.5B-Instruct under the Apache 2.0 license. Read the card before using the weights in a product.

Clean up

To stop GPU charges while keeping the deployment and cache:

uv run lazycloud deployment stop openai-server

Restart with uv run lazycloud deployment start openai-server. To remove it entirely, delete the deployment first. Delete the cache only if this example created it and you no longer need the model files:

uv run lazycloud deployment delete openai-server
uv run lazycloud volume delete vllm-model-cache
unset VLLM_URL

Keep the volume if you want the next deployment to reuse the download. LazyCloud bills the volume for as long as it exists.

Getting started

Core concepts

CLI reference

Examples

Account

OpenAI-compatible LLM service

Prerequisites

The pod

Deploy

Use it

Troubleshoot a request

Cost and licensing

Clean up

​Prerequisites

​The pod

​Deploy

​Use it

​Troubleshoot a request

​Cost and licensing

​Clean up

Prerequisites

The pod

Deploy

Use it

Troubleshoot a request

Cost and licensing

Clean up