Scaling and concurrency

Capacity is containers times what each container handles at once, and both start at one. Without scaling settings, a .map() over 500 inputs runs them one after another. Per-kind defaults are in Resources and options.

Concurrency inside one container

Functions

concurrency is how many calls one container serves at once. Each call gets its own process, so CPU-bound calls run in parallel and a crash stays contained. in_process=True runs them as threads in one interpreter instead, so a model loaded once in on_start serves every slot:

from lazycloud import App, GpuType

app = App("inference")


@app.function(gpu=GpuType.L4, memory="8Gi", concurrency=4, in_process=True)
def classify(image_bytes: bytes) -> str:
    ...

Threads share the GIL, so this pays off when calls spend their time in a GPU kernel, a network call, or a C extension. Cancelling one call in a shared interpreter stops the container, and the other calls in it retry.

Endpoints and ASGI apps

A container runs workers server processes, and each takes concurrency requests at once, or concurrent_requests on ASGI and realtime. Workers don’t share memory, so each loads its own copy of a model in on_start.

@app.endpoint(cpu=2, memory="1Gi", workers=2, concurrency=16)
def lookup(key: str) -> dict[str, str]:
    ...

A GPU model fits workers=1 with higher concurrency. A CPU-bound handler fits one worker per CPU and a concurrency of one or two. Async and I/O-bound code takes a high concurrency in one worker.

More containers

Autoscaler sets how many containers a workload may run:

from lazycloud import Autoscaler


@app.function(
    cpu=2,
    memory="2Gi",
    concurrency=4,
    max_pending_tasks=1000,
    autoscaler=Autoscaler(max_containers=20, tasks_per_container=4),
)
def transform(path: str) -> int:
    ...

The autoscaler runs one container per tasks_per_container queued calls, up to max_containers. It fits best at the container’s real concurrency, which is concurrency for a function and workers * concurrency for an endpoint. Here, 500 inputs from .map() run on 20 containers, 4 at a time each. Without an autoscaler, a workload runs at most one container. With one, tasks_per_container defaults to 1, so it’s worth setting. Pods scale by hand with lazycloud deployment scale. See Pods.

Warm containers

An idle container stays up for keep_warm seconds so the next call skips the start, which can take a minute for a model server. Functions default to 10 seconds, endpoints to 180, and scheduled functions to 0. You pay for warm containers while they run. min_containers keeps containers running even with no work, so the first call after a quiet hour doesn’t wait:

@app.endpoint(
    gpu=GpuType.L4,
    memory="16Gi",
    concurrency=8,
    keep_warm=600,
    autoscaler=Autoscaler(min_containers=1, max_containers=4, tasks_per_container=8),
)
def generate(prompt: str) -> dict[str, str]:
    ...

When the queue is full

max_pending_tasks, 100 by default, caps queued and running calls per workload. Past it, a function call fails with a capacity error and an endpoint returns 429. .map() submits every input up front, so a batch of 500 needs a cap of at least 500. A queued function call waits up to its timeout_seconds for a container, and an endpoint request waits up to its timeout_seconds, 180 by default, for a free slot. After five seconds, the CLI and dashboard show why a call is waiting.

Patterns

Batch over many inputs

@app.function(
    cpu=1,
    memory="1Gi",
    timeout_seconds=600,
    retries=2,
    max_pending_tasks=5000,
    autoscaler=Autoscaler(max_containers=50, tasks_per_container=1),
)
def process_page(url: str) -> int:
    ...


results = list(process_page.map([(url,) for url in urls]))

One call per container keeps a crash or an out-of-memory kill to one input.

GPU model over HTTP

MODEL = {}


def load_model() -> None:
    MODEL["embedder"] = ...  # load weights once per worker process


@app.endpoint(
    gpu=GpuType.A10G,
    memory="16Gi",
    workers=1,
    concurrency=8,
    keep_warm=900,
    on_start=load_model,
    autoscaler=Autoscaler(min_containers=1, max_containers=6, tasks_per_container=8),
)
def embed(texts: list[str]) -> list[list[float]]:
    ...

One worker loads the model once, eight requests share it, and one container stays warm for the first request of the day.

CPU API with steady traffic

service = app.asgi(
    name="api",
    cpu=4,
    memory="2Gi",
    workers=4,
    concurrent_requests=32,
    autoscaler=Autoscaler(max_containers=10, tasks_per_container=128),
)(fastapi_app)

Four workers on four CPUs, 32 async requests each, and a new container per 128 in flight.

Long-running job

@app.function(cpu=4, memory="16Gi", timeout_seconds=6 * 3600, retries=0, preemptible=False)
def train(config: dict[str, float]) -> str:
    ...

retries=0 avoids repeating six hours of work, and preemptible=False keeps the machine from being reclaimed mid-run. Checkpoints in a volume let a new run resume.

Getting started

Core concepts

CLI reference

Examples

Account

Scaling and concurrency

Concurrency inside one container

Functions

Endpoints and ASGI apps

More containers

Warm containers

When the queue is full

Patterns

Batch over many inputs

GPU model over HTTP

CPU API with steady traffic

Long-running job

​Concurrency inside one container

​Functions

​Endpoints and ASGI apps

​More containers

​Warm containers

​When the queue is full

​Patterns

​Batch over many inputs

​GPU model over HTTP

​CPU API with steady traffic

​Long-running job

Concurrency inside one container

Functions

Endpoints and ASGI apps

More containers

Warm containers

When the queue is full

Patterns

Batch over many inputs

GPU model over HTTP

CPU API with steady traffic

Long-running job