> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lazycloud.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Scaling and concurrency

> Concurrency, autoscaling, idle windows, and queue limits, with sizing patterns for batch jobs, GPU models, and APIs.

Capacity is containers times what each container handles at once, and both
start at one. Without scaling settings, a `.map()` over 500 inputs runs them one
after another. Per-kind defaults are in
[Resources and options](/concepts/resources#defaults).

## Concurrency inside one container

### Functions

`concurrency` is how many calls one container serves at once. Each call gets
its own process, so CPU-bound calls run in parallel and a crash stays contained.
`in_process=True` runs them as threads in one interpreter instead, so a model
loaded once in `on_start` serves every slot:

```python theme={null}
from lazycloud import App, GpuType

app = App("inference")


@app.function(gpu=GpuType.L4, memory="8Gi", concurrency=4, in_process=True)
def classify(image_bytes: bytes) -> str:
    ...
```

Threads share the GIL, so this pays off when calls spend their time in a GPU
kernel, a network call, or a C extension. Cancelling one call in a shared
interpreter stops the container, and the other calls in it retry.

### Endpoints and ASGI apps

A container runs `workers` server processes, and each takes `concurrency`
requests at once, or `concurrent_requests` on ASGI and realtime. Workers don't
share memory, so each loads its own copy of a model in `on_start`.

```python theme={null}
@app.endpoint(cpu=2, memory="1Gi", workers=2, concurrency=16)
def lookup(key: str) -> dict[str, str]:
    ...
```

A GPU model fits `workers=1` with higher `concurrency`. A CPU-bound handler fits
one worker per CPU and a `concurrency` of one or two. Async and I/O-bound code
takes a high `concurrency` in one worker.

## More containers

`Autoscaler` sets how many containers a workload may run:

```python theme={null}
from lazycloud import Autoscaler


@app.function(
    cpu=2,
    memory="2Gi",
    concurrency=4,
    max_pending_tasks=1000,
    autoscaler=Autoscaler(max_containers=20, tasks_per_container=4),
)
def transform(path: str) -> int:
    ...
```

The autoscaler runs one container per `tasks_per_container` queued calls, up to
`max_containers`. It fits best at the container's real concurrency, which is
`concurrency` for a function and `workers * concurrency` for an endpoint. Here,
500 inputs from `.map()` run on 20 containers, 4 at a time each.

Without an `autoscaler`, a workload runs at most one container. With one,
`tasks_per_container` defaults to `1`, so it's worth setting. Pods scale by hand
with `lazycloud deployment scale`. See [Pods](/concepts/pods).

## Warm containers

An idle container stays up for `keep_warm` seconds so the next call skips the
start, which can take a minute for a model server. Functions default to 10
seconds, endpoints to 180, and scheduled functions to 0. You pay for warm
containers while they run.

`min_containers` keeps containers running even with no work, so the first call
after a quiet hour doesn't wait:

```python theme={null}
@app.endpoint(
    gpu=GpuType.L4,
    memory="16Gi",
    concurrency=8,
    keep_warm=600,
    autoscaler=Autoscaler(min_containers=1, max_containers=4, tasks_per_container=8),
)
def generate(prompt: str) -> dict[str, str]:
    ...
```

## When the queue is full

`max_pending_tasks`, 100 by default, caps queued and running calls per
workload. Past it, a function call fails with a capacity error and an endpoint
returns `429`. `.map()` submits every input up front, so a batch of 500 needs a
cap of at least 500.

A queued function call waits up to its `timeout_seconds` for a container, and
an endpoint request waits up to its `timeout_seconds`, 180 by default, for a
free slot. After five seconds, the CLI and dashboard show
[why a call is waiting](/concepts/tasks-and-logs#why-a-task-is-pending).

## Patterns

### Batch over many inputs

```python theme={null}
@app.function(
    cpu=1,
    memory="1Gi",
    timeout_seconds=600,
    retries=2,
    max_pending_tasks=5000,
    autoscaler=Autoscaler(max_containers=50, tasks_per_container=1),
)
def process_page(url: str) -> int:
    ...


results = list(process_page.map([(url,) for url in urls]))
```

One call per container keeps a crash or an out-of-memory kill to one input.

### GPU model over HTTP

```python theme={null}
MODEL = {}


def load_model() -> None:
    MODEL["embedder"] = ...  # load weights once per worker process


@app.endpoint(
    gpu=GpuType.A10G,
    memory="16Gi",
    workers=1,
    concurrency=8,
    keep_warm=900,
    on_start=load_model,
    autoscaler=Autoscaler(min_containers=1, max_containers=6, tasks_per_container=8),
)
def embed(texts: list[str]) -> list[list[float]]:
    ...
```

One worker loads the model once, eight requests share it, and one container
stays warm for the first request of the day.

### CPU API with steady traffic

```python theme={null}
service = app.asgi(
    name="api",
    cpu=4,
    memory="2Gi",
    workers=4,
    concurrent_requests=32,
    autoscaler=Autoscaler(max_containers=10, tasks_per_container=128),
)(fastapi_app)
```

Four workers on four CPUs, 32 async requests each, and a new container per 128
in flight.

### Long-running job

```python theme={null}
@app.function(cpu=4, memory="16Gi", timeout_seconds=6 * 3600, retries=0, preemptible=False)
def train(config: dict[str, float]) -> str:
    ...
```

`retries=0` avoids repeating six hours of work, and `preemptible=False` keeps
the machine from being reclaimed mid-run. Checkpoints in a
[volume](/concepts/volumes) let a new run resume.