Queues and maps

A queue passes work items between processes, and a map holds temporary shared values. Within a workspace, a queue or map with the same name refers to the same data.

Queue

Queue is a FIFO queue of JSON-serializable values:

from lazycloud import Queue

q = Queue("work-items")

q.put({"path": "a.wav"})
print(len(q))
print(q.peek())
item = q.pop()
print(item)

pop() hands each item to exactly one caller, or returns None when empty. Work that needs results and retries fits a function call better.

Coordinate containers

.map() fits inputs known up front. A queue fits work discovered along the way, drained by a few long-lived workers, with a map to share progress:

from lazycloud import App, Autoscaler, Map, Queue

app = App("crawler")
frontier = Queue("crawl-frontier")
seen = Map("crawl-seen")


@app.function(
    cpu=1,
    memory="512Mi",
    timeout_seconds=3600,
    retries=0,
    autoscaler=Autoscaler(max_containers=8),
)
def crawl_worker() -> int:
    handled = 0
    while (url := frontier.pop()) is not None:
        if url in seen:
            continue
        seen.set(url, True, ttl=86400)
        for link in fetch_links(url):   # your fetch code
            frontier.put(link)
        handled += 1
    return handled


if __name__ == "__main__":
    frontier.put("https://example.com/")
    calls = crawl_worker.spawn_map([() for _ in range(8)])
    print(sum(call.get() for call in calls))

Eight workers drain one queue in parallel. The in check and set are two calls, so two workers can occasionally fetch the same URL. A crawler shrugs that off, and a payment can’t, so exactly-once work belongs in an idempotent handler or a function call. A worker that crashes after pop() loses the item, since the queue has no acknowledgement.

Map

Map is a key/value store with the MutableMapping interface:

from lazycloud import Map

state = Map("pipeline-state")

state["cursor"] = 42
state.get("cursor")
state.set("cursor", 43, ttl=3600)   # per-key TTL in seconds
"cursor" in state
list(state.keys())
len(state)
del state["cursor"]

Keys expire after seven days, the longest timed expiry, and ttl=0 keeps a key until deleted.

Edit from the dashboard

The dashboard inspects and edits both by hand.

Delete a queue or map

q.delete() removes the queue and its remaining items. state.delete() removes the map and all its keys.

Getting started

Core concepts

CLI reference

Examples

Account

Queue

Coordinate containers

Map

Edit from the dashboard

Delete a queue or map

​Queue

​Coordinate containers

​Map

​Edit from the dashboard

​Delete a queue or map

Queue

Coordinate containers

Map

Edit from the dashboard

Delete a queue or map