Skip to main content
Process one Parquet file per task, then combine the results into a JSON summary in your bucket. The batch fails if any partition fails, so a partial run never overwrites the summary. Source is parallel_parquet/app.py. PyArrow installs in the remote image, and your machine only runs the coordinator. The CLI installs once with uv tool install lazycloud-client. The commands below download a standalone project and install its dependencies, with no repository checkout. The rest of the guide runs from that directory through uv run, which uses the project’s pinned SDK.

​
Choose the input and output paths

Use a bucket you own and a new prefix for the first run:
Replace the bucket and region with your own. The bucket must already exist. A successful run writes the output key and replaces any summary already there. Keep that key outside the input prefix, and pick a new key to preserve earlier results. Paths must be relative object keys without .., backslashes, or empty path segments. The output key must end in .json.

​
Give the functions bucket access

Store an access key pair with permission to list and read the input prefix and write the output key. The seed step also needs permission to write its input files.
configure.py reads the two credential values from your environment and uses Secret(...).set(...) to store them under the names in the app’s configuration. Running it again replaces those values. The LAZYCLOUD_PARQUET_* variables hold bucket settings and secret names. Keep them exported in the shell that runs the batch. The app declares one CloudBucket and attaches it to all four functions with volumes=[data_bucket]. Edit that definition to change the mount path or access mode. For another S3-compatible store, also set its HTTPS endpoint:
The endpoint cannot include credentials or a query string. If the workspace lives in a connected AWS account, you can omit both secret-name variables and use the instance role. Grant the role the same bucket permissions. See cloud buckets.

​
Run with sample data

The arguments tell the batch to seed the input, write four partitions of ten rows each, and refuse to overwrite existing seed files. The coordinator runs locally and submits the remote functions as needed, so you do not need a separate deployment. The result includes partition_count: 4, row_count: 40, the total amount_cents, per-partition counts, task IDs, and the output key. The functions read and write through the same CloudBucket mount.

​
Process your own files

Each input file needs record_id, amount_cents, and category columns. Set the input prefix to those files and run without seeding:
The coordinator calls spawn_map() to start one task per partition and checks every result before it calls the summary writer. To rerun the sample on input that already exists, pass false as well. For a fresh sample, use a new prefix. Passing true as the fourth argument lets the run replace seed files, so use it only on disposable data.

​
Read the summary

With the AWS CLI configured for your bucket:
For another S3-compatible store, add --endpoint-url "$LAZYCLOUD_PARQUET_ENDPOINT". The AWS CLI uses its own credentials, not the LazyCloud workspace secrets.

​
Investigate a failed partition

The coordinator reports failed task IDs and leaves the previous summary unchanged. Read the named task’s error and logs:
Fix the input or bucket access, then rerun without seeding.

​
Clean up sample data

Function containers shut down after their idle window. Your bucket keeps the input and summary, and its storage and request charges continue. List what the input prefix contains first:
For the four-file sample seeded above only, remove its exact seed keys and the summary:
Before deleting, compare the listed filenames with the seed result. Keep your own datasets and any summary you still need. For another S3-compatible store, add --endpoint-url to these commands too. If you created the two workspace secrets only for this example, delete them: