> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lazycloud.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Process Parquet files in parallel

> Remote Python over the Parquet files in your own S3 bucket, mounted as a directory.

Process one Parquet file per task, then combine the results into a JSON summary
in your bucket. The batch fails if any partition fails, so a partial run never
overwrites the summary.

Source is `parallel_parquet/app.py`. PyArrow installs in the remote image, and
your machine only runs the coordinator.

The CLI installs once with `uv tool install lazycloud-client`. The commands
below download a standalone project and install its dependencies, with no
repository checkout. The rest of the guide runs from that directory through
`uv run`, which uses the project's pinned SDK.

```bash theme={null}
lazycloud example download parallel-parquet
cd parallel-parquet
uv sync
uv run lazycloud login
```

## Choose the input and output paths

Use a bucket you own and a new prefix for the first run:

```bash theme={null}
export LAZYCLOUD_PARQUET_BUCKET="my-analytics-bucket"
export LAZYCLOUD_PARQUET_REGION="us-east-1"
export LAZYCLOUD_PARQUET_INPUT_PREFIX="examples/parallel-parquet/first-run/input"
export LAZYCLOUD_PARQUET_OUTPUT_KEY="examples/parallel-parquet/first-run/output/summary.json"
```

Replace the bucket and region with your own. The bucket must already exist. A
successful run writes the output key and replaces any summary already there.
Keep that key outside the input prefix, and pick a new key to preserve earlier
results.

Paths must be relative object keys without `..`, backslashes, or empty path
segments. The output key must end in `.json`.

## Give the functions bucket access

Store an access key pair with permission to list and read the input prefix
and write the output key. The seed step also needs permission to write its
input files.

```bash theme={null}
read -rsp "S3 access key ID: " PARQUET_S3_ACCESS_KEY_ID; echo
read -rsp "S3 secret access key: " PARQUET_S3_SECRET_ACCESS_KEY; echo
export PARQUET_S3_ACCESS_KEY_ID PARQUET_S3_SECRET_ACCESS_KEY

export LAZYCLOUD_PARQUET_ACCESS_KEY_SECRET="PARQUET_S3_ACCESS_KEY_ID"
export LAZYCLOUD_PARQUET_SECRET_KEY_SECRET="PARQUET_S3_SECRET_ACCESS_KEY"

uv run python -m parallel_parquet.configure
unset PARQUET_S3_ACCESS_KEY_ID PARQUET_S3_SECRET_ACCESS_KEY
```

`configure.py` reads the two credential values from your environment and uses
`Secret(...).set(...)` to store them under the names in the app's configuration.
Running it again replaces those values. The `LAZYCLOUD_PARQUET_*` variables
hold bucket settings and secret names. Keep them exported in the shell that
runs the batch.

The app declares one `CloudBucket` and attaches it to all four functions with
`volumes=[data_bucket]`. Edit that definition to change the mount path or
access mode.

For another S3-compatible store, also set its HTTPS endpoint:

```bash theme={null}
export LAZYCLOUD_PARQUET_ENDPOINT="https://objects.example.com"
```

The endpoint cannot include credentials or a query string.

If the workspace lives in a connected AWS account, you can omit both
secret-name variables and use the instance role. Grant the role the same bucket
permissions. See [cloud buckets](/concepts/volumes#mount-an-existing-cloud-bucket).

## Run with sample data

```bash theme={null}
uv run lazycloud run parallel_parquet.app:run_batch true 4 10 false
```

The arguments tell the batch to seed the input, write four partitions of ten
rows each, and refuse to overwrite existing seed files. The coordinator runs
locally and submits the remote functions as needed, so you do not need a
separate deployment.

The result includes `partition_count: 4`, `row_count: 40`, the total
`amount_cents`, per-partition counts, task IDs, and the output key.
The functions read and write through the same `CloudBucket` mount.

## Process your own files

Each input file needs `record_id`, `amount_cents`, and `category` columns.
Set the input prefix to those files and run without seeding:

```bash theme={null}
uv run lazycloud run parallel_parquet.app:run_batch false
```

The coordinator calls `spawn_map()` to start one task per partition and checks
every result before it calls the summary writer.

To rerun the sample on input that already exists, pass `false` as well. For a
fresh sample, use a new prefix. Passing `true` as the fourth argument lets the
run replace seed files, so use it only on disposable data.

## Read the summary

With the AWS CLI configured for your bucket:

```bash theme={null}
aws s3 cp \
  "s3://$LAZYCLOUD_PARQUET_BUCKET/$LAZYCLOUD_PARQUET_OUTPUT_KEY" - \
  --region "$LAZYCLOUD_PARQUET_REGION"
```

For another S3-compatible store, add
`--endpoint-url "$LAZYCLOUD_PARQUET_ENDPOINT"`. The AWS CLI uses its own
credentials, not the LazyCloud workspace secrets.

## Investigate a failed partition

The coordinator reports failed task IDs and leaves the previous summary
unchanged. Read the named task's error and logs:

```bash theme={null}
uv run lazycloud task show <task-id>
uv run lazycloud task logs <task-id>
```

Fix the input or bucket access, then rerun without seeding.

## Clean up sample data

Function containers shut down after their idle window. Your bucket keeps the
input and summary, and its storage and request charges continue.

List what the input prefix contains first:

```bash theme={null}
aws s3 ls "s3://$LAZYCLOUD_PARQUET_BUCKET/$LAZYCLOUD_PARQUET_INPUT_PREFIX/" \
  --recursive --region "$LAZYCLOUD_PARQUET_REGION"
```

For the four-file sample seeded above only, remove its exact seed keys and the
summary:

```bash theme={null}
for index in 0 1 2 3; do
  key="$(printf '%s/part-%03d.parquet' "$LAZYCLOUD_PARQUET_INPUT_PREFIX" "$index")"
  aws s3 rm "s3://$LAZYCLOUD_PARQUET_BUCKET/$key" --region "$LAZYCLOUD_PARQUET_REGION"
done
aws s3 rm "s3://$LAZYCLOUD_PARQUET_BUCKET/$LAZYCLOUD_PARQUET_OUTPUT_KEY" \
  --region "$LAZYCLOUD_PARQUET_REGION"
```

Before deleting, compare the listed filenames with the seed result. Keep your
own datasets and any summary you still need. For another S3-compatible store,
add `--endpoint-url` to these commands too.

If you created the two workspace secrets only for this example, delete them:

```bash theme={null}
uv run lazycloud secret delete PARQUET_S3_ACCESS_KEY_ID
uv run lazycloud secret delete PARQUET_S3_SECRET_ACCESS_KEY
unset LAZYCLOUD_PARQUET_BUCKET LAZYCLOUD_PARQUET_REGION
unset LAZYCLOUD_PARQUET_INPUT_PREFIX LAZYCLOUD_PARQUET_OUTPUT_KEY
unset LAZYCLOUD_PARQUET_ENDPOINT
unset LAZYCLOUD_PARQUET_ACCESS_KEY_SECRET LAZYCLOUD_PARQUET_SECRET_KEY_SECRET
```