photocli/README.md

# photoscli

`photoscli` is a macOS-only command-line exporter for Apple Photos. It uses a small Objective-C PhotoKit bridge from Go to list albums, inspect assets, export optimized previews or originals, keep resumable manifests, and produce machine-readable logs for backup automation.

The tool is designed for repeatable, resumable Photos backups rather than one-off drag-and-drop exports.

For a practical step-by-step manual with recommended backup workflows, recovery steps, automation examples, and troubleshooting, see `USERGUIDE.md`.

## Highlights

- List albums, photos, and the Photos folder/album tree.
- Export one album or back up the whole Photos tree.
- Export JPEG previews with configurable size and quality, or original files.
- Skip already-exported assets through a JSONL or SQLite manifest.
- Resume interrupted runs safely.
- Parallel export with live worker progress and ETA.
- Optional structured logging to `export.log` or SQLite `logs` table.
- Dry-run mode for planning large backups.
- Filters for favorites, media type, date, estimated size, and excluded albums.
- Failure tracking with `failures.jsonl` and `retry-failed`.
- Deduplicated failure management with `failures list`, `failures clear`, and `retry-failed --clear-on-success`.
- Verification, reporting, and diff commands for backup integrity.
- Optional XMP sidecar verification with `verify --sidecar`.
- Optional SHA-256 manifest checksums with `--checksum sha256`.
- Metadata-only XMP refresh for manifest-backed exports with `--metadata-only`.
- Status command for quick backup summaries.
- Opt-in rich XMP sidecar metadata with `--sidecar xmp`.
- Optional Apple MapKit reverse geocoding for GPS assets on macOS 26+ with `--reverse-geocode`.
- Script-friendly exit codes and optional JSON summaries.
- 100% test coverage for the Go CLI and parsing layers.

## Requirements

- Apple Silicon Mac, M1/M2/M3/M4 or newer (`darwin/arm64`).
- macOS with Apple Photos.
- Go 1.25 or newer.
- Xcode command-line tools.
- Photos privacy permission for the built binary or terminal app.

The provided release binary is Apple Silicon only. Intel Macs are not currently a supported release target. The production build uses cgo and links against Apple frameworks through the Objective-C bridge in `bridge/`.

## Release Assets

Release zip files are named with the supported architecture:

```text
photoscli-<version>-macos-arm64.zip
```

The zip contains the Apple Silicon binary plus README, USERGUIDE, and CHANGELOG.

## Build

```bash
make build
```

The binary is written to:

```bash
./bin/photoscli
```

Run the full local release pipeline:

```bash
make pipeline
```

The pipeline cleans artifacts, builds the test bridge, runs vet, runs race-enabled tests with coverage, builds the real PhotoKit bridge, builds the CLI, and verifies the embedded version.

## Quick Start

List albums:

```bash
./bin/photoscli albums
```

Inspect an album by title or PhotoKit local identifier:

```bash
./bin/photoscli photos --album-id "Vacation"
```

Export preview JPEGs from one album:

```bash
./bin/photoscli export --album-id "Vacation" --out ./export
```

Export higher-quality previews:

```bash
./bin/photoscli export --album-id "Vacation" --out ./export --size 2048 --quality 92
```

Export originals:

```bash
./bin/photoscli export --album-id "Vacation" --out ./originals --originals
```

Back up the entire Photos album tree:

```bash
./bin/photoscli backup-all --out ./photos-backup --manifest sqlite --log
```

Preview what would happen before writing anything:

```bash
./bin/photoscli backup-all --out ./photos-backup --dry-run --json
```

Verify a backup later:

```bash
./bin/photoscli verify --out ./photos-backup --manifest sqlite
./bin/photoscli verify --out ./photos-backup --deep
./bin/photoscli manifest repair --out ./photos-backup --checksum sha256 --dry-run
./bin/photoscli cleanup --out ./photos-backup --dry-run
```

## Commands

### `albums`

Lists user-created albums as tab-separated ID and title.

```bash
photoscli albums
```

Example output:

```text
5E9F.../L0/001	Vacation
8A1B.../L0/001	Work
```

### `photos`

Lists assets in an album. `--album-id` accepts a PhotoKit local identifier or an exact album title.

```bash
photoscli photos --album-id "Vacation"
```

Output includes ID, filename, cloud state, media type, dimensions, optional creation date, optional duration, and a `*` marker for favorites.

### `tree`

Prints the Photos folder and album hierarchy.

```bash
photoscli tree
```

### `export`

Exports assets from one album.

```bash
photoscli export --album-id <id-or-title> --out <dir> [flags]
```

Useful examples:

```bash
photoscli export --album-id "Family" --out ./family --size 2560 --quality 90
photoscli export --album-id "Favorites" --out ./favorites --only-favorites
photoscli export --album-id "Videos" --out ./videos --media videos --include-videos
photoscli export --album-id "Archive" --out ./archive --originals --retry 3
photoscli export --album-id "Vacation" --out ./vacation --date-template YYYY/MM/DD
photoscli export --album-id "Vacation" --out ./vacation --sidecar xmp --reverse-geocode
```

### `backup-all`

Walks the Photos folder/album tree and exports every album to a matching folder structure.

```bash
photoscli backup-all --out <dir> [flags]
```

Duplicate sibling album names are disambiguated by adding the album ID to the generated folder name.

Useful examples:

```bash
photoscli backup-all --out ./backup --manifest sqlite --log
photoscli backup-all --out ./backup --checksum sha256
photoscli backup-all --out ./backup --exclude-album "Recently Deleted" --exclude-album "Temp*"
photoscli backup-all --out ./backup --since 2024-01-01 --sort newest
photoscli backup-all --out ./backup --concurrency 8 --retry 3
photoscli backup-all --out ./backup --dry-run --json
photoscli backup-all --out ./backup --sidecar xmp --reverse-geocode
```

### `report`

Shows manifest and failure counts for an output directory.

```bash
photoscli report --out ./backup --manifest sqlite
```

Example output:

```text
entries	12345
failures	2
```

### `diff`

Compares an album against the manifest and prints assets missing from the manifest.

```bash
photoscli diff --album-id "Vacation" --out ./backup --manifest jsonl
```

Exit code is `2` when missing assets are found.

### `verify`

Checks that manifest entries have corresponding files on disk.

```bash
photoscli verify --out ./backup --manifest sqlite
```

Exit code is `2` when files are missing.

### `retry-failed`

Retries assets recorded in `failures.jsonl`.

```bash
photoscli retry-failed --out ./backup
```

This is useful after transient iCloud or network failures.

Use `--clear-on-success` to remove successfully retried assets from `failures.jsonl`.

### `failures`

Lists or clears deduplicated failure records.

```bash
photoscli failures list --out ./backup
photoscli failures clear --out ./backup
```

### `status`

Shows a quick backup summary.

```bash
photoscli status --out ./backup --manifest sqlite
photoscli status --out ./backup --manifest sqlite --json
```

## Export Flags

Common flags for `export` and `backup-all`:

- `--out <dir>`: destination directory.
- `--size <px>`: longest-side target for preview export, default `1024`.
- `--quality <1-100>`: JPEG preview quality, default `85`.
- `--originals`: export original files instead of previews.
- `--concurrency <N>`: parallel workers, default `3`, capped by bridge slot count.
- `--retry <N>`: retry failed exports with a small backoff.
- `--dry-run`: print planned exports without writing files, manifests, or logs.
- `--json`: print a machine-readable summary to stdout.
- `--verify`: run manifest/file verification after export.
- `--sidecar` with `verify`: also verify expected `.xmp` sidecars.
- `--log`: enable structured export logging.
- `--manifest jsonl|sqlite`: choose manifest backend, default `jsonl`.
- `--no-manifest`: disable manifest reads/writes.
- `--sort oldest|newest`: asset sort order where supported, default `oldest`.
- `--since <date>`: only include assets on or after `YYYY-MM-DD` or RFC3339 date.
- `--only-favorites`: only export favorite assets.
- `--media photos|videos|all`: media filter, default `photos`.
- `--include-videos`: compatibility flag that includes videos/audio.
- `--min-size <n>`: filter by estimated pixel count.
- `--max-size <n>`: filter by estimated pixel count.
- `--format jpeg|heic|png`: preview format hint. Current bridge output is still the existing preview path; non-JPEG bridge output is future work.
- `--sidecar none|xmp`: write opt-in XMP metadata sidecars next to exported files.
- `--metadata-only`: with `--sidecar xmp`, refresh XMP sidecars for manifest-backed files without exporting media.
- `--reverse-geocode`: with `--sidecar xmp`, add cached Apple MapKit address metadata for GPS assets on macOS 26+.
- `--date-template <template>`: append date folders based on creation date, for example `YYYY/MM/DD`.

`backup-all` also supports:

- `--exclude-album <pattern>`: repeatable exact or glob-style album-name exclusion.

`export` also requires:

- `--album-id <id-or-title>`: PhotoKit local identifier or exact album title.

## Manifests

Manifests prevent re-exporting assets on subsequent runs.

JSONL backend:

```text
downloads.jsonl
```

SQLite backend:

```text
downloads.db
```

The manifest stores asset ID, filename, size, cloud state, and export timestamp. SQLite is useful for very large libraries and for keeping logs in the same database.

Use no manifest only when you intentionally want stateless export behavior:

```bash
photoscli export --album-id "Scratch" --out ./scratch --no-manifest
```

## Logging

Enable logs with:

```bash
photoscli backup-all --out ./backup --log
```

With JSONL/no-manifest mode, logs are written to:

```text
export.log
```

With SQLite manifest mode, logs are written to the `logs` table in `downloads.db`.

Logged events include session start/end, completed exports, skipped assets, and failures. Each event includes timestamp, level, event name, asset ID, album, filename, size, cloud state, duration, and message where available.

## XMP Sidecars

Write archival metadata sidecars with:

```bash
photoscli export --album-id "Vacation" --out ./Vacation --sidecar xmp
photoscli export --album-id "Vacation" --out ./Vacation --sidecar json
photoscli export --album-id "Vacation" --out ./Vacation --sidecar xmp,json
```

Sidecars are opt-in and use the exported file basename:

```text
IMG_0001.jpg -> IMG_0001.xmp
IMG_0001.HEIC -> IMG_0001.xmp
IMG_0001.jpg -> IMG_0001.json
```

The XMP contains photoscli metadata such as asset ID, filenames, album, manifest path, media type, dimensions, favorite state, cloud state, export mode, version, exported timestamp, size, and creation date when available. If `--sidecar xmp` is explicitly selected and the sidecar cannot be written, that asset is treated as failed.

Sidecars also include richer public PhotoKit metadata where available: modification date, duration, hidden state, adjustment state, media subtypes, source type, playback style, burst data, GPS coordinates, adjustment info, structured asset resources, standard XMP dates, EXIF GPS coordinates, favorite rating, and album/folder keywords. Add `--reverse-geocode` to include cached address fields from Apple MapKit for assets with GPS coordinates. Reverse geocoding requires macOS 26 or newer; on older macOS versions the export continues and XMP still includes GPS coordinates.

Control XMP location metadata with `--xmp-privacy keep|strip-location|strip-address`. The default is `keep`. Use `strip-address` to omit reverse-geocoded address fields while keeping GPS coordinates, or `strip-location` to omit both GPS and address fields.

Control generated XMP keywords and ratings with `--xmp-keywords album-path|album|none` and `--xmp-rating favorite|none`. Defaults preserve existing behavior: album/folder keywords and favorite assets mapped to `xmp:Rating="5"`.

Verify generated sidecars with:

```bash
photoscli verify --out ./backup --sidecar
```

For stricter checks against recent photoscli-generated XMP sidecars:

```bash
photoscli verify --out ./backup --sidecar --strict
```

Inspect one generated sidecar with:

```bash
photoscli sidecar inspect ./backup/IMG_0001.xmp
photoscli sidecar inspect ./backup/IMG_0001.xmp --json
```

Refresh sidecars for files already present in a manifest-backed export without rewriting media files:

```bash
photoscli backup-all --out ./backup --sidecar xmp --metadata-only
```

## Failure Tracking

Failed exports are deduplicated by asset ID and stored in:

```text
failures.jsonl
```

Retry them with:

```bash
photoscli retry-failed --out ./backup
```

Clear successful retries automatically:

```bash
photoscli retry-failed --out ./backup --clear-on-success
```

Use `--retry N` during normal exports to handle transient iCloud or network failures automatically.

## Configuration File

Defaults can be loaded from:

```text
~/.photoscli.toml
```

Or from a custom path:

```bash
PHOTOSCLI_CONFIG=/path/to/photoscli.toml photoscli backup-all --out ./backup
```

Supported keys mirror long flag names without the leading dashes:

```toml
size = 2048
quality = 90
concurrency = 8
manifest = "sqlite"
log = true
sort = "newest"
media = "photos"
retry = 3
```

Command-line flags override config-file values.

## Exit Codes

- `0`: success.
- `1`: command/config/runtime error.
- `2`: partial failure, such as some exports failed or diff/verify found missing items.
- `3`: Photos access denied.

## Permissions

On first use, macOS may prompt for Photos access. If access is denied, grant access in:

```text
System Settings > Privacy & Security > Photos
```

Depending on how you launch the binary, macOS may associate the permission with Terminal, iTerm, your shell, or the built executable.

## Development

Run tests with the stub bridge:

```bash
make test
```

Run all package coverage:

```bash
go test -tags=test -race -count=1 -coverprofile=coverage.out ./...
go tool cover -func=coverage.out
```

Run the release pipeline:

```bash
make pipeline
```

Create a release after committing and tagging:

```bash
make release
```

## Architecture

- `cmd/photoscli`: CLI, argument handling, filtering, manifests, progress, reporting, and command orchestration.
- `internal/photos`: Go interface and JSON parsing for PhotoKit responses.
- `internal/manifest`: JSONL/SQLite manifest backends and log writers.
- `bridge/`: Objective-C PhotoKit implementation plus a C stub used by tests.

Objective-C returns JSON to Go. Tests use the stub bridge and do not require real Photos access.

## Known Notes

- This project is macOS-only.
- Preview export currently follows the existing bridge preview implementation; `--format heic|png` is validated as an output-format hint but still needs bridge support for true non-JPEG previews.
- Album title lookup uses exact title matching. Use PhotoKit local identifiers when names are ambiguous.
- iCloud-backed assets may trigger downloads and can fail due to network or account state.
- `--min-size` and `--max-size` currently use estimated pixel count from dimensions, not encoded file size.

## Roadmap

- `0.8.x`: make XMP sidecars as rich, standard-compatible, and complete as possible from public PhotoKit metadata.
- `0.9.0` through `0.9.5`: checksums, deep verification, manifest repair, cleanup, doctor, and backup-integrity hardening.
- After `0.9.5`: opt-in Vision/Core ML analysis for photoscli-generated face/object/animal/scene metadata.