v0.5.0: manifests, filters, logging, docs

README.md

@@ -1,43 +1,33 @@
 # photoscli
 
-`photoscli` is a small macOS-only CLI written in Go that reads data from Apple Photos through a PhotoKit bridge.
+`photoscli` is a macOS-only command-line exporter for Apple Photos. It uses a small Objective-C PhotoKit bridge from Go to list albums, inspect assets, export optimized previews or originals, keep resumable manifests, and produce machine-readable logs for backup automation.
 
-It supports five tasks:
+The tool is designed for repeatable, resumable Photos backups rather than one-off drag-and-drop exports.
 
-- listing albums
-- listing asset IDs, filenames, and cloud status in an album
-- showing the folder and album tree
-- backing up all albums into the Photos folder tree
-- exporting resized JPEG previews or original files from an album
+## Highlights
 
-## What The Code Does
-
-The executable lives in `cmd/photoscli` and calls into `internal/photos`, which wraps an Objective-C bridge in `bridge/`.
-
-Current behavior:
-
-- `albums` prints one line per album as `<album-id>\t<title>`
-- `photos --album-id <id-or-title>` prints one asset per line as `<id>\t<filename>\t<cloud>`; accepts either a PhotoKit local identifier or an album title
-- `tree` prints the human-readable folder and album hierarchy from Apple Photos
-- `backup-all --out <dir> [--size <px>] [--originals]` exports every album into a matching folder tree, showing per-asset progress with filename, size, and cloud status
-- `export --album-id <id-or-title> --out <dir> [--size <px>] [--originals]` exports either JPEG previews or original files with a progress bar showing filename, size, and cloud status; `--album-id` accepts either a PhotoKit local identifier or an album title
-
-The bridge uses PhotoKit to:
-
-- request access to the user's Photos library
-- fetch album collections by local identifier or album title
-- fetch album assets sorted by `creationDate` ascending
-- render resized images with `PHImageManager`
-- write JPEG files with compression `0.85`
-- support graceful cancellation via a cancel flag checked between file exports
+- List albums, photos, and the Photos folder/album tree.
+- Export one album or back up the whole Photos tree.
+- Export JPEG previews with configurable size and quality, or original files.
+- Skip already-exported assets through a JSONL or SQLite manifest.
+- Resume interrupted runs safely.
+- Parallel export with live worker progress and ETA.
+- Optional structured logging to `export.log` or SQLite `logs` table.
+- Dry-run mode for planning large backups.
+- Filters for favorites, media type, date, estimated size, and excluded albums.
+- Failure tracking with `failures.jsonl` and `retry-failed`.
+- Verification, reporting, and diff commands for backup integrity.
+- Script-friendly exit codes and optional JSON summaries.
+- 100% test coverage for the Go CLI and parsing layers.
 
 ## Requirements
 
-- macOS
-- Go 1.22+
-- Xcode command-line tools
+- macOS with Apple Photos.
+- Go 1.25 or newer.
+- Xcode command-line tools.
+- Photos privacy permission for the built binary or terminal app.
 
-The project builds with cgo and links against `Photos`, `Foundation`, `AppKit`, and `UniformTypeIdentifiers`.
+The production build uses cgo and links against Apple frameworks through the Objective-C bridge in `bridge/`.
 
 ## Build
 
@@ -45,41 +35,79 @@ The project builds with cgo and links against `Photos`, `Foundation`, `AppKit`,
 make build
 ```
 
-Output binary:
+The binary is written to:
 
 ```bash
 ./bin/photoscli
 ```
 
-## Test
+Run the full local release pipeline:
 
 ```bash
-make test
+make pipeline
 ```
 
-Tests run against a stub bridge, so they do not require real Photos access.
+The pipeline cleans artifacts, builds the test bridge, runs vet, runs race-enabled tests with coverage, builds the real PhotoKit bridge, builds the CLI, and verifies the embedded version.
 
-## Usage
+## Quick Start
+
+List albums:
 
 ```bash
 ./bin/photoscli albums
-./bin/photoscli photos --album-id "<album-local-identifier>"
-./bin/photoscli photos --album-id "Vacation"
-./bin/photoscli tree
-./bin/photoscli backup-all --out ./backup
-./bin/photoscli backup-all --out ./backup --originals
-./bin/photoscli export --album-id "<album-local-identifier>" --out ./export
-./bin/photoscli export --album-id "Vacation" --out ./export
-./bin/photoscli export --album-id "<album-local-identifier>" --out ./export --size 2048
-./bin/photoscli export --album-id "<album-local-identifier>" --out ./export --originals
 ```
 
-### Commands
+Inspect an album by title or PhotoKit local identifier:
 
-`albums`
+```bash
+./bin/photoscli photos --album-id "Vacation"
+```
 
-- Requests Photos access
-- Lists albums as tab-separated album ID and title
+Export preview JPEGs from one album:
+
+```bash
+./bin/photoscli export --album-id "Vacation" --out ./export
+```
+
+Export higher-quality previews:
+
+```bash
+./bin/photoscli export --album-id "Vacation" --out ./export --size 2048 --quality 92
+```
+
+Export originals:
+
+```bash
+./bin/photoscli export --album-id "Vacation" --out ./originals --originals
+```
+
+Back up the entire Photos album tree:
+
+```bash
+./bin/photoscli backup-all --out ./photos-backup --manifest sqlite --log
+```
+
+Preview what would happen before writing anything:
+
+```bash
+./bin/photoscli backup-all --out ./photos-backup --dry-run --json
+```
+
+Verify a backup later:
+
+```bash
+./bin/photoscli verify --out ./photos-backup --manifest sqlite
+```
+
+## Commands
+
+### `albums`
+
+Lists user-created albums as tab-separated ID and title.
+
+```bash
+photoscli albums
+```
 
 Example output:
 
@@ -88,128 +116,285 @@ Example output:
 8A1B.../L0/001	Work
 ```
 
-`photos --album-id <id-or-title>`
+### `photos`
 
-- Requests Photos access
-- If the value looks like a PhotoKit local identifier, uses it directly
-- Otherwise searches album titles for a match and resolves the identifier
-- Lists asset local identifiers and cloud status for the given album
+Lists assets in an album. `--album-id` accepts a PhotoKit local identifier or an exact album title.
+
+```bash
+photoscli photos --album-id "Vacation"
+```
+
+Output includes ID, filename, cloud state, media type, dimensions, optional creation date, optional duration, and a `*` marker for favorites.
+
+### `tree`
+
+Prints the Photos folder and album hierarchy.
+
+```bash
+photoscli tree
+```
+
+### `export`
+
+Exports assets from one album.
+
+```bash
+photoscli export --album-id <id-or-title> --out <dir> [flags]
+```
+
+Useful examples:
+
+```bash
+photoscli export --album-id "Family" --out ./family --size 2560 --quality 90
+photoscli export --album-id "Favorites" --out ./favorites --only-favorites
+photoscli export --album-id "Videos" --out ./videos --media videos --include-videos
+photoscli export --album-id "Archive" --out ./archive --originals --retry 3
+photoscli export --album-id "Vacation" --out ./vacation --date-template YYYY/MM/DD
+```
+
+### `backup-all`
+
+Walks the Photos folder/album tree and exports every album to a matching folder structure.
+
+```bash
+photoscli backup-all --out <dir> [flags]
+```
+
+Duplicate sibling album names are disambiguated by adding the album ID to the generated folder name.
+
+Useful examples:
+
+```bash
+photoscli backup-all --out ./backup --manifest sqlite --log
+photoscli backup-all --out ./backup --exclude-album "Recently Deleted" --exclude-album "Temp*"
+photoscli backup-all --out ./backup --since 2024-01-01 --sort newest
+photoscli backup-all --out ./backup --concurrency 8 --retry 3
+photoscli backup-all --out ./backup --dry-run --json
+```
+
+### `report`
+
+Shows manifest and failure counts for an output directory.
+
+```bash
+photoscli report --out ./backup --manifest sqlite
+```
 
 Example output:
 
 ```text
-1F2A.../L0/001	IMG_0001.JPG	local
-9C4D.../L0/001	IMG_0002.JPG	cloud
+entries	12345
+failures	2
 ```
 
-`backup-all --out <dir> [--size <px>] [--originals]`
+### `diff`
 
-- Requests Photos access
-- Walks the Photos folder and album hierarchy
-- Builds a complete index of all assets before exporting, skipping files already on disk
-- Creates directories as `out/folder/album/files`
-- Exports previews by default, originals when `--originals` is present
-- Shows a progress display with:
-  - Scroll log of completed files (✅ copied, ☁ downloaded, ⏭ skipped, ❌ failed)
-  - Worker status lines with live download progress bars for cloud files
-  - Total and Album progress bars with color gradient (red → yellow → green)
-- Uses `--size` only for preview export
+Compares an album against the manifest and prints assets missing from the manifest.
 
-Example layout:
+```bash
+photoscli diff --album-id "Vacation" --out ./backup --manifest jsonl
+```
+
+Exit code is `2` when missing assets are found.
+
+### `verify`
+
+Checks that manifest entries have corresponding files on disk.
+
+```bash
+photoscli verify --out ./backup --manifest sqlite
+```
+
+Exit code is `2` when files are missing.
+
+### `retry-failed`
+
+Retries assets recorded in `failures.jsonl`.
+
+```bash
+photoscli retry-failed --out ./backup
+```
+
+This is useful after transient iCloud or network failures.
+
+## Export Flags
+
+Common flags for `export` and `backup-all`:
+
+- `--out <dir>`: destination directory.
+- `--size <px>`: longest-side target for preview export, default `1024`.
+- `--quality <1-100>`: JPEG preview quality, default `85`.
+- `--originals`: export original files instead of previews.
+- `--concurrency <N>`: parallel workers, default `3`, capped by bridge slot count.
+- `--retry <N>`: retry failed exports with a small backoff.
+- `--dry-run`: print planned exports without writing files, manifests, or logs.
+- `--json`: print a machine-readable summary to stdout.
+- `--verify`: run manifest/file verification after export.
+- `--log`: enable structured export logging.
+- `--manifest jsonl|sqlite`: choose manifest backend, default `jsonl`.
+- `--no-manifest`: disable manifest reads/writes.
+- `--sort oldest|newest`: asset sort order where supported, default `oldest`.
+- `--since <date>`: only include assets on or after `YYYY-MM-DD` or RFC3339 date.
+- `--only-favorites`: only export favorite assets.
+- `--media photos|videos|all`: media filter, default `photos`.
+- `--include-videos`: compatibility flag that includes videos/audio.
+- `--min-size <n>`: filter by estimated pixel count.
+- `--max-size <n>`: filter by estimated pixel count.
+- `--format jpeg|heic|png`: preview format hint. Current bridge output is still the existing preview path; non-JPEG bridge output is future work.
+- `--date-template <template>`: append date folders based on creation date, for example `YYYY/MM/DD`.
+
+`backup-all` also supports:
+
+- `--exclude-album <pattern>`: repeatable exact or glob-style album-name exclusion.
+
+`export` also requires:
+
+- `--album-id <id-or-title>`: PhotoKit local identifier or exact album title.
+
+## Manifests
+
+Manifests prevent re-exporting assets on subsequent runs.
+
+JSONL backend:
 
 ```text
-backup/
-  Trips/
-    Italy 2024/
-      Venice/
-        0000_....jpg
-  Favorites/
-    0000_....jpg
+downloads.jsonl
 ```
 
-`export --album-id <id-or-title> --out <dir> [--size <px>] [--originals]`
-
-- Requests Photos access
-- Resolves `--album-id` by local identifier first, then by album title if not found
-- Creates the output directory if needed
-- Exports assets with a progress bar
-- Shows file size and cloud/local status for each exported asset
-- Exports resized JPEG previews by default
-- Exports original files when `--originals` is present
-- Writes a summary like `exported 10 photos to ./export` or `exported 10 original files to ./export` to stderr
-
-`--size` is the target bounding box passed to PhotoKit for preview export. Default: `1024`.
-
-`--originals` switches export mode to original-file export. In that mode, `--size` is ignored.
-
-`--include-videos` includes video and audio assets in the export. By default, videos and audio are filtered out.
-
-`tree`
-
-- Requests Photos access
-- Prints folders and albums as an indented tree
-- Omits internal album IDs for human-readable output
-
-Example output:
+SQLite backend:
 
 ```text
-Trips
-  Italy 2024
-    Venice
-Favorites
+downloads.db
 ```
 
+The manifest stores asset ID, filename, size, cloud state, and export timestamp. SQLite is useful for very large libraries and for keeping logs in the same database.
+
+Use no manifest only when you intentionally want stateless export behavior:
+
+```bash
+photoscli export --album-id "Scratch" --out ./scratch --no-manifest
+```
+
+## Logging
+
+Enable logs with:
+
+```bash
+photoscli backup-all --out ./backup --log
+```
+
+With JSONL/no-manifest mode, logs are written to:
+
+```text
+export.log
+```
+
+With SQLite manifest mode, logs are written to the `logs` table in `downloads.db`.
+
+Logged events include session start/end, completed exports, skipped assets, and failures. Each event includes timestamp, level, event name, asset ID, album, filename, size, cloud state, duration, and message where available.
+
+## Failure Tracking
+
+Failed exports are appended to:
+
+```text
+failures.jsonl
+```
+
+Retry them with:
+
+```bash
+photoscli retry-failed --out ./backup
+```
+
+Use `--retry N` during normal exports to handle transient iCloud or network failures automatically.
+
+## Configuration File
+
+Defaults can be loaded from:
+
+```text
+~/.photoscli.toml
+```
+
+Or from a custom path:
+
+```bash
+PHOTOSCLI_CONFIG=/path/to/photoscli.toml photoscli backup-all --out ./backup
+```
+
+Supported keys mirror long flag names without the leading dashes:
+
+```toml
+size = 2048
+quality = 90
+concurrency = 8
+manifest = "sqlite"
+log = true
+sort = "newest"
+media = "photos"
+retry = 3
+```
+
+Command-line flags override config-file values.
+
+## Exit Codes
+
+- `0`: success.
+- `1`: command/config/runtime error.
+- `2`: partial failure, such as some exports failed or diff/verify found missing items.
+- `3`: Photos access denied.
+
 ## Permissions
 
-On first use, macOS may prompt for Photos access.
+On first use, macOS may prompt for Photos access. If access is denied, grant access in:
 
-If access is denied, the CLI returns an error telling you to grant access in:
+```text
+System Settings > Privacy & Security > Photos
+```
 
-`System Settings > Privacy & Security`
+Depending on how you launch the binary, macOS may associate the permission with Terminal, iTerm, your shell, or the built executable.
 
-## Export Details
+## Development
 
-Exported files currently:
+Run tests with the stub bridge:
 
-- are JPEGs when exporting previews
-- keep their original filenames when exporting originals when possible
-- fall back to a sanitized asset identifier if an original filename is unavailable
-- prefix duplicate original filenames with the asset index to avoid collisions
-- name preview exports like `0000_<asset-local-identifier>.jpg`
-- replace `/` and `\` in asset IDs with `_` for generated preview filenames
-- replace `/` and `\` in folder and album names with `_` when creating backup directory names
-- preserve ordering based on ascending asset creation date
+```bash
+make test
+```
 
-If some assets fail but at least one succeeds, the command still succeeds and reports the number exported.
+Run all package coverage:
 
-If all exports fail, the command returns an error.
+```bash
+go test -tags=test -race -count=1 -coverprofile=coverage.out ./...
+go tool cover -func=coverage.out
+```
 
-## Signal Handling
+Run the release pipeline:
 
-Sending `SIGINT` (Ctrl+C) or `SIGTERM` during export or backup triggers a graceful shutdown:
+```bash
+make pipeline
+```
 
-1. The CLI prints `received signal, finishing current file...` to stderr
-2. The current file export is allowed to complete
-3. No further files are started
-4. The process exits after the in-progress file finishes
+Create a release after committing and tagging:
 
-A second signal forces an immediate exit.
+```bash
+make release
+```
 
 ## Architecture
 
-- `cmd/photoscli`: CLI entrypoint, argument parsing, progress display, and album name resolution
-- `internal/photos`: Go bridge interface, JSON parsing, and error mapping
-- `bridge/`: Objective-C PhotoKit implementation plus a C test stub
+- `cmd/photoscli`: CLI, argument handling, filtering, manifests, progress, reporting, and command orchestration.
+- `internal/photos`: Go interface and JSON parsing for PhotoKit responses.
+- `internal/manifest`: JSONL/SQLite manifest backends and log writers.
+- `bridge/`: Objective-C PhotoKit implementation plus a C stub used by tests.
 
-Data passed from Objective-C to Go is serialized as JSON and unmarshaled into Go structs.
+Objective-C returns JSON to Go. Tests use the stub bridge and do not require real Photos access.
 
-## Known Limitations
+## Known Notes
 
-- The tree view only shows user collections exposed through PhotoKit's top-level user collections API
-- Album title resolution matches the first album with that title; if multiple albums share a title, use the local identifier instead
-- `photos` only prints asset IDs and filenames, not dates or metadata
-- Preview export uses PhotoKit preview rendering, not original file export
-- Original export currently writes the first PhotoKit asset resource for each asset, which may not capture every related representation for complex assets
-- iCloud-backed assets may require network download during export
-- A second interrupt signal forces an immediate exit without waiting for the current file
-- Partial export failures are not listed individually
+- This project is macOS-only.
+- Preview export currently follows the existing bridge preview implementation; `--format heic|png` is validated as an output-format hint but still needs bridge support for true non-JPEG previews.
+- Album title lookup uses exact title matching. Use PhotoKit local identifiers when names are ambiguous.
+- iCloud-backed assets may trigger downloads and can fail due to network or account state.
+- `--min-size` and `--max-size` currently use estimated pixel count from dimensions, not encoded file size.