# LedgerSprout

LedgerSprout is a privacy-first toolkit that converts bank and card statements into clean ledger files.

## Current Status

v0.3.0 release candidate in progress with CSV/XLSX exports, parser scaffolding, parse diagnostics, and a local review app.

Implemented:
- Parser registry loader (plugin-style parsers).
- CLI parse command for PDF input to CSV/XLSX output.
- Multi-format export support (CSV, XLSX).
- Parse quality reports (JSON + human summary).
- File-level parse diagnostics for threshold misses and parser failures.
- Local review app for inspecting matches, warnings, and parsed rows.
- Drag-and-drop review workflow with file/folder intake and session export.
- Parser scaffolding command.
- Canonical output schema with parser metadata and review flags.
- Example parser package in parser_registry/example_bank_checking.

## Product Vision

Users should be able to provide one or many statement PDFs and receive structured exports in CSV or XLSX with minimal setup.

## Why This Matters

- Statement formats are inconsistent across banks.
- Existing tools are often cloud-only, expensive, or locked to one institution.
- Users need transparent parsing with confidence scoring and manual review paths.

## How LedgerSprout Compares

| Feature | Cloud tools | OSS projects | LedgerSprout |
|---|---|---|---|
| **Privacy / local-only** | No | Partial | Yes |
| **Zero per-page cost** | No | Yes | Yes |
| **Auto-detects bank format** | Yes | Rarely | Yes |
| **Tax category mapping (Sched. A/C)** | No | No | Yes |
| **YAML plugin registry** | No | No | Yes |
| **Portable single-file install** | No | Java required | Yes |
| **Confidence scoring / review app** | No | No | Yes |

## Core Principles

- Local-first processing by default.
- Explainable parsing (rules and parser definition are visible).
- Extensible parser registry for institution-specific adapters.
- Deterministic exports with auditable source references.

## Planned v1 Outcomes

- Multi-file PDF ingestion.
- Auto-detection of known institutions.
- Unified transaction schema.
- Export to CSV and XLSX.
- Confidence scoring and uncertainty report.
- Pluggable parser registry.

## Near-Term Priorities

- Add an unknown-template fallback parser for files that do not clear an institution-specific threshold.
- Add a first-class CLI parser verification command built on the existing fixture workflow.
- Clarify supported templates, known limits, and parser maturity in the docs.

## Quickstart

1. Create and activate a Python environment.
2. Install project dependencies.
3. Run the CLI.

Example:

	python -m venv .venv
	source .venv/bin/activate
	pip install -e .
	ledgersprout --help

For XLSX support:

	pip install -e ".[xlsx]"

For all formats:

	pip install -e ".[all]"

## Building for Distribution

### Linux/macOS Python Packages

Build source and wheel distributions:

	./scripts/build.sh

Include XLSX extra dependencies before building:

	./scripts/build.sh --xlsx

### Windows Portable Executable

Create a standalone Windows portable zip (no Python required) using PyInstaller:

	powershell -File scripts/windows/build_portable.ps1

Optional parameters:

	powershell -File scripts/windows/build_portable.ps1 -Version "0.3.0"

**Output:**
- `dist/LedgerSproutPortable-{VERSION}-win64.zip` - Ready for distribution
- Contains the executable, parser registry, category registry, and launcher batch file
- Users simply unzip and double-click "Start LedgerSprout.bat" to launch the review app

### Linux Portable Executable

Create a standalone Linux portable tarball (XLSX support included by default):

	./scripts/build_portable_linux.sh

Exclude XLSX support:

	./scripts/build_portable_linux.sh --no-xlsx

**Output:**
- `dist/LedgerSproutPortable-{VERSION}-linux.tar.gz` — ready for distribution
- Contains the executable, parser registry, category registry, and a `start_ledgersprout.sh` launcher
- Users extract and run `./start_ledgersprout.sh` — no Python installation needed

### Release Distributable Builds

Build the Linux portable and upload both Linux and Windows artifacts to the production downloads directory (XLSX support included by default):

	./scripts/release.sh

Exclude XLSX support from the Linux build:

	./scripts/release.sh --no-xlsx

Skip the PyInstaller build (upload existing `dist/` artifacts only):

	./scripts/release.sh --skip-build

**Windows portable workflow:**
1. Build on Windows: `powershell -File scripts/windows/build_portable.ps1`
2. Copy `dist/LedgerSproutPortable-{VERSION}-win64.zip` to your Linux machine's `dist/`
3. Run `./scripts/release.sh --skip-build` to upload both artifacts together

The script reads the version from `pyproject.toml` automatically and uploads files to `https://ledgersprout.superdavelab.com/downloads/`, writing a `version.json` manifest alongside the artifacts.

### Deploy Landing Page (ledgersprout.superdavelab.com)

Deploy the static site in `site/` to Apache on `root@davekoons.com`:

	./scripts/deploy_site.sh

Preview changes without uploading:

	./scripts/deploy_site.sh --dry-run

Override defaults via environment variables when needed:

	SSH_TARGET=root@other-host REMOTE_DIR=/var/www/ledgersprout ./scripts/deploy_site.sh

Default script settings:
- `SSH_TARGET=root@davekoons.com`
- `REMOTE_DIR=/var/www/ledgersprout`
- `SITE_DOMAIN=ledgersprout.superdavelab.com`
- `VERIFY_URL=https://ledgersprout.superdavelab.com`

### Docker

Build the image:

	docker build -t ledgersprout .

Run without any license (Community edition, built-in categories):

	docker run -p 5000:5000 ledgersprout

Then open **http://localhost:5000** in your browser.

#### Persisting your license and category customizations

Mount `~/.config/ledgersprout` from the host into `/config` inside the container. This directory holds your license key, public key, and any saved category overrides:

	docker run -p 5000:5000 \
	  -v "$HOME/.config/ledgersprout:/config" \
	  ledgersprout

#### Pre-loading statements

Mount a folder of PDF statements as `/statements` (read-only is fine):

	docker run -p 5000:5000 \
	  -v "$HOME/.config/ledgersprout:/config" \
	  -v "$HOME/statements:/statements:ro" \
	  ledgersprout

#### Supplying a license via environment variable

If you prefer not to mount a file, pass the raw license token directly:

	docker run -p 5000:5000 \
	  -e LEDGERSPROUT_LICENSE_TOKEN="<your-token>" \
	  ledgersprout

#### Environment variables

| Variable | Default in image | Purpose |
|---|---|---|
| `LEDGERSPROUT_BIND_HOST` | `0.0.0.0` | Network interface to bind (change to `127.0.0.1` to restrict to localhost) |
| `LEDGERSPROUT_LICENSE_PATH` | `/config/license.key` | Path to the license key file inside the container |
| `LEDGERSPROUT_LICENSE_PUBLIC_KEY_PATH` | `/config/license_public_key.pem` | Path to the public key PEM |
| `LEDGERSPROUT_LICENSE_TOKEN` | _(unset)_ | Supply license token directly without a file |

## CLI Usage

### Parse Statements

Parse a single PDF to CSV:

	ledgersprout parse /path/to/statement.pdf -o output.csv --registry parser_registry

Parse all PDFs in a folder to CSV:

	ledgersprout parse /path/to/statements --glob "*.pdf" -o output.csv --registry parser_registry

Parse to XLSX:

	ledgersprout parse /path/to/statements -o output.xlsx --registry parser_registry

Or specify format explicitly:

	ledgersprout parse /path/to/statements -o output.csv --format xlsx --registry parser_registry

**Output:** After parsing, you'll get:
- Ledger file (CSV/XLSX) with transaction data.
- `output_report.json` with machine-readable parse metrics.
- Console summary with parse quality statistics and mismatch diagnostics.

### Review Parse Results Locally

The review app is a small local Python HTTP server started by the CLI. It runs on `127.0.0.1`, opens a browser tab, and never uploads files anywhere.

Launch the local review app against one file or a folder:

	ledgersprout review /path/to/statements --registry parser_registry
	ledgersprout review /path/to/statements --registry parser_registry --categories category_registry

Or start it empty and drag PDFs into the browser window:

	ledgersprout review --registry parser_registry
	ledgersprout review --registry parser_registry --categories category_registry

Optional flags:

	ledgersprout review /path/to/statements --port 8765 --no-open --registry parser_registry

	# Load custom YAML categories/overrides from a directory tree
	ledgersprout review --registry parser_registry --categories category_registry

Help:

	ledgersprout --help
	ledgersprout -?

The review app shows:
- per-file match status
- selected parser and threshold score
- warning/error diagnostics for unmatched files
- parser candidate ranking
- parsed transaction rows with search and review filters
- drag-and-drop PDF upload or file chooser intake
- export buttons for the current review session in CSV or XLSX

### Scaffold a New Parser

Create a new parser package with stub files:

	ledgersprout scaffold my_bank_checking --institution "My Bank" --account-type checking --registry parser_registry

This generates:
- `parser_registry/my_bank_checking/manifest.yaml` – parser metadata
- `parser_registry/my_bank_checking/parser.py` – implementation template
- `parser_registry/my_bank_checking/tests/test_parser.py` – test template

Overwrite existing parser with `--force`:

	ledgersprout scaffold my_bank_checking --institution "My Bank" --account-type checking --registry parser_registry --force

### Verify a Statement Against an Answer Sheet

Point `verify-statement` at a real PDF and a human-authored answer sheet (`.xlsx` or `.csv`) to confirm the parser output is correct:

	ledgersprout verify-statement /path/to/statement.pdf answer.xlsx --registry parser_registry

To force a specific parser instead of auto-detecting:

	ledgersprout verify-statement /path/to/statement.pdf answer.xlsx --registry parser_registry --parser ally_checking

**Answer sheet format** — the answer sheet must have a header row containing `post_date`, `description`, `amount`, and `balance` columns (the `.answer.xlsx` files created by `scripts/new_fixture.py` are already in this format). Rows where `description` starts with `EXAMPLE` are automatically skipped.

**Description matching** is intentionally loose: all whitespace (including newlines from multi-line PDF text) is collapsed to a single space, the value is trimmed and lowercased, and spacing around `,`, `-`, and `/` is tightened — on **both** the parser output and the answer sheet before comparing.

**Amount and balance** are compared as exact floats rounded to 2 decimal places.

**Exit codes:**
- `0` — 100% accurate, all rows match
- `1` — one or more row mismatches, missing, or extra rows
- `2` — setup/config error (bad paths, no parser found, …)



Output columns include:
- statement_start
- statement_end
- account_number
- account_last4
- post_date
- description
- amount
- balance
- direction
- source_file
- parser_id
- parser_version
- row_confidence
- review_flag
- review_reason

## Parse Quality Report

After each parse run, a JSON report is generated with:
- Total input files
- Matched files
- Unmatched files
- Total rows exported
- Rows flagged for review
- Parser usage counts
- File-level diagnostics with parser scores, warnings, and parse errors

Example report structure:

```json
{
  "timestamp": "2024-01-15T10:30:00.123456",
  "total_input_files": 10,
  "matched_files": 8,
  "unmatched_files": 2,
  "total_rows_exported": 342,
  "rows_flagged_for_review": 5,
  "parser_usage_counts": {
    "example_bank_checking": 200,
    "other_bank": 142
  },
	"unmatched_file_list": ["file1.pdf", "file2.pdf"],
	"file_diagnostics": []
}
```

## Parser Registry

Parsers live under parser_registry/<parser_id>/ and include:
- manifest.yaml
- parser.py

See parser_registry/example_bank_checking for a reference implementation and specs/PARSER_REGISTRY_SPEC.md for contract details.

## Repository Layout

- docs: product and engineering docs.
- specs: parser contracts and schema definitions.
- parser_registry: parser definitions and examples.
- .github: issue templates and workflow automation.
- src: CLI and core runtime package.
- tests: automated test coverage.

## Parser Accuracy

> **Disclaimer:** LedgerSprout makes best efforts to accurately extract transactions from PDF
> statements, and results are generally reliable for supported institutions.  However, PDF text
> extraction is inherently imperfect and output **may not be 100% accurate**.  Always review
> exported data before using it for financial, tax, or legal purposes.

Different statement formats produce different levels of parsing reliability depending on how many
numeric columns the PDF contains.  See [docs/PDF_PARSING_ACCURACY.md](docs/PDF_PARSING_ACCURACY.md)
for a plain-English summary and a technical breakdown of expected error rates by statement type.

## Private Preview Notes

- This repository is currently used for private preview and evaluator feedback.
- Distribution terms are controlled by private evaluator agreements.
- Security reporting guidance remains in SECURITY.md.
- CI workflow is in .github/workflows/ci.yml.

## Name Note

LedgerSprout was selected as a distinctive working brand name. This does not constitute trademark clearance. Perform a trademark and package-name check before public launch.
