Implement Gitea CI and real statement imports

.kilo/plans/1789727434954-gitea-ci-real-statement-imports.md

@@ -0,0 +1,84 @@
+# Gitea CI and real statement imports
+
+## Goal and constraints
+
+- Move CI to the canonical Gitea Actions location and make the currently failing pipeline pass on the existing `vm-runner`.
+- Store only CI-only credentials in Gitea repository Actions secrets; never copy local or future production secrets into Gitea.
+- Replace guessed behavior for Alfa debit XLSX, Ozon debit PDF, and Rocketbank debit PDF with contracts proven against the ignored files in `examples/`.
+- Never commit real statements, extracted text, full parser snapshots, or private expected values.
+- Keep `alfa_credit_xlsx` and `ozon_credit_pdf` code untouched but remove them from the active adapter registry until real fixtures exist.
+- Do not implement or redesign frontend UI. Retain its existing CI checks, and document the future HOLD UI contract in ADR 0003.
+
+## Confirmed decisions
+
+- `examples/ozon.pdf` is an Ozon debit statement.
+- Repository-local `examples/` is an allowed private-fixture location because it is ignored; external `PRIVATE_FIXTURES_DIR` remains supported.
+- Alfa `HOLD` rows are immutable source records, but they do not create ledger operations during normal import confirmation.
+- An unresolved HOLD blocks closing the budget month containing its `occurred_on` date.
+- A later confirmed statement can resolve a HOLD automatically when exactly one conservative match becomes posted.
+- Manual HOLD resolution has two backend actions, both with a required reason: `confirm_as_posted` creates a normal operation; `discard_as_cancelled` creates none.
+- Gitea secrets to create are `CI_POSTGRES_PASSWORD`, `CI_RLS_DATABASE_PASSWORD`, `CI_MINIO_ROOT_PASSWORD`, and `CI_IMPORTS_KEK_B64`.
+
+## Implementation plan
+
+1. **Lock the private-data boundary before parser work.**
+   - Preserve the existing user change adding `/examples/` to `.gitignore`; verify `git ls-files examples` remains empty.
+   - Update `docs/parsers/private-fixture-gate.md` to allow an ignored repo-local fixture directory as well as an external one.
+   - Add a private manifest contract (`manifest.json` beside the statements) containing adapter ID, parser version, period, row counts, control totals, balances, and HOLD totals. Create the current manifest under ignored `examples/`; never commit it.
+   - Ensure private-test failures report only filenames, adapter IDs, counts, dates, and aggregate mismatches, not raw descriptions, names, account/card numbers, or extracted pages.
+
+2. **Move and repair Gitea Actions CI.**
+   - Move `.github/workflows/ci.yml` to `.gitea/workflows/ci.yml` and rename the workflow to make Gitea ownership explicit. Keep compatibility names such as `${{ github.ref }}` and `$GITHUB_OUTPUT` where Gitea deliberately exposes the GitHub-compatible context.
+   - Use the configured `origin` (`k1nq/finance` on `git.k1nq.space`) and installed `tea` login. List existing Actions secret names first, generate new independent CI-only values, and create/update the four agreed secrets through `tea actions secrets`; do not print, persist, or reuse values from `infra/.env`.
+   - Replace hard-coded PostgreSQL, restricted-role, MinIO, and KEK values in the workflow with `${{ secrets.* }}`. Keep non-sensitive usernames, database names, endpoints, and bucket names in YAML.
+   - Retrieve the authenticated logs for Gitea run `246` before changing job mechanics. The observable failures are frontend dependency installation, backend MinIO startup, and Compose infrastructure startup; fix the concrete causes rather than assuming GitHub runner behavior.
+   - Make Docker resources safe on the persistent VM runner: unique container/project names, an `if: always()` cleanup step, and either serialized backend/Compose jobs or non-overlapping host ports if logs confirm shared-port collisions. Do not weaken tests to obtain a green run.
+   - Keep the frontend job limited to its existing install/lint/type/test/build checks. If its install failure is lockfile/tooling-related, make only the minimal package-manager/CI correction; do not change UI code.
+
+3. **Make the parser boundary explicit and internally consistent.**
+   - In `backend/app/imports/adapters/base.py`, make adapter output canonical: `amount_kopecks` is always positive, `direction` carries the sign, and each row has an immutable bank status (`posted` or `hold`).
+   - Replace the shared global parser version for changed adapters with per-contract versions. Use a new version for the three rewritten contracts so existing parse runs can never be confused with the old guessed output.
+   - Extend `StatementDraft` with independent bank control totals: declared inflow, declared outflow, and optional pending/HOLD total. Persist them in the existing parse-run draft JSON.
+   - Refactor reconciliation in `imports/service.py`: compare parsed posted inflow/outflow with declared totals; compare HOLD totals separately; where both balances exist, also require `opening + inflow - outflow == closing`. A mismatch stays review-blocking with stable diagnostic fields.
+   - Preserve raw descriptions only in encrypted/import-domain storage already designed for them; logs and test diagnostics must remain redacted.
+
+4. **Rewrite the three evidenced adapters against their actual layouts.**
+   - **Alfa debit XLSX:** detect the exact workbook structure (`Выписка по счету`, account metadata, period fields, and operation headers) rather than a nonexistent `Альфа`/`дебет` marker. Parse operation date, posting date, code, bank category, description, amount, status, MCC, and location. Treat a `HOLD` posting-date cell as `bank_status=hold`; stop before signatures/footer. Parse current balance, posted totals, and unconfirmed total from the header.
+   - **Ozon debit PDF:** detect the legal-bank/title/account/table structure without requiring a nonexistent debit marker. Extract table rows by PDF table/word geometry so multiline document IDs and descriptions remain attached to one transaction. Parse period, account fingerprint input, opening/closing balances, declared totals, external document ID, timestamp date, direction, and amount.
+   - **Rocketbank debit PDF:** replace inheritance from `OzonPdfAdapterBase` with an independent state-machine parser. Track Russian day headings across pages, accumulate each `time + operation code + multiline description + signed RUR amount` record, ignore repeated headers/footers and daily totals, and extract MCC/location where present. Parse statement-level period, account, balances, inflow/outflow controls.
+   - Keep the Alfa/Ozon credit classes present but unregistered. `detect_adapter` must only select the three verified-format development contracts and must still fail closed for unknown layouts.
+
+5. **Model HOLD lifecycle without mutating extracted facts.**
+   - Add an Alembic migration and model field `SourceRecord.bank_status` with constrained values `posted|hold`, defaulting existing records to `posted`.
+   - Add a tenant-scoped, one-per-HOLD `HoldResolution` record rather than overwriting the extracted source row. Store resolution kind (`matched_posted`, `confirmed_as_posted`, `discarded_as_cancelled`), optional resolving source record/operation/user, required manual reason, and timestamp.
+   - During candidate building, match a new posted record to unresolved confirmed HOLD records only within the same space and account/statement contract. Prefer a stable external ID; otherwise require one unambiguous date, amount, direction, and normalized-description fingerprint match. Ambiguity remains review-required. Create the resolution only when the new statement is confirmed.
+   - Skip unresolved HOLD rows when applying a confirmed import to the ledger, while retaining them in the active parse run and API detail response.
+   - Add a CSRF-protected imports endpoint/service action for `confirm_as_posted` and `discard_as_cancelled`. Require a mapped account and reason; use ledger services for operation creation, add provenance links, honor closed-period rules, and write audit events.
+   - Add unresolved active HOLDs from confirmed imports to budgeting close blockers for their `occurred_on` month. Manual confirmation may then expose the normal uncategorized-expense blocker until the created operation is categorized.
+
+6. **Build layered tests without leaking fixtures.**
+   - Add tracked synthetic tests for canonical amount semantics, stable detection failures, exact Alfa header mapping, Ozon multiline table grouping, Rocket day/page state transitions, totals reconciliation, HOLD parsing, and malformed/oversized/no-text cases. Generate sanitized XLSX data in tests and test PDF text/table cores with sanitized extracted structures; do not commit derivatives of the real statements.
+   - Add `private_fixtures` to pytest markers and a separately collected private test module. It reads `PRIVATE_FIXTURES_DIR/manifest.json`, fully parses each real file twice, asserts deterministic output and private expectations, and never runs in normal CI.
+   - Add service/database tests for: HOLD does not create an operation; confirmed unresolved HOLD blocks only its month; a unique later posted record resolves it on confirmation; ambiguous overlap remains unresolved; both manual actions require a reason; manual confirmation creates exactly one operation and late reimport links rather than duplicates it; tenant isolation and closed-period rules hold.
+   - Keep the beta gate as **not passed** for all three adapters after this work because only one real statement file per contract is available. Record that one private fixture passes and that a second independent real layout/period is still required.
+
+7. **Document the product contract and defer UI deliberately.**
+   - Amend `docs/adr/0003-versioned-statement-imports.md` with posted/HOLD source status, append-only resolution metadata, automatic settlement matching, close blocking, both manual actions, and the future UI requirement to show and resolve HOLD rows.
+   - Update parser gate/status documentation with the new parser versions and local validation result, without embedding private counts, totals, identifiers, or descriptions.
+   - Make no changes under `frontend/src`. Do not regenerate or redesign frontend API/UI in this workstream; backend response additions remain backward-compatible optional fields for the later design-first frontend project.
+
+## Validation and rollout
+
+1. Run focused adapter synthetic tests, HOLD service tests, import service tests, migration upgrade, Ruff, formatting, and mypy.
+2. Run the private suite locally with `PRIVATE_FIXTURES_DIR` pointing to `examples/`; confirm the manifest and all statement files remain ignored and untracked afterward.
+3. Run the complete backend suite. Run existing frontend checks only to validate the CI tooling correction, with no UI modifications.
+4. Validate Compose and image builds on the same Docker topology used by the Gitea VM runner, including cleanup after forced failure.
+5. Push only after local validation, then inspect the new Gitea Actions run with `tea`: all jobs green, secrets present by name, no secret values or private fixture paths/content in logs, and no `.github/workflows` workflow left active.
+6. Do not mark any adapter beta-supported until a second independent private fixture for that contract passes the full gate.
+
+## Principal risks
+
+- PDF text order can drift even when the visual table looks unchanged; table/word grouping must fail closed and bump parser version on layout drift.
+- Alfa HOLD rows may receive a new code or changed description when posted; only unique conservative matching may auto-resolve them.
+- Gitea VM runners can retain Docker containers and share host ports across jobs; cleanup and isolation are acceptance criteria, not optional hygiene.
+- Assertions, action logs, parser exceptions, and private manifests are all possible financial-data leak paths and must be reviewed before the first Gitea run.