mirror of
https://github.com/mauriceboe/TREK.git
synced 2026-10-19 05:26:47 +00:00
cc1c16ec34f4e2d92e3ddda1140cc13ebfed0d39
16 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Maurice
|
bcdb931a34 |
fix(docsync): hand a cursor on only from a run that left both sides as listed
A run stored the cursor of the listing it started from, whatever it did afterwards. Paperless, Papra and Synology build that cursor from what they list, so a copy TREK uploaded and somebody deleted at the provider before the next run left the scope exactly as the stored listing had it. The next plain run (scheduler, webhook, Sync now) read the cursor as unchanged, planned only the TREK side, and neither flagged the copy as missing nor asked the mass-delete guard until something else moved in the scope or a full run came along. A copy TREK binned and somebody restored at the provider read the same way. WebDAV was not affected: its root ETag never returns to an earlier value. The same held for a run that stopped short. What the transfer budget left over, and a download that failed, was never planned again under an unchanged cursor, so on a quiet binding those documents did not arrive at all. A run now keeps the listing's cursor only when it ended ok and asked the provider to change nothing: no upload, no rename, no trash. Any other run stores none, and the next one compares both sides in full and hands a cursor on once it ends clean. On Paperless, Papra and Synology that costs nothing, since they walk the scope on every run anyway; on WebDAV it is one Depth:1 listing instead of the Depth:0 probe. |
||
|
Maurice
|
ec2711cec0 |
fix(docsync): name the store of a binding whose provider is switched off
Both shells keep a binding on screen after an admin switches its provider off, and name its store through storeName(), which falls back to a providerName on the link before the raw id. The server never sent one, so such a binding was labelled with the provider id. publicLink now carries providerName from document_providers.name whatever its enabled flag, and every route and the MCP listing that return links go through it. The comments on docsync:changed in the realtime schema and the client's event policy say that the event now also marks a binding created or removed, and that the Files screen re-asks whether to offer the sync button when it arrives. |
||
|
Maurice
|
233f46575c |
fix(docsync): make the admin switch pause a binding, and retry a failed update
The check that was meant to stand a binding down when its provider is switched off asked AddonsService.isAddonEnabled(link.provider_id). The addons table has no rows for paperless, papra and the other document providers, so it answered "off" for every binding, and every run came back disabled before it listed anything. The unit test stubbed that answer and the e2e never runs a sync, so nothing caught it. isSwitchedOff() now asks the two things an admin actually switches: the Documents addon, and document_providers.enabled for the binding's provider. dueLinks leaves such bindings out in SQL, since one that never runs never gets a last_sync_at and twenty of them would take every slot of a tick. A run, the webhook on arrival and when its timer fires, "Sync now" and the MCP sync tool all refuse before anything is written, so a paused binding keeps its state and its documents' attempt counters and resumes where it stopped. The REST route answers 409 provider_disabled, the MCP tool reports the same code, status() carries providerOff per link, and both shells show one line saying the binding is paused because its provider is switched off, in all 23 languages. A failed pull_update also left the row stuck. upsertItem wrote the listing's version, size and time next to the old bytes, so the next run saw no upstream change, planned touch, which leaves error alone, and the document stayed in error on stale bytes for good, "Sync now" included. A failed transfer now leaves the pairing describing the copy TREK holds; a new row keeps what pairs it but no version, so it is fetched again. |
||
|
Maurice
|
9de8490176 |
style(docsync): drop dash punctuation from the added comments and test titles
The lines this branch adds used em and en dashes as sentence punctuation across comments, JSDoc, CSS comments and test titles. Each one is rewritten with a comma, a colon, parentheses or a second sentence, keeping what the sentence said. Only lines this branch adds are touched; everything that already exists on dev stays as it is. No logic, error code or logged message changes. The only string literals affected are describe and it titles, which now read "X: topic" like the rest of the provider tests. |
||
|
Maurice
|
843582863c |
fix(docsync): offer the sync button only where something is behind it
The Documents addon put a "Document sync" button on every trip's Files tab, and the seed ships every provider switched off, so after the upgrade each of those buttons led into "No document providers are available". The button now shows on a trip that is already bound, to every member, and otherwise only to somebody who may bind the trip on an instance that offers a provider. Without it the Files toolbar is the one from before the feature, trash flush right, on both shells. /api/addons carries no document providers for a normal user, so the answer comes from the docsync routes through a new online-only docSyncRepo and is kept per session in docSyncOfferStore. A revisit renders the last answer at once and checks it again; offline nothing is asked and the last answer stands. Creating a binding, removing one and removing a connection now ping the trip with docsync:changed and zero counts, so a member already on the Files tab sees the button come and go without leaving it. Both shells keep listing a binding whose provider the admin has since switched off instead of claiming there are no providers, and name its store through one storeName() helper. A load that answers 403 or 404 clears the bindings as well, so they do not linger once the Documents addon is gone. The phone admin screen gets the Documents shelf the desktop one has: the document providers as sub rows under the addon, toggled through the same admin route and re-read after Documents itself is toggled, because the server switches the providers off with it. isTripOwner and the isOwner prop become canManageSync and canManage, which is what canManageDocSync answers. Both folder pickers take loadScopes out of the sync object before their effect, so exhaustive-deps is satisfied without the effect running on every render. |
||
|
Maurice
|
25d4e34959 |
fix(docsync): keep the Synology device token across a restart, tied to its row
The DSM trusted-device token lived in process memory only. After a restart every token was gone and the stored one-time code long spent, so every two-factor NAS binding stopped syncing until its owner typed a fresh code, and nothing on screen said why. The token is now kept in the connection's encrypted secrets, together with a digest of the instance and account it was issued for. A saved connection's ref carries a saveSecret callback that writes through DocSyncConfigService.saveEarnedSecret, which rereads the row inside a transaction so a sync run holding an old ref cannot undo a password the owner changed meanwhile. The key belongs to no form field, so maskSecrets never reports it and mergeSecrets never touches it: the GET and PUT payloads do not carry it, an edit keeps it, and a client cannot plant one. It survives a password edit and a spent or cleared code. A connection repointed at another NAS or account drops it on its next call, and a token DSM refuses with 403 or 404 is dropped in memory and in storage. A probe of form values and an unsaved form may use what is stored but never write. A refused token no longer shadows a fresh code: the code in the form gets its turn in the same login call. In memory the slot is keyed by the connection id together with the row's created_at. A backup restored in place or the demo reset rolls the id sequence back without a restart, and a new connection that reused an id must not log in on the token of the row it replaced. The Synology adapter receives its client through DI like Paperless and WebDAV, instead of building a second instance beside the one the module registers. Two readings of a failed request are settled as well: - INVALID_PURPOSE, INVALID_CA and PATH_LENGTH_EXCEEDED count as an untrusted certificate again, so the settings screen offers the self-signed switch. Papra used to find them through the message text, and the shared classifier reads codes only. - A host that never completes the TCP handshake reads as unreachable on every client. WebDAV said timeout before the shared transport; the change is now documented and tested. |
||
|
Maurice
|
71400b7e85 |
fix(docsync): follow a rename on Synology, and settle restores and purges in one run
Where the id is the path, a rename or a move upstream lists the same copy under a new id. The upstream-only loop re-paired it with a touch, which went wrong twice. The pairing had already been counted as vanished, so the same run flagged it missing as well and counted it towards the mass-delete guard. And touch wrote the provider's new name into remote_name while the trip file kept the old one, so the next run read that as a rename made in TREK and renamed the provider's copy back. The heuristic also matched on a content hash only, which Synology never sends: on the one provider without stable ids it did not fire at all, and every rename there ended as a missing flag plus a second download of the same bytes. The planner now moves such a pairing first, with a relocate action that updates remote_id and the version and leaves the agreed name alone. The pair loop then treats it like any listed document, so the new name comes into TREK through rename_local exactly as it does where ids are stable, and nothing counts as vanished. A copy is recognised by its content hash where the listing has one, otherwise by size and modification time, which a rename or a move leaves untouched. Only a one-to-one match is followed, and a copy TREK binned itself never ties with a live twin of the same bytes. The rows now load remote_size and remote_modified_at, which were written on pull and never read; touch keeps them current and a push records them. rename_local and the provider branch of resolveConflict take the provider name in its cleaned form, since a raw name with a colon or a slash read as a rename made in TREK on the next run. Restores and purges: - A file taken back out of TREK's trash whose copy somebody else had deleted meanwhile resumed as synced and was then counted as vanished, so a few such restores tripped the mass-delete guard on every run until the circuit opened. It now lands as remote_missing directly, outside the guard, and the service asks for a full listing while such a restore is pending. - A row in error that once held a file now reads as a deletion once the file is purged, told apart from a first download that never landed by the agreed hash. "Sync now" used to download the deleted document again. - Moves are followed before a restore is judged, so a binned copy restored at the provider under a new path resumes its pairing instead of being duplicated on both sides. - relocate and both renames lift a remote_missing flag the way touch does. - A double rename under trek_wins or provider_wins is settled with a rename instead of a transfer. |
||
|
Maurice
|
b88b92add7 |
fix(docsync): a paused binding no longer resets its own settings
PATCH links validated its body with a schema derived from the create contract through .partial(). In Zod 4 a .default() still fires inside an optional field, so a patch naming one setting arrived with every other one filled in: pausing a pull-only binding switched it to two-way, reset both policies and blanked its label. The update contract is now written out without defaults, and an e2e test drives a patch through the real pipe, which the controller unit tests never see. Also, the cosmetic leftovers from the review: - POST connections/:connectionId/scopes no longer wants the connection in the body as well. It was required, never read, and let one request name two connections; a client that still sends it has the key stripped rather than refused. - createLink loses the scope argument every caller passed as null. - The controller's owner check is now assertCanManage, because it lets instance admins through just as canManageDocSync does on the client. Who may manage a binding is unchanged. - document_provider_fields drops settings_key and payload_key. Nothing read them: the connect form takes its fields from /docsync/providers and the admin shelf sends none for document providers. - The e2e test imports the provider classes under their new names. |
||
|
Maurice
|
01a7224f29 |
fix(docsync): key the Synology device token per connection and share one transport
The DSM trusted-device token was cached per NAS and account. A second trip whose owner knew the account password but never had the authenticator could bind the same account and log in on the first trip's token, a two-factor bypass across trip owners. Nothing ever cleared the entry either, so a token DSM stopped trusting went out ahead of any fresh code on every attempt until the process restarted. The token is now filed per connection, with instance and account still in the key. An edit keeps the connection id, so the token survives a password change and a stale code in the form. An unsaved form (connection 0) keeps what its probe earns on the session only, and the saved connection adopts it from there. A token DSM refuses with 403 or 404 is dropped for that connection alone. The four clients each carried their own copy of the transport, and the copies had drifted. provider-http.ts now holds the SSRF-guarded fetch with its timeout and certificate switch, the reading of a request that got no answer, the base status table and the guard on a streamed download. Timeouts, caps, auth headers and each client's error class and wording are unchanged. Where the copies differed by accident they now agree: - Paperless recognises UNABLE_TO_GET_ISSUER_CERT_LOCALLY (a private CA) as tls_untrusted instead of unreachable. - ERR_SSL_* handshake failures read as unreachable on WebDAV too, since the self-signed switch cannot fix them. - Synology maps HTTP 409, 415 and 507 like the others, Paperless maps 415. - Every download is refused on an announced length over its ceiling and counted as it streams. WebDAV and Paperless had no ceiling, Synology only read the header, and Papra's limiter left the source pinned when it tripped. - Papra reported a download without Content-Length as zero bytes, so the size from the listing was never used. PapraProvider and SynologyDriveProvider are renamed PapraDocumentProvider and SynologyDriveDocumentProvider, matching the other adapters. |
||
|
Maurice
|
dd3d97589b |
fix(docsync): act on a deletion once, and know what to do when it is undone
Both planner branches planned local_deleted for as long as a file sat in TREK's trash. Under unlink nothing showed, until the binding was switched to trash: the next run then binned every document ever deleted in TREK, months back included. A binding on trash from the start asked the provider to bin the same copy on every run. A local_deleted row is now left alone, so the policy is applied once, under the policy in force at that moment. That guard also meant a file taken back out of TREK's trash stayed on local_deleted for good, which it already did before: no transfer, no upload, no flag, and a second deletion of the same file would never have propagated. The row now leaves that state by what became of the provider copy: - still there: the pairing counts again and the row is judged like any synced one, so an upstream edit made meanwhile comes down; - binned by TREK: the pairing is dropped and the file goes up again as a new document through the ordinary push, keeping its trek_doc_uid; - deleted by somebody else: flagged remote_missing, never uploaded over their deletion. Telling the last two apart needs a record of what TREK did, so the new remote_trashed_at column (migration 239) is written when TREK bins a copy and cleared when the pairing is back, or when the binned copy is listed again because somebody restored it at the provider. A file purged from the trash came straight back as well: file_id is ON DELETE SET NULL, and the row read as a download that never landed. A tombstone now stays a tombstone, and a purge no run had seen yet is treated as the deletion it is. Also drops snapshotHash, which nothing read. |
||
|
Maurice
|
9f531d1cd9 |
feat(docsync): make the conflict answer real and give a conflict a way out
The binding asked at setup what should happen when both sides changed, stored the answer, and nothing ever read it: every conflict was manual whatever the binding said. The planner now acts on it — provider_wins pulls, trek_wins pushes, anything else parks the row — and never in the direction the binding forbids. Conflicts themselves had no exit in either shell. The server has resolved them since the first version and the strings have been in the locales just as long, but neither the panel nor the phone sheet ever listed the rows, so the only way to answer one was the MCP tool. Both shells now list them through one shared hook with the same three answers. Also: - The desktop connect dialog shows a refused save instead of looking as though nothing happened. - An error the server phrases as a sentence no longer reaches the screen as a raw translation key. - The scope pickers depend on the stable callback rather than the hook object, so opening one no longer re-lists the provider's folders on every render above it. |
||
|
Maurice
|
e6a5ca1521 |
fix(docsync): stop a push from stealing another file's pairing
Papra answers a push of bytes it already holds with the document that has them. Two trip files with the same content therefore pushed to one document, and the unique index on (link_id, remote_id) aborted that run and every run after it with a constraint error. The pairing now stays with the file that owns it and the second file is recorded as a conflict. Also in this pass: - Paperless no longer adopts a same-bytes document that lacks the scope's tag when a consume outlives the poll budget; that document belongs to someone else's filing and delete-through would have trashed it. - The pull path writes the file row, the retirement of the copy it replaces and the pairing in one transaction, and announces them only once committed. - PATCH links stops accepting remoteRootId/remoteRootPath, which it read and silently dropped. - The MCP sync tool clears shelved items before running, as the REST route does. |
||
|
Maurice
|
7237a1881d |
fix(docsync): switches that did not switch off, a webhook nobody could authenticate, and a counter that only grew
More from the audit, each checked against the code before it was touched. The webhook obeyed neither the Documents addon nor the app_settings kill switch, so "switch document sync off" stopped the poll while every provider holding a webhook carried on driving full runs. Both are now read on arrival and again when the debounce timer fires, so a switch thrown mid-window still takes effect. Nextcloud's subscription registered its secret under `X-TREK-Docsync-Signature` while the endpoint reads `x-trek-docsync-secret`: the subscription existed, the calls arrived, and every one of them was dropped for carrying no secret the controller could find. The two names now agree, and the test pins the one the endpoint actually reads. A WebDAV create was an unconditional PUT, so a name already taken upstream was silently overwritten — two trips bound to one folder, or a `receipt.pdf` a person put there by hand, is enough. Creates now send `If-None-Match: *`. The per-item retry bookkeeping was stuck at both ends: the backoff was computed at step 1 every time, so three of the curve's four steps were unreachable, and the attempt counter only ever grew, which shelved a row that failed once a month after six months of otherwise healthy syncing. A successful pass now clears it, which is what makes the limit mean six failures in a row. Also: a manual run on an orphaned binding is refused with 409 (its credential belongs to somebody no longer on the trip, and the scheduler already refuses it, so the button was a way around that); a document deleted while a run is in flight is no longer uploaded anyway; a path-shaped WebDAV id containing `..` is rejected instead of addressing a file outside the bound folder; and Papra's multipart part header no longer takes an unescaped media type, which a CR could end early. Client: three writes in useDocSync swallowed their failure into `void` call sites, a load failure blanked the panel to "no providers available" whatever the cause, and overlapping loads let the slowest response win. Writes now report, only 403/404 blanks the panel, and a generation stamp keeps the newest answer. The dead lastRun state went with them. |
||
|
Maurice
|
4fa06c3483 |
fix(docsync): an idle WebDAV binding read as an emptied folder, and two ids nobody checked
The audit's worst find, reproduced against a live Nextcloud before and after the
fix: the WebDAV adapters answer "nothing changed here" by returning an EMPTY
listing plus `cursorUnchanged`, and nothing read that flag — `grep -rn
cursorUnchanged src/` found it set by four adapters and consulted by none. The
core therefore saw a folder that had lost everything. Run 1 and 2 look fine
because the run's own uploads move the ETag; from run 3 on, every run of an
idle binding ends `partial` with `mass_delete_guard` and six documents reported
missing, forever, until failure_count opens the circuit and the scheduler drops
the binding entirely. That is the ordinary case — a binding nobody is touching.
planReconcile now takes `remoteUnchanged` and plans only what happened in TREK.
Two ids were authorised by trip and never checked against it:
- resolveConflict took an item id on trust, so the owner of any trip could
decide a conflict belonging to somebody else's. Both are plain integers and
nothing compared them.
- The connection test merged the STORED credential into a probe of whatever
baseUrl the caller named — a way to have TREK post another account's API
token at a server of the caller's choosing. Stored secrets are now reused
only for the address they were stored against.
The phone's connect form was wrong in three ways at once, each of which the
desktop had right: it read the address as `baseUrl` while the field is
`base_url` (so every connection it saved carried an empty address), it drew
`allow_insecure_tls` as a text box and always sent false, and it built label
keys as `docsync.provider${cap(label)}` when the label column already holds the
full suffix — `docsync.providerProviderApiKey`, rendered raw on screen. Those
rules are not markup, so they now live in `useConnectForm`, which both shells
use. Its test fixture invented the field shape and so agreed with the broken
sheet; it now mirrors document_provider_fields.
Also: the scheduler read its interval once at bootstrap while this file's own
comment promised a per-tick read, so changing it did nothing until a restart;
and dueLinks ordered by a column that a successful run clears, which left every
binding tied and SQLite answering in insertion order — on a database with 170
bindings the same twenty ran on every tick and nothing created after them was
ever synced automatically.
|
||
|
Maurice
|
1430056d24 |
fix(docsync): read the retry limit back out, debounce the webhook, and cover the UI
The attempt counter and the backoff were written on every failure and never consulted when planning the next run: rows were found in a live database at 46 attempts against a limit of 6, so a document a provider refuses — one it reads as corrupt, a type it will not take, a full quota — was re-uploaded on every cron tick forever. planReconcile now skips a row that is shelved or still inside its window, and a manual run clears the counters, which is what "waits for a person" has to mean. The webhook had the same shape of bug: WEBHOOK_NUDGE_DEBOUNCE_SECONDS existed, with a comment explaining it, and nothing read it. Providers fire once per document, so twenty files dropped into a watched folder were twenty runs — the in-flight guard answered `busy` to nineteen of them, throwing away nineteen announcements, and the one run that did start had begun before most of the changes landed. A burst now folds into one trailing run, and the binding is re-read when the timer fires so a sync switched off in between does not get a last run out of a stale row. Four defects the new suites turned up, all fixed: the sync button lost its accessible name below sm, where its label is hidden; the stored address was never shown in the connect dialog although the form kept submitting it; the allow-self-signed switch only read the connection at mount, so a stored "on" showed as off and the next save wrote that back; and a refused folder create escaped as an unhandled rejection with nothing said to anybody. Also here: the phone shell gets the same feature (one sheet, four views, the same hook as the desktop), the marks come from the product's own icon files rather than redrawn approximations, and the 22 non-English locales carry the new keys. ADMIN-SVC-069 asserted that `documents` has no MCP surface. It does now, so the test takes its example from MCP_GATED_ADDON_IDS instead of naming one. |
||
|
Maurice
|
2d7cb124b3 |
feat(documents): two-way sync with Paperless, Papra, Nextcloud, OpenCloud and Synology
A trip can be bound to a folder, tag or space in a self-hosted document store, and documents then move in both directions: what someone uploads in TREK appears there, and what someone files there appears in TREK. Three decisions shape the rest. The connection belongs to the TRIP, not to a person. Every other integration here is per user, and photos are shared only through an opt-in flag; neither can deliver "everyone on the trip sees the same documents", because visibility would depend on whose credentials fetched a file, and a member without their own Paperless account would see nothing. So the trip owner binds the trip once, the server talks to the provider under that one identity, and TREK's own membership decides who sees what. The provider never learns that TREK has members. When the credential holder leaves the trip the binding goes to `orphaned` rather than quietly carrying on with an ex-member's token. TREK keeps the bytes. A synced document is an ordinary trip_files row, so download, trash, offline cache, PDF export and backup keep working untouched, and a provider outage does not blank out a trip that starts tomorrow. Deletion never crosses the boundary on its own. A document that disappears upstream is recorded as missing and waits for a human, and a run that sees more than 40 percent of its known documents vanish at once refuses the whole listing rather than acting on it: an unmounted share answers exactly like a mass delete. The core reconciles by enumerating both sides, because the weakest provider sets the model. A changed-since query cannot see a deletion, Papra does not move updatedAt when a tag changes, a Paperless bulk edit leaves `modified` alone, and Synology FileStation has no change feed at all. A webhook is therefore only ever "look now", never a source of truth. Echo is caught by comparing content, not timestamps: pushed_sha256 records what TREK itself wrote, so a provider reporting TREK's own upload is not mistaken for a stranger's edit. Provider differences are answered by capability data rather than by branching on an id: stable ids where they exist (Paperless document id, oc:fileid, driveId) and a normalised path where they do not, replace-in-place where the API offers it, and a content hash from the listing only where the provider really supplies one. Admins switch providers on under Admin, Addons, Documents, mirroring how photo providers hang off Journey, including the cascade and the 409 when the parent addon is off. The per-trip setup lives in the file manager, where the person has the context that makes the folder choice obvious. |