Files
millerson-overlay.nix/docs/research_marmel_upstream.md
T
alex e569243ae4
CI / check (push) Has been cancelled
chore(marmel): bump to afcc1dc (main), give checks a usable TMPDIR
Track upstream main at afcc1dc971ae4acd59b157f031d4bdbc9d52a7d0
(2026-11-07), 22 commits past the de3e99c8 pin. Upstream still ships no
tags or releases and Cargo.toml still reads version 1.0.0, so the dated
0-unstable-* string stays. cargoHash moves on its own: the new
crates/zim-search workspace member grew Cargo.lock from 399 to 621
entries, and cargo vendor hashes the whole workspace.

The real packaging work is the TMPDIR override appended to preCheck.
Upstream's new harness::fs::path_tests scratch-root helper puts its
scratch dirs under <cwd>/target/ and first asserts that root lies
outside std::env::temp_dir(). The build sandbox exports TMPDIR=/build,
which is an ancestor of the unpacked source at /build/source, so that
premise check tripped and all 8 path-security tests failed with
"scoped root /build/source/target/marmel-path-tests-N sits inside the
temp allow-list entry /build". Pointing TMPDIR at
/tmp/marmel-check-tmp, outside $NIX_BUILD_TOP, restores upstream's
premise, so all 8 tests keep running for real instead of being filtered
out of cargoTestFlags. Build-time only; the installed binary is
untouched.

Two derivation comments carried stale facts and are corrected:

- meta.license claimed the repository shipped no LICENSE file and that
  Cargo.toml had no license field. Upstream c98179d023 (Na1w/marmel#40)
  added a root LICENSE file and declared license = "MIT" in Cargo.toml,
  so the comment now records that MIT is asserted in the README,
  Cargo.toml and LICENSE alike; lib.licenses.mit was already the right
  value.
- the cargoTestFlags comment quoted "the lib suite passes at 1055
  tests". Roughly 500 test fns landed since (TUI, fs-path, PTY and
  dispatch-integrity suites), so the figure is dropped rather than
  refreshed: nothing checks it and it rots on every bump.

Also adds the two research notes behind the bump under docs/: the
de3e99c8..afcc1dc9 upstream delta audit and the update-tooling findings.

Verified: clean nix build .#marmel including the serialised test suite
(--test-threads=1) — 2146 passed, 0 failed.
2026-11-07 16:54:02 +03:00

16 KiB
Raw Blame History

Upstream research: Na1w/marmel — pin bump discovery

Date of research: 2026-11-07 (all API calls made live; network was reachable). Package under bump: packages/marmel/package.nix (attr marmel, crate name marmennill).

✅ HEAD TO USE

field value
rev (full 40-char SHA) afcc1dc971ae4acd59b157f031d4bdbc9d52a7d0
version 0-unstable-2026-11-07
commit date (ISO 8601) 2026-11-07T05:55:53Z (author and committer date identical)
commit subject feat: isolate git worktrees per task, plan merge steps, and commit as Marmel Coder (#46)
default branch main (confirmed: GET /repos/Na1w/marmel → default_branch: "main", pushed_at: 2026-11-07T05:55:58Z)

Both src hash and cargoHash must be recomputed — the source tree changed and Cargo.lock grew from 399 to 621 packages.

Still no tags and no releases (GET /repos/Na1w/marmel/tags → [], GET /repos/Na1w/marmel/releases → []), so the 0-unstable-* / branch-head scheme stays correct.


1. HEAD of main

Evidence (GitHub REST API, HTTP 200):

  • GET https://api.github.com/repos/Na1w/marmel/commits/HEAD → sha: afcc1dc971ae4acd59b157f031d4bdbc9d52a7d0, commit.author.date = commit.committer.date = 2026-11-07T05:55:53Z, PGP-verified commit (verification.verified: true), tree 5b2e615fba13f69a2e2866a492a950fc185c1c72.
  • GET https://api.github.com/repos/Na1w/marmel/commits?sha=main&per_page=100 → first entry is the same SHA (the current pin is the 23rd entry, see §2).

Version-string convention check: the current pin de3e99c8… has author/committer date 2026-10-23T00:43:14Z and the derivation says 0-unstable-2026-10-23, i.e. the date is the commit date of the pinned HEAD (UTC). Applying the same rule to the new HEAD gives 0-unstable-2026-11-07.

2. Is the current pin an ancestor of HEAD? How far behind?

GET /repos/Na1w/marmel/compare/de3e99c89b18bf43fbb471930bd9f7a40e1ca1b2...afcc1dc971ae4acd59b157f031d4bdbc9d52a7d0

status:      ahead
ahead_by:    22
behind_by:   0
total_commits: 22

→ Yes, de3e99c89b18bf43fbb471930bd9f7a40e1ca1b2 is a strict ancestor of main HEAD, and the pin is exactly 22 commits behind. No force-push / history rewrite happened (behind_by: 0, and the pin also appears in the linear commits?sha=main listing at index 22).

3. Commits between the pin and HEAD (oldest → newest)

# SHA (abbrev) date subject
1 b66f669257 2026-10-26 Fix/arbitrator reply and adhoc validation (#26)
2 70fa3078ba 2026-11-05 Feature/new UI (#27)
3 c6a975f39d 2026-11-05 Feature/new UI (#27) (#29)
4 0dc722296e 2026-11-05 feat(orchestration): integrate zim-search as workspace subcrate + expose kiwix__* tools (#28)
5 8cf2195a06 2026-11-05 fix: mcp initializes asynchronously (#30)
6 d1581295aa 2026-11-05 Feature/agents md project instructions (#31)
7 3619ed21d7 2026-11-05 feat: nested AGENTS.md per directory and CLAUDE.md/GEMINI.md fallback (#32)
8 00a05a031a 2026-11-05 Clean up stray research left-overs (#33)
9 ca363cca6b 2026-11-06 fix: make test scratch directory names process-unique (#34)
10 15c3f3ae06 2026-11-06 docs: document zim-search/kiwix integration and refresh README (#35)
11 a0da04bed0 2026-11-06 Feature/regex replace (#36)
12 1081776bf0 2026-11-06 Fix/prevent autonomous pause (#37)
13 dd293465c5 2026-11-06 Fix/windows support (#38)
14 e1e9418b19 2026-11-06 Feature/git worktree policy (#39)
15 c98179d023 2026-11-06 docs: add MIT license and update license metadata (#40)
16 f036a68fb8 2026-11-06 Fix/log issues and speedup (#41)
17 385b97bcc0 2026-11-06 fix: fall back to USERPROFILE for home directory lookup on Windows (#42)
18 cb9c0310b3 2026-11-06 Feature/composer arrow scroll (#43)
19 04b81bc6ea 2026-11-07 Feature/resume by default plan (#44)
20 26464aaa3b 2026-11-07 feat: clean up .marmel runtime files on plan completion by default (#45)
21 5f12a32ebf 2026-11-07 Docs/readme cleanup (#47)
22 afcc1dc971 2026-11-07 feat: isolate git worktrees per task, plan merge steps, and commit as Marmel Coder (#46)

139 paths changed. The big functional blocks: a UI rewrite (src/ui/tui/** new components/, layout.rs, state.rs, theme.rs), a new crates/zim-search workspace member, Windows support work, AGENTS.md/CLAUDE.md project-instruction support, and prompt updates.

4. Derivation impact checklist

4.1 postInstall example configs — ✅ still fine

Root listing at HEAD (GET /repos/Na1w/marmel/contents/?ref=afcc1dc9…, plus the fetched source tarball):

.github/  AGENTS.md  Cargo.lock  Cargo.toml  LICENSE  README.md
config/  crates/  marmel.toml.cloud  marmel.toml.df  marmel.toml.example
prompts/  scripts/  src/  tests/
  • marmel.toml.example — present (modified in range: +75 lines, content only).
  • marmel.toml.cloud — present (untouched in the range).
  • Both are in the git archive, so both install -Dm644 lines keep working.
  • Side note: marmel.toml.df also exists (already existed at the pin; the derivation has never installed it). Not a breakage, just an uninstalled extra if you ever want it.

4.2 Skipped sandbox test — ✅ still present, --skip filter still valid

  • src/harness/sandbox.rs:1800 at HEAD: fn test_internal_sandbox_exec_dev_null_and_dns() (same path, same line number as at the pin).
  • diff of src/harness/sandbox.rs pin → HEAD: byte-identical.
  • The cargoTestFlags --skip harness::sandbox::tests::test_internal_sandbox_exec_dev_null_and_dns therefore still matches a real test (no "filter matched no tests" error, no silent no-op).
  • The file's own maintainer note is unchanged and still describes exactly the Nix situation: the exec half runs on a store host, the /etc/resolv.conf half is the fixture question.
  • Caveat (soft, not a breakage): these sandbox tests use integration_host() / candidate_is_current() and print SKIP <test>: no current executable implementing --internal-sandbox-exec was found next to the test binary (run cargo buildfirst,cargo test --lib does not rebuild bin targets) when the bin is stale or built from a different tree. buildRustPackage builds the bin in buildPhase and runs tests from the same tree in checkPhase, so this should keep passing as before — but re-read the checkPhase log after the bump, because the derivation comment currently asserts "no SKIP line is printed".

4.3 Cargo.toml version / license — ⚠️ one comment in the derivation is now stale

[package] at HEAD vs at the pin:

key pin de3e99c8 HEAD afcc1dc9
name marmennill marmennill
version 1.0.0 1.0.0 (unchanged → the "Cargo.toml still says 1.0.0" comment stays accurate)
edition 2024 2024
rust-version 1.98 1.98 (unchanged → no new toolchain floor)
license (absent) "MIT" (NEW)
authors (absent) ["Fredrik Andersson"] (NEW)
[workspace] (absent) resolver = "2", members = ["crates/zim-search"] (NEW)

Consequences:

  1. A LICENSE file now ships at the repo root (added by c98179d023 "docs: add MIT license and update license metadata (#40)"). The meta.license comment in package.nix — "Upstream README states MIT, but the repository ships no LICENSE file and Cargo.toml has no license field" — is now factually wrong and should be rewritten (upstream now declares MIT both in Cargo.toml and as a shipped LICENSE). lib.licenses.mit itself remains the correct value. Since the source is fetched from git, you may optionally keep it as-is or switch to a license = { file = "LICENSE"; … } style declaration — SPDX hash not required for the bump.
  2. version = "1.0.0" is unchanged, so version = "0-unstable-YYYY-MM-DD" (i.e. ignoring the crate version) is still the right call.

4.4 New native build inputs / system deps / submodules / build-time network — ✅ none needed, ⚠️ but cargoHash changes and there is a latent trap

No new build-time requirements for the package as built today:

  • [dependencies] of the root marmennill package is byte-for-byte identical pin → HEAD (28 deps + 2 dev-deps + the cfg(unix)/cfg(target_os = "linux") libc/landlock blocks). Lockfile reverse-check: the direct dep set of marmennill is identical.
  • No build.rs anywhere in the tree (find -name build.rs → 0 hits), no .gitmodules, no vendored vendor/ dir → no submodules to fetch, no custom build script needing extra tools.
  • All include_str! / include_bytes! targets (prompts, embedded modules) resolve inside the source archive — checked programmatically: 0 missing. So the "binary needs no runtime data files" claim still holds. New embedded prompts (prompts/tool_reasoning.md, prompts/skills/git_workflow.md) are present in the tree.
  • No new build-time network need: nothing downloads at build time in the default graph.

Cargo.lock grew a lot — cargoHash must be regenerated:

  • 399 → 621 lock entries (+196 new package names, 0 removed).
  • Cause: the new workspace member crates/zim-search (ZIM archive reader with Tantivy BM25 search): tantivy 0.26.2, zim 0.5.0, clap, rayon, image/rav1e (via zim), zstd-sys, lzma-sys, plus an optional semantic = ["dep:fastembed"] feature that drags in fastembed/ort/ort-sys (ONNX), tokenizers, hf-hub, ureq.
  • cargo vendor (what cargoHash/fetchCargoTarball hashes) resolves the whole workspace, so the vendor tarball gets noticeably bigger even though none of it is compiled. Expect a new cargoHash and a longer vendor fetch.

Why no extra nativeBuildInputs are required (verified, not assumed):

  • buildRustPackage runs bare cargo build/test --release --locked, and with a root package present the default member is the root package only — crates/zim-search is not built or tested. Upstream states this deliberately in Cargo.toml: "crates/zim-search is reached over MCP stdio only, so it is a member here but is deliberately NOT a path dependency of marmennill (no tantivy/zim in this graph)."
  • Lockfile graph analysis (name+version edges): zim, tantivy, clap, rayon, ureq, hf-hub, ort, ort-sys, fastembed, tokenizers, zstd-sys, lzma-sys are reachable from zim-search only — not from marmennill.
  • openssl / openssl-sys / native-tls / hyper-tls / tokio-native-tls / vcpkg appear in the lockfile but are reachable only via zim-search → fastembed (optional semantic, off by default) → hf-hub 0.5.0 → reqwest 0.12.28 → native-tls → openssl. marmennill itself uses reqwest 0.13.5, whose deps are rustls-only (rustls, rustls-platform-verifier, tokio-rustls, quinn) — no OpenSSL, no cmake/pkg-config.
  • Latent trap to record in the derivation comment: if anyone ever adds --workspace / --all-features to cargoBuildFlags/cargoTestFlags, or upstream makes zim-search a path dep, the build immediately needs OpenSSL (openssl-sys, vcpkg) and, with semantic, libonnxruntime. Keep bare cargo invocations.

TLS/trust-store note still valid: rustls-native-certs is still in marmennill's graph (via reqwest 0.13.5 → rustls-platform-verifier), so the nativeCheckInputs = [ cacert ] + preCheck SSL_CERT_FILE workaround is still required — do not drop it.

4.5 Test-suite size — ⚠️ comment is stale (low priority)

Grep proxy over #[test] + #[tokio::test] (cfg-gated tests are counted regardless of platform, so treat as an upper bound; this also explains why it overshoots the "1055" figure recorded in the derivation):

suite pin de3e99c8 HEAD afcc1dc9
src/ (lib test binary) 1036 #[test] + 169 #[tokio::test] ≈ 1205 1562 #[test] + 197 #[tokio::test] ≈ 1759
tests/ (integration binaries) ≈ 348 ≈ 397

→ The lib suite grew materially (~+46%, roughly +500 test fns, mostly the new src/ui/tui/** tests, src/harness/fs_path_tests.rs (+1616 lines), pty_tests.rs (+775 lines), dispatch_integrity_tests.rs (+853 lines)). The derivation comment's "the lib suite passes at 1055 tests with that single filter" is stale; re-measure the real cargo test --lib count after the bump and refresh the number.

4.6 Other checkPhase-relevant observations (informational)

  • ca363cca6b "make test scratch directory names process-unique (#34)" adds src/test_scratch.rs — friendlier for sandboxed/parallel runs.
  • The new PTY tests self-gate at runtime via pty_or_skip() / pty_allocation_blocker() (src/harness/pty_tests.rs) and print SKIP <test>: this host cannot allocate a PTY (…) instead of failing when openpty() returns EACCES (the Nix sandbox ptmxmode=000 case). Upstream deliberately avoids #[ignore] here. So new PTY tests should degrade to visible SKIPs, not failures — check the log for the SKIP lines.
  • Loopback/HTTP test surface is unchanged in shape (4 files reference TcpListener at both pins, 1 file references resolv.conf at both) — the cacert preCheck rationale is intact.
  • --test-threads=1 is still warranted: the shared process-global orchestrator state the comment describes (src/orchestrator/workers.rs, bus.rs, preemption.rs) is untouched in spirit and src/orchestrator/worker_pause.rs adds more shared state. Keep the flag.
  • Windows-support commits (#38, #42) are cfg-gated; meta.platforms = lib.platforms.unix is unaffected.

5. Concrete to-do for the bump (not applied — this file is the only deliverable)

  1. rev = "afcc1dc971ae4acd59b157f031d4bdbc9d52a7d0", version = "0-unstable-2026-11-07".
  2. Rehash src (fetchFromGitHub) and cargoHash (lockfile changed). nix-update --flake marmel --use-update-script should handle both; note nix-update's --version=branch=main scheme yields exactly 0-unstable-2026-11-07 for this commit date.
  3. Update the stale meta.license comment (MIT is now declared in Cargo.toml and shipped as LICENSE at the repo root).
  4. Keep postInstall as-is (both example configs still exist).
  5. Keep cargoTestFlags as-is (skip filter still matches a real test); keep preCheck SSL_CERT_FILE + cacert.
  6. After the build, refresh the "1055 tests" figure in the comment and confirm no unexpected SKIP lines; consider adding a one-line note that crates/zim-search is intentionally not built (and why no OpenSSL/ONNX inputs are needed).

6. Evidence / commands used

# HEAD + default branch
curl -s https://api.github.com/repos/Na1w/marmel/commits/HEAD
curl -s "https://api.github.com/repos/Na1w/marmel/commits?sha=main&per_page=100"
curl -s https://api.github.com/repos/Na1w/marmel            # default_branch=main, license=MIT
# ancestry + commit list + changed files
curl -s https://api.github.com/repos/Na1w/marmel/compare/de3e99c89b18bf43fbb471930bd9f7a40e1ca1b2...afcc1dc971ae4acd59b157f031d4bdbc9d52a7d0
# tags / releases (both empty)
curl -s "https://api.github.com/repos/Na1w/marmel/tags?per_page=5"
curl -s "https://api.github.com/repos/Na1w/marmel/releases?per_page=5"
# full trees of both revisions, compared locally
curl -sL https://codeload.github.com/Na1w/marmel/tar.gz/<sha>

Local analysis on the two extracted trees: diff of Cargo.toml/Cargo.lock/ src/harness/sandbox.rs, TOML-parsed dependency-graph reachability from marmennill vs zim-search, include_str! target resolution, and #[test]/#[tokio::test] counts. Reference file for the claims above: packages/marmel/package.nix (read-only, not modified) — version line 14, rev line 19, cargoHash line 23, preCheck line 41, the "no SKIP line … 1055 tests" comment lines 60–61, cargoTestFlags line 62, postInstall line 72, meta.license comment lines 95–96.