e569243ae4
CI / check (push) Has been cancelled
Track upstream main at afcc1dc971ae4acd59b157f031d4bdbc9d52a7d0 (2026-11-07), 22 commits past the de3e99c8 pin. Upstream still ships no tags or releases and Cargo.toml still reads version 1.0.0, so the dated 0-unstable-* string stays. cargoHash moves on its own: the new crates/zim-search workspace member grew Cargo.lock from 399 to 621 entries, and cargo vendor hashes the whole workspace. The real packaging work is the TMPDIR override appended to preCheck. Upstream's new harness::fs::path_tests scratch-root helper puts its scratch dirs under <cwd>/target/ and first asserts that root lies outside std::env::temp_dir(). The build sandbox exports TMPDIR=/build, which is an ancestor of the unpacked source at /build/source, so that premise check tripped and all 8 path-security tests failed with "scoped root /build/source/target/marmel-path-tests-N sits inside the temp allow-list entry /build". Pointing TMPDIR at /tmp/marmel-check-tmp, outside $NIX_BUILD_TOP, restores upstream's premise, so all 8 tests keep running for real instead of being filtered out of cargoTestFlags. Build-time only; the installed binary is untouched. Two derivation comments carried stale facts and are corrected: - meta.license claimed the repository shipped no LICENSE file and that Cargo.toml had no license field. Upstream c98179d023 (Na1w/marmel#40) added a root LICENSE file and declared license = "MIT" in Cargo.toml, so the comment now records that MIT is asserted in the README, Cargo.toml and LICENSE alike; lib.licenses.mit was already the right value. - the cargoTestFlags comment quoted "the lib suite passes at 1055 tests". Roughly 500 test fns landed since (TUI, fs-path, PTY and dispatch-integrity suites), so the figure is dropped rather than refreshed: nothing checks it and it rots on every bump. Also adds the two research notes behind the bump under docs/: the de3e99c8..afcc1dc9 upstream delta audit and the update-tooling findings. Verified: clean nix build .#marmel including the serialised test suite (--test-threads=1) — 2146 passed, 0 failed.
275 lines
16 KiB
Markdown
275 lines
16 KiB
Markdown
# Upstream research: `Na1w/marmel` — pin bump discovery
|
||
|
||
Date of research: 2026-11-07 (all API calls made live; network was reachable).
|
||
Package under bump: `packages/marmel/package.nix` (attr `marmel`, crate name `marmennill`).
|
||
|
||
> ## ✅ HEAD TO USE
|
||
>
|
||
> | field | value |
|
||
> |---|---|
|
||
> | **`rev` (full 40-char SHA)** | `afcc1dc971ae4acd59b157f031d4bdbc9d52a7d0` |
|
||
> | **`version`** | `0-unstable-2026-11-07` |
|
||
> | commit date (ISO 8601) | `2026-11-07T05:55:53Z` (author **and** committer date identical) |
|
||
> | commit subject | `feat: isolate git worktrees per task, plan merge steps, and commit as Marmel Coder (#46)` |
|
||
> | default branch | `main` (confirmed: `GET /repos/Na1w/marmel` → `default_branch: "main"`, `pushed_at: 2026-11-07T05:55:58Z`) |
|
||
>
|
||
> Both `src` hash and `cargoHash` **must** be recomputed — the source tree changed and
|
||
> `Cargo.lock` grew from 399 to 621 packages.
|
||
>
|
||
> Still no tags and no releases (`GET /repos/Na1w/marmel/tags` → `[]`,
|
||
> `GET /repos/Na1w/marmel/releases` → `[]`), so the `0-unstable-*` / branch-head
|
||
> scheme stays correct.
|
||
|
||
---
|
||
|
||
## 1. HEAD of `main`
|
||
|
||
Evidence (GitHub REST API, HTTP 200):
|
||
|
||
- `GET https://api.github.com/repos/Na1w/marmel/commits/HEAD`
|
||
→ `sha: afcc1dc971ae4acd59b157f031d4bdbc9d52a7d0`,
|
||
`commit.author.date = commit.committer.date = 2026-11-07T05:55:53Z`,
|
||
PGP-verified commit (`verification.verified: true`), tree `5b2e615fba13f69a2e2866a492a950fc185c1c72`.
|
||
- `GET https://api.github.com/repos/Na1w/marmel/commits?sha=main&per_page=100`
|
||
→ first entry is the same SHA (the current pin is the 23rd entry, see §2).
|
||
|
||
Version-string convention check: the current pin `de3e99c8…` has author/committer date
|
||
`2026-10-23T00:43:14Z` and the derivation says `0-unstable-2026-10-23`, i.e. the date is the
|
||
**commit date of the pinned HEAD** (UTC). Applying the same rule to the new HEAD gives
|
||
**`0-unstable-2026-11-07`**.
|
||
|
||
## 2. Is the current pin an ancestor of HEAD? How far behind?
|
||
|
||
`GET /repos/Na1w/marmel/compare/de3e99c89b18bf43fbb471930bd9f7a40e1ca1b2...afcc1dc971ae4acd59b157f031d4bdbc9d52a7d0`
|
||
|
||
```
|
||
status: ahead
|
||
ahead_by: 22
|
||
behind_by: 0
|
||
total_commits: 22
|
||
```
|
||
|
||
→ **Yes, `de3e99c89b18bf43fbb471930bd9f7a40e1ca1b2` is a strict ancestor of `main` HEAD,
|
||
and the pin is exactly 22 commits behind.** No force-push / history rewrite happened
|
||
(`behind_by: 0`, and the pin also appears in the linear `commits?sha=main` listing at index 22).
|
||
|
||
## 3. Commits between the pin and HEAD (oldest → newest)
|
||
|
||
| # | SHA (abbrev) | date | subject |
|
||
|---|---|---|---|
|
||
| 1 | `b66f669257` | 2026-10-26 | Fix/arbitrator reply and adhoc validation (#26) |
|
||
| 2 | `70fa3078ba` | 2026-11-05 | Feature/new UI (#27) |
|
||
| 3 | `c6a975f39d` | 2026-11-05 | Feature/new UI (#27) (#29) |
|
||
| 4 | `0dc722296e` | 2026-11-05 | feat(orchestration): integrate zim-search as workspace subcrate + expose kiwix__* tools (#28) |
|
||
| 5 | `8cf2195a06` | 2026-11-05 | fix: mcp initializes asynchronously (#30) |
|
||
| 6 | `d1581295aa` | 2026-11-05 | Feature/agents md project instructions (#31) |
|
||
| 7 | `3619ed21d7` | 2026-11-05 | feat: nested AGENTS.md per directory and CLAUDE.md/GEMINI.md fallback (#32) |
|
||
| 8 | `00a05a031a` | 2026-11-05 | Clean up stray research left-overs (#33) |
|
||
| 9 | `ca363cca6b` | 2026-11-06 | fix: make test scratch directory names process-unique (#34) |
|
||
| 10 | `15c3f3ae06` | 2026-11-06 | docs: document zim-search/kiwix integration and refresh README (#35) |
|
||
| 11 | `a0da04bed0` | 2026-11-06 | Feature/regex replace (#36) |
|
||
| 12 | `1081776bf0` | 2026-11-06 | Fix/prevent autonomous pause (#37) |
|
||
| 13 | `dd293465c5` | 2026-11-06 | Fix/windows support (#38) |
|
||
| 14 | `e1e9418b19` | 2026-11-06 | Feature/git worktree policy (#39) |
|
||
| 15 | `c98179d023` | 2026-11-06 | **docs: add MIT license and update license metadata (#40)** |
|
||
| 16 | `f036a68fb8` | 2026-11-06 | Fix/log issues and speedup (#41) |
|
||
| 17 | `385b97bcc0` | 2026-11-06 | fix: fall back to USERPROFILE for home directory lookup on Windows (#42) |
|
||
| 18 | `cb9c0310b3` | 2026-11-06 | Feature/composer arrow scroll (#43) |
|
||
| 19 | `04b81bc6ea` | 2026-11-07 | Feature/resume by default plan (#44) |
|
||
| 20 | `26464aaa3b` | 2026-11-07 | feat: clean up .marmel runtime files on plan completion by default (#45) |
|
||
| 21 | `5f12a32ebf` | 2026-11-07 | Docs/readme cleanup (#47) |
|
||
| 22 | `afcc1dc971` | 2026-11-07 | feat: isolate git worktrees per task, plan merge steps, and commit as Marmel Coder (#46) |
|
||
|
||
139 paths changed. The big functional blocks: a UI rewrite (`src/ui/tui/**` new
|
||
`components/`, `layout.rs`, `state.rs`, `theme.rs`), a new `crates/zim-search` workspace
|
||
member, Windows support work, AGENTS.md/CLAUDE.md project-instruction support, and prompt
|
||
updates.
|
||
|
||
## 4. Derivation impact checklist
|
||
|
||
### 4.1 `postInstall` example configs — ✅ still fine
|
||
|
||
Root listing at HEAD (`GET /repos/Na1w/marmel/contents/?ref=afcc1dc9…`, plus the fetched
|
||
source tarball):
|
||
|
||
```
|
||
.github/ AGENTS.md Cargo.lock Cargo.toml LICENSE README.md
|
||
config/ crates/ marmel.toml.cloud marmel.toml.df marmel.toml.example
|
||
prompts/ scripts/ src/ tests/
|
||
```
|
||
|
||
- `marmel.toml.example` — **present** (modified in range: +75 lines, content only).
|
||
- `marmel.toml.cloud` — **present** (untouched in the range).
|
||
- Both are in the git archive, so both `install -Dm644` lines keep working.
|
||
- Side note: `marmel.toml.df` also exists (already existed at the pin; the derivation has
|
||
never installed it). Not a breakage, just an uninstalled extra if you ever want it.
|
||
|
||
### 4.2 Skipped sandbox test — ✅ still present, `--skip` filter still valid
|
||
|
||
- `src/harness/sandbox.rs:1800` at HEAD: `fn test_internal_sandbox_exec_dev_null_and_dns()`
|
||
(same path, same line number as at the pin).
|
||
- `diff` of `src/harness/sandbox.rs` pin → HEAD: **byte-identical**.
|
||
- The `cargoTestFlags` `--skip harness::sandbox::tests::test_internal_sandbox_exec_dev_null_and_dns`
|
||
therefore still matches a real test (no "filter matched no tests" error, no silent no-op).
|
||
- The file's own maintainer note is unchanged and still describes exactly the Nix situation:
|
||
the exec half runs on a store host, the `/etc/resolv.conf` half is the fixture question.
|
||
- Caveat (soft, not a breakage): these sandbox tests use `integration_host()` /
|
||
`candidate_is_current()` and print `SKIP <test>: no current executable implementing
|
||
--internal-sandbox-exec was found next to the test binary (run `cargo build` first,
|
||
`cargo test --lib` does not rebuild bin targets)` when the *bin* is stale or built from a
|
||
different tree. `buildRustPackage` builds the bin in `buildPhase` and runs tests from the
|
||
same tree in `checkPhase`, so this should keep passing as before — but re-read the
|
||
`checkPhase` log after the bump, because the derivation comment currently asserts
|
||
"no SKIP line is printed".
|
||
|
||
### 4.3 `Cargo.toml` `version` / `license` — ⚠️ one comment in the derivation is now stale
|
||
|
||
`[package]` at HEAD vs at the pin:
|
||
|
||
| key | pin `de3e99c8` | HEAD `afcc1dc9` |
|
||
|---|---|---|
|
||
| `name` | `marmennill` | `marmennill` |
|
||
| `version` | `1.0.0` | `1.0.0` (unchanged → the "Cargo.toml still says 1.0.0" comment stays accurate) |
|
||
| `edition` | `2024` | `2024` |
|
||
| `rust-version` | `1.98` | `1.98` (unchanged → no new toolchain floor) |
|
||
| `license` | *(absent)* | **`"MIT"` (NEW)** |
|
||
| `authors` | *(absent)* | `["Fredrik Andersson"]` (NEW) |
|
||
| `[workspace]` | *(absent)* | `resolver = "2"`, `members = ["crates/zim-search"]` (NEW) |
|
||
|
||
Consequences:
|
||
|
||
1. **A `LICENSE` file now ships at the repo root** (added by `c98179d023` "docs: add MIT
|
||
license and update license metadata (#40)"). The `meta.license` comment in
|
||
`package.nix` — *"Upstream README states MIT, but the repository ships no LICENSE file and
|
||
Cargo.toml has no license field"* — is now **factually wrong** and should be rewritten
|
||
(upstream now declares MIT both in `Cargo.toml` and as a shipped `LICENSE`).
|
||
`lib.licenses.mit` itself remains the correct value. Since the source is fetched from
|
||
git, you may optionally keep it as-is or switch to a `license = { file = "LICENSE"; … }`
|
||
style declaration — SPDX hash not required for the bump.
|
||
2. `version = "1.0.0"` is unchanged, so `version = "0-unstable-YYYY-MM-DD"` (i.e. ignoring
|
||
the crate version) is still the right call.
|
||
|
||
### 4.4 New native build inputs / system deps / submodules / build-time network — ✅ none needed, ⚠️ but `cargoHash` changes and there is a latent trap
|
||
|
||
**No new build-time requirements for the package as built today:**
|
||
|
||
- `[dependencies]` of the root `marmennill` package is **byte-for-byte identical** pin → HEAD
|
||
(28 deps + 2 dev-deps + the `cfg(unix)`/`cfg(target_os = "linux")` `libc`/`landlock`
|
||
blocks). Lockfile reverse-check: the direct dep *set* of `marmennill` is identical.
|
||
- **No `build.rs` anywhere** in the tree (`find -name build.rs` → 0 hits), **no `.gitmodules`**,
|
||
**no vendored `vendor/` dir** → no submodules to fetch, no custom build script needing
|
||
extra tools.
|
||
- All `include_str!` / `include_bytes!` targets (prompts, embedded modules) resolve inside
|
||
the source archive — checked programmatically: **0 missing**. So the "binary needs no
|
||
runtime data files" claim still holds. New embedded prompts
|
||
(`prompts/tool_reasoning.md`, `prompts/skills/git_workflow.md`) are present in the tree.
|
||
- No new build-time network need: nothing downloads at build time in the default graph.
|
||
|
||
**`Cargo.lock` grew a lot — `cargoHash` must be regenerated:**
|
||
|
||
- 399 → **621** lock entries (+196 new package names, 0 removed).
|
||
- Cause: the new workspace member `crates/zim-search` (ZIM archive reader with Tantivy BM25
|
||
search): `tantivy 0.26.2`, `zim 0.5.0`, `clap`, `rayon`, `image`/`rav1e` (via zim),
|
||
`zstd-sys`, `lzma-sys`, plus an **optional** `semantic = ["dep:fastembed"]` feature that
|
||
drags in `fastembed`/`ort`/`ort-sys` (ONNX), `tokenizers`, `hf-hub`, `ureq`.
|
||
- `cargo vendor` (what `cargoHash`/`fetchCargoTarball` hashes) resolves the whole workspace,
|
||
so the vendor tarball gets noticeably bigger even though none of it is compiled. Expect a
|
||
new `cargoHash` and a longer vendor fetch.
|
||
|
||
**Why no extra `nativeBuildInputs` are required (verified, not assumed):**
|
||
|
||
- `buildRustPackage` runs bare `cargo build/test --release --locked`, and with a root package
|
||
present the **default member is the root package only** — `crates/zim-search` is *not*
|
||
built or tested. Upstream states this deliberately in `Cargo.toml`:
|
||
*"`crates/zim-search` is reached over MCP stdio only, so it is a member here but is
|
||
deliberately NOT a path dependency of `marmennill` (no tantivy/zim in this graph)."*
|
||
- Lockfile graph analysis (name+version edges): `zim`, `tantivy`, `clap`, `rayon`, `ureq`,
|
||
`hf-hub`, `ort`, `ort-sys`, `fastembed`, `tokenizers`, `zstd-sys`, `lzma-sys` are reachable
|
||
from `zim-search` **only** — **not** from `marmennill`.
|
||
- `openssl` / `openssl-sys` / `native-tls` / `hyper-tls` / `tokio-native-tls` / `vcpkg` appear
|
||
in the lockfile but are reachable only via
|
||
`zim-search → fastembed (optional `semantic`, off by default) → hf-hub 0.5.0 → reqwest 0.12.28 → native-tls → openssl`.
|
||
`marmennill` itself uses `reqwest 0.13.5`, whose deps are rustls-only
|
||
(`rustls`, `rustls-platform-verifier`, `tokio-rustls`, `quinn`) — no OpenSSL, no `cmake`/`pkg-config`.
|
||
- **Latent trap to record in the derivation comment:** if anyone ever adds `--workspace` /
|
||
`--all-features` to `cargoBuildFlags`/`cargoTestFlags`, or upstream makes `zim-search` a
|
||
path dep, the build immediately needs OpenSSL (`openssl-sys`, `vcpkg`) and, with
|
||
`semantic`, `libonnxruntime`. Keep bare cargo invocations.
|
||
|
||
**TLS/trust-store note still valid:** `rustls-native-certs` is still in `marmennill`'s graph
|
||
(via `reqwest 0.13.5 → rustls-platform-verifier`), so the `nativeCheckInputs = [ cacert ]` +
|
||
`preCheck` `SSL_CERT_FILE` workaround is still required — do not drop it.
|
||
|
||
### 4.5 Test-suite size — ⚠️ comment is stale (low priority)
|
||
|
||
Grep proxy over `#[test]` + `#[tokio::test]` (cfg-gated tests are counted regardless of
|
||
platform, so treat as an upper bound; this also explains why it overshoots the "1055" figure
|
||
recorded in the derivation):
|
||
|
||
| suite | pin `de3e99c8` | HEAD `afcc1dc9` |
|
||
|---|---|---|
|
||
| `src/` (lib test binary) | 1036 `#[test]` + 169 `#[tokio::test]` ≈ **1205** | 1562 `#[test]` + 197 `#[tokio::test]` ≈ **1759** |
|
||
| `tests/` (integration binaries) | ≈ 348 | ≈ 397 |
|
||
|
||
→ The lib suite grew **materially** (~+46%, roughly +500 test fns, mostly the new
|
||
`src/ui/tui/**` tests, `src/harness/fs_path_tests.rs` (+1616 lines), `pty_tests.rs`
|
||
(+775 lines), `dispatch_integrity_tests.rs` (+853 lines)). The derivation comment's
|
||
*"the lib suite passes at 1055 tests with that single filter"* is stale; re-measure the real
|
||
`cargo test --lib` count after the bump and refresh the number.
|
||
|
||
### 4.6 Other `checkPhase`-relevant observations (informational)
|
||
|
||
- `ca363cca6b` "make test scratch directory names process-unique (#34)" adds
|
||
`src/test_scratch.rs` — friendlier for sandboxed/parallel runs.
|
||
- The new PTY tests self-gate at runtime via `pty_or_skip()` / `pty_allocation_blocker()`
|
||
(`src/harness/pty_tests.rs`) and print `SKIP <test>: this host cannot allocate a PTY (…)`
|
||
instead of failing when `openpty()` returns `EACCES` (the Nix sandbox `ptmxmode=000`
|
||
case). Upstream deliberately avoids `#[ignore]` here. So new PTY tests should degrade to
|
||
visible SKIPs, not failures — check the log for the `SKIP ` lines.
|
||
- Loopback/HTTP test surface is unchanged in shape (4 files reference `TcpListener` at both
|
||
pins, 1 file references `resolv.conf` at both) — the cacert `preCheck` rationale is intact.
|
||
- `--test-threads=1` is still warranted: the shared process-global orchestrator state the
|
||
comment describes (`src/orchestrator/workers.rs`, `bus.rs`, `preemption.rs`) is untouched
|
||
in spirit and `src/orchestrator/worker_pause.rs` adds *more* shared state. Keep the flag.
|
||
- Windows-support commits (#38, #42) are `cfg`-gated; `meta.platforms = lib.platforms.unix`
|
||
is unaffected.
|
||
|
||
## 5. Concrete to-do for the bump (not applied — this file is the only deliverable)
|
||
|
||
1. `rev = "afcc1dc971ae4acd59b157f031d4bdbc9d52a7d0"`, `version = "0-unstable-2026-11-07"`.
|
||
2. Rehash `src` (`fetchFromGitHub`) **and** `cargoHash` (lockfile changed).
|
||
`nix-update --flake marmel --use-update-script` should handle both; note nix-update's
|
||
`--version=branch=main` scheme yields exactly `0-unstable-2026-11-07` for this commit date.
|
||
3. Update the stale `meta.license` comment (MIT is now declared in `Cargo.toml` **and**
|
||
shipped as `LICENSE` at the repo root).
|
||
4. Keep `postInstall` as-is (both example configs still exist).
|
||
5. Keep `cargoTestFlags` as-is (skip filter still matches a real test); keep `preCheck`
|
||
`SSL_CERT_FILE` + `cacert`.
|
||
6. After the build, refresh the "1055 tests" figure in the comment and confirm no unexpected
|
||
`SKIP ` lines; consider adding a one-line note that `crates/zim-search` is intentionally
|
||
not built (and why no OpenSSL/ONNX inputs are needed).
|
||
|
||
## 6. Evidence / commands used
|
||
|
||
```bash
|
||
# HEAD + default branch
|
||
curl -s https://api.github.com/repos/Na1w/marmel/commits/HEAD
|
||
curl -s "https://api.github.com/repos/Na1w/marmel/commits?sha=main&per_page=100"
|
||
curl -s https://api.github.com/repos/Na1w/marmel # default_branch=main, license=MIT
|
||
# ancestry + commit list + changed files
|
||
curl -s https://api.github.com/repos/Na1w/marmel/compare/de3e99c89b18bf43fbb471930bd9f7a40e1ca1b2...afcc1dc971ae4acd59b157f031d4bdbc9d52a7d0
|
||
# tags / releases (both empty)
|
||
curl -s "https://api.github.com/repos/Na1w/marmel/tags?per_page=5"
|
||
curl -s "https://api.github.com/repos/Na1w/marmel/releases?per_page=5"
|
||
# full trees of both revisions, compared locally
|
||
curl -sL https://codeload.github.com/Na1w/marmel/tar.gz/<sha>
|
||
```
|
||
|
||
Local analysis on the two extracted trees: `diff` of `Cargo.toml`/`Cargo.lock`/
|
||
`src/harness/sandbox.rs`, TOML-parsed dependency-graph reachability from `marmennill` vs
|
||
`zim-search`, `include_str!` target resolution, and `#[test]`/`#[tokio::test]` counts.
|
||
Reference file for the claims above: `packages/marmel/package.nix` (read-only, not modified)
|
||
— `version` line 14, `rev` line 19, `cargoHash` line 23, `preCheck` line 41, the
|
||
"no SKIP line … 1055 tests" comment lines 60–61, `cargoTestFlags` line 62, `postInstall`
|
||
line 72, `meta.license` comment lines 95–96.
|