Files
alex 44e6d90a77
CI / check (push) Has been cancelled
feat(magg): add magg 1.3.0 MCP aggregator package
Add magg, an MCP aggregator / meta-MCP server, to the overlay. It is a
Python package on the hatchling backend, built with
python3Packages.buildPythonApplication from the v1.3.0 source tarball,
so no upstream release artifacts are fetched. Ships both console
scripts declared upstream: magg (the aggregator CLI/server) and mbro
(the MCP browser). License is AGPL-3.0-or-later.

The install check runs the upstream pytest suite with
-m "not integration", because the integration-marked tests need a live
server. A forced nix build .#magg --rebuild passes with 329 passed,
3 skipped, 3 deselected, 0 failed, and both binaries smoke-test clean.

Also adds the README and AGENTS.md package rows, the upstream and build
research notes, and the verification report in docs/magg_build_report.md.
2026-11-08 12:48:30 +03:00

309 lines
18 KiB
Markdown

# Build/packaging requirements: `sitbon/magg`
Companion to `docs/research_magg_upstream.md` (upstream fact sheet). This file answers only
the build/packaging questions an implementer needs answered before picking a build helper.
Evidence snapshot (all verified, nothing inferred):
| item | value | source |
| --- | --- | --- |
| upstream | <https://github.com/sitbon/magg>, default branch `main`, head tree `dafa9c9ccbfc7209f7bd8e5f303d51598d19f59c` | `GET /repos/sitbon/magg/git/trees/main?recursive=1` (146 entries) |
| latest release / PyPI | `v1.3.0` (published 2026-09-26) == PyPI `magg` 1.3.0 | `GET /repos/sitbon/magg/releases/latest`, `GET /pypi/magg/json` |
| language | Python only (no Rust/Go/Node sources, no `Makefile`, no `Cargo.toml`, no `go.mod`, no `package.json` anywhere in the 146-entry tree) | git tree listing |
| nixpkgs pin used for version checks | `nixos-unstable` rev `eaad089433ca2bb662274377d33df3d0e51ef28b` (== this overlay's `nixpkgs` **and** `nixpkgs-latest`, see `flake.lock` nodes `nixpkgs`/`nixpkgs-latest`) | `flake.lock` |
---
## 1. Which nixpkgs build helper
**`python3Packages.buildPythonApplication` with `pyproject = true` and `build-system = [ hatchling ]`.**
Evidence:
- `pyproject.toml:120-122` — `[build-system] requires = ["hatchling"] / build-backend = "hatchling.build"`.
The pin is unpinned (`"hatchling"`, no version specifier), so no `sed` pin-relax hack like
`packages/graphify/package.nix:48-50` is needed.
- Pure-python wheel: PyPI ships `magg-1.3.0-py3-none-any.whl` (62 files, no compiled extension of its own)
and an sdist `magg-1.3.0.tar.gz` (121 entries).
- Version is static in `pyproject.toml` (no `dynamic = ["version"]`, no `uv-dynamic-versioning`), so no
`writableTmpDirAsHomeHook`/setuptools-scm dance is required.
- None of the other helpers apply: no `Cargo.toml` (→ not `buildRustPackage`), no `package.json`/`bun.lock`
(→ not `buildNpmPackage`/`bun2nix`), no `go.mod`/`go.sum` (→ not `buildGoModule`), no `Makefile`
(→ not `stdenv.mkDerivation`).
- In-repo precedent to copy verbatim: `packages/skillsmcp/package.nix:7-30` already does
`python3Packages.buildPythonApplication` + `build-system = [ hatchling ]` + `python3Packages.fastmcp`.
`packages/nftables-analyzer/package.nix` and `packages/freetoken/package.nix:82-104` are the same shape.
Secondary option (deliberately **not** recommended): the repo commits `uv.lock`, so `uv2nix` /
`pyproject-nix` (`pyproject.buildApplication` + `pyparsix`/`fetchPypiTree`) is technically possible.
It buys nothing here because every dependency exists in nixpkgs (see §3), and the overlay has no
`uv2nix` precedent.
## 2. Lockfiles
- **Committed: `uv.lock`** (387 038 bytes at head, also included in the PyPI sdist).
Header: `version = 1`, `revision = 3`, `requires-python = ">=3.12"`, 92 resolved `[[package]]`
entries, all from `https://pypi.org/simple` with sdist+wheel hashes.
- **Not present:** `Cargo.lock`, `package-lock.json`, `bun.lock`, `pnpm-lock.yaml`, `go.sum`,
`requirements.txt`, `poetry.lock`.
- Consequence: there is **no `cargoHash` / `npmDepsHash` / `vendorHash` equivalent to fill in** when using
`buildPythonApplication` — dependencies come from `python3Packages`, so the only hash in the
derivation is the `src` hash. The offline build is automatic (no network at build time).
`uv.lock` would only matter for a `uv sync --locked`-based build (uv2nix route), where each of the
92 locked artifacts must be prefetched — another argument for the nixpkgs-deps route.
## 3. Toolchain version requirements
Declared upstream:
- `pyproject.toml:4` → `requires-python = ">=3.12"`; classifiers list 3.12 / 3.13 / 3.14.
- `.python-version` → `3.14` (upstream's own preferred interpreter).
- `.github/workflows/test.yml` matrix → `python-version: ['3.12', '3.13', '3.14']`, installed via
`astral-sh/setup-uv@v8.2.0` + `uv python install`.
- No Rust / Go / Node toolchain requirement. `nodejs`/`npm` appear only in `dockerfile:10`
(`apk add … nodejs npm`) so that *runtime*-mounted `npx` MCP servers work inside the container image —
not a build requirement.
Fit against this overlay's pinned nixpkgs (`eaad0894…`):
- `pkgs/top-level/all-packages.nix:4484` at that rev says `python3 = python314;`, and
`pkgs/development/interpreters/python/cpython/` ships `3.11 … 3.15`. So the default `python3Packages`
already matches upstream's `.python-version = 3.14`. **No `nixpkgs-latest`, `nixpkgs-rust195` or
`nixpkgs-torch211` instance is needed.**
Direct-dependency floors vs. what the pinned nixpkgs ships (versions read from
`pkgs/development/python-modules/<name>/default.nix` at `eaad0894…`):
| magg requirement (`pyproject.toml:31-42`) | nixpkgs attr | nixpkgs version | ok |
| --- | --- | --- | --- |
| `fastmcp>=3.4,<4` | `fastmcp` (thin wrapper over `fastmcp-slim`) | 3.4.7 | ✅ |
| `mcp>=1.28` | `mcp` | 1.29.0 | ✅ |
| `aiohttp>=3.12.13` | `aiohttp` | 3.14.3 | ✅ |
| `art>=6.5` | `art` | 6.5 | ✅ (exactly at the floor) |
| `cryptography>=45.0.4` | `cryptography` | 50.0.0 | ✅ |
| `pydantic>=2.11.7` | `pydantic` | 2.13.4 | ✅ |
| `pydantic-settings>=2.10.0` | `pydantic-settings` | 2.12.0 | ✅ |
| `pyjwt>=2.10.1` | `pyjwt` | 2.13.0 | ✅ |
| `prompt-toolkit>=3.0.51` | `prompt_toolkit` | 3.0.52 | ✅ |
| `rich>=14.0.0` | `rich` | 15.0.0 | ✅ |
| `watchdog>=6.0.0` | `watchdog` | 6.0.0 | ✅ (exactly at the floor) |
| test extra: `pytest>=8.4`, `pytest-asyncio>=1.0` | `pytest`, `pytest-asyncio` | 9.1.1, 1.4.0 | ✅ |
All 11 runtime deps + both test deps exist as `pkgs/development/python-modules/<name>/`. `fastmcp-slim`
is present and its own `doCheck = pythonAtLeast "3.14"` confirms 3.14 support.
**Bonus evidence that the nixpkgs dep set reproduces upstream's locked environment** — the versions
`uv.lock` resolved on upstream's own machine are the same ones nixpkgs ships:
| package | `uv.lock` @ head | nixpkgs `eaad0894…` |
| --- | --- | --- |
| pytest | 9.1.1 | 9.1.1 |
| pytest-asyncio | 1.4.0 | 1.4.0 |
| mcp | 1.29.0 | 1.29.0 |
| rich | 15.0.0 | 15.0.0 |
| aiohttp | 3.14.3 | 3.14.3 |
| art | 6.5 | 6.5 |
| watchdog | 6.0.0 | 6.0.0 |
| fastmcp | 3.4.5 | 3.4.7 (only drift, still inside `<4`) |
This is a good signal that `doCheck = true` is realistic rather than optimistic.
Undeclared-but-imported modules to keep in mind (they arrive transitively via `fastmcp`/`mcp`/`rich`,
so no extra `dependencies` entry is normally needed, but they explain a `pythonImportsCheck` failure if
a transitive dep is ever cut): `httpx`, `pygments`, `readline`/`tomllib` (stdlib).
## 4. Pre-build steps / build-time network
**None.** Verified:
- No frontend/JS asset, no protobuf/gRPC/ANTLR codegen, no WASM, no `Makefile`, no `build.sh`,
no asset download script in the tree. `scripts/` holds only `bump_version.py` and
`validate_manual_release.py` (release tooling; CI uses the former to check `server.json` ↔
`pyproject.toml` version parity — irrelevant to the build).
- Upstream's own build is `uv sync --locked` + `uv build`/`uv publish`
(`.github/workflows/publish.yml:86-110`), i.e. plain hatchling.
- Hatchling is a pure-python backend; with `dependencies` supplied from `python3Packages` the build is
fully offline. Nothing to prefetch (no `cargoHash`-style extra fetch).
- The only "asset" handling is hatchling copying non-`.py` files that live inside the `magg/` package
directory (see §6). `[tool.hatch.build.targets.sdist]` (`pyproject.toml:100-119`) only affects the
sdist, not the wheel.
## 5. Test suite & `doCheck` policy
Upstream has a real suite: `test/magg/*.py` — 35 files, 328 `def test_*` (180 `async def`).
CI (`.github/workflows/test.yml`) runs `uv sync --all-groups --locked` then `uv run pytest -v`
(no `-m` filter) on 3.12/3.13/3.14; last run of that workflow: **success** (2026-09-26T16:28Z,
`GET /actions/workflows/test.yml/runs`). No `conftest.py`; `[tool.pytest.ini_options]`
(`pyproject.toml:86-89`) declares only the `integration` marker.
Sandbox-safety audit of the suite:
- **No outbound network in tests.** The discovery clients that *do* hit the network
(`magg/discovery/search.py:32` `registry.modelcontextprotocol.io`, `:144` `glama.ai`,
`:264` `api.github.com`, `:318` `registry.npmjs.org`) are exercised only through canned fixtures
(`test/magg/test_registry_search.py:7-56` `SAMPLE_RESPONSE`) and `monkeypatch`ed methods
(`test/magg/test_cli_fixes.py:118-124`). No `httpx`/`aiohttp`/`requests` call appears in `test/`.
- **Loopback only.** The two e2e tests bind `localhost:54322` (hard-coded,
`test/magg/test_e2e_simple.py:65,91`, `test/magg/test_e2e_mounting.py:129`) and poll for up to 10 s.
Loopback works in the Nix sandbox, but the fixed port + wall-clock polling is the main flake risk.
- **Subprocess tests need `python` on PATH**: `test/magg/test_integration.py:37` and
`test/magg/test_mounting_real.py` spawn `python <generated server.py>` stdio MCP backends;
`test/magg/test_server_robustness.py:374` and `test/magg/test_cli_fixes.py:140` run
`sys.executable -m magg …`. `buildPythonPackage`'s `checkPhase` runs inside its venv, so `python`
resolves and `magg` is importable — satisfied automatically.
- **`$HOME` is read, not written**: `test/magg/test_config.py:95`, `test_auth.py:25`,
`test_config_safety.py:76` assert `Path.home()/".magg"` / `Path.home()/".ssh"/"magg"` defaults;
`test_cli_fixes.py:25` monkeypatches `HOME` to a tmp dir. Scratch space is
`tempfile.TemporaryDirectory()` (→ `$TMPDIR`, writable in the sandbox).
- **`sh` is required**: `test/magg/test_cli_fixes.py:78` runs `subprocess.run(["sh", "-c", …])`
(bash provides `bin/sh` in the build env — fine).
- Timing-sensitive groups: `test_config_reload.py` (watchdog/polling), `test_server_robustness.py`
(hung-backend timeouts).
Recommended policy (mirrors the `packages/marmel/package.nix:33-60` comment style — explain each
workaround and what would let it be dropped):
```nix
nativeCheckInputs = with python3Packages; [ pytestCheckHook pytest-asyncio ];
# Upstream runs the whole suite (`.github/workflows/test.yml`: `uv run pytest -v`), but the three
# `integration`-marked tests (test/magg/test_basic.py:97, test_e2e_mounting.py:18,
# test_e2e_simple.py:16) spawn real stdio/HTTP backends and hard-code localhost:54322 with a
# 10 s poll loop (test_e2e_simple.py:65,91), which is the only flake source in an otherwise
# network-free suite (registry/glama/github/npm clients are monkeypatched in the unit tests).
# Drop this once upstream picks an ephemeral port.
pytestOptions = [ "-m" "not integration" ];
preCheck = ''
# pydantic-settings defaults read $HOME (magg/settings.py:72,203) and the config-reload tests
# poll files under $TMPDIR; keep both inside the writable sandbox tmp dir.
export MAGG_AUTO_RELOAD=false
mkdir -p /tmp/magg-check
export TMPDIR=/tmp/magg-check
export HOME=/tmp/magg-check
'';
pythonImportsCheck = [ "magg" ]; # import magg pulls fastmcp/mcp/aiohttp/cryptography
```
Notes for the implementer:
- `pytest-asyncio` is mandatory in `nativeCheckInputs`: essentially every async test carries
`@pytest.mark.asyncio` (178 marker lines for 180 `async def test_*` definitions) and pytest-asyncio
defaults to strict mode, so unmarked async tests are only skipped. The 2 apparently unmarked ones are
therefore silently skipped upstream too — do not add `-W error`.
- Only the `[project.optional-dependencies] test` extra is needed; the `[dependency-groups] dev`
group additionally pulls `keyring`, `anthropic`, `ruff` — do **not** add those.
- If the suite turns out to be flaky in practice, the repo-wide fallback is
`doCheck = false; pythonImportsCheck = [ "magg" ];` as in `packages/skillsmcp/package.nix:33-34`.
## 6. Install phase
Standard wheel install — **no custom `installPhase` needed.**
- The published wheel contains exactly `magg/**` + `magg-1.3.0.dist-info/` (62 files), so
`buildPythonApplication` puts `$out/lib/python3.14/site-packages/magg/` plus two console scripts
from `entry_points.txt`: `magg = magg.cli:main`, `mbro = magg.mbro.cli:main`
(same as `pyproject.toml:65-67`) → `$out/bin/magg`, `$out/bin/mbro`. Set `mainProgram = "magg";`.
- **Runtime-loaded data files live inside the package dir and must not be stripped**:
`magg/contrib/magg.mbro`, `magg/contrib/readme.md`, `magg/contrib/kit.d/example.json` (confirmed
present in the wheel). `magg/contrib` is a **namespace package with no `__init__.py`**; hatchling
still ships it, and it is discovered at runtime via `magg/util/paths.py:12-15`
(`contrib.__path__`), which feeds the `MAGG_PATH` search list at `magg/settings.py:203`
(`./.magg`, `~/.magg`, contrib dirs) used for `config.json` and `kit.d/`.
⇒ Do not add `disallowedReReferences`/`find -delete` rules for `*.json`/`*.mbro`, and do not
"fix" the missing `__init__.py`.
- Everything else in the repo is *not* installed and must not be: `server.json` (MCP-registry manifest),
`compose.yaml`, `dockerfile`, `docs/`, `examples/`, `scripts/`, and `etc/glama-mcp-api.json`
(a static API dump — `grep -rn "glama-mcp-api" magg/` has no hits; the live code calls
`https://glama.ai/api/mcp/v1/servers` instead).
- No `share/` payloads, no shell completions, no man pages, no `py.typed`.
- Hatchling needs `readme.md` and `license.md` present (`pyproject.toml:6-7`); both are in git and in
the sdist.
## 7. Runtime dependencies
- The Python interpreter wrapper comes from `buildPythonApplication` (shebang points at
`python3`'s `$out`), plus the 11 declared python deps and their transitive closure
(`openssl` via `cryptography`, `readline` — `magg/mbro/arepl.py` imports `readline`, and nixpkgs'
CPython already carries it, `pcre2`/`multidict`/`yarl` C extensions from `aiohttp`).
- **TLS trust store** for the search features: `magg/discovery/search.py:40` builds
`aiohttp.ClientSession(trust_env=True)` against `registry.modelcontextprotocol.io`, `glama.ai`,
`api.github.com`, `registry.npmjs.org`. nixpkgs' OpenSSL/CPython resolve the bundle from
`SSL_CERT_FILE` / `/etc/ssl/certs`, so it works out of the box on NixOS; on non-NixOS hosts users
need `SSL_CERT_FILE` (same class of issue documented in `packages/marmel/package.nix:33-45`).
Adding `cacert` to the *runtime* wrapper is optional, not required.
- **Writable state at runtime**: `CatalogManager` writes `<cwd>/.magg/search_cache.json`
(`magg/discovery/catalog.py:15-16`) and the config lives at `./.magg/config.json` or
`~/.magg/config.json` (`magg/settings.py:203`, `MAGG_CONFIG_PATH`, `MAGG_PATH`, `MAGG_READ_ONLY`).
Worth a `meta` note / NixOS-module hint, nothing build-time.
- **Optional external programs** (only when a user mounts such a server): `npx`/`node`, `uvx`,
`python` for stdio backends — spawned by `fastmcp`, not by magg itself
(`grep -rn "subprocess|Popen" magg/` → no hits). Upstream's image installs `nodejs`/`npm`
(`dockerfile:10`) for that reason. Not a build or hard runtime dep.
- `magg --version` reads `importlib.metadata.version("magg")` (`magg/__init__.py:9-12`) → works
because the dist-info is installed; no source-tree assumption.
## 8. Prior art
**`magg` is not packaged in nixpkgs, and upstream ships no Nix expression.**
- Complete (non-truncated) listing of `pkgs/development/python-modules` at `eaad0894…` → 9949 entries,
none named `magg` (nearest neighbours in that listing: `magic`, `magic-filter`, `magicgui`,
`magic-wormhole`, `magika`).
- `pkgs/top-level/all-packages.nix` at that rev → `grep -i magg` returns nothing (so it is not a
top-level tool either).
- The magg repo (146-entry tree) contains no `.nix` file and no `flake.nix`; upstream's documented
install paths are `uv tool install magg`, `uvx --from git+https://github.com/sitbon/magg.git magg`,
`pip`/`poetry`, and GHCR Docker images.
- Nothing reusable to mirror; write from scratch. Best in-repo template:
`packages/skillsmcp/package.nix` (hatchling + `python3Packages.fastmcp`), with
`packages/nftables-analyzer/package.nix` as the `fetchFromGitHub` + dated-version variant.
## 9. Update mechanism
Use the plain form (repo-wide list style, cf. `packages/agnix/package.nix:34-38`):
```nix
passthru.updateScript = [
"nix-update"
"--flake"
".#magg"
];
```
- Upstream **does** tag releases: `v1.3.0` (2026-09-26) matching PyPI `magg 1.3.0`, and `pyproject.toml`
carries a static `version`, so nix-update's default version detection works — no
`--version=branch=main` (that form is only used in this repo for untagged upstreams, e.g.
`packages/ds4/package.nix:78`, `packages/skillsmcp/package.nix:39`).
- **Caveat if you use `fetchFromGitHub` + `tag = "v${version}"`:** the full tag list (30 tags) also
contains two-part alias tags — `v1.3`, `v1.2`, `v1.1`, `v0.10`, `v0.9`, `v0.8`, `v0.7`, `v0.6` — plus
non-version tags `root`, `latest-release`, `latest-publish` and legacy `magg/v0.5.1.53`-style tags, so
a tag-based bump can pick the `v1.3` alias (a valid tag, but an ugly version string) or a junk tag.
Two clean options:
1. **Preferred:** build from PyPI — `src = fetchPypi { pname = "magg"; inherit version; format = "pyproject"; }`.
The sdist is complete (121 entries: `magg/`, `test/`, `pyproject.toml`, `uv.lock`, `readme.md`,
`license.md`), so `checkPhase` still works, and nix-update's PyPI updater resolves clean versions
deterministically. Ready-made hash for 1.3.0:
`sha256-2jzBtka4YWp4yEGvJorHN/JQpI6sWCrmRu/TtvRnqpU=`
(sha256 `da3cc1b646b8616a78c841af268ac737f250a48eac582ae646efd3b6f467aa95`).
2. Keep `fetchFromGitHub` (rev = tag `v1.3.0`) and pin the updater with `--version=tag`, then verify
the picked version is the 3-part tag.
---
### One-paragraph summary for the implementer
`magg` is a pure-python hatchling project: use `python3Packages.buildPythonApplication` with
`pyproject = true`, `build-system = [ python3Packages.hatchling ]`, and all 11 runtime deps taken from
`python3Packages` (all present and new enough on the overlay's pinned `nixos-unstable`, whose default
`python3` is 3.14 — exactly upstream's `.python-version`). No lockfile hash, no vendorHash, no
pre-build codegen, no asset fetch, no custom install phase; two console scripts (`magg`, `mbro`) and
the `magg/contrib/**` data files come along via the wheel. Run the test suite (`pytest` +
`pytest-asyncio`) with `-m "not integration"` and a tmp `$HOME`/`$TMPDIR` in `preCheck`, keep
`pythonImportsCheck = [ "magg" ]`, set `mainProgram = "magg"`, license `agpl3Plus`
(`license = {text = "AGPL-3.0-or-later"}`), and update with plain `nix-update --flake .#magg`
(prefer `fetchPypi` over `fetchFromGitHub` because upstream also publishes `v1.3`-style alias tags).
MISSION COMPLETE (t-002)