Files
alex 44e6d90a77
CI / check (push) Has been cancelled
feat(magg): add magg 1.3.0 MCP aggregator package
Add magg, an MCP aggregator / meta-MCP server, to the overlay. It is a
Python package on the hatchling backend, built with
python3Packages.buildPythonApplication from the v1.3.0 source tarball,
so no upstream release artifacts are fetched. Ships both console
scripts declared upstream: magg (the aggregator CLI/server) and mbro
(the MCP browser). License is AGPL-3.0-or-later.

The install check runs the upstream pytest suite with
-m "not integration", because the integration-marked tests need a live
server. A forced nix build .#magg --rebuild passes with 329 passed,
3 skipped, 3 deselected, 0 failed, and both binaries smoke-test clean.

Also adds the README and AGENTS.md package rows, the upstream and build
research notes, and the verification report in docs/magg_build_report.md.
2026-11-08 12:48:30 +03:00

18 KiB

Build/packaging requirements: sitbon/magg

Companion to docs/research_magg_upstream.md (upstream fact sheet). This file answers only the build/packaging questions an implementer needs answered before picking a build helper.

Evidence snapshot (all verified, nothing inferred):

item value source
upstream https://github.com/sitbon/magg, default branch main, head tree dafa9c9ccbfc7209f7bd8e5f303d51598d19f59c GET /repos/sitbon/magg/git/trees/main?recursive=1 (146 entries)
latest release / PyPI v1.3.0 (published 2026-09-26) == PyPI magg 1.3.0 GET /repos/sitbon/magg/releases/latest, GET /pypi/magg/json
language Python only (no Rust/Go/Node sources, no Makefile, no Cargo.toml, no go.mod, no package.json anywhere in the 146-entry tree) git tree listing
nixpkgs pin used for version checks nixos-unstable rev eaad089433ca2bb662274377d33df3d0e51ef28b (== this overlay's nixpkgs and nixpkgs-latest, see flake.lock nodes nixpkgs/nixpkgs-latest) flake.lock

1. Which nixpkgs build helper

python3Packages.buildPythonApplication with pyproject = true and build-system = [ hatchling ].

Evidence:

  • pyproject.toml:120-122 — [build-system] requires = ["hatchling"] / build-backend = "hatchling.build". The pin is unpinned ("hatchling", no version specifier), so no sed pin-relax hack like packages/graphify/package.nix:48-50 is needed.
  • Pure-python wheel: PyPI ships magg-1.3.0-py3-none-any.whl (62 files, no compiled extension of its own) and an sdist magg-1.3.0.tar.gz (121 entries).
  • Version is static in pyproject.toml (no dynamic = ["version"], no uv-dynamic-versioning), so no writableTmpDirAsHomeHook/setuptools-scm dance is required.
  • None of the other helpers apply: no Cargo.toml (→ not buildRustPackage), no package.json/bun.lock (→ not buildNpmPackage/bun2nix), no go.mod/go.sum (→ not buildGoModule), no Makefile (→ not stdenv.mkDerivation).
  • In-repo precedent to copy verbatim: packages/skillsmcp/package.nix:7-30 already does python3Packages.buildPythonApplication + build-system = [ hatchling ] + python3Packages.fastmcp. packages/nftables-analyzer/package.nix and packages/freetoken/package.nix:82-104 are the same shape.

Secondary option (deliberately not recommended): the repo commits uv.lock, so uv2nix / pyproject-nix (pyproject.buildApplication + pyparsix/fetchPypiTree) is technically possible. It buys nothing here because every dependency exists in nixpkgs (see §3), and the overlay has no uv2nix precedent.

2. Lockfiles

  • Committed: uv.lock (387 038 bytes at head, also included in the PyPI sdist). Header: version = 1, revision = 3, requires-python = ">=3.12", 92 resolved [[package]] entries, all from https://pypi.org/simple with sdist+wheel hashes.
  • Not present: Cargo.lock, package-lock.json, bun.lock, pnpm-lock.yaml, go.sum, requirements.txt, poetry.lock.
  • Consequence: there is no cargoHash / npmDepsHash / vendorHash equivalent to fill in when using buildPythonApplication — dependencies come from python3Packages, so the only hash in the derivation is the src hash. The offline build is automatic (no network at build time). uv.lock would only matter for a uv sync --locked-based build (uv2nix route), where each of the 92 locked artifacts must be prefetched — another argument for the nixpkgs-deps route.

3. Toolchain version requirements

Declared upstream:

  • pyproject.toml:4 → requires-python = ">=3.12"; classifiers list 3.12 / 3.13 / 3.14.
  • .python-version → 3.14 (upstream's own preferred interpreter).
  • .github/workflows/test.yml matrix → python-version: ['3.12', '3.13', '3.14'], installed via astral-sh/setup-uv@v8.2.0 + uv python install.
  • No Rust / Go / Node toolchain requirement. nodejs/npm appear only in dockerfile:10 (apk add … nodejs npm) so that runtime-mounted npx MCP servers work inside the container image — not a build requirement.

Fit against this overlay's pinned nixpkgs (eaad0894…):

  • pkgs/top-level/all-packages.nix:4484 at that rev says python3 = python314;, and pkgs/development/interpreters/python/cpython/ ships 3.11 … 3.15. So the default python3Packages already matches upstream's .python-version = 3.14. No nixpkgs-latest, nixpkgs-rust195 or nixpkgs-torch211 instance is needed.

Direct-dependency floors vs. what the pinned nixpkgs ships (versions read from pkgs/development/python-modules/<name>/default.nix at eaad0894…):

magg requirement (pyproject.toml:31-42) nixpkgs attr nixpkgs version ok
fastmcp>=3.4,<4 fastmcp (thin wrapper over fastmcp-slim) 3.4.7 ✅
mcp>=1.28 mcp 1.29.0 ✅
aiohttp>=3.12.13 aiohttp 3.14.3 ✅
art>=6.5 art 6.5 ✅ (exactly at the floor)
cryptography>=45.0.4 cryptography 50.0.0 ✅
pydantic>=2.11.7 pydantic 2.13.4 ✅
pydantic-settings>=2.10.0 pydantic-settings 2.12.0 ✅
pyjwt>=2.10.1 pyjwt 2.13.0 ✅
prompt-toolkit>=3.0.51 prompt_toolkit 3.0.52 ✅
rich>=14.0.0 rich 15.0.0 ✅
watchdog>=6.0.0 watchdog 6.0.0 ✅ (exactly at the floor)
test extra: pytest>=8.4, pytest-asyncio>=1.0 pytest, pytest-asyncio 9.1.1, 1.4.0 ✅

All 11 runtime deps + both test deps exist as pkgs/development/python-modules/<name>/. fastmcp-slim is present and its own doCheck = pythonAtLeast "3.14" confirms 3.14 support.

Bonus evidence that the nixpkgs dep set reproduces upstream's locked environment — the versions uv.lock resolved on upstream's own machine are the same ones nixpkgs ships:

package uv.lock @ head nixpkgs eaad0894…
pytest 9.1.1 9.1.1
pytest-asyncio 1.4.0 1.4.0
mcp 1.29.0 1.29.0
rich 15.0.0 15.0.0
aiohttp 3.14.3 3.14.3
art 6.5 6.5
watchdog 6.0.0 6.0.0
fastmcp 3.4.5 3.4.7 (only drift, still inside <4)

This is a good signal that doCheck = true is realistic rather than optimistic.

Undeclared-but-imported modules to keep in mind (they arrive transitively via fastmcp/mcp/rich, so no extra dependencies entry is normally needed, but they explain a pythonImportsCheck failure if a transitive dep is ever cut): httpx, pygments, readline/tomllib (stdlib).

4. Pre-build steps / build-time network

None. Verified:

  • No frontend/JS asset, no protobuf/gRPC/ANTLR codegen, no WASM, no Makefile, no build.sh, no asset download script in the tree. scripts/ holds only bump_version.py and validate_manual_release.py (release tooling; CI uses the former to check server.json ↔ pyproject.toml version parity — irrelevant to the build).
  • Upstream's own build is uv sync --locked + uv build/uv publish (.github/workflows/publish.yml:86-110), i.e. plain hatchling.
  • Hatchling is a pure-python backend; with dependencies supplied from python3Packages the build is fully offline. Nothing to prefetch (no cargoHash-style extra fetch).
  • The only "asset" handling is hatchling copying non-.py files that live inside the magg/ package directory (see §6). [tool.hatch.build.targets.sdist] (pyproject.toml:100-119) only affects the sdist, not the wheel.

5. Test suite & doCheck policy

Upstream has a real suite: test/magg/*.py — 35 files, 328 def test_* (180 async def). CI (.github/workflows/test.yml) runs uv sync --all-groups --locked then uv run pytest -v (no -m filter) on 3.12/3.13/3.14; last run of that workflow: success (2026-09-26T16:28Z, GET /actions/workflows/test.yml/runs). No conftest.py; [tool.pytest.ini_options] (pyproject.toml:86-89) declares only the integration marker.

Sandbox-safety audit of the suite:

  • No outbound network in tests. The discovery clients that do hit the network (magg/discovery/search.py:32 registry.modelcontextprotocol.io, :144 glama.ai, :264 api.github.com, :318 registry.npmjs.org) are exercised only through canned fixtures (test/magg/test_registry_search.py:7-56 SAMPLE_RESPONSE) and monkeypatched methods (test/magg/test_cli_fixes.py:118-124). No httpx/aiohttp/requests call appears in test/.
  • Loopback only. The two e2e tests bind localhost:54322 (hard-coded, test/magg/test_e2e_simple.py:65,91, test/magg/test_e2e_mounting.py:129) and poll for up to 10 s. Loopback works in the Nix sandbox, but the fixed port + wall-clock polling is the main flake risk.
  • Subprocess tests need python on PATH: test/magg/test_integration.py:37 and test/magg/test_mounting_real.py spawn python <generated server.py> stdio MCP backends; test/magg/test_server_robustness.py:374 and test/magg/test_cli_fixes.py:140 run sys.executable -m magg …. buildPythonPackage's checkPhase runs inside its venv, so python resolves and magg is importable — satisfied automatically.
  • $HOME is read, not written: test/magg/test_config.py:95, test_auth.py:25, test_config_safety.py:76 assert Path.home()/".magg" / Path.home()/".ssh"/"magg" defaults; test_cli_fixes.py:25 monkeypatches HOME to a tmp dir. Scratch space is tempfile.TemporaryDirectory() (→ $TMPDIR, writable in the sandbox).
  • sh is required: test/magg/test_cli_fixes.py:78 runs subprocess.run(["sh", "-c", …]) (bash provides bin/sh in the build env — fine).
  • Timing-sensitive groups: test_config_reload.py (watchdog/polling), test_server_robustness.py (hung-backend timeouts).

Recommended policy (mirrors the packages/marmel/package.nix:33-60 comment style — explain each workaround and what would let it be dropped):

  nativeCheckInputs = with python3Packages; [ pytestCheckHook pytest-asyncio ];

  # Upstream runs the whole suite (`.github/workflows/test.yml`: `uv run pytest -v`), but the three
  # `integration`-marked tests (test/magg/test_basic.py:97, test_e2e_mounting.py:18,
  # test_e2e_simple.py:16) spawn real stdio/HTTP backends and hard-code localhost:54322 with a
  # 10 s poll loop (test_e2e_simple.py:65,91), which is the only flake source in an otherwise
  # network-free suite (registry/glama/github/npm clients are monkeypatched in the unit tests).
  # Drop this once upstream picks an ephemeral port.
  pytestOptions = [ "-m" "not integration" ];

  preCheck = ''
    # pydantic-settings defaults read $HOME (magg/settings.py:72,203) and the config-reload tests
    # poll files under $TMPDIR; keep both inside the writable sandbox tmp dir.
    export MAGG_AUTO_RELOAD=false
    mkdir -p /tmp/magg-check
    export TMPDIR=/tmp/magg-check
    export HOME=/tmp/magg-check
  '';

  pythonImportsCheck = [ "magg" ];   # import magg pulls fastmcp/mcp/aiohttp/cryptography

Notes for the implementer:

  • pytest-asyncio is mandatory in nativeCheckInputs: essentially every async test carries @pytest.mark.asyncio (178 marker lines for 180 async def test_* definitions) and pytest-asyncio defaults to strict mode, so unmarked async tests are only skipped. The 2 apparently unmarked ones are therefore silently skipped upstream too — do not add -W error.
  • Only the [project.optional-dependencies] test extra is needed; the [dependency-groups] dev group additionally pulls keyring, anthropic, ruff — do not add those.
  • If the suite turns out to be flaky in practice, the repo-wide fallback is doCheck = false; pythonImportsCheck = [ "magg" ]; as in packages/skillsmcp/package.nix:33-34.

6. Install phase

Standard wheel install — no custom installPhase needed.

  • The published wheel contains exactly magg/** + magg-1.3.0.dist-info/ (62 files), so buildPythonApplication puts $out/lib/python3.14/site-packages/magg/ plus two console scripts from entry_points.txt: magg = magg.cli:main, mbro = magg.mbro.cli:main (same as pyproject.toml:65-67) → $out/bin/magg, $out/bin/mbro. Set mainProgram = "magg";.
  • Runtime-loaded data files live inside the package dir and must not be stripped: magg/contrib/magg.mbro, magg/contrib/readme.md, magg/contrib/kit.d/example.json (confirmed present in the wheel). magg/contrib is a namespace package with no __init__.py; hatchling still ships it, and it is discovered at runtime via magg/util/paths.py:12-15 (contrib.__path__), which feeds the MAGG_PATH search list at magg/settings.py:203 (./.magg, ~/.magg, contrib dirs) used for config.json and kit.d/. ⇒ Do not add disallowedReReferences/find -delete rules for *.json/*.mbro, and do not "fix" the missing __init__.py.
  • Everything else in the repo is not installed and must not be: server.json (MCP-registry manifest), compose.yaml, dockerfile, docs/, examples/, scripts/, and etc/glama-mcp-api.json (a static API dump — grep -rn "glama-mcp-api" magg/ has no hits; the live code calls https://glama.ai/api/mcp/v1/servers instead).
  • No share/ payloads, no shell completions, no man pages, no py.typed.
  • Hatchling needs readme.md and license.md present (pyproject.toml:6-7); both are in git and in the sdist.

7. Runtime dependencies

  • The Python interpreter wrapper comes from buildPythonApplication (shebang points at python3's $out), plus the 11 declared python deps and their transitive closure (openssl via cryptography, readline — magg/mbro/arepl.py imports readline, and nixpkgs' CPython already carries it, pcre2/multidict/yarl C extensions from aiohttp).
  • TLS trust store for the search features: magg/discovery/search.py:40 builds aiohttp.ClientSession(trust_env=True) against registry.modelcontextprotocol.io, glama.ai, api.github.com, registry.npmjs.org. nixpkgs' OpenSSL/CPython resolve the bundle from SSL_CERT_FILE / /etc/ssl/certs, so it works out of the box on NixOS; on non-NixOS hosts users need SSL_CERT_FILE (same class of issue documented in packages/marmel/package.nix:33-45). Adding cacert to the runtime wrapper is optional, not required.
  • Writable state at runtime: CatalogManager writes <cwd>/.magg/search_cache.json (magg/discovery/catalog.py:15-16) and the config lives at ./.magg/config.json or ~/.magg/config.json (magg/settings.py:203, MAGG_CONFIG_PATH, MAGG_PATH, MAGG_READ_ONLY). Worth a meta note / NixOS-module hint, nothing build-time.
  • Optional external programs (only when a user mounts such a server): npx/node, uvx, python for stdio backends — spawned by fastmcp, not by magg itself (grep -rn "subprocess|Popen" magg/ → no hits). Upstream's image installs nodejs/npm (dockerfile:10) for that reason. Not a build or hard runtime dep.
  • magg --version reads importlib.metadata.version("magg") (magg/__init__.py:9-12) → works because the dist-info is installed; no source-tree assumption.

8. Prior art

magg is not packaged in nixpkgs, and upstream ships no Nix expression.

  • Complete (non-truncated) listing of pkgs/development/python-modules at eaad0894… → 9949 entries, none named magg (nearest neighbours in that listing: magic, magic-filter, magicgui, magic-wormhole, magika).
  • pkgs/top-level/all-packages.nix at that rev → grep -i magg returns nothing (so it is not a top-level tool either).
  • The magg repo (146-entry tree) contains no .nix file and no flake.nix; upstream's documented install paths are uv tool install magg, uvx --from git+https://github.com/sitbon/magg.git magg, pip/poetry, and GHCR Docker images.
  • Nothing reusable to mirror; write from scratch. Best in-repo template: packages/skillsmcp/package.nix (hatchling + python3Packages.fastmcp), with packages/nftables-analyzer/package.nix as the fetchFromGitHub + dated-version variant.

9. Update mechanism

Use the plain form (repo-wide list style, cf. packages/agnix/package.nix:34-38):

  passthru.updateScript = [
    "nix-update"
    "--flake"
    ".#magg"
  ];
  • Upstream does tag releases: v1.3.0 (2026-09-26) matching PyPI magg 1.3.0, and pyproject.toml carries a static version, so nix-update's default version detection works — no --version=branch=main (that form is only used in this repo for untagged upstreams, e.g. packages/ds4/package.nix:78, packages/skillsmcp/package.nix:39).
  • Caveat if you use fetchFromGitHub + tag = "v${version}": the full tag list (30 tags) also contains two-part alias tags — v1.3, v1.2, v1.1, v0.10, v0.9, v0.8, v0.7, v0.6 — plus non-version tags root, latest-release, latest-publish and legacy magg/v0.5.1.53-style tags, so a tag-based bump can pick the v1.3 alias (a valid tag, but an ugly version string) or a junk tag. Two clean options:
    1. Preferred: build from PyPI — src = fetchPypi { pname = "magg"; inherit version; format = "pyproject"; }. The sdist is complete (121 entries: magg/, test/, pyproject.toml, uv.lock, readme.md, license.md), so checkPhase still works, and nix-update's PyPI updater resolves clean versions deterministically. Ready-made hash for 1.3.0: sha256-2jzBtka4YWp4yEGvJorHN/JQpI6sWCrmRu/TtvRnqpU= (sha256 da3cc1b646b8616a78c841af268ac737f250a48eac582ae646efd3b6f467aa95).
    2. Keep fetchFromGitHub (rev = tag v1.3.0) and pin the updater with --version=tag, then verify the picked version is the 3-part tag.

One-paragraph summary for the implementer

magg is a pure-python hatchling project: use python3Packages.buildPythonApplication with pyproject = true, build-system = [ python3Packages.hatchling ], and all 11 runtime deps taken from python3Packages (all present and new enough on the overlay's pinned nixos-unstable, whose default python3 is 3.14 — exactly upstream's .python-version). No lockfile hash, no vendorHash, no pre-build codegen, no asset fetch, no custom install phase; two console scripts (magg, mbro) and the magg/contrib/** data files come along via the wheel. Run the test suite (pytest + pytest-asyncio) with -m "not integration" and a tmp $HOME/$TMPDIR in preCheck, keep pythonImportsCheck = [ "magg" ], set mainProgram = "magg", license agpl3Plus (license = {text = "AGPL-3.0-or-later"}), and update with plain nix-update --flake .#magg (prefer fetchPypi over fetchFromGitHub because upstream also publishes v1.3-style alias tags).

MISSION COMPLETE (t-002)