Files
millerson-overlay.nix/README.md
T
alex e061d3c170
CI / check (push) Has been cancelled
feat(kyojin): add Kyojin ExLlamaV3 engine for AMD Strix Halo
Kyojin (Yamz-Labs/kyojin) is a fork of ExLlamaV3 that runs 100 GB-class
EXL3 MoE packs on one 128 GB Ryzen AI Max box. It ships a torch C++
extension that torch's cpp_extension cross-compiles to HIP, and upstream
supports gfx1151 only, so the kernels are built for gfx1151 alone and
the package is marked Linux-only. Pinned to the v1.3 release.

The extension must be compiled against a ROCm build of torch. The flake's
nixos-unstable carries torch 2.13, whose ROCm build fails in nixpkgs (the
CK SDPA configure step runs a script through /bin/bash) and has no binary
on the cache, so the build would compile PyTorch from source and die.
python3Packages.torchWithRocm from the already pinned nixpkgs-torch211
input gives torch 2.11 with ROCm 7.2.2, gfx1151 in its target list and a
cached binary, so default.nix builds through that input the way freetoken
does.

nixpkgs splits the single devel tree the AMD wheels ship, so two
symlinkJoins stand in for it: one as ROCM_HOME (hipcc, headers, amdgcn
bitcode) and one handed to setup.py as EXL3_ROCM_DEV_INCLUDE, which wants
hipsparse/, rocsparse/, rocrand/, thrust/ and pybind11 (nixpkgs torch no
longer exports pybind11 headers). hipsolver is in there because torch's
own HIPContextLight.h includes it.

setup.py gets one patch in postPatch. It imports exllamav3 to reach
build_config, that runs the package __init__, which imports ext.py, which
JIT-compiles the entire extension whenever no precompiled exllamav3_ext is
importable, meaning every wheel build, into a $HOME the sandbox does not
grant. Registering a stub package keeps the two leaf modules importable
without that detour.

doCheck = false: tests/ drives a real gfx1151 GPU and the published
packs. Inference on a card is not verified here, this builder has no
/dev/kfd, so the wheel-only LD_PRELOAD workaround for torch's bundled HSA
runtime is untested. It should not be needed, the store torch links the
store rocm-runtime.

Verified with nix build .#kyojin at v1.3: exllamav3_ext loads, torch
reports hip 7.2.53211, the closure holds one torch (ROCm), and
kyojin-serve-qwen plus kyojin-serve-glm print their usage.

Refs nix-overlay-du9
2026-10-07 16:14:09 +03:00

11 KiB

millerson.name Nix Overlay

A custom Nix overlay and flake providing additional packages not found in upstream nixpkgs, built using numtide/blueprint for streamlined flake development.

Features

  • Custom packages - Additional software packaged for Nix
  • Two overlay strategies - Choose between binary-cache-friendly or dependency-sharing overlays
  • Blueprint-based - Uses modern Nix flake patterns with numtide/blueprint

Available Packages

Package Description Category
awg-tool CLI for AmneziaWG self-hosting — generates 1.0/1.5/2.0/3.0 obfuscation parameters, exports .conf and vpn:// configs, installs servers over SSH Networking
aionui Free, open-source, Cowork app with AI Agents AI Coding Agents
codeaf Coding harness and software factory for open models: hand work off, then watch every project from one terminal AI Coding Agents
container-use Containerized environments for coding agents AI Coding Agents
desloppify Multi-language codebase health scanner and technical debt tracker for AI agents AI Coding Agents
ds4 DeepSeek 4 Flash and PRO local inference engine for ROCm (Strix Halo) AI Inference
freebuff The world's strongest free coding agent AI Coding Agents
freetoken Local MoE-offload LLM inference runtime with OpenAI- and Anthropic-compatible APIs AI Inference
graphify Turn any folder of code, docs, papers, images, or videos into a queryable knowledge graph AI Coding Agents
g3 AI coding agent that writes code and executes commands, with multi-provider LLM support, context compaction, and desktop automation AI Coding Agents
haivemind Multi-model AI consensus, aggregation, and fusion runner for popular AI CLIs AI Coding Agents
hipengine ROCm-native local LLM inference engine with torch-free runtime for AMD RDNA GPUs AI Inference
marmel Autonomous agentic coding assistant with Manager + specialist subagent orchestration over any OpenAI-compatible LLM backend AI Coding Agents
mcp-gateway Universal Model Context Protocol gateway that sits between AI client and MCP tools/servers MCP Servers
nftables-analyzer Analyze nftables firewall rules and evaluate traffic queries Networking
nftablesbuilder Web interface to manage nftables rules with drag-and-drop rule creation Networking
shardr Decentralized LLM repository: content-addressed model store, BitTorrent-based sync, OpenAI-compatible serving via llama-server AI Inference
skillsmcp MCP server that exposes Agent Skills to AI agents via the Model Context Protocol MCP Servers
kubernetes-mcp-server Model Context Protocol (MCP) server for Kubernetes and OpenShift MCP Servers
kyojin ExLlamaV3-based inference engine with speculative decoding for 100 GB-class MoE packs on one 128 GB AMD Strix Halo (gfx1151, ROCm 7) AI Inference
llama-cpp llama.cpp pinned to build b11439, built for Strix Halo (gfx1151) with ROCm + Vulkan backends AI Inference
loop Corporate messenger for your team Communication
radar Modern Kubernetes visibility — topology, event timeline, service traffic, resource browsing, Helm management, and GitOps support Kubernetes
omniroute Unified AI router with 160+ providers, auto fallback, MCP/A2A, OpenAI-compatible APIs AI LLM Gateway
open-code-review AI-powered code review CLI combining deterministic pipelines with an LLM agent for line-level review comments AI Coding Agents
relay-free-llm RESTful API to route user prompts to various AI model providers with automatic failover and intent-based routing AI LLM Gateway
stakpak DevOps AI agent that generates infrastructure code, debugs Kubernetes, configures CI/CD, and automates deployments AI Agents
traycer Open-source AI orchestration app for agentic coding AI Coding Agents

Usage

As a Flake

Add to your flake.nix inputs:

inputs.millerson-nix-overlay.url = "git+https://git.millerson.name/alex/millerson-overlay.nix.git";

Then use in your outputs:

outputs = { nixpkgs, millerson-nix-overlay, ... }: {
  nixosConfigurations.your-host = nixpkgs.lib.nixosSystem {
    system = "x86_64-linux";
    modules = [
      millerson-nix-overlay.nixosModules.default
      # ... your other modules
    ];
  };
};

As an Overlay

Option 1: Default Overlay (Binary Cache Friendly)

Uses the packages built against this flake's nixpkgs revision. Binary cache hits work best when using the same nixpkgs revision.

nixpkgs.overlays = [ millerson-nix-overlay.overlays.default ];

Option 2: Shared Nixpkgs Overlay (Dependency Sharing)

Builds packages against your system's nixpkgs, sharing dependencies with the rest of your system. No second nixpkgs evaluation, but binary cache only hits when your nixpkgs revision matches ours.

nixpkgs.overlays = [ millerson-nix-overlay.overlays.shared-nixpkgs ];

Installing Packages

With flakes enabled:

# Try a package
nix run git+https://git.millerson.name/alex/millerson-overlay.nix.git#mcp-gateway

# Install permanently
nix profile install git+https://git.millerson.name/alex/millerson-overlay.nix.git#mcp-gateway

nftablesbuilder Runtime Notes

nftablesbuilder is a web interface for managing nftables. It consists of a root-launcher (nftablesbuilder) that spawns the unprivileged webserver binary and pipes encrypted commands between it and the nft binary.

Before running it you must:

  1. Copy the settings template to /etc/nftablesbuilder (the only fixed path):
    nix build .#nftablesbuilder
    cp result/share/nftablesbuilder/settings.example /etc/nftablesbuilder
    
    Adjust paths inside (html, webserver, nft, savepath, TLS files).
  2. Create a TLS key/certificate pair at the paths referenced by tlskey/tlscert (e.g. openssl req -x509 -newkey rsa:2048 -nodes ...).
  3. Run the launcher as root (it needs to write /etc/nftables.conf and run nft); the web interface listens on https://<server>:1969.

Note: upstream publishes the GUI only as a prebuilt tarball on nftablesbuilder.eu (served with a self-signed certificate, hence the curlOpts = "-k" in the derivation); the source repository is missing the settings crate, which this overlay patches in.

OpenCodeReview OpenCode Plugin (Home Manager)

The open-code-review package bundles its OpenCode plugin (the plugin file plus runtime node_modules) as passthru.opencode-plugin, built from the same source revision as the CLI. A Home Manager module symlinks it into ~/.config/opencode/plugins/ and puts the ocr CLI on PATH:

inputs.millerson-nix-overlay.url = "git+https://git.millerson.name/alex/millerson-overlay.nix.git";
inputs.home-manager.url = "github:nix-community/home-manager";
# in your home-manager configuration
imports = [ millerson-nix-overlay.homeManagerModules.default ];

programs.open-code-review = {
  enable = true;
  opencode.enable = true; # symlink plugin into ~/.config/opencode
};

Without Home Manager, symlink manually:

plugin=$(nix build --print-out-paths \
  git+https://git.millerson.name/alex/millerson-overlay.nix.git#open-code-review.passthru.opencode-plugin)
mkdir -p ~/.config/opencode/plugins
ln -sf $plugin/share/open-code-review/opencode/open-code-review.ts ~/.config/opencode/plugins/
ln -sfn $plugin/share/open-code-review/opencode/node_modules ~/.config/opencode/plugins/node_modules

Note: the node_modules symlink is shared across everything in the plugins directory; other manually-installed plugins with conflicting npm dependencies cannot coexist with it.

Development

Prerequisites

  • Nix with flakes enabled
  • treefmt-nix for formatting (optional)

Building Packages

# Build all packages for current system
nix build .#packages

# Build specific package
nix build .#mcp-gateway

# Enter development shell
nix develop

Project Structure

nix-overlay/
├── flake.nix                    # Flake definition
├── treefmt.toml                 # Code formatting configuration (nixfmt)
├── overlays/                    # Overlay implementations
│   ├── default.nix             # Binary-cache-friendly overlay
│   └── shared-nixpkgs.nix      # Dependency-sharing overlay
├── packages/                    # Package definitions
│   ├── awg-tool/               # AmneziaWG parameter generation and SSH deployment CLI
│   ├── aionui/                 # AionUi - AI Cowork desktop app
│   ├── codeaf/                 # Coding harness and software factory for open models
│   ├── container-use/          # Containerized environments for coding agents
│   ├── default/                # Meta-package listing all packages
│   ├── desloppify/             # Codebase health scanner for AI agents
│   ├── ds4/                    # DeepSeek V4 Flash/PRO inference engine (ROCm)
│   ├── flake-inputs/           # Utility for caching flake inputs
│   ├── freebuff/               # Free coding agent (Codebuff)
│   ├── freetoken/             # Local MoE inference runtime (NVIDIA CUDA)
│   ├── graphify/               # Knowledge graph generator for code folders
│   ├── g3/                     # AI coding agent (g3 + g3-studio)
│   ├── haivemind/              # Multi-model AI consensus runner
│   ├── hipengine/              # ROCm-native LLM inference engine for AMD GPUs
│   ├── kubernetes-mcp-server/  # MCP server for Kubernetes and OpenShift
│   ├── kyojin/                 # ExLlamaV3-based inference engine for Strix Halo (gfx1151)
│   ├── loop/                   # Corporate messenger for your team
│   ├── mcp-gateway/            # MCP protocol gateway
│   ├── omniroute/              # Unified AI router with 160+ providers
│   ├── open-code-review/       # AI code review CLI (deterministic + LLM agent)
│   ├── radar/                  # Kubernetes UI (topology, timeline, Helm, GitOps)
│   ├── relay-free-llm/        # AI model provider routing gateway
│   ├── skillsmcp/              # MCP server for Agent Skills
│   ├── stakpak/                # DevOps AI agent for infrastructure automation
│   └── traycer/                 # AI orchestration desktop app (agentic coding)
└── README.md

Adding New Packages

  1. Create a new directory under packages/<package-name>/
  2. Add a package.nix with the package definition
  3. Create a default.nix that imports package.nix with proper arguments
  4. The package will be automatically picked up by blueprint

Example structure:

packages/my-package/
├── default.nix    # Import wrapper
└── package.nix    # Actual package definition

License

See LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.