feat(llm): package upstream inference engines
Centralize engine pins, compiler wrappers and NPU packages. Use upstream Kyojin v1.8.10 and drop the Thunderbolt dependency. Offline evaluation passed; workstation builds remain pending. Bead: nix-overlay-v09
This commit is contained in:
@@ -37,7 +37,14 @@ A custom Nix overlay and flake providing additional packages not found in upstre
|
||||
| `shardr` | Decentralized LLM repository: content-addressed model store, BitTorrent-based sync, OpenAI-compatible serving via llama-server | AI Inference |
|
||||
| `skillsmcp` | MCP server that exposes Agent Skills to AI agents via the Model Context Protocol | MCP Servers |
|
||||
| `kubernetes-mcp-server` | Model Context Protocol (MCP) server for Kubernetes and OpenShift | MCP Servers |
|
||||
| `kyojin` | ExLlamaV3-based inference engine with speculative decoding for 100 GB-class MoE packs on one 128 GB AMD Strix Halo (gfx1151, ROCm 7) | AI Inference |
|
||||
| `kyojin` | Upstream Kyojin 1.8.10 with AMD ROCm SDK wheels and a host C compiler for Triton JIT | AI Inference |
|
||||
| `hipfire` | AMD RDNA inference engine with TheRock compiler wrappers and quantization tools | AI Inference |
|
||||
| `hipfire-kernels` | Precompiled hipfire kernels for gfx1151 | AI Inference |
|
||||
| `gufo` | Gufo with XDNA2 NPU support and the Unsloth UD 3.1 compatibility patch | AI Inference |
|
||||
| `xrt-amdxdna` | XRT with the amdxdna plugin, built against gufo's nixpkgs | AI Inference |
|
||||
| `therock-rocm` | TheRock 10.0 SDK for gfx1151 | AI Inference |
|
||||
| `pwilkin-llama-cpp` | pwilkin llama.cpp fork with ROCm for gfx1151 | AI Inference |
|
||||
| `strix-llama-cpp` | halo-box llama.cpp fork with Vulkan | AI Inference |
|
||||
| `loop` | Corporate messenger for your team | Communication |
|
||||
| `radar` | Modern Kubernetes visibility — topology, event timeline, service traffic, resource browsing, Helm management, and GitOps support | Kubernetes |
|
||||
| `omniroute` | Unified AI router with 160+ providers, auto fallback, MCP/A2A, OpenAI-compatible APIs | AI LLM Gateway |
|
||||
@@ -102,29 +109,29 @@ nix run git+https://git.millerson.name/alex/millerson-overlay.nix.git#mcp-gatewa
|
||||
nix profile install git+https://git.millerson.name/alex/millerson-overlay.nix.git#mcp-gateway
|
||||
```
|
||||
|
||||
### Kyojin JIT Toolchain
|
||||
### Inference Engines
|
||||
|
||||
Kyojin's serving wrappers use the packaged ROCm compiler by default. Supply a
|
||||
pinned SDK to compile HIP JIT kernels with another compiler without rebuilding
|
||||
PyTorch or the HIP extension:
|
||||
The engine packages target AMD Strix Halo (`gfx1151`) on `x86_64-linux`.
|
||||
`nixpkgs-engine` pins the shared engine toolchain independently of the other
|
||||
packages. Gufo and Kyojin retain their upstream nixpkgs pins.
|
||||
|
||||
```nix
|
||||
kyojin.override { jitRocmSdk = myJitSdk; }
|
||||
```
|
||||
Kyojin uses the upstream 1.8.10 flake with its AMD ROCm SDK wheels. The serving
|
||||
wrappers also set `CC` for Triton's runtime C compilation. The upstream derivation
|
||||
still reports version `1.5.8`; the source pin is the `v1.8.10` tag.
|
||||
`kyojin.tests.jit` and `kyojin.tests.cpu` expose the upstream HIP compiler smoke
|
||||
check and CPU tests. The former nixpkgs
|
||||
PyTorch/ROCm package and its `jitRocmSdk` override have been removed.
|
||||
|
||||
The SDK must provide `bin/hipcc` and matching `amdgcn/bitcode`. Its compiler
|
||||
wrapper must set `HIP_DEVICE_LIB_PATH` to that bitcode **in the compiler
|
||||
subprocess only**, along with any required `ROCM_PATH` and Nix C++ toolchain
|
||||
flags. Keep the compiler wrapper's real path inside the SDK prefix so upstream's
|
||||
compiler provenance check recognizes it.
|
||||
Hipfire exports `kernels`, `hipcc`, `rocm`, `llvmReadobj` and `llvmObjdump` as
|
||||
package attributes. Set the model directory with
|
||||
`hipfire.override { modelDir = "/path/to/models"; }`.
|
||||
The upstream service module is available as `nixosModules.hipfire`, and its
|
||||
ROCm overlay as `overlays.hipfire`. Neither is enabled by the default module.
|
||||
|
||||
The serving process keeps PyTorch's ROCm device libraries. Setting TheRock's
|
||||
Clang 23 bitcode globally makes the ROCm 7.2 runtime crash on a basic PyTorch
|
||||
matrix multiplication. The serving wrappers also disable unsupported expandable
|
||||
allocator segments. `nix build .#kyojin.tests.jit` checks compilation of five HIP
|
||||
kernel modules without requiring a GPU; `packages/kyojin/check-jit.py --gpu`, run
|
||||
with the package's Python environment and serving variables, additionally checks
|
||||
PyTorch matrix multiplication and module loading through its existing HIP runtime.
|
||||
Gufo exports its matching `xrt`, `npuDir` and `fabricClock` as package attributes.
|
||||
The consumer must configure NPU power and fabric clocks before using the NPU.
|
||||
TheRock and XRT use individual recipes from the pinned `nix-strix-halo` source
|
||||
tree. Its flake and Thunderbolt dependencies are not loaded.
|
||||
|
||||
### nftablesbuilder Runtime Notes
|
||||
|
||||
|
||||
Reference in New Issue
Block a user