Commit Graph

12 Commits

Author SHA1 Message Date
alex 635f9f7dfc fix(kyojin): reuse queued prefix pages across agent turns
Bead: nix-overlay-7p6
2026-11-19 01:28:42 +03:00
alex 7843d79e76 fix(kyojin): release prefix buffers before the writer waits
Bead: nix-overlay-9hu
2026-11-19 00:41:57 +03:00
alex 41a50393f9 perf(kyojin): allow blocking HIP waits in the Qwen server
CI / check (push) Has been cancelled
Bead: nix-overlay-j2g
2026-11-18 21:16:42 +03:00
alex 2b8eddf7d2 fix(kyojin): Stop direct expert reads at unaligned EOF
Bead: nix-overlay-qy9
2026-11-18 18:34:42 +03:00
alex 6156a0d904 perf(kyojin): Skip prefix segments that cannot fit on disk
Bead: nix-overlay-tb3
2026-11-18 18:31:54 +03:00
alex afd0c4b84d perf(kyojin): Let short replies finish before a long prefill
CI / check (push) Has been cancelled
Give two-session Qwen serving a bounded, opt-in decode grace for large cold prompts. Returning to normal interleaving at the deadline bounds the wait when the running reply takes longer than expected.

Bead: nix-overlay-qfi
2026-11-18 15:19:23 +03:00
alex 85bd7421de feat(llm): package upstream inference engines
Centralize engine pins, compiler wrappers and NPU packages.
Use upstream Kyojin v1.8.10 and drop the Thunderbolt dependency.

Offline evaluation passed; workstation builds remain pending.

Bead: nix-overlay-v09
2026-11-12 20:36:52 +03:00
alex d05ca58ee6 chore(kyojin): bump 1.7.1 -> 1.8.5
CI / check (push) Has been cancelled
Upstream 1.8.4/1.8.5 are iGPU decode and returning-session speedups with no
packaging change: setup.py still reads EXL3_ROCM_DEV_INCLUDE/EXL3_HIP_DEFINES,
the pyproject runtime deps are unchanged, and tools/{glm,mimo,qwen}/serve.py
are still the serve entry points. Only the pin and source hash move.

Verified on gfx1151: nix build .#kyojin, importing exllamav3_ext against the
pinned ROCm torch 2.11, kyojin-serve-qwen --help, and nix build
.#kyojin.tests.jit.
2026-11-02 16:16:00 +03:00
alex e2f24b0876 chore(packages): bump pins to current upstream releases
- awg-tool 0.4.0 -> 0.5.0
- awgen 0.1 -> 0.2
- codeaf 0.7.1 -> 0.8.2
- ds4 -> ee7e840 (2026-10-21)
- freebuff 0.2.19 -> 0.2.34
- graphify 0.9.77 -> 0.9.95 (add pyyaml dependency)
- hipengine 0.6.1 -> 0.8.3
- kubernetes-mcp-server 0.0.67 -> 0.0.69 (build with go_1_27)
- kyojin 1.3 -> 1.7.1 (drop obsolete setup.py/pyproject patches, ship
  gr_mix_hip from the package, add ninja runtime dependency)
- marmel -> de3e99c (2026-10-23)
- omniroute 3.8.50 -> 3.8.52
- open-code-review 1.12.12 -> 1.13.3 (build with go_1_27)
- pilotdeck 2026.10.17 -> 2026.10.20
- radar 1.16.2 -> 1.18.0
- traycer 1.4.2 -> 1.5.1

Verified: nix build passes for every bumped package; kyojin.tests.jit
compiles all five JIT kernels.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
2026-10-24 14:09:34 +03:00
alex 0813f55762 fix(kyojin): Isolate JIT and runtime toolchains
CI / check (push) Has been cancelled
Allow a pinned JIT SDK without rebuilding PyTorch or the extension.
Keep runtime device bitcode unchanged: exporting TheRock bitcode into
the serving process crashes CLR 7.2 on a basic tensor operation.

Disable unsupported expandable segments and check all five HIP kernel
modules, with an optional GPU matmul and module-loading regression test.
2026-10-08 17:17:56 +03:00
alex 4c24610aae fix(kyojin): serve wrappers get a C compiler, hipcc and the JIT sources
CI / check (push) Has been cancelled
env.sh assumes a distro box: a system cc for Triton's runtime C builds, a hipcc
for the JIT HIP kernels and the repo root on PYTHONPATH. The wrappers now export
all three (, ,  for the amdgcn bitcode,
PYTHONPATH=${src}/gr for gr_mix_hip).

The wheel ships only the files pyproject lists, so the .hip kernel sources, the
qsa_proof marker gating the QSA prefill kernel, the dense-GEMM tuning seed and
gr/ never reached site-packages: kernels fell back to Triton and every start
re-tuned dense GEMM for 7 minutes.

Verified on the Strix Halo box: server READY, start-up 713 s -> 103 s, warm-up
418 s -> 4 s, 584-token prompt served over /v1/chat/completions with no
'unavailable' fallbacks.
2026-10-08 15:57:52 +03:00
alex e061d3c170 feat(kyojin): add Kyojin ExLlamaV3 engine for AMD Strix Halo
CI / check (push) Has been cancelled
Kyojin (Yamz-Labs/kyojin) is a fork of ExLlamaV3 that runs 100 GB-class
EXL3 MoE packs on one 128 GB Ryzen AI Max box. It ships a torch C++
extension that torch's cpp_extension cross-compiles to HIP, and upstream
supports gfx1151 only, so the kernels are built for gfx1151 alone and
the package is marked Linux-only. Pinned to the v1.3 release.

The extension must be compiled against a ROCm build of torch. The flake's
nixos-unstable carries torch 2.13, whose ROCm build fails in nixpkgs (the
CK SDPA configure step runs a script through /bin/bash) and has no binary
on the cache, so the build would compile PyTorch from source and die.
python3Packages.torchWithRocm from the already pinned nixpkgs-torch211
input gives torch 2.11 with ROCm 7.2.2, gfx1151 in its target list and a
cached binary, so default.nix builds through that input the way freetoken
does.

nixpkgs splits the single devel tree the AMD wheels ship, so two
symlinkJoins stand in for it: one as ROCM_HOME (hipcc, headers, amdgcn
bitcode) and one handed to setup.py as EXL3_ROCM_DEV_INCLUDE, which wants
hipsparse/, rocsparse/, rocrand/, thrust/ and pybind11 (nixpkgs torch no
longer exports pybind11 headers). hipsolver is in there because torch's
own HIPContextLight.h includes it.

setup.py gets one patch in postPatch. It imports exllamav3 to reach
build_config, that runs the package __init__, which imports ext.py, which
JIT-compiles the entire extension whenever no precompiled exllamav3_ext is
importable, meaning every wheel build, into a $HOME the sandbox does not
grant. Registering a stub package keeps the two leaf modules importable
without that detour.

doCheck = false: tests/ drives a real gfx1151 GPU and the published
packs. Inference on a card is not verified here, this builder has no
/dev/kfd, so the wheel-only LD_PRELOAD workaround for torch's bundled HSA
runtime is untested. It should not be needed, the store torch links the
store rocm-runtime.

Verified with nix build .#kyojin at v1.3: exllamav3_ext loads, torch
reports hip 7.2.53211, the closure holds one torch (ROCm), and
kyojin-serve-qwen plus kyojin-serve-glm print their usage.

Refs nix-overlay-du9
2026-10-07 16:14:09 +03:00