perf(kyojin): Let short replies finish before a long prefill
CI / check (push) Has been cancelled

Give two-session Qwen serving a bounded, opt-in decode grace for large cold prompts. Returning to normal interleaving at the deadline bounds the wait when the running reply takes longer than expected.

Bead: nix-overlay-qfi
This commit is contained in:
2026-11-18 15:19:23 +03:00
parent f07d234baf
commit afd0c4b84d
3 changed files with 122 additions and 0 deletions
+5
View File
@@ -5,6 +5,11 @@
cpuCheck,
}:
kyojin.overrideAttrs (old: {
# Upstream installs tools directly from $src, bypassing the unpacked tree.
src = pkgs.applyPatches {
src = old.src;
patches = [ ./short-reply-prefill-grace.patch ];
};
nativeBuildInputs = (old.nativeBuildInputs or [ ]) ++ [ pkgs.makeWrapper ];
meta = (old.meta or { }) // {
homepage = "https://github.com/Yamz-Labs/kyojin";