r/Julia • u/cafedude • 21h ago
I accidentally a Julia-based inference engine that can run Qwen3.8-27B
(There's more to Luminal.jl than this, see the github link for more, but this post focuses on running Qwen3.8-27B)
Serve Qwen3.8-27B (GGUF)
examples/qwen35_server.jl serves a Qwen3.5 / 3.6 / 3.8 GGUF through an
OpenAI-compatible API (streaming chat completions, tool calls, reasoning
content), for harnesses such as opencode. Weights stay in the GGUF's own
quantization on the GPU (LUMINAL_GGUF_NATIVE=1; Q4_K_XL, about 17 GB), and the
model's MTP head drafts tokens for speculative decoding.
bash
LUMINAL_GGUF_NATIVE=1 julia -t auto,1 --project=. examples/qwen35_server.jl \
/path/to/Qwen3.8-27B-UD-Q4_K_XL.gguf --max-seq=262144 --port=8080
- Use
-t auto,1: the server needs the extra interactive thread. The HTTP tasks run on the main thread and the GPU work in a task pinned to another one; without that, nothing is streamed while a prefill or a decode step blocks the main thread. - The first start takes a few minutes (reading the GGUF, compiling the graphs).
The log prints
serving ...when it is ready. --max-seqsets the context and the KV cache allocated up front (Float16, 64 KB per token: 17 GB at 262144). Other flags:--mtp=<gguf>(a separate file with the MTP head),--host,--name(the model id),--no-spec,--int8.- Point the client at
http://127.0.0.1:8080/v1(GET /v1/models,GET /health,POST /v1/chat/completions). For opencode, add an@ai-sdk/openai-compatibleprovider with thatbaseURLand the--nameas the model id. Long prompts can take minutes before the first token, so raise or disable its request timeouts. --reasoning-budget=Nlimits the thinking: after N reasoning tokens the server closes the<think>block with a short wrap-up and the model answers from what it has (the resume costs a re-prefill of the new turn, about 10-20 s at N = 4096). A request overrides it withthinking_budget(alsothinking.budget_tokens,reasoning_budget; 0 = no limit). In opencode, put"thinking_budget": Nin the model'soptions.- Prompts are cached: the system prompt, each turn, and state checkpoints every
4096 tokens (
LUMINAL_CKPT_EVERY), so a follow-up turn reprocesses only its new tokens. A cold prefill runs at 330-480 tok/s (about 3 minutes for 75k tokens); a warm turn answers in 3-25 s. Decode is 11-15 tok/s at 70-130k tokens of context (2.3-2.9 tokens per speculation cycle). - It handles one request at a time, and it needs about 35-60 GB of unified memory at the largest context, depending on how many prompts are cached. The log reports prompt processing, generation speed and speculation per request.
For a one-shot generation from a GGUF, examples/qwen35_gguf.jl <file.gguf> "prompt".
https://github.com/philtomson/Luminal.jl
NOTE: right now I'm only sure that this works well on AMD Strix Halo GPUs. It might work on Nvidia GPUs via CUDA.jl, but I have not tested it.

