Qwen 3.6-27B is a state-of-the-art large language model suitable for local deployment. With technologies such as quantization, MTP, and TurboQuant, this language model can be deployed and run smoothly on an RTX 2080 Ti with 22GiB VRAM.

The following steps have been successfully tested on the latest version of CachyOS (installed using the 260426 ISO).

Installing dependencies

sudo pacman -Syu cmake cuda nodejs npm
paru -S python-modelscope

Configure npm mirror if necessary:

npm config set registry https://mirrors.cloud.tencent.com/npm/
npm config set strict-ssl false

Rebooting or relogging in is recommended for refreshing environment variables.

Compiling llama-cpp-turboquant

cd
git clone https://github.com/TheTom/llama-cpp-turboquant.git
# Following three lines for compiling Web UI frontend
cd llama-cpp-turboquant/tools/ui
npm i
npm run build

cd ../..
cmake -B build -DGGML_CUDA=ON -DLLAMA_BUILD_WEBUI=ON && cmake --build build --config Release -j --target llama-server llama-cli

Executables are in build/bin directory.

Downloading the model

modelscope download --model Tariel/Qwen3.6-27B-4bpw-MTP.gguf --local_dir ~/Qwen3.6

Deploying

cd ~/llama-cpp-turboquant/build/bin
./llama-server -m ~/Qwen3.6/Qwen3.6-27B-4bpw-MTP.gguf \
    -mm ~/Qwen3.6/mmproj-Qwen-Qwen3.6-27B-Q6_K.gguf \
    -fa on --spec-type draft-mtp --spec-draft-n-max 3 -np 1 -kvu \
    -c 190208 \
    -ngl all -ctk q8_0 -ctv turbo3 \
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 \
    --host 0.0.0.0 \
    --port 22345 \
    -a qwen3.6:27b

Usage

Visit port 22345 (e.g., http://127.0.0.1:22345) in the Web browser for llama.cpp Web UI.

Allow port 22345 in the ufw firewall in CachyOS for visiting from other machines in the network:

sudo ufw allow 22345

llama-server supports OpenAI and Anthropic APIs, allowing user to use many AI clients with it.

Notes on Cherry Studio

To turn on/off thinking mode properly in Cherry Studio, use “nvidia” provider and config API Host to llama.cpp address (see Issue #14981).

Performance

The environment for performance testing is: CPU i5-12500H (using integrated graphics for video output), GPU RTX 2080 Ti 22GiB, Kernel Linux 7.0.10-1-cachyos.

MTP significantly improves the decoding performance and overall performance of the model at the cost of slightly increased VRAM usage and a small reduction in prefill speed.

Performance on non-thinking mode:

Performance on thinking mode: