FinOps model for vLLM (4-bit)
Qwen3 4B fine-tuned to write FinOps audit notes on GPU sizing and serving cost, compressed to 4 bits (GPTQ W4A16) so one GPU can serve it.
No account needed. A quick check keeps bots away, then each file comes straight from Cloudflare. Every file shows its size and fingerprint so you can check it, and the guide below takes you from download to a first answer.
Each download is one zip with one folder inside, taken from our Hugging Face page at a fixed version. Pick the one you need, or see which file do I need?
Step 1: confirm you are a person
Loading the downloads...
Qwen3 4B fine-tuned to write FinOps audit notes on GPU sizing and serving cost, compressed to 4 bits (GPTQ W4A16) so one GPU can serve it.
A small add-on (LoRA, rank 8) for Qwen3 4B that teaches the same FinOps audit format. Runs on a Mac with Apple silicon, on an NVIDIA GPU or, slowly, on a plain CPU.
A 20 KB file: a linear probe that reads the model's layer-18 hidden state and flags a made-up FinOps claim, plus a steering vector. Try it on the claims dataset.
1,000 short FinOps claims labelled true or made up, with the model's layer-18 activations (1,000 × 2,560 values) saved beside them, so you can test a probe without running the model.
It depends on your computer and what you want to try.
Steps 1 to 3 are the same for every file. Then follow the part for your file. Commands go in a terminal: Terminal on a Mac, any terminal on Linux, PowerShell on Windows (or the Ubuntu window of WSL2 for the 4-bit model).
Every card shows a SHA-256 fingerprint: a 64-character code that changes if even one byte of the file is different. Make the code for your file and compare it with the card. If they differ, the download is broken or not ours: delete it and download again.
Macshasum -a 256 ~/Downloads/qwen3-4b-finops-w4a16-vllm-21288a6.zipLinux and WSL2
sha256sum qwen3-4b-finops-w4a16-vllm-21288a6.zipWindows (PowerShell)
Get-FileHash $HOME\Downloads\qwen3-4b-finops-w4a16-vllm-21288a6.zip -Algorithm SHA256
Use the name of the file you downloaded; the examples show the 4-bit model. Windows prints the code in capitals, which is the same code.
Double-click the zip on a Mac or Windows, or on Linux run:
unzip qwen3-4b-finops-w4a16-vllm-21288a6.zip
You get one folder, for example qwen3-4b-finops-w4a16-vllm, holding the files and an ABOUT-THIS-DOWNLOAD.txt that says where they came from. The 4-bit model needs about 2.7 GB of free disk to unzip.
A "virtual environment" is a private folder of Python packages for this project only, so nothing else on your computer changes. You need Python 3.10 or newer (python3 --version tells you; get it from python.org if needed). In the folder where you unzipped the file:
python3 -m venv .venv source .venv/bin/activateWindows (PowerShell)
py -m venv .venv .venv\Scripts\Activate.ps1
Your prompt now starts with (.venv). Run the activate line again whenever you open a new terminal. Leave with deactivate.
vLLM is a free server that loads the model on your GPU and answers in the same format as the OpenAI API, so most chat tools can talk to it. It runs on Linux with an NVIDIA GPU. On Windows, install WSL2 first (wsl --install in PowerShell as administrator, then restart) and the current NVIDIA driver for Windows, and run everything below in the Ubuntu window.
Check the GPU first: nvidia-smi should list it with 12 GB (12,288 MiB) or more.
pip install --upgrade pip pip install vllm
vllm serve ./qwen3-4b-finops-w4a16-vllm --served-model-name finops --max-model-len 8192
The first start takes a minute or two. It is ready when it prints Application startup complete, and it keeps running in that terminal (stop it with Ctrl+C).
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "finops",
"messages": [{"role": "user", "content": "A team serves a 7B model on four A100 GPUs that sit at 15% use. Write a short FinOps audit note."}],
"temperature": 0.6, "top_p": 0.95, "max_tokens": 800}'
Or from Python, with the small openai package (pip install openai); no OpenAI account is used, it only talks to your own server:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")
reply = client.chat.completions.create(
model="finops",
messages=[{"role": "user", "content": "A team serves a 7B model on four A100 GPUs that sit at 15% use. Write a short FinOps audit note."}],
temperature=0.6, top_p=0.95, max_tokens=800,
)
print(reply.choices[0].message.content)
--max-model-len 4096, or let it take less of the GPU, --gpu-memory-utilization 0.80.<think> part: Qwen3 models can think before they answer. For a direct answer add "chat_template_kwargs": {"enable_thinking": false} to the request (in Python: extra_body={"chat_template_kwargs": {"enable_thinking": False}}).--host 0.0.0.0, and only on a private network: the server has no password unless you add --api-key.The adapter is a small add-on: Python loads the public Qwen3 4B model from Hugging Face (about 8 GB, once, then kept in a cache) and puts our training on top. It works on a Mac with Apple silicon, on an NVIDIA GPU, or on a plain CPU (slow: minutes per answer).
pip install torch transformers peft accelerate
ask.py next to the unzipped folderimport torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
ADAPTER = "./qwen3-4b-finops-lora-week1" # the unzipped folder
tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, ADAPTER)
messages = [{"role": "user", "content": "A team serves a 7B model on four A100 GPUs that sit at 15% use. Write a short FinOps audit note."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_dict=True, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=800, do_sample=True, temperature=0.6, top_p=0.95, top_k=20)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
python ask.py
torch_dtype=torch.bfloat16 instead of dtype=.enable_thinking=False inside apply_chat_template(...).Download both the probe and the dataset and unzip them in the same folder. The dataset holds 1,000 claims, each with the hidden state the model had at layer 18 while reading it. The probe turns one hidden state into a score from 0 to 1: above 0.5 means "this claim looks made up".
pip install numpy pandas pyarrow safetensors
probe.py and run python probe.pyimport numpy as np
import pandas as pd
from safetensors.numpy import load_file
probe = load_file("qwen3-4b-activation-probes-week2/layer18_probe_and_steering.safetensors")
states = load_file("finops-residual-activations-1k/layer18_activations_1000x2560.safetensors")["layer18_hidden_states"]
claims = pd.read_parquet("finops-residual-activations-1k/data/train-00000-of-00001.parquet")
logit = (states * probe["probe_weight"]).sum(axis=1) + probe["probe_bias"] # each value times its weight, added up
score = 1 / (1 + np.exp(-logit)) # 0 = looks true, 1 = looks made up
claims["flagged"] = (score > 0.5).astype(int)
test = claims[claims["split"] == "test"]
print("Right on the", len(test), "test claims:", (test["flagged"] == test["label"]).mean())
print(claims[["text", "label_name", "flagged"]].head())
We ran this on the published files: it prints a score of 1.0 (every one of the 200 test claims right). Read that with care: the claims were built from templates, so true and made-up ones may differ in easy ways. The research page says more.
The probe file also holds v_truth, a direction in the same 2,560-value space that the model card uses to nudge an answer back towards the truth. Using it means changing the model while it runs, which this guide does not cover.