POC exploration
One Extra Correct Answer Is Not a Model Win: What the QLoRA Sprint Actually Proved
30 held-out rows · 100% schema-valid · 43.3% → 46.7%
The headline number is one. On the same 30 held-out resume examples, the base Qwen3-4B model got 13 decisions right and the LoRA adapter got 14. Both produced strict, schema-valid JSON every time. That is a 3.3-point lift, and it sits comfortably inside the noise of such a small eval. I would not use it to claim the adapter is better.
That was never the useful claim. I built this sprint to close a different gap: train an open-weight model on hardware I could actually afford, recover from notebook failures, publish the artifact, and compare it with the base model under controlled settings. A hosted API would have been faster, but the vendor would still own most of the model lifecycle. RAG would have been the right answer for changing facts, but it would not prove that I could train or govern an adapter. A full fine-tune was too heavy for a free Colab T4 and too blunt for the question I was asking.
QLoRA fit the job. The base model stayed in 4-bit form while a small adapter trained on top. I pinned the prompt shape, tokenizer, held-out rows, and generation settings so the before-and-after run meant something. The output contract was strict JSON because a resume screener eventually has software on the other side of it. If the model wraps a decision in a paragraph or changes the schema, that is an integration failure even when the prose sounds sensible.
Open models move work into your own platform. The base revision, adapter checkpoint, tokenizer, dataset snapshot, prompt, and eval set all need versions. A serving gateway needs to log which combination answered each request. Promotion needs gates for schema validity, task metrics, regressions, and bias. Rollback should mean detaching an adapter, not rebuilding the product around another model.
The sprint proved that operating loop. It did not prove that a 4B model should make hiring decisions. A production evaluation would need hundreds of reviewed examples, slices by role and seniority, adversarial resumes, and adverse-impact testing. A human would stay in the decision path. Small POCs are allowed to be small; the mistake is pretending they measured more than they did.
The decision
Use QLoRA to prove the model lifecycle, then judge the sprint as an operating-loop POC rather than a production resume screener.
Alternatives considered
- ·Use a hosted model API. Faster, but it would not prove open-weight training, artifact handling, or adapter rollback.
- ·Run a full fine-tune. More control over the weights, but too heavy for the T4 budget and unnecessary for this proof.
- ·Use RAG only. Useful for fresh facts, but it does not test whether I can train and govern an adapter.
Tech stack
The challenge
Close a real open-weight model gap on a free Colab T4, publish a reproducible adapter, and report what happened without turning one extra correct answer in a 30-row eval into a quality claim.
Architecture approach
- ·Loaded Qwen3-4B in 4-bit form and trained a small LoRA adapter instead of changing the base model.
- ·Pinned the prompt, tokenizer, generation settings, held-out rows, and base model so the comparison could be repeated.
- ·Required strict, schema-valid JSON because downstream software cannot govern prose-shaped guesses.
- ·Published the notebook, script, captured outputs, screenshots, and adapter instead of asking readers to trust a summary.
Results
- Base model: 13/30 correct (43.3%)
- QLoRA adapter: 14/30 correct (46.7%)
- Both runs: 100% strict JSON and 100% schema-valid
- Conclusion: the lifecycle works; the eval is too small to claim the adapter is better
What I'd do differently at production scale
- ·Expand the held-out set beyond 200 examples and report slices by role, seniority, and resume format.
- ·Run bias and adverse-impact checks before the output touches an employment decision.
- ·Keep a human in the decision path and make adapter rollback independent from the base-model deployment.