Low-Cost LoRA Fine-Tuning and Deployment-Efficiency Profiling of a 0.5B-Parameter Language Model for Closed-Vocabulary Structured Extraction on Consumer Apple Silicon

We present a fully reproducible case study in fine-tuning and deploying a sub-billion-parameter language model for closed-vocabulary structured extraction on commodity hardware. Using low-rank adaptation (LoRA), we fine-tune Qwen2.5-0.5B-Instruct to extract a strict four-field JSON object (sentiment, a bounded topic set, a price-mention flag, and an implied 1–5 rating) from unstructured mobile-application reviews, and we evaluate it with a fully deterministic parse-and-compare protocol that uses no model-based judge. Training is performed entirely on an Apple M4 Mac mini using the Metal Performance Shaders (MPS) backend in 32-bit precision; because 4-bit bitsandbytes quantization is CUDA-only, this configuration deviates from canonical QLoRA and the associated quantization-recovery control cannot be produced, a limitation we state explicitly rather than omit. On a held-out test set of 200 examples, fine-tuning improves the unweighted aggregate score from 62.3% to 76.6% (+14.2 points), with the largest gain on the closed-vocabulary topic field (26.1% to 73.5%). We then profile the quantized deployment (4-bit weights, served via Ollama) using a measurement methodology adapted from the low-bit-inference literature: peak-memory decomposition, key–value (KV) cache-quantization tolerance, the memory-bandwidth throughput ceiling, and energy per token. The central deployment finding is a qualitative collapse under aggressive KV-cache quantization: an 8-bit KV cache is near-lossless (77.95% to 77.02%), whereas a 4-bit KV cache reduces aggregate accuracy to 0.38% and valid-JSON parse rate to 10.5%. We discuss the label-circularity threat that limits the interpretation of three of the four fields, and we release all code, data splits, model artifacts, and raw measurements

paper.pdf

Files (340.1 kB)

• E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, 2021.
• T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314, 2023.
• Qwen Team. Qwen2.5 Technical Report. arXiv:2412.15115, 2024.
• G. Grano, A. Di Sorbo, F. Mercaldo, C. A. Visaggio, G. Canfora, and S. Panichella. Android Apps and User Feedback: A Dataset for Software Evolution and Quality Improvement. Proc. 2nd ACM SIGSOFT International Workshop on App Market Analytics (WAMA), 2017.
• T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv:2208.07339, 2022.
• S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. Hugging Face, 2022. https://github.com/huggingface/peft
• PrismML. Bonsai 27B: Full 27B-class reasoning in binary and ternary transformer weights. Technical report, 2026. https://github.com/PrismML-Eng/Bonsai-demo
• W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. Proc. SOSP, 2023.
• G. Gerganov et al. llama.cpp: LLM inference in C/C++. 2023–2026. https://github.com/ggml-org/llama.cpp