Small LLM Image Processing using NuExtract3
Python
Large Language Models
More Small LLMs ?
For this post, I’m using another small model called NuExtract3. This model is actually just a fine-tuned version of a Qwen3.5-4B model, which I had previously run locally on my own machine before. I’ve already spoken quite a bit about the smaller Qwen and Gemma models, and how they have a lot of utility for well-defined, targeted tasks. Because they have a small footprint and don’t require a lot of VRAM, they are easy to deploy in a low cost-environment.
Let’s get down to brass tacks. What got me specifically interested in NuExtract is that it covers a use-case that I run across a lot in my day job: document processing. We are often asked to process documents in a variety of ways - including extraction of information, validating fields on the document, and performing more complex tasks (like classification). We typically use vision LLMs with structured prompts and tools to perform these tasks. However, cost is always a concern. While a single, 2-3 page document might not cost much to process (likely in the range of less than 1 cent), when you multiply this across hundreds of thousands of documents, the dollars start to add up. It is typically prudent to start with the smallest viable model you can, and work up from there. In many cases, the tasks are simple enough that you don’t need a bleeding-edge model to process it!
The Data
Let’s illustrate a simple scenario here: extracting information from parking tickets. For this example I rely on NYC Open Data’s collection of moving violations and tickets. This set of data contains a number of structured fields in a tabular format about the violation (plate number, issue date, etc…). However, they also include a field named summons_image which is a url link to an image of the ticket. These image copies look like this:
This is a good example for testing because we have an image with a mixture of different field types, as well as some ground truth defined in the dataframe.
My goal is to set up a small model and extract values off of these tickets. While many of the fields are relatively straightforward, like plate number or date of incident, some other fields are more ambiguous. For example, the summons number is an unlabeled vertical number in the bottom right-hand corner. Getting the LLM to reliably extract this will require some extra work.
Running the Model
Setting up the local LLM
For my local setup I use a 4-bit quantized version of the base model. This involves downloading both the .gguf file and the vision component. I deploy this model the same way as I have before, using llama.cpp and exposing a local endpoint via llama-server. One unique element of NuExtract is that it is tuned for a specific JSON extraction schema using their own named types. All this means is when we invoke the model, we should pass a schema object to the model along with any ancillary instructions, and conform to the expected schema rules. While we could certainly run the model using a different schema (it is, after all, just a fine-tuned base model) it is likely to degrade performance if we diverge from the fine-tuned expectations.
I typically structure extraction rules as Pydantic schemas, because it helps formalize and standardize extraction rules, as well as making it easier for me to validate the output. Luckily, NuExtract has a helper function that converts Pydantic schema directly to their own expected style:
from pydantic import Field, BaseModel
from numind.nuextract_utils import convert_json_schema_to_nuextract_template
class TicketExtract(BaseModel):
date: str = Field(description="date")
time: str = Field(description="time")
car_make: str
car_body_type: str
car_color: str
summons_number: str
violation_code: str
location: str
license_plate: str
fine_amount: float
comments: str
template, _ = convert_json_schema_to_nuextract_template(
TicketExtract.model_json_schema()
)The actual template is then converted to the following below, which is passed directly to the model as its extraction schema.
{'date': 'date',
'time': 'time',
'car_make': 'string',
'car_body_type': 'string',
'car_color': 'string',
'summons_number': 'string',
'violation_code': 'string',
'location': 'string',
'license_plate': 'string',
'fine_amount': 'number',
'comments': 'string'}