In 2024 I fine-tuned an LLM. Jev could have removed the side quests.

In November 2024 I tried fine-tuning a small model on r/SkincareAddiction. It was the thing to do at the time. The first version answered like a redditor: ask it for a routine and it told you to read the sidebar. Sometimes it insulted you.

A fine-tuned model is only as good as the pairs it’s trained on, and mine were reddit comments with nothing but basic cleanup. So the real work was deciding which of 120,633 comments deserved to be in the training set at all.

That is a classification problem, and this is what one decision looks like:

I built it the way these things were built at the time. Today it makes a good demonstration of Jev, so let’s run through the details.

How good is Reddit as training data, really?

Fine-tuning an instruct model needs an input and an output. I used the post title and body as the input and any comment with at least two upvotes as the output, which gives you an SFT dataset in an afternoon. It also means whatever a well-liked Reddit comment does, the model learns to do.

The base model’s answer was objective and from the point of view of a robot. My fine-tuned model was responding like a person with their own skin and a favorite lotion! This is a data filtering problem.

So the data had to be filtered. Not by upvotes or length, but by what a good comment is for our purpose. Deciding “good” is subjective, and at scale it’s usually a classifier’s job, though these days people mostly just throw it at an LLM.

Inside a classifier

A classifier is a function that answers a yes/no question about a piece of data, preferably with a confidence level.

For this dataset I wanted three questions answered about every comment:

1. Useful? Does it actually contribute to skincare discussion and education?
2. Objective? Is it a personal anecdote? Someone’s own routine or concerns?
3. Quality? Is it spam? Is it answering the question? Is it respectful?

I picked these categories from digging through the actual data, finding clusters, testing out iterations of the chatbot, and personal opinion.

Ways to answer a yes/no question about text

Roughly in order of how much machinery you end up building.

Regex

The scary but often default solution. You build a list of words (or stems) you want to exclude, write a pattern, iterate.

It’s a very long-tail solution that I consider too much of a cat-and-mouse game for this. For cleaner problems on other datasets I do still use it.

Negative: leaky, and never reliable for semantic differences.

Math

You turn each comment into a vector, cluster them into multi-dimensional categories, and try to find where the junk lives.

You can vectorize with classical NLP methods or with embeddings.

Negative: the cost of vectorizing, and you can’t fully interpret the results.

Give it to an LLM

Take every comment and send it to an LLM with a prompt, asking for a verdict or a JSON output.

The easiest thing to do, but you’ll need to evaluate different models and thinking levels and find the Pareto frontier for cost and timing.

At the time (November 2024) Claude 3.5 Sonnet was $3 per million input tokens and $15 per million output, and Haiku was $1 and $5. A full pass would have cost somewhere around $70 to $180 depending on the models used.

Negative: the cost of iteration, and price (though that’s getting better these days.)

Teacher and student

Have a big, expensive LLM label a large sample of the data, train a tiny transformer to learn the same labels, then process your full dataset through that new model.

This is what I tend to default to, and the way I did this project. It scales a lot better and is dirt cheap for multiple runs (as long as you don’t change the definitions.) Acquiring the teacher labels cost me around $7 each time.

Negative: dev time and distraction from the main mission, and drift between the teacher and the classifier.

The way I did it

I’m gonna be honest with you: it sucked.

I went through a funnel of concerns (useful → objective → quality), manually graded a few hundred comments myself, iterated on the prompts with Claude until it matched my labels,I tried a few labeling schemes. Initially I wanted a 1–5 score for each comment, but a binary choice with a confidence rating turned out to be much easier to train, and easier to set a cutoff on. I also tried a single classifier that could spit out all three criteria versus three specific models. Like I said, this thing is its own whole project. trained the classifier, fine-tuned the LLM, tested it, then repeated the whole flow.

After I decided on the architecture, the iteration loop was:

You can start to appreciate why companies hire full-time staff for data cleanup. Each step was its own separate project: determining the best settings to use, deciding if the teacher model was strong enough, and so on.

I chose different teacher and classifier models for the different steps, mostly for cost and performance. These days there are much better models to pick from.

This setup sounds like a dream, and it’s quite fun!

But of course the classifier can’t be 100% in line with the teacher, and of course the teacher can be wrong too:

At the time I considered this good enough and went back to my actual main project, but I clearly could have spent weeks just perfecting this.

Enter Jev

OK, getting to the meat of it. Jev came out recently and has caused a lot of buzz and confusion. I even had to confirm I understand it’s not an LLM before I could get my Jev API key!

From my understanding it’s a general decision model, which means a lot of things, one of them being a generalized classifier. No labeling, no fine-tuning, just a request. Here’s the interaction:

That’s the entire integration. I sent all 120,633 pairs through it with the three questions above, five records per request.Jev can technically take a lot more data per request, but bigger batches were timing out for me, and the answers get worse as the batch grows. Five agreed with single-record calls about 97% of the time. It took 23 minutes and cost $3.47, and returned labels and confidence scores just like my old models.

The nice thing is that instead of focusing on the accuracy of the metrics, we can now focus on optimizing the confidence cutoffs:

Side by side

But how is the quality?

Against my old teacher-labeled dataset, Jev agrees 86% of the time on usefulness, 73% on objectivity, and 88% on quality. My handmade classifier models agreed higher (up to 97% for usefulness), but the teacher itself was not producing perfect labels either.

I asked Gemini 3.8 Flash to compare, and in general it said Jev is better on objectivity and similar on the rest.

To be clear, the key benefit is that with Jev I can iterate on all of this with just a prompt change, so it’s much quicker to get to “good enough.”

Costs

At the scale I was working with, the cost isn’t a huge differentiator, and of course there are ways to optimize the LLM price with batch processing and so on.

Conclusion

Like most of my projects I really didn’t have a goal for this other than learning and having fun, and I got a lot of both.

If I hadn’t gotten so lost in the weeds on data filtering, I could have focused on other problems I noticed in the chatbot, like out-of-distribution data around high-end luxury products or non-Western brands. It’s the distractions that get in the way, and in the future Jev can completely remove this specific distraction for me on other projects.

If you’re interested in talking about more details like synthetic data generation or actual edge hosting, hit me up. If you’re working on anything and need help, please reach out to kasra@abstract.partners. Thanks for reading!