I’ve actually found the opposite. At work, we went from a fine-tuned model to a RAG system for internal and external documentation and a generic coding-focused model for code.

Fine tuning against in-house code seems like a small gain over a base model and search. It’s unlikely your code is unique and special and big enough that it’s hard to get results from a base model. You’ll be pinned to a certain version of a certain model, and you won’t be able to upgrade to future models nearly as quickly. Of course, you’re also fighting time again on each commit changing the code unless you continually fine tune it.

A RAG model might still struggle with a super vague question like “where does the foo cal bar with bax set” but it’s unlikely that this would work for fine tuning as well. This is where static code search by symbols really should be used.

> I'm interested to know if anyone is using fine-tuning to train a model on proprietary or in-house codebases and documentation.

I've done it, 1/2 the team though it was great 20% of the time, 1/2 the team hated it from day 0. I used roughly 500K lines of code.

> How much effort is required to turn code into something one can use for fine-tuning?

Very little to moderate, less than 200 lines of python, QWEM FIM, HF, LLAMA.CPP, LLAMA.CPP code extension.

> RAG solutions seem to have their limitations, and fine-tuning might be a more effective approach.

The only problem either way is keeping the information up to date, RAG just adds more cost to the inference process (which at my dev speed is pretty important).

> How much effort is required to turn code into something one can use for fine-tuning?

Fine tuning "fill in the middle" process is the process of taking a file, cutting out a some text in the middle and asking AI to guess what was there - there is a hugging face example that will have you doing it in an hour or less - your OPs team saying "No you cant litreally copy all code to a single folder" is probably the biggest hurdle (advise them you'll do it in CI and then they can stand up a FIM training endpoint that accepts a csv, pretty easy)

I would like to see more knowledgeable people with experience talk about this.

Is it just a matter of assembling Q/A pairs like: “What’s class X?”, “class X { … }”

Do you really need to do this training on the base model instead, which means you have to fine tune chat on it afterward?

How does this work?

Generally it is recommended that you fine-tune if you want to shape the output style. If you eant ro only output json, or just output jsdoc, etc. To add knowledge, RAG is a better idea and easier to keep updated on code changes. If rag does not give back good results then that's a problem with the retrieval part, which can be measured and improved. We're building documentation + rag systems for enterprise code bases and this is what we saw works best.

We are at Scribe[1]. We do it to make sense of knowledge workflows on computers to predict the next step in the process (our software points out where in the DOM a user might need to interact with next). We fine tune with tons of JSON data and DOM data. I’m sure doing it with code no more complicated.

[1] https://scribehow.com/library/scribe-agent

We see a lot of this in large orgs! The main issue imo is actually the selection of chat templates - there's a lot of people who use a template for finetuning then totally forget to use it for finetuning.

A lot of financial, legal and health companies do fine-tuning! Reasoning finetuning via GRPO is also very powerful since you don't need any cot data in between! Just inputs and outputs!

Not sure but you can just implement tool calling with some custom prompting yourself, it's really not too hard and is what we do

Function calling docs: https://ai.google.dev/gemma/docs/capabilities/function-calli...

It takes a significant amount of time (few hours) on a single consumer GPU, even 4090 / 5090, on personal machines. I think most people use online services like runpod, vast ai, etc to rent out high-powered H100 and similar GPUs for a few cents per hour, run the fine-tuning / training there, and just use local GPUs for inference on those fine-tuned models generated on cloud-rented instances.

Google Colab is quite easy to use and has the benefit of not making your local computer feel sluggish while you run the training. The linked Unsloth post provides a notebook that can be launched there and I've had pretty good luck adapting their other notebooks with different foundational models. As a sibling noted, if you're using LORA instead of a full fine-tune, you can create adapters for fairly large models with the VRAM available in Colab, especially the paid plans.

If you have a Mac, you can also do pretty well training LORA adapters using something like Llama-Factory, and allowing it to run overnight. It's slower than an NVIDIA GPU but the increased effective memory size (if you say have 128GB) can allow you more flexibility.

Take a look at the hardware requirements at https://github.com/hiyouga/LLaMA-Factory?tab=readme-ov-file#...

A 'LoRA' is a memory-efficient type of fine tuning that only tunes a small fraction of the LLM's parameters. And 'quantisation' reduces an LLM to, say, 4 bits per parameter. So it's feasible to fine-tune a 7B parameter model at home.

Anything bigger than 7B parameters and you'll want to look at renting GPUs on a platform like Runpod. In the current market, there are used 4090s selling on ebay right now for $2100 while runpod will rent you a 4090 for $0.34/hr - you do the math.

It's certainly possible to scale model training to span multiple nodes, but generally scaling through bigger GPUs and more GPUs per machine is easier.

For experimentation and smaller models, single gpu is the way to go! Tbh I normally find most people to spend the majority of their time on datasets, training loss convergence issues etc!

But if its helpful I was thinking about spinning up a platform for something like that!

For experimentation? Absolutely. It can often be done overnight for smaller models and reasonably sized GPUs (24GB+).

It'd become a lot less practical with huge datasets, but I'd guess that a lot of fine tuning tasks aren't really that large.

I've had trouble getting a great answer to this question - I ask it in various places every month or so, most recently here: https://nitter.net/simonw/status/1895301139819860202

On paper fine tuning smaller models can greatly reduce the cost for a specific task, but I've not heard many real-world success stories around that.

I think vision LLMs are one of the most interesting applications here - things like fine-tuning for better results extracting data from a specific paper form or report structure. Again, not many public examples of that.

Finetuning is easy and worthwhile, especially with LoRAs as these Unsloth demos do. The bottleneck then becomes how to self-host the finetuned model in a way that's cost-effective and scalable.

In practice prompt engineering and few-shot prompting with modern LLMs, due to their strong-and-only-getting-better-over-time prompt adherence, tends to be more pragmatic.

I've been finetuning these models since before chatGPT, and the one lesson I've learned is that by the time you have set up everything to fine-tune a model, you can expect a newer model to do as well with prompt-tuning.

So, unless you hope to stay at the fore front (e.g. to be ahead of competitors), there has been no real reason to finetune for the last 4 years, at best you could hope to stay about 1-3 months ahead, depending on how fast you were at setting up your training. And if that is what you did hope to achieve, you needed to automate on a higher level, i.e. automate data collection and the collection of eval cases.

I have documents from the last 50 years that I need to digitalize, millions of them written in old Arabic. The OCR is not accurate due to handwritten documents, so I need to fine-tune a model on around 300k pairs of texts (OCR output and manually corrected versions)

IMHO the biggest factor holding that back is how rushed and distanced these model releases are, still.

Both Phi-4-mini and Gemma 3 were released recently. Phi-4's damn close to a good, real, model release. Microsoft's done a great job of iterating.

Gemma 3's an excellent, intelligent, model, but it's got a gaping blind spot: tool-calling / JSON output. There was a vague quick handwave about it in some PR, a PM/eng on the Gemma team commented here in response to someone else that TL;DR "it's supported in Ollama!", which is Not Even Wrong, i.e. in the Pauli sense of the phrase.

- Ollama uses a weak, out of date llama.cpp thing where the output tokens are constrained to match a JSON schema. This falls apart almost immediately, i.e. as soon as there is more than one tool.

- The thing that matters isn't whether we can constrain output tokens, any model can do that, I've had Llama 3 1B making tool calls that way. The thing that matters is A) did you train that in and B) if you did, tell us the format

All that to say, IMHO we're still 6 months to a year out from BigCo understanding enough about their own stuff to even have a good base for it. Sure, tool calling and fine-tuning are orthogonal, in a sense, but in practice, if I'm interested in getting a specific type of output, odds are I wanted that formatted a specific way.

We were but with the models becoming so good, so large, and so cheap, we've largely abandoned it in our long-term roadmap.

I’m trying right now. The combination of small models, qlora and grpo has made it accessible to experimenters. I’m not using unsloth yet, but I will probably start checking it out pretty soon so that I can train larger models or increase the number of generations for grpo.

I am. I have some use cases related to data extraction where using a fine tuned small model outperforms the best-in-class closed source models and at a fraction of the cost.

Oh hey! For now we don't have a platform, so we generally tell folks to use Colab free gpus! Kaggle also has 30 hours for free per week! I put links for kaggle here: https://docs.unsloth.ai/get-started/unsloth-notebooks

> This also means Colab Notebooks with free Tesla T4 GPUs also work!

My understanding is that they don't charge these by themselves although you might have to pay Colab fee to Google.

They charge higher end models it seems https://unsloth.ai/pricing