Small language models are not a plug-and-play way to get generative AI. But they do have their uses. I’ve learned some lessons about making them work smoothly.
For almost a year now, I’ve been tinkering around with LLMs (AI tools you run on your PC; also called small language models, or SLMs). In the process, I’m finding myself in territory I haven’t traversed in decades.
I remember the days when acquiring a computer program meant buying a box filled with stiffy drives and then inserting those reverently, one by one, into a slot in your computer until the whole program loaded. (Yes, that is what the drives were called. They were preceded by floppy drives. I am not making this up. See the picture at the top of the post: floppies to the left, stiffy on the right.)
Things often went wrong. Often.
Since I’ve always been tech-adventurous at heart, I found myself footling around in MS-DOS command lines, trying to figure out how to make the frigging thing do what I wanted it to do.
(Also – of course – doing that for bewildered friends and family members.)
In the shiny heights of the 21st century, we just download an app and start using it. Even big apps like Microsoft’s Office. When we buy a new computer, we turn it on, have it do a small song-and-dance about ways it can help us (probably not, if it is Windows) and start working.
I’ve got used to this convenience, and I really do like it – just as I like the super-quick results I can get out of a commercial cloud-based generative AI tool.
Learning to do things the hard way
Nevertheless, something in me was called to try something else: AI I can run locally on my machine, the aforementioned small language models. Dear reader, I again find myself in the land of figuring out how the frigging thing works. And I confess I quite like being here. There’s something satisfying about applying my long-buried problem-solving skills to opaque tech issues.
This blog post is a catch-up for those who might want to go down this path, and a guide to some new wrinkles I have discovered in the process.
I’ll summarise what I’ve learned and done so far, and then fill you in on the quirks and oddities that I have uncovered.
What I have learned and done with SLMs to date
Full disclosure: I put two previous blog posts into Gemini Notebook and asked it to summarise the story to date.
The two posts:
Gen AI that isn’t big tech – an exploration
Why – and how – to ensure privacy when using GenAI (it’s in the anonymising!)
Here’s the Gemini Notebook summary, edited and added to by me:
Reasons for needing an SLM
Data privacy and control: Cloud AI tools (like ChatGPT or Claude) act as “black boxes” where your inputs are processed on external servers and often used to train models by default. Small language models run on your own hardware.
Legal compliance and confidentiality: Uploading sensitive client details, bank statements, or proprietary business documents to cloud tools can expose private data or violate protection laws like POPIA.
Zero subscription costs: Local setup tools and models are free to download and run completely offline, eliminating recurring subscription fees.
The “killer use case” (safe anonymisation): An SLM lets you remove all confidential or identifying information from client documents and PDFs before feeding them into cloud AI platforms.
How to set up a small language model
1. Choose an app: First, put your computer’s system specs into a commercial AI tool and ask it:
- is my system able to run a local LLM
- to suggest models that work with whatever your needs are (stash that list somewhere, you’ll need it later).
Then go download a free desktop application like LM Studio (that’s the one I’ve settled on; note that you need to scroll down. You don’t want LM Studio Bionic yet; go for LM Studio further down the page).
2. Install the software: Open the downloaded installer file (e.g., the *.exe file on Windows) to install the program on your computer.
3. Select and download a model: Inside the application, search for and download an SLM suited to your PC specs (such as Gemma 4). LM Studio will recommend one, or you can try one of the models suggested by commecial GenAI in your earlier research.
4. Start using it: Open the built-in chat interface to run prompts just as you would in a tool like ChatGPT.
Small language model quirks and oddities
This is where I started to learn things that are simply not evident at first glance.
First: do not mention the attachment. I wanted to upload two PDFs that contained information about my retirement annuities, so that I could get the tool to compare the performance of said annuities between 2025 and 2026. These are, obviously, private documents and so keeping them inside the purview of an SLM is the common sense thing to do.
I found an SLM that had document upload properties (they don’t all offer it), leaning on Gemini to give me a list of possibilities.
Then, I uploaded the retirement annuity PDFs. They were there, inside the chat window. Summarise these attachments, I said. Nope, said the LLM: there are no attachments.
This went back and forth for a bit until I lost patience. I took myself off to Gemini (again) and asked what the hell was going on. Ah, said Gemini:
Models like Gemma and Qwen are pre-trained by Google and Alibaba to tell users, ‘I cannot browse the web or read files.’ If you prompt the model by saying, ‘Please read this attachment,’ its safety training triggers an automatic refusal – even though LM Studio has already successfully fed it the text.
So the fix is this: Do not mention “attachments”, “uploads”, or “files”. Instead, prompt it with: “Summarise the text provided above” or “Based on the information given, answer this question.”
I know this makes no sense, but it does work. I told the SLM to summarise the text and it did. Not very well, but that is part of the larger learning curve of experimenting in which I search for a model that does what I want it to do.
Second: not all documents are equal. I also learned that the kind of document you upload matters. For PDFs, a scanned document (in other words, an image) won’t work. To figure out what kind of PDF you have, try highlighting text in the original. If you cannot highlight individual words, it is an image. You must run the PDF through Optical Character Recognition (OCR) software before uploading it. One of my PDFs was indeed a scanned document so I used PDF OCR to convert it and deleted it from there as soon as the job was done.
If the document is heavily formatted (complex Word tables for instance) or password-protected, you will also have problems. The fix is to open your document, copy all the text, and paste it into a plain .txt file using your computer’s notetaking app or Sublime Text. Upload the .txt file instead, which will give the model unformatted text.
Is it all worth it?
Every time I get a sub-par answer (that is, all the time) from an SLM I have a moment of wondering why I am bothering. But I figure that this learning curve is worth the time if I can, eventually, free myself from the clutches of the unsavoury companies that give us the current cloud offerings.
My next task is to identify the particular SLM that I want to use, and then figure out how to train it on my own data. And, for me, this will, in fact, be fun.
Main picture shows 8-inch, 5.25-inch, and 3.5-inch discs from the ancient past. I don’t think I ever encountered an 8-inch disc but I remember both 5.25 and 3.5-inch discs. By George Chernilevsky, Wikimedia Commons, Public Domain
Previous Sensible Guide articles
The diary of an AI user: practical tools for everyday work – ChatGPT launched as a consumer tool in November 2022. Nearly four years later, here’s how generative AI has changed my working life (or not).
A guide to the oddities of Claude – Usability is not a strong point in generative AI tools. Claude is there with the worst of them, but it’s worth persevering.
Where did that document go? Hints and tips for finding things in your folders – One of the best ways to use generative AI is as a thinking partner. Here’s how it helped me get more organised about finding documents on my PC.
If any of this is relevant to your work…
You might find one of these useful:
