Large language model, small language model – what’s the difference? A step away from big tech, that’s what.
Sometime there in the first decade of the 21st century, I was working on a big South African news website.
The internet was upon us. Google had started to become the search engine of choice, and springing up around us were the beginnings of social media – even though we didn’t know then that that was what it was called.
At the time, I was running a subsidiary website called Babynet – a parenting product that suited me well because I was the mother of a toddler. And Babynet had a thing called a forum: a place where parents could talk to each other and share their thoughts and struggles. I was responsible for the wellbeing of the forum – moderating discussions, sorting out tech issues, participating every now and then.
(The forum was run on phpBB software – which is available to this day.)
The Babynet “social platform” ran successfully for several years – while Facebook and Twitter and YouTube were starting to extend their tentacles into our lives.
Today, forums still exist (there’s a South African one here, for instance) but they are certainly not what anyone would regard as a dominant form of social media. There’s Reddit, which is big and has some similarities to the forums of old, and that’s about it.
And that is a pity.
Because the thing about them is that they are open source and community-based: the people who use the forum can run the forum. Think about that the next time you are staring dismally at your Facebook feed, wondering what happened to posts from your actual friends and family. (Answer: an algorithm, made somewhere far, far away by people you will never know, decides what you see.)
We are at the start of something now
This, however, is not a nostalgic walk down bulletin board lane. Instead, I’ve been thinking: there are ways in which the advent of Generative AI is similar to the tidal wave that was the internet in the 2000s. There’s flexing and hyping and Big Tech finding ways to make money out of us just as there was then. So – are there kinds of AI that are smaller, better, more personal, just like those forums?
Where that thought came from
My first thoughts about this were prodded into being by an article on the Daily Maverick website by Michael Power, billed as a recently retired global strategist at financial services company Ninety One.
He makes a distinction between closed and open AI models. Closed models are delivered to you on a plate (think ChatGPT), while open models are transparent and available to be worked on by many people (they fall into the category of open source). Broadly, Power says, the United States is investing in closed models while China is the place to find open models. He predicts that China will win the AI race.
Now, I’ve read more predictions like that, one way or another, than I care to think about. While that jury remains out, I was interested in the platforms and systems and models Power describes as open. I thought: why don’t I see what these things are?
I fished all the names out of the article and did a bit of poking around. Here’s what I found.
Basic concepts
We are largely looking at things called Small Language Models (SLMs). Wikipedia says SLMs are artificial intelligence language models designed for human natural language processing. That puts them in the same class as Large Language Models (LLMs) – which are the foundations for tools like ChatGPT or Claude.
But SLMs are small. An LLM’s training parameters might number in the hundreds of billions. That means they need enormous computational power. SLMs, on the other hand, are based on training parameters that range from a few thousand to a few hundred million. And that means you could, if you wanted, host them on your PC or even your phone.
It follows that if your AI tool is sitting on your PC, it is much more private than a tool like ChatGPT – which, for me, would be a drawcard. (See my investigation into privacy and Gen AI here: Privacy and Generative AI – what you need to know.)
Kinds of SLMs and open source models
As I worked through the Daily Maverick article, I saw that there are different kinds of “open” tools. There are some that are set up so that developers and coders can use them as the basis of making their own tools –Meta’s Llama, Google’s Gemma, Microsoft’s Phi-3 and Mistral’s Mixtral 8x22B. I am not a developer or coder, so I didn’t go down this road.
But these two kinds of AI are accessible to ordinary mortals:
AI tools like ChatGPT (that aren’t big tech): One of these is relatively well-known: DeepSeek. There’s also Qwen – both of these are Chinese. I was more interested in Le Chat, made by a company called Mistral and based in France, which means it would be subject to EU legislation and probably pretty good on the privacy front. I’ve signed up for an account and will now start using it alongside my other AI tools. A more detailed look at it is in the Sensible Woman pipeline!
AI tools that you can download to your machine: Subsequent to reading the Daily Maverick article, an email newsletter with the title My Private, Free AI Setup by Jeremy Caplan found its way into my inbox. He’s the person who referred me to my preferred notetaker Granola, and always makes good recommendations. So I used his text as my starting point.
He says that these tools are free and private. “Use [them] offline without any subscription cost and avoid the risk of having sensitive info ingested into a large language model like ChatGPT, Claude, or Gemini.”
What’s not to like?
I got to work looking at the four tools he mentions: Jan, Msty, Anything LLM or LM Studio. As usual there were four long websites to scroll and a confusing range of offerings. So I kept things simple and downloaded Jan – the one recommended by Kaplan.
What I learned about private AI
The process was more-or-less simple. You download the program as you would, say, an app from the Microsoft Store or from the internet. Then you find the .EXE file and double click on that. The program then installs itself on your machine.
You need to choose your model (the SLM which is going to power your AI tool). There are a lot to choose from, so I took the one Kaplan uses, called Jan V1 4B GGUF (note – this link is not to the download, you get that inside the Jan program. I included it for the more technically minded reader).
I then did several test AI prompts, asking it for how-to advice on connecting itself to my Gmail (because that seems like a good thing to do inside the confines of my own PC). But the AI interface exhibited all of the problems that ChatGPT used to show two years ago: not understanding the question, giving steps that bore no relation to reality. Underwhelming.
I thought I’d give it a second chance and tried a different model (Jan-Nano-GGUF) that has web search built in. There were quite a few setup steps required but I persevered and got it connected.
Then I asked it: “Find me venues for breakfast in Cape Town southern suburbs.” It gave me a list which contained a venue in Sea Point and one in Tyger Valley (for non-Capetonians, those are both many kilometres away from the southern suburbs). I tried that same prompt on ChatGPT and Gemini, and got sensible lists out of both.
It’s possible, of course, that if I persevered with this version of AI it would improve over time. Or that if I had tried something other than Jan, I might have had a better result.But by the time I had got to this point, I had decided I wasn’t going to continue. As I went along, I uncovered a number of wrinkles that, for me, mean none of these tools are likely to be workable.
Problems with private AI
These were the things that became apparent to me as I went along:
1. The initial download and setup is relatively simple and familiar – but anything beyond requires ever-increasing levels of skill. I love doing these kinds of things, but I’m not sure the end result justifies the effort.
2. As far as I could tell, there’s no way to collaborate with other people – which is a growing and really useful capability in the commercial AI tools. More importantly, I could not figure out how I would connect my own devices to a central instance of the programme (if I were to be using this as a serious tool, I’d want it on my main desktop PC, my laptop and my phone, and for all of those to be talking to each other).
3. The program is using your machine, so it will be as fast or as slow as your machine is.
4. I tried connecting Jan to my OpenAI account, which I thought would overcome some of the problems listed above. This proved complicated. You have to get an API key, for instance. That’s easy enough to do – but then it wanted me to “verify your organisation”. I gave up – again, it wasn’t apparent to me that what I would get at the end was worth investing the time.
FINAL VERDICT
If privacy is your over-riding (perhaps only) concern then this is the way to go.
For myself, I will continue using the commercial tools, with all my usual cautionary practices. For the vast majority of users, I can’t see that a private, single-machine-based AI is any competition for the cutting edge models now available in ChatGPT, Claude and Gemini.
And yet…
I am hoping that Mistral’s Le Chat will offer a path away from the Big Tech players, and will be stress testing that extensively. And perhaps more of us should consider decentralising our digital social lives and looking for some old-fashioned forums? I’ll be browsing this list to see where I might fit: Find A Forum. See you there?
Main picture: Lateiner, Wikimedia Commons, Licence: CC BY-SA 3.0
Other things I have written
Privacy and Generative AI – what you need to know – I’ve done some research on the question of privacy and Generative AI. My findings and thoughts…
The diary of an AI user: the bad and the good – Generative AI can be overwhelming. But over time, it’s possible to find ways to make it work for you. Here’s how AI use works in practice for me…
How can I help you make order from chaos?
Join the Safe Hands AI Explorer learning circle!
Sign up for my Sensible Woman’s Guide to AI and Content Creation, which comes out fortnightly.
Or sign up for my personal fortnightly email newsletter In Safe Hands (it’s a place to pause and think about things).
Book a free half hour of my time here. Find out how I can help with writing, editing, project management, general clear thinking, technical hand holding, or an introduction to AI.
Contact me via email.