Aardvark Daily

Please visit the sponsor!

I have my own AI LLM now

There was a story on the newswires earlier this week which claimed that a US company had ended up with a half-billion dollar bill as the result of "enthusiastic" IA usage.

On further investigation it appears that this was more of an anecdotal story than one based on actual facts. As a result, many news publishers seem to have "vanished" their original reports so as to cover their gullibility -- although some remain online.

Although AI tokens are expensive, it is very unlikely that any company will have racked up half a billion dollars worth of use in a single 30 day period -- hence the rapid backtracking.

However, commercial-scale use of AI is expensive and growing moreso by the day.

Companies such as Anthropic are reporting very strong revenues so obviously a lot of people are paying for AI services already.

Being the scrooge that I am, I've relied solely on "free" AI LLM access such as that provided to Google's Gemini chatbot. I use AI but I'm not reliant on it.

As I mentioned in a previous column however, it was my plan to set up my own local AI LLM in order to better acquaint myself with the technology and to perhaps create a resource that I could use without the need for internet connectivity or a monthly subscription.

That is exactly what I did yesterday.

Now, as I type this, I have an SSH session in a window on my Linux desktop. That session connects to the 8GB Rasberry Pi 5 that now sits, tucked out of the way, under a desk on the other side of the room.

Running on that Pi is the smallest (2 billion parameter) model of Gemma 4, a "free" AI LLM model from Google.

I guess what most people reading this will want to know is "how well does it work?"

To be honest, the results could be considered disappointing or outstanding, depending on your expectations.

This LLM runs at less than 10 tokens per second (which is pretty slow in the AI world) but it still produces some very useful results at a pace which is not intollerably slow. For instance, I asked it to create a "hello world" program in Python and it took less than 20 seconds to do so.

The outstanding thing is that it was able to create that code. The disappointing thing is that I could probably have typed it out by hand in less time.

However, this LLM assures me that it can do some stuff I had never expected -- vibe-coding in Z80 assembler for instance.

The biggest strength of the system, as currently configured, is also its biggest weakness -- the fact that it's not "connected" to the wider internet. This means that its knowledgebase is frozen in time at about 18 months ago. It also means it's not going to quickly search the internet for information it needs but which isn't already part of that knowledgebase. So long as you're aware of this it's really not that much of a handicap though.

You won't get news analysis or insight into any contemporary internet content -- but it still seems pretty good at solving math problems, doing pretty basic coding and problem-solving.

The single biggest problem performance-wise however, is the way it seems to need to re-evaluate every query you've entered into a session before processing a subsiquent request. This means that it starts out quite sprightly but after half a dozen queries or more, it slows dramatically as it labouriously re-processes all the previous queries to create "context" for the current one. Regularly clearing the context-space avoids this but it does limit the complexity of what it can process.

I'll be doing an awful lot more work and experimentation over the coming weeks but, given that I bought my RPi5 before RAM became more valuable than gold, it certainly seems an excellent use for an SBC that was previously just gathering dust.

Since having this energy-efficient Pi and LLM running 24/7 is not going to make any perceptable difference in my power bill, there's no usage limit or subscription and that it's always on tap when I need it, I'm pretty happy with the outcome.

Carpe Diem folks!

Please visit the sponsor!

Here is a PERMANENT link to this column

Rank This Aardvark Page

Change Font