In 2023,I spent a lot of time playing with LLMs. Over the course of the year I got more confident in the idea that anything worth doing can be done locally. LLMs are so cool and fun to play around with, I can see a huge amount of potential, and them being a core part of all computer usage over the next 10 years. This is the exact sort of thing I desperately do not want to have to depend on someone else for!
Fortunately, progress. There has been plenty of innovations now allowing perfectly usable LLMs to work on local hardware, so I’ve been experimenting with them.
I’m currently running ollama on what’s normally my gaming laptop, and then open-webui on the server.
Trying to get some 100% local offline copilot like code complete in vscode is frustratingly hard. Most docs and link talk about using either Cody, or Code GPT or similar plugins, but pretty much all of them have moved to a model of having a centralised service you must sign up for first, albeit for free (so probs evil data thieves!)
However, this is all running locally. I shouldn’t have to login to an external service to make the local IDE speak to a local server!
Currently, twinny looks like a good option to look at, but that looks quite small.