I trained a local LLM on my Obsidian, here's what I learned
How I did it in an hour and what to try next
It’s amazing what you can do for free with open source models on a 4 year old computer.
In less than an hour I had a locally running LLM which could access my notes and talk to me about myself.
Here’s what I learned:
AI is not magic, you need to know what problem you are trying to solve.
I’ve been on a journey these past few months exploring various interests in writing and podcasting. I wanted an LLM to summarize these efforts and help me figure out what I’m most interested in.
Basically, I wanted the robot to tell me what to do.
My LLM told me to go focus on learning AI and apply it to my various interests. Which I find suspicious, but I’ll go with it for now 😆.
It’s awesome what you can do with default settings and an hour of your time. But for AI to really be useful, you need to know what you want it to do.
If you can define success, you can train it for your desired outcome.
Data is the most important thing.
Base models are great, but when using RAG you need useful and relevant data to add to the context window.
I got more interesting and useful results when I loaded less data into my vector store.
If I manually selected only notes that were relevant to the question I wanted to answer, the responses from my LLM were more potent.
One thing that’s great about Obsidian and following a Zettelkasten note taking method is you can take a MOC (Map of Content) note and walk the tree of links to only load those notes. Now you have a RAG specific to that MOC.
This approach has been very useful so far.
We need to collect better data.
I along with every company in the world trying to operationalize AI seem to be coming to the same realization.
For AI to be useful to us, we need to collect more useful data (or at least label it).
By this I mean largely making use of metadata on any documents we want to use in a RAG pipeline.
Here’s how I did it:
I used PrivateGPT which is a configurable API that abstracts the details of a RAG (Retrieval Augmented Generation) pipeline.
PrivateGPT needs a model. For that, I used Ollama to pull in Mistral 7B.
Drop in your documents.
Next Steps
There are still plenty of things to try from this point.
My main focus will be using a LLM to assist in labeling my data for improved RAG. And separately I’d like to actually fine-tune a base model using Hugging Face.