Local AI Server - Preface and Goals
Part 1
Preface
A couple of weeks ago, I came home from Michigan, finishing up my final semester on campus, and I had a stark realization - I should now have the tools to create (or learn how to create) pretty much whatever I can think of in the software world, and yet my most impressive creations feel a bit lackluster.
I decided to go to Bird Rock (aside: pretty cool coffee shop in SD) and brainstorm how I could apply my newly acquired skills in a practical context; I came up with the following personal requirements to guide my journey to discover a cool technical project:
Must incorporate networking + os fundamentals (current fields of interest)
Must strive to go past the boundary of projects accomplishable by AI tools
Must have a long-term focused vision, making the project more meaningful than a one-off coding sprint
Must help me work towards my goal of becoming a purposeful and clean programmer
Must feel cool to me (I need to be motivated to complete this)
After creating this list, I hit a roadblock - my requirements had overconstrained me, but I was motivated to find a meaningful project. That’s when I remembered a conversation I had with my roommate (shoutout Ohm Patel). We had briefly talked about Pewdiepie and his LLM council, and the gears started turning.
This summer, I spent a good bit of time working (mainly thinking deeply) about Nomie, a software layer allowing users to grant granular, conditional access of personal information to AI agents. The main issue was always data persistence. Once an agent ingests information, it tends to cache pieces of it, whether that lies in embeddings, system memory, or even in logs; The user has no real visibility or control over where that data lives. Although we didn’t figure out a great way to fix this gap (aside: these guys might have figured it out), this knowledge convinced me to create my own AI stack.
In this blog, I hope to share my learnings, struggles, and successes so that anyone else interested can get an idea of what it was like to make something like this…
Goals
Before I dive into the nitty-gritty, here are a couple of project-specific goals I noted down to guide my development. I plan to reference/make decisions based on these goals while I build to structure my progress.
Thorough Technical Understanding of the Hardware and Software
I have a barebones understanding of computer hardware, as I used to build computers. I also have a basic understanding of AI agents, as I’ve used AI API’s frequently (picking up things like tokens, etc). What I don’t understand are things like model size, how I can customize RAG for my personal purposes, and how I can split prompt workload across my consumer GPUs.
By the end of this project, I want to deeply understand the ins and outs of building a technical system.
Open Source Tools && Deep technical understanding >= Mindless Dev work
If I have the option to use open source tools that accomplish my goals more efficiently than I can develop on my own, I plan to use them wholeheartedly.
I don’t want to mindlessly use other people’s software, though, as that would defeat the point of doing a personal project, so I want to make sure that I thoroughly understand the tools that I use.
Guide + Open Source
I always see open source projects, and I would love to contribute to the community. My goal with this project is to (eventually) build an open-source guide and software that users with adequate hardware can use very easily.
Data Control & Trust
No third-party exposure: Inputs and outputs never leave my machine
Performance
Local inference should be faster, as I should not need to deal with shared cloud GPU queues and rate limits.
Local inference should also (if possible) be cheaper than paying for ChatGPT. I want to track power consumption/etc.
Practical Use
I want to use this instead of ChatGPT. This means I need a UI, I need to figure out networking to access my local machine from anywhere, etc.
Miscellaneous
Since I have so many GPUs that will be used in this rig, and I won’t be prompting 24/7, I want to see if I can mine cryptocurrency when I’m not using LLMs.
I also want to look into renting out my infrastructure when it is not being used by me.
I want to host a group Minecraft server on here :)
Well, if you made it this far, I applaud your attention span. Finally, I’m done defining everything and setting goals. Now that the boring part is out of the way, let’s start looking into the hardware 😏.