okay so marin v0.1 is finally here 🥀 after way too many cuda kernels, memory bugs and endless optimization loops, i can finally show what i've been building lol marin is my own local llm inference runtime written in rust + cuda. no llama.cpp or python underneath. the whole idea was to make local inference usable on regular consumer hardware without needing some crazy setup. it has cpu/cuda inference, int8, streaming chat, context reuse, batching, persistent sessions and even a tiny background assistant that can suggest help based on approved activity. here's a lil demo of me actually using marin from the terminal. some benchmarks and repo link below ↓

gm guys, it’s sunday and i’m bored out of my mind and sleepy for absolutely no reason 😭 anyways, dropping an article on marin, my llm inference runtime v soon.

okay so marin v0.1 is finally here 🥀 after way too many cuda kernels, memory bugs and endless optimization loops, i can finally show what i've been building lol marin is my own local llm inference runtime written in rust + cuda. no llama.cpp or python underneath. the whole idea was to make local inference usable on regular consumer hardware without needing some crazy setup. it has cpu/cuda inference, int8, streaming chat, context reuse, batching, persistent sessions and even a tiny background assistant that can suggest help based on approved activity. here's a lil demo of me actually using marin from the terminal. some benchmarks and repo link below ↓

disappeared for a week cuz some stuff happened and I got busy + Marin also kept me busy w endless optimization loop and my own sanity marin is almost wrapped up tho i have SO much to yap about. so will drop it tonight 🥀

hi idk why but i’ve turned into a sloth lately. all i wanna do is sleep the whole day

another marin update been a while since the last one. quite a bit has changed. - fixed the extra vram usage from batched decoding. 3 concurrent reqs now get ~1.9x throughput without needing more gpu memory. - also first-token latency dropped from 333ms to 112ms. - built marin into an actual interactive terminal. just run marin, pick a model once and chat. there's also a read-only coding mode that can explore your repo. - convos can now survive restarts. marin saves the chat history and can optionally save the model's internal state too. resuming a 2.2k-token convo went from 25.5s to 9.2s by restoring that state instead of recomputing everything. all of this is running locally on my 8gb laptop gpu. still a lot left to build esp around memory management. but marin is finally starting to feel like something i can actually use.

another marin update been working on making convos faster and handling multiple reqs on my 8gb laptop gpu. - multi-turn chats: marin was recomputing parts of previous convos. fixed that by saving and reusing the model's internal state. two-turn convos are now ~26% faster. - request scheduling: a long prompt used to block smaller requests for ~11.5s. now marin processes prompts in smaller chunks so other reqs can get a turn. wait dropped to ~0.47s. - batched decoding: realized faster responses don't necessarily mean higher throughput. so instead of generating tokens separately for each request, marin now processes up to 3 together, reusing model weights. total throughput went from ~36 to ~64 tok/s. - correctness: each convo still has its own memory and generated outputs remain identical even when requests run together. currently: working on reducing the extra vram batching needs. it's faster, but it exceeded the memory budget i set, so there's still some work to do.

bengaluru has potholes, i have blood loss. neither of us is doing well but at least i'm hot

marin update after a while. got sick for a bit right after my last update, so things slowed down 😭 but quite a lot has changed since then. marin can now run qwen3.5-4b on my 8gb laptop gpu, getting around 38–40 tokens/sec. also added a way to store model weights in a smaller format and load them directly, so they take less memory. then i added marin plan. it checks your hardware and available memory before running a model, figures out what can fit and picks b/w cpu and gpu instead of just crashing. also started exploring mixture-of-experts models with ibm granite. basically, instead of using every part of the model for every token, it picks a few smaller networks to do the work. got it running on cpu and gpu (gpu still needs some numerical work). also made cpu prompt processing over 4x faster by grouping tokens that use the same experts. and the latest thing: marin now has a local server. you can load a model once, send requests to it and get responses streamed back without reloading the model every time. still a lot left to figure out especially memory management and making convos faster. so will update more

marin update: it can now run qwen3.5-2b on both cpu and gpu this was a pretty big jump from qwen3. instead of relying only on attention, qwen3.5 also has layers that keep a running memory of previous tokens. so i had to implement a diff way of processing tokens and managing that memory. got the outputs matching hugging face. then started making it faster. some results: cpu prompt processing went from ~8 to ~93 tokens/sec, around 11x faster int8 brought cpu ram usage from 3.9gb to 2.08gb cuda now generates ~58 tokens/sec on my laptop gpu the fun part is that the same runtime now handles two pretty different model architectures.

It's here! Today I'm officially launching Yomu, an app that makes your daily reading easy. - Bring your own blogs and articles and read them inside Yomu - Uses your local agents (Codex, Claude, OpenCode, etc.) so you can ask any question mid-reading - Bookmark and save an article to the Yomu folder in your browser, and it automatically appears in the app -Upload any research paper PDF and get a stable UI for long reading sessions -Customize the theme and make it look the way you like And many more! (yomu.lowkeydev.me)

marin update: finally started optimizing the cpu side of my rust inference engine after spending so much time on cuda. turns out it was almost 10x slower than llama.cpp at processing prompts on a single core 😭 so i rewrote how marin handles calculations, fixed a bunch of unnecessary memory reads and made it use multiple cpu cores. results so far: - processing prompts is now ~5x faster with 8 cores - generating tokens is ~1.5-1.8x faster - int8 weights save up to 1.7gb ram and make generation another 1.5-1.85x faster - single-core generation is now around 90% of llama.cpp's speed on my laptop also working on making memory usage grow only when needed instead of reserving it all upfront. the goal is to make marin run well on low-end pcs too and eventually support much bigger models.

marin update: spent the last few days trying to squeeze more speed out of my rust + cuda inference engine and the results are kinda crazy lol here's what changed: 8k token prompts went from 25.4s -> 5.9s before the first word appears. turns out my attention code wasn't even using the gpu's tensor cores properly. rewrote it and got ~4.3x faster prefill while using 77% less gpu energy. long convos no longer slow down generation nearly as much. found that the gpu was reading the kv cache in tiny chunks. fixed those reads and went from 18 -> 35 tokens/sec at 16k context. almost 2x faster. marin can now remember convos across restarts. instead of processing thousands of old tokens again, it saves the kv cache to disk and restores it. resuming a ~4k token conversation went from ~6.5s -> 0.22s to first token. kv cache now uses half the memory with fp16 without noticeably changing the answers in my tests. int8 weights are working in normal chat too. ~1.7gb less vram and up to 63% faster generation. still optional since i'm not fully happy with the quality tradeoffs yet. also tried a bunch of things that didnt work including using a smaller model to guess the next tokens. (will try this later on after some more minor optimizations) honestly the funniest part is how many bottlenecks turned out to be my own cuda kernels.

marin update: last time i said prefill was next, so spent the last few hours making long prompts and chats suck less - first fixed prefill attention basically the work marin does on your prompt before it can reply. got attention ~2.25x faster and a 3.5k token prompt now takes ~7s to first token instead of ~11.7s. - then found something kinda stupid: every new msg was making marin process the whole chat again from scratch. now it keeps the kv cache from previous turns and only processes the new stuff. at ~4.1k chat history. first token went from ~9s to ~0.25s around 35x faster. - also tested this with speculative decoding including wrong guesses getting rolled back to make sure the reused cache doesnt get messed up. same outputs across bf16/int8. - then added paged kv. before, an 8k context reserved ~1.88gb of kv memory upfront even for a tiny chat. now memory grows with the chat: 64 tokens: ~117mb 4k tokens: ~940mb 8k tokens: ~1.88gb so an 8k context no longer means wasting almost 2gb vram from the start. costs only a few % speed at long context and outputs stay identical. next up: figuring out where kv pages should live when they don't need to sit on the gpu.