Has anyone tried in organization to use self hosted llm models for agentic programming?
Im curious if it makes any sense. My organization spends fortune on tokens from us companies. I want to recommend something…
I would recommend Qwen4-27B + CoDoc
What is CoDoc? Have a look at the recent post I published so that you have all context (no pun intended) ;)
IMO, massive costs savings ahead for those that adopt the set-up above.
My organization spends fortune on tokens
Perhaps recommend that they spend the money on hiring competent staff instead.
Do you realistically think that’s gonna work?
no, there’s no possible way that hiring professional software engineers will result in software engineering being performed.
/s
There’s a useful split lurking in this. For narrow agentic work like retrieval over internal docs, structured classification, test scaffolding, deterministic refactor passes, a self-hosted 30B-class model can be fine and the inference economics work out at team scale. For multi-step planning and the harder agent loops, the frontier gap still shows up in the number of retries and the time-to-correct-answer.
The honest test is to pick the prompt category that’s costing you the most and benchmark something like Qwen 2.5 Coder 32B or DeepSeek V3 against whatever you’re paying for now. If the gap is small you’ve found your candidate. If it isn’t, you’ve at least costed the gap accurately rather than guessing at it.
The two costs people underestimate are the GPU box (plus a second one for the eval/staging path) and the maintenance overhead. Model picks go stale fast and someone on the team has to own that, or you end up shipping a Llama 3.1 stack into 2026 because nobody rebuilt the harness for whatever’s current.
Qwen 3.6 and gemma4 models are the only ones usable for agentic prog sessions that I and my employer run locally. It’s less stable and slower than third-party services, even on much better hardware (as it’s with my employer). The best way is to go with a provider hosting deepseek flash/pro if your privacy policy allows though. It’s going to be hard to beat their price.
If you just want to avoid U.S. company, you can try mistrialAI.
As far as I understand, the only way to get anything resembling usable output for coding is with massive, expensive, labouriously hand-tuned models, not local ones.
I see that qwen 3.5 has pretty good performance and can be run on macbook with 64GB ram
Qwen 3.6 is even better
Qwen 3.6 27B dense is really good. Very usable coding output
GLM is pretty good in mg experience, the company I currently freelance at runs it locally (in house server room) for compliance reasons. But it needs very beefy hardware.
Models running within the constraints of a dev machine have no chance
If you want this, you need a company AI server with enough performance to support the entire team at once, and it will probably still be worse than using a cloud one. Though it MIGHT pay for itself in… A while
How about qwen 3.6 and MacBook with 64GB ram?
I thought about that AI server, but idk how to calculate how long it pay for itself…
I mean… RAM? Don’t you need mass VRAM for this kind of thing? Or are they shared on Mac?
idk how to calculate how long it pay for itself…
You don’t… Not in this industry. You guess and hope it goes in your favor.
No calculations matter if the market can jump or drop by 300% in a few months… And that applies to programming, hardware prices, AI subscription prices, regulations between countries when Trump is in office…
Apple unified memory shares all over CPU, GPU and NPU, you can assign a lot of memory to run local models and there bandwidth is good, depending on the model.
AMD has something similar with their something something AI CPUs and they go up to 128GB at the moment. Apple can be way faster though. And you were able to buy a Mac Studio with 512GB back when RAM wasn’t worth more than unicorn pee. For… I guess 10k though.
Apple unified memory shares
That’s cool asf.
Apple engineers with better leadership could change the fucking world… But instead they’re used to screw over their own user base.
If my GPU starts falling back to RAM my game fps drops to 1 lol.
I am running qwen 3.5 locally using llama.cpp on 8gb of VRAM and 16 gigs of RAM. It works well enough with a 4B to 9B parameter model along with quantization and MTP. More optimizations are on the way with turboquant and possibly other tech.
It’s just there to assist me, not do all the work, so I am happy as long as I can self host it.
I can’t say how well my specs would work in a professional setting but for personal use a MacBook should be sufficient in my opinion.
