I wrote a post about a specific problem with running AI agents on local models: they're unreliable with structured output. You ask for JSON, they mostly deliver, but then they forget a closing brace, invent a tool name, or add a paragraph of text after the JSON object. Hosted APIs like OpenAI handle this server-side. Locally, you're on your own.
My approach: llama.cpp supports GBNF grammars that constrain which tokens the model can produce. I wrote a compiler that turns each tool's schema into grammar rules, so the model's output is constrained at every token position. It literally cannot produce malformed JSON. Then I narrow the grammar per-turn so the model only sees the 3-5 tools that are relevant instead of all 50.
The post is a deep dive with real code from the project (Eris, a local agent in Rust that uses your Markdown notes as memory, runs entirely on your machine).
https://eris-system.dev/blog/gbnf-grammars
Repo: https://github.com/janpauldahlke/eris (Apache 2.0)
ps. i wanted to share how i solve the problem, it is related to my project, but not self advertisement.
[link] [comments]