Ran into this problem: a flat markdown file (or even a GraphRAG "brain") gives agents better context, but when a chain of actions fails, you cannot point to which stpe broke or what to fix. No explicability, a slow convergence, and no ceiling.
So I modeled it as a partially observable MDP instead. States the agent can observe become nodes, actions become edges, and each edge is weighted by success probability, token cost, and outcome. The agent explores to complete the graph, then exploits it like standard RL.
Ran the same task batch 3 ways (flat file, brain, this) across Claude Opus 4.8 and Codex GPT-5.5. The succcess rate went from about 0.78 average to about 0.95, at a lower token cost than the flat file.
Curious if others have tried framing agent policies this way!
Wrote up the math and the full comparison in the link in the comments:
[link] [comments]