We were blocked on evaluation of our multi agentic AI for a while because we assumed we needed a complete dataset before we could trust any results. What finally unblocked us was starting with something much smaller and more practical. We picked one wo…
I have been working on an agent style system where behavior changes often as we adjust tools, prompts, and control flows. One recurring problem is evaluation. If the system keeps evolving, when is a good time to invest in a proper evaluation dataset An…
I spent the week diagramming the full path documents take through my RAG system. Visualizing it clarified something I’d been feeling for a while. Most retrieval issues don’t start at retrieval. They start much earlier. The moment ingestion or segmentat…
RAG failures often look mysterious: Relevant info appears missing, unrelated chunks show up, top-k results wobble week to week. Based on what we observed the real culprit is usually your metadata tags no longer describe the chunks you actually embedded…
We sometimes think RAG breaks because the model isn’t good enough. But the failures are almost always systemic. Here’s the uncomfortable bit: RAG collapses because the preprocessing pipeline is unmonitored, not because the LLM lacks intelligence. We us…
Our RAG stack degraded slowly over months. Text-shape differences created different embedding vectors Hidden characters slipped in from OCR Partial updates mixed old and new embeddings Incremental index rebuilds drifted from ground truth Retrieval lo…
Chunking is one of the most repetitive parts of a RAG pipeline, but it quietly decides whether retrieval holds up or falls apart. I keep running into the same failure modes: boundary drift, semantic fragmentation, inconsistent overlaps, context dilutio…
We’ve been working on an Autonomous Agentic AI, and the thing that keeps surprising me is how often performance drops come from ingestion changing quietly in the background, not from embeddings or the retriever. Sometimes the extractor handles a doc di…
Working with AI engineering teams for years has shown me a consistent pattern. Most of the time isn’t spent on model. It’s spent on repetitive workflow steps. – Ingestion: data formats vary, cleaning rules stay the same – Chunking: simple segmentation …
