Utilizing small language model, coupled with RAG, to suggest appropriate course of action for system administrators

Greetings all,

Apologies if I stumble here in my description of what I'm trying to accomplish. I'm a newbie in this field with only about a month under my belt.

I've been tasked with researching an 'AI solution' where we can throw 'system message logs' to an AI model, that's been agumented with our own internal knowledge articles on administering our propertiary system, and have the AI model tell the user what article describes how to address issues going on in the system.

The desire is to have this run on the system itself (aka - no cloud), therefore we're using a small language model. I've been focusing on Phi4-mini thus far.

The knowledge base articles are all local and I'm using the techniques described here to make use of them (RAG using Ollama. Below is a step-by-step guide on how to… | by Diwakar Kumar | Medium).

The message logs from our propritary system are limited to critical errors, all of which are described in the knowledge base articles indexed via RAG.

So far the responses from the model have been poor and haven't yielded anything meaningful.

Open to suggestions on how to improve this.

Thanks!