The job is a lot closer to backend engineering than people expect.
most days aren't spent training models. it's retrieval, wiring tools the model can call, building evals, handling failures, adding guardrails, and figuring out why something that worked yesterday quietly got worse today.
the biggest mental shift from normal backend work is that you're building around a probabilistic component. a request can succeed technically and still produce a bad result, so tests become datasets, graders, regression cases, and production monitoring instead of just pass/fail assertions.
some roles include fine-tuning or model work, but most of the hard part is making existing models reliable inside a real product.
the API call is the easy part. getting confident that the system still works after the next prompt, model, or retrieval change is where most of the engineering goes.
[link] [comments]