tl;dr: treat llm output as a pointer to the answer, not the answer
Everyone who leans on these tools has been burnt: an invented flag, a test that passes for the wrong reason. Abstaining isn't a serious option for me either ... the leverage is too large to leave on the table. So the question is how to keep the leverage without inheriting the failure modes.
Constructing the proof was always the expensive part
The rule I've settled on, and would suggest to anyone who asks: every piece of LLM output is a pointer to an answer, not the answer.
The mental model is proof versus verification. Constructing a proof is the expensive act; checking a candidate proof is cheap by comparison.
That asymmetry is what makes delegation work, and also provides the "rule of thumb" here: if there's no cheap, decisive way to check an output, don't delegate it!
The model does the search and the first draft; the check stays with me. And "cheap" still has to be paid: acting on a faulty diagnosis costs far more than confirming it.
In practice: when Claude hands me a diagnosis for a misbehaving cluster, I confirm it (however I do that) before acting on it. When it writes a test, I do my bit, however small, to ... "test the test." When it writes a change, I read the diff the way I'd read one from someone with zero context on our systems.
"The LLM said so" is not a root cause
There's nothing wrong with a failed check, that's the loop we have to drive now. Feed the failure back with a nudge (whatever it is, a contradicting log-line, a skeptical question) and the next candidate is usually better.
Finally, ownership: every PR or runbook that we ship has a human owner, and in a postmortem, "the LLM said so" is not (yet?!) an acceptable root cause (though that phrase is a pointer too ... usually to something that got skipped).
So, the wrong answers are priced in, and it's the right ones that should worry you!
Every unchecked answer that turns out right ... makes the next check feel a little more optional.
And yes, I've caught myself skipping already, more and more these days.