Data point: A couple years ago I worked for a company that calculated sales tax for customers. They had developed expertise in knowing where each jurisdiction kept it's tax code and how to turn that tax code into software. I was peripherally involved in a proof-of-concept where a reasonably well skilled team trained some sort of model on tax code for one US state. They demoed it to the board of directors where one of the directors asked a somewhat complex question.

The model returned an answer that looked legit, but after the board meeting someone pointed out that the answer was wrong and had we given this answer to a paying customer, we might have been criminally liable. I'm not sure who would have been criminally liable. I don't think they would arrest the entire company.

This was over two years ago and Claude seems to be getting MUCH better over the last year, so maybe things are better now. But, as the Russians say, "доверяй, но проверяй" (trust, but verify.)