I think at least one chance for them to clarify their position, or genuinely answer to critique should be given. If it's clear that it's just a drive-by low-effort comment, or if they just have an axe to grind, only then should they be downvoted. If they are serial offenders, they typically get shadow-banned.
However, I disagree with you on a second point. If you comment, don't downvote at all. Your words make your opinions known and downvoting feels both disrespectful to your conversational partner and arrogant. Like voting for yourself for prom king.