Easy enough to ignore them to be fair, or a quick offer of the whistle and ask if they want to take over usually makes them pipe down.

Did have an incident when my son (14 at the time) was reffing and a parent ran on the pitch to remonstrate with him, I was running the line and charged on after him, luckily the home coach saw me making a beeline and managed to get in between us in time otherwise my flag may have never seen the light of day again