If you think refs get a beating on these boards, wait until you see the one you give yourself. Most refs I know are their own harshest critics. The difference is that referees a quite a lot more knowledgeable on the rules. Much of the critique from fans in based in (understandable) ignorance not just of the rules but off all the work off the mat referees put in to get it right.
Which brings up my second con. It takes constant work during the season to stay conversant on *all* the rules. I had a regime where I read the rulebook cover to cover every two weeks. (I kept a copy by the toilet.) You also need to be in frequent communication with you fellow referees to make sure you are interpreting and applying the rule consistently amongst yourselves.
It may not be a deal for you but my least favorite part of refereeing is dealing with Seattle traffic to get to the gigs.
You have nailed one of the biggest pros for refereeing, staying connected. You will get to know more people in the sport than just about any other way.
One of the biggest advantages to reffing is that you get the best seat in the house. You may be too busy sometimes to appreciate all the action but you will see some great wrestling like never before.
It is a good workout. While there is not much running involved, wrestling refs use there bodies more than other official to keep in the right position. (Ice hockey strikes me as another tough sport to ref.) A long day at a tourney usually means you don't have to watch your caloric intake for a couple of days.
And don't forget the mental workout. A good referee should keep the scoring sequence in his head at least back to the beginning of the period and how many penalties and warnings and cautions each wrestler has accumulated. My favorite mental exercise is simultaneously keeping a 5 count going for a near fall while doing a 2 count for the pin.
Despite the common image, it is not a thankless job. You will get more "good job" comments than complaints.
Overall, it is very gratifying. Knowing that you have completed a very challenging task is a big satisfaction. You also know you are helping to keep alive a sport very dear to you.