I'd hedge a bet that the majority of infractions that are handed our are based on other users reporting posts they feel are inappropriate. While mods do browse, there's just not enough hours in the day to just browse everything, especially not in small talk.
Sometimes a mod chooses to take action, sometimes they do not. Either way, if an infraction is handed out there's a pretty good chance that it was because a user broke the rules "enough" to warrant someone else reporting it and a moderator taking action.
Sometimes mods screw up, but in general I think most have a pretty good idea of what flies and what doesn't. It might seem nitpicky to you, but that's the way it is. There is a bit of a grey area, and leeway is given based on moderator discretion, but it's a fine line and if was any bigger it would not bode well for the site.
It stands to reason that if you're "popular" in small talk (and probably "unpopular" elsewhere on the site), that you might be popping up on the radar of mods more often than you like if you can't stay within the confines of the rules.