Referee who's seen some stuff, here: Don't do it. For one thing, you can seriously mess up your ear if you get shot without it on. Permanent hearing loss is certainly a good possibility. Second, you'll have no temple coverage. If that rather sensitive part of your head gets hit, there's a good chance you'll be taking an unplanned nap, with the potential of your goggles falling off when you hit the ground, hitting your head, and all manner of unpleasant things. TL;DR, just don't do it. Many reputable fields won't let you on without ear coverage, as well.
~Keep yo' goggles on yo' face~