Odds are the drill will "walk" a little and your hole will be a little oversized. It won't drill small so worst case you get a snug fit and best case you get a nice slip fit. Go ahead and drill the hole if its small I'll mail ya a 4.1mm
reamer. I used to be a Toolmaker and I built about a zillion aluminum prototype molds, trust me you will be fine.