I had similar problem on mine. 2 Things cause his as far as I know -
1. the silver locking pin isnt screwed in all the way due to gunk/dirt/debris stuck in the hole where it screws into. This one is hard to tell but if you unscrew it out of the back cap and get a flash light and shine it into the hole, you may see a buncha crap in there that needs to get cleaned out.
2. Your magnet fell out, you fixed it but put too much glue and now it doesnt sit flush with the back cap... this one is probrably more likely I suppose... I've run into this particular issue on 10+ guns probrably. I've always just dremeled down the magnet to make it flush with the back cap if I couldnt get enough grip on the magnet to pull it out and do over... either way works...
hope that helps.
Proud owner of:
Bob Long G6r - 4c/Lp Poppet - Custom Dust White/Polished Gold
Bob Long VCOM - 4c - Dust White/Polished Gold
Awesome Proto PM8