Safe Z should have no effect while g-code is running.
From the sound of things, you may be losing steps when the router is retracting. When it's cutting the .05 deep groove, what does the DRO show? If it show .3, then your losing steps. If it shows -.05, then your code is wrong.