The only time I can get that error is when entering depths 0f 0.004 or less. I have tried with the machine in inches and working in mm, mm - mm, mm - inches and inches - inches. It is all the same.
The formatting must ignore 4 thous or less. Over 4 thou to 8 thou the formatting interprets this as 0.01. I cannot get the DRO to accept 0.009 - and from 0.01 onward it seems OK.
Reading the formatting instructions, this seems to indicate that Z has two numbers to the right of the decimal place. If more numbers are entered, then the formatting rounds up (or down) to two decimal places - which seems to be in line with my results above
You do not need to put a negative number in the hole depth - the formatting changes the number to a negative number anyway. Just enter the depth you want. If you are in inches - then 1.5, 2.75 etc - to 2 decimal places. If you are metric then 5mm is 5.0 etc. - also to 2 decimal places - but I don't think metric drills go to that (not the ones I use anyway)
Maybe the DRO could be changed to accept only two figures after the decimal place.