Ray,
As you describe the camera rotates around one pixel. Does that one pixel rotate around the crosshairs in Mach3 as you rotate the spindle? If so, as it rotates is it the same distance from the crosshairs in each quadrant? Does your camera have adjustment for centering.
I am guessing that if it doesn't have adjustment it may not be that good for Mach3, the software that came with it probably will let you adjust the crosshair to compensate for the center of the camera not being concentric with the camera shaft axis/spindle axis.
The key to the precision is being able to align the camera view axis precisely with the spindle axis, this is easily verified by rotation, if the target mark moves from the crosshair as you rotate it is not properly aligned.
I saw those cameras on Ebay, it was funny just after I had found and mounted a camera that worked real good I felt really lucky because the camera I chose was so easy to mount and focused really well up close, I was cruising ebay and saw those and discovered it looks like it was the exact same camera I used on my mount. If you need just a new mount let me know.
Arnie