There's a couple of things you need to configure to get that 0 - 10V DC signal working. I'm not familiar with the ESS Smoothstepper, so I'll have to discuss in generic terms.
Firstly, you need to know which pin maps to this 0 - 10 volt output. That should be in the documentation for the Smoothstepper itself.
Then you need to configure the Spindle in Ports and Pins | Spindle tab. You should have both the "Use Spindle Motor Output" and "PWM Control" boxes checked. Then set the PWMBase Freq. to an appropriate value (usually 50Hz will work). You can also set up control relays on this screen to allow start/stop operation of the spindle via that mechanism if needed, but that is out of the scope of this post.
Then you need to go to the Motor Outputs tab and set Spindle to "Enabled" (green check in that column), the Step Pin# to the pin number of your BOB that maps to the 0 - 10V output and the Step Port to the correct port for your Smoothstepper.
This should get your spindle running at varying speeds, but it may operate backwards (full RPM setting in Mach will result in minimum RPM on the spindle and vice-versa). If that is the case, you'll also need to change the "Step Low Active" to the opposite of what it is (change it from a red X to a green check or from a green check to a red X) and that should get you going with spindle control under Mach3. I'm not sure how different this may be in Mach4 but it should be similar so if you're using Mach4 then you'll need to consult the user's guide for where these settings are made.
Hope that helps,
Stephen "Highspeed" Kruse