After looking at the technique described by Isaac Botkin, "Painting with polygons", I decided to attempt to duplicate this in 3dmax 6 for my next project. After a bit of tweaking with my standard 'tooning' technique that I used in Archon Defender, I was able to come up with a pretty cool look:
As you can see the new technique improves on the old 'tooning' method by introducing light color control. The new tooning material responds more to light color, as opposed to the old technique which didn't at all.
The sketch like effect is created using the multipass camera effect, in 'motion blur' mode. This subdivides a frame into 5 passes, which are different slices in time of the same frame, blended together (you can do 1 to 32 passes, each one taking more rendering time) The base object (here the teapot) has a noise modifier which distorts the polygons of the object, the cycling of the distortion is set to be synchronized with the frame rate. This means that the 'paint effect' will be consistent from frame to frame, otherwise the sketch effect would 'jitter' from frame to frame. (which could be handy for fire and effects... I'll have to experiment)
Now, because the camera is set to multipass, and the mode is motion blur, what I found is that any object in motion will blur instead of having this effect applied to it. To fix THIS, I set the frequency of the distortion to a ridiculously high value, so instead of one distortion cycle per frame, you get 100 or so. Then, I set the interval between the passes to a very short period, 1/100th of a frame, so you have pass 1 at 1 second, pass 2 at 1.001, pass 3 at 1.002 etc. This eliminates the 'smearing' of unwanted motion blur, the high frequency of the distortion 'cycle' maintains the sketch effect.
The only drawback is that this method takes 5 times as long to render. (because I am doing 5 camera passes.) After experimenting around, I found 5 passes to be the ideal trade off between artistic look and rendering time.