I have read the paper "Digital Bas-Relief from 3D Scenes" from which this process is taken. It is mathematical involving calculus. Instead of using the points of the 3d surface and changing their heights, the process uses the slopes (gradient field) of the surface. Maybe I can explain basically what I read that they do.
First, the gradient field:
Imagine a simple 3d bump like a hill. Think of a 2d contour map of the hill. It is a family of curves with each one representing where the surface has a certain height. Each contour curve has a height number attached to it. The perpendicular distance (in 2d now) between the curves on the contour map indicates the slope of the surface. Places where contours are close together means the surface is steep and places where the contours are far apart means the surface is flatter. Now imagine standing on the surface and looking at your contour map. If you wanted to climb up the hill the fastest, the direction to take would be the direction which takes you straight to the next contour line. At each point on the surface, this unique direction is called the gradient direction. It is the direction which gets you up the hill the fastest. Now at each point on the map, draw an arrow (vector) which points in this fastest direction. Make the length of the arrow equal to the rate at which the height in that direction increases (the steepest slope) and you have what is called the gradient vector field. It is this gradient vector field that he process uses.
The process keeps all of the gradient field directions the same and changes only the lengths. That is, the slopes of the surface are decreased but those steepest directions are not changed. (That field of steepest directions is really the esthetic geometry of the hill.) The slopes are now decreased using a logarithm function (sorry for any flash backs!). The logarithm is used because it reduces the larger slopes more than the smaller slopes. The result is the vector field of a new, lower profile hill. The new hill has to now be recovered from the new vector field data. This is a process called integration. After integration, we have a new hill with the same basic shape but is not as steep or high.
The process, of course, involves may other details and complications. What I have described is just the very basic idea. The paper, if you want to read it is at:
http://gfx.cs.princeton.edu/pubs/Wey...DBF/relief.pdf