Close
After a couple hours in a debugger, I managed to fix this.

In CS4 32-bit (Windows only, and you must have the 11.0.1 patch installed), change byte 009631D5 in Photoshop.exe from 01 to 02. (If it's not already 01, then you're either in the wrong place or have a different binary.) Back up your executable, obviously.

This changes the base for the rotation tool from 1/10 to 1/100 (in other words, it's .1^n). This affects both free transform and transform selection.

This also affects mousewheel changes. Before, mousewheel in the rotation field would move in 0.1 steps, and shift-mousewheel would move by 1.0. Now it's 0.01 and 0.1. There's no in-between (sorry, best I can do with a hex editor). You can still drag-rotate the selection for rough alignment.

You can change it to 03 if you want 0.001 steps, but I'm not sure that's useful (and it makes the mousewheel useless).