Ken is right, there is no difference in screw size for the hammer screw. The originals had a wider piece below the head and acted as a washer.
The screw size is 4x40.
If you are lucky enough that a small portion of the broken screw is sticking out a pair of pliers should get it out, if broken flush then a bigger job.
Most hammers that I have taken off were hard to remove almost like a tapered fit near the bottom and usually have to be removed by rocking back and forth with two thin blade screwdrivers.
It is very possible that your fit was too loose and the hammer was sliding off the fit and cocking it every time rotated the screw until it finally broke. That is just my guess.