I wear a WM in AA. I just got my first zip-up hoodie (S) and it fits perfectly. For pullovers, I have both S and M. The small fits correctly and is more flattering than the medium; the medium is better for that "it's raining, the dishes need to be washed, and I'm grumpy" fit.
The pullover hoodies seem to be fairly forgiving in their fit; Narfcake wears a ML in AA and we both order M pullover hoodies. The zip-ups are slightly less forgiving because the band around the bottom is tight when the hoodie is zipped, but folks who wear the same size in t-shirts may prefer different sizes of zip-up hoodies because the fit depends on where the bottom of the hoodie falls on your body and your body shape.