Knitting For Adults versus Babies

I can't believe how I am struggling with making adult hats.  The baby hats have long been in production so I can eyeball the size.  But adult hats to me are huge.  I am making a plain hat, 12 rows of 2x2 ribbing and I either take the ribbing all the way up or change to stockinette stitch.  When I think I am not going to have enough yarn, at the end of the ribbing, I add about 5 rows of a different color.  I like this look and I get to use up every piece of yarn.  With my first hat, after the different color band (approx 17 rows), I finished the hat with another 5 inches of knitting.  Holy Moley!!!  When DH tried it on, it barely covered his ears.  DH is a medium size man.  I was shocked because I thought it was huge.  I am now bringing the hat up 6 to 7 inches past my color band.  This fits DH perfectly and if it is too big, it can be folded up and still look good.  I am finally happy with the result and getting use to the size.  I am using Knitpicks, Wool of the Andes, Worsted.  I probably could use a little heavier weight but this is the yarn I have in my stash and it will keep someone's head warm.