Here are some thoughts about how to solve this problem:
For a perfect finish, check out my golf club covers – done exactly like a hat top, using a garter bar to make nicely even decreases, here:
Although my photo has them lying on a table, so there are some wrinkles, they're smooth with the proper circular decreases for a fitted hat. No gathers!
This is a lot of stitches to move, though. By the time you finish a golf club cover, you're have had a garter bar workout.
4. You can make your hat sideways and short-row the crown, as I did in this short-rowed baby hat, pictured in shades of light blue and green.