Depends on the model and aluminum/fiberglass. It will make a bigger difference in Aluminum because of the extra weight will help the boat ride better seeing how the rider position will differ (Closer to the transom the better the ride). Fiberglass length won't make much of a difference ride wise.
I have fished 14' boats to 23', fiberglass and aluminum and I can say it is nice to have the extra room but the longer boat is harder to handle by yourself unless you have autopilot. Aluminum will almost always ride a little harder because it lacks the weight no matter what hull design/manufacturer.
When I bought my boat 2 years ago I had a choice between a smaller aluminum (18' i think) boat and chose my 23' fiberglass based on size and the ability to fish 5 guys not including me comfortably. Not to mention my father-in-law has the same make boat and the ride is comparable to a Grady.
Do you have to buy new? There are a lot of boats in that range used for a good price.