You can hit it far. You're probably not limited by your size or fitness as much as you think. The secret is not leaking any energy, not a single drop, at the top of the backswing. If one little part is loose or out of place, all the good tension you've created is undone.
Here are the TOP 3 Driver tips of golf professional Rickie Fowler:
1.) THE BACK KNEE STAYS BENT
If my right leg straightens at any point in the backswing, I give up a ton of leverage. I'll end up swinging the club back farther, but what I lose is a strong connection to the ground with my right foot. Without stable footing, it's impossible to hit the ball anywhere near your potential. You want to turn your weight on top of that knee, like a screw.
2.) THE ARMS RIDE ON THE CHEST
See how my left shoulder is snug across my chest? If I were to let my upper arms stray any farther from my body, so they're flapping around, I'd have a hard time syncing things up again on the downswing. Don't be overly strict about this, because you want to stay relaxed, but it's kind of like you keep your armpits more closed.
3.) THE LEFT WRIST SETS FLAT
Butch Harmon and I have worked on getting my hands a little higher at the top of the backswing so my arm plane is less flat—I want my hands higher than my shoulders. I really like how my left wrist is perfectly flat in this photo. It mirrors the clubface, which is square. From here, the chances of hitting the ball square go way up. You don't have to worry about saving the swing on the way down. You can just uncoil.
BONUS TIP: Hitting driver off the fairway takes precision, but it's not impossible. In fact, the shot is a little easier than it was five years ago because a lot of the new driver heads have a lower center of gravity. I'd say most single-digit handicappers should at least mess around with hitting it off the deck. Even if you don't have the confidence to do it in competition, just practicing this shot does wonders for your swing by training your shoulders to be level through impact. If you tip back and try to help the ball up, you'll top it. Come in too steep, and you'll drive it into the ground.