There's nothing wrong with starting small and working your way up.
When I first decided to start walking several years ago, I could barely make it around the block. After I built up my strength and began to lose weight, I walked in a March of Dimes walk. Sure, 2 or 3 miles is nothing for long-distance walkers, but for a former couch-potato it was a pretty good accomplishment!
