This is part of the Vorkosigan series by Lois McMaster Bujold. It's the first I've read in the series (and by this author) but I think it isn't technically the first in the series. There's a number of different recommendations for where to start the series and this was one of them even though it wasn't the first thing published.
The main character wants very much to join the military as an officer but has some serious physical limitations (something like osteogenesis imperfecta except caused by outside forces before he was born instead of genetic) and isn't able to pass the physical tests to get into officer school. He ends up traveling since he doesn't know what to do with himself now and has adventures! He collects other "desperate" people along the way and gives them a purpose.
There's a lot of class and hereditary political power stuff. More what I'd think of as a medieval political/social structure, except all sci-fi-y. With liege-lords being responsible for those who have sworn oaths to them and that sort of thing. Very serious about not marrying outside your class...
And there's some mention of superstitions regarding "defective" people. I think the planet he's from has had a lot of change in the last hundred years and there's still a lot of older attitudes that don't quite jive with the new attitudes/technology of the rest of the planets. There's this whole idea of him succeding despite the expectations of everyone around him.