Our "best athletes" are built for other sports. Many of our best athletes I wouldn't consider to be losses for the sport of soccer. The only ones I'd really put in the loss category are point guards, RB/CB/S, and the occasional baseball player. A LeBron James is more of a loss to rugby (I like that sport too) than soccer.
Even the huge soccer players, guys like Ibrahimovic and Vieira, are small by American athletic standards. Ibrahimovic is 6'4" and can't weight more than 190 lbs, Vieira is half an inch shorter and maybe slightly heavier, if at all. Some guys, Kevin Garnett comes to mind, could have been astounding goalkeepers, but when we already have Tim Howard as one of the world's best, that's no loss.
A wikipedia search of FIFA World Players of the Year shows that Marco Van Basten was the tallest winner ever and he's all of 6'2". Also consider the commonly accepted greatest players ever, Pele (5'8), Maradona (5'5), Cruyff (5'11), Best (5'9), Beckenbauer (5'11.5), Zidane (6'1), fat Ronaldo (6'0), Puskas (5'7), Eusebio (5'9.5).
The real losses for soccer are the inner city kids. We just don't have any around and that's where most of the world's best players come from. The sport just doesn't exist there. A lot of those kids grow up playing basketball and end up as those 5'11-6'2 shooting guards that are astounding high school and college players but have no shot at being really significant in the NBA.
Jerryd Bayless is a loss for soccer. Not LeBron James.