As much as I would like to think there is not some subconscious racism involved in the debate over NBA age limits, there's really no way around it. When you consider that no one seems to have the same problem with Baseball and Hockey, or even non-black foreign basketball players (Darko Milicic rarely seems to come up in these discussions, for some reason), the common denominator is that we're uncomfortable with black kids making basketball a priority over school.
And while there is probably a lengthy discussion to be had about educational priorities in African American culture, an enforced Age Limit amounts to reverse Affirmative Action against kids who don't need to go to college at the age of 18 to earn a living.
Plus, as Manny pointed out, there's no time table for going to college. The implied "straight to college after high school" rule is overrated... and I believe it causes a lot of burnout with kids who, frankly, aren't ready for a higher education. That, also, is a topic ripe for discussion.
But back to the NBA... if the league gets to a point where High School players are adversely affecting the level of play in the league, owners will see the effects on their bottom line, and they will draft accordingly. But I don't think we've seen any higher a percentage of High School NBA failures than 4-years-of-college failures.