You’re right, Jane, that this is a common question. But the answer, I think, is pretty straightforward: Discrimination on the basis of skin color is different — constitutionally, statutorily, historically, morally — than discrimination on the basis of, say, whether someone is from Wyoming.
I think that conservatives and libertarians are generally happy to leave it to schools to decide whom to admit. Of course the decision can be based on considerations other than test scores and grades (the recommendation of a particularly trustworthy teacher), and people can agree to disagree about how heavily different criteria can be weighed (athletic ability), whether it is appropriate for some to be weighed at all (Dad’s donations), and whether the answer is always the same for every school (a public/private distinction). Personally, for instance, I don’t like alumni preferences, especially for public universities. And I oppose sex discrimination, too (which, of course, is also illegal, not just bad policy).
But racial discrimination is uniquely divisive, unfair, and ugly.