This is from a NYT profile of Baker, published in 2001:

And yet, as Mr. Baker sees it, in some ways he has more in common with the black inmates in America’s jails than he does with his white colleagues down the hall. In his new book, “Turning South Again: Rethinking Modernism/Rereading Booker T.” (Duke University Press), he elaborates his view that to be a black American — no matter how successful or well off — amounts to a kind of prison sentence.
From slavery through segregation, he points out, blacks were physically confined, first held against their will on plantations and then relegated to black communities. Today, he argues, the confinement of blacks continues. Only now, instead of slave quarters and the backs of buses, they inhabit jails and public housing projects.
Or else, he writes, more insidiously, blacks are psychologically constrained, trapped in anxiety-ridden relationships with white culture. “American ‘history,’ ” he concludes, “thus reads out, in black-majority vocabularies, as enslavement, incarceration, imprisonment.”
“It’s not that white academics don’t work extraordinarily hard,” said Mr. Baker, impeccably turned out in a gray wool suit and tie. “But what they have that I lack is a sense of leisure, an absence of endangerment, a look of being unconcerned that at any moment they could die.”

From his tenured post at Duke, of course, Baker has no leisure, is constantly endangered, and must always fear for his very life.