Back at my Ivy League university in the late 1990s, a well heeled alumnus and his two children sat next to me one day at the local coffee shop. They were white, solidly upper middle class, Dad wore a navy blazer and his kids – two teenage boys – wore slacks with a shirt and tie.

Dad, in his early 50's, was busy painting a picture of the campus life as he had known it in the late 60's. Vietnam, Martin Luther King, the counter culture, long hair, sandals, arm bands, protests, all that stuff.

He hadn't had time for any of it, he said. He stayed away from the hippies. They had agitated for dramatic reforms for years he said, but they had been mostly unsuccessful.

Now, it was the tail end of the Clinton boom and Wall Street was triumphant, economic growth was historic, unemployment was at its lowest level in 30 years. He was speaking with an authority that suggested he was partly responsible for it all. He had backed the right horse.

“Of course, one of the very worst things about the 60s was affirmative action,” he told his sons, who sat wordlessly, awed by the venerable college campus. “That was always a very bad idea. Instead of appointing people based on merit, the country started filling racial and gender quotas. We need some women here, we need some blacks” – he said blacks – “there.” Then he paused for dramatic effect. “That meant that you had suddenly had people, lots of people, wholly unqualified for their jobs in positions of trust.”

Affirmative action was a transparently flawed premise, he said, toying with his gold cufflinks. Any system that gives unqualified people career breaks that they did nothing to earn should be vigorously opposed, he said.

All the while he spoke in loud and commanding tones, as though he expected to have an audience who would listen and agree with him. Had I applauded I am certain he would not have minded.

It emerged in his conversation that both his father and his grandfather had attended the college. He had just given his eldest son the grand tour, including a brief meeting with the college dean, safe in the knowledge that he could now find his way around when the new term commenced in the fall.

But that day for some reason it was the evil legacy of affirmative action that was most on his mind. “William F. Buckley wrote all about it,” he added, “I read a column by him in the National Review in 1969 that made a great impression on me in college. An authoritative study by Arthur Jensen found that blacks are generally less intelligent than whites and Asians. Do you know what that means?”

I sat in stunned silence like the man's two sons. “That means that we're doing them no favors, we're doing us no favors, pretending that things are other than what they are.”

Now I could tell you that I stood up at that point and confronted him, appalled by his odious speech, but I was still reeling from hearing my first ever example of unvarnished racism. I felt like the room had started to spin, but I was still new to life in the United States and like many a callow immigrant, I was still unsure of my social footing. I said nothing.

Many times since I wish I would have told him – and his two teenage sons - that his whole life was an example of affirmative action. But he was so complacent, so certain of his place at the top of the evolution chain, that I am sure I would have wasted my breath: handouts and hands-up were for undeserving minorities, not for people like himself who lived in eight bedroomed houses with two sports utility vehicles and a 100 foot yacht, and who had worked hard for their prosperity with help from no one okay?

Men like that almost comically entitled alumnus believe that their riches are simply an expression of their personal value. This is nonsense, of course, but it feels like the truth. The real truth is that no one anywhere – not even cosseted billionaires – ever make it on their own.

Every American city, even that absurdly rich college town, has a sharp divide between the haves and have-nots. Some will get golden handshakes and many more will get the heave-ho. But I was stunned by the scale of the contrast between rich and poor at my university. Within a block on some streets staggering wealth and privilege gave way instantly to decades of dispossession and despair. It turned out to be the perfect introduction to the 21st century here.

Racism, I have learned, is often an expression of unconscious anxiety. There's a lot more to unpack than that of course, but there's no question it's a motivating factor. If you live in a system that strongly favors you, you have no need to consciously concern yourself with the fate of those it does not.

Powerful figureheads will emerge to reassure you that you are right to look away. They'll even do the rough work at night to ensure you can. You need not worry at all.

Until your cities are burning or you roads are blocked, that is.

Last week in Baltimore, the streets were burning. In 2015, there are 15 neighborhoods in Baltimore with lower life expectancies than North Korea, 8 of them are currently worse than Syria. If you're born in the wealthy part of the city you can expect to live 19 years longer than those born in the poorest, who – statistics show – often won't live to draw their own social security.

I still sometimes think of that father handing on a toxic legacy to his all too credulous sons. Perhaps they learned what he refused to, but I doubt it. Why do anything to disrupt a system that places you near the top? It would upend most of history.

No one wrote about racism and its complications in America better than James Baldwin. In a now famous debate with William F. Buckley at Cambridge in the 1960's he uttered the following unforgettable words:

“It comes as a great shock around the age of 5, 6, or 7 to discover that the flag to which you have pledged allegiance, along with everybody else, has not pledged allegiance to you. It comes as a great shock to see Gary Cooper killing off the Indians, and although you are rooting for Gary Cooper, that the Indians are you.”

Conservatives scorn the idea of ‘white privilege,’ which they say does not exist, whilst simultaneously calling for the end of affirmative action, which they say should not exist.

It seems incredible that some people would want to undo the few attempts that are made to redress these longstanding racial inequalities, when you consider that the only thing standing between us and that sad fate of the Indians that Baldwin spoke of is the luck to be born in the right zip code.