Barack Obama’s election to the presidency highlights a profound paradox at the heart of American race relations. After centuries of exclusion, black Americans have been almost wholly accepted into the public sphere of American life, and they are central to the nation’s definition of itself as a political and social community. Obama’s presidency is the culmination of this amazing national transformation. Today we find racial equality in its political, civic, and cultural forms at a level that far exceeds any other advanced society, or even any of the large plural societies of the developing world. The National Urban League, one of the nation’s preeminent African American interest groups, has documented this achievement in its annual State of Black America’s Equality Index, a composite “measure of the relative status between blacks and whites in America.” The one bright spot is in the area of “Civic Engagement,” which it claims “is the only area in which African Americans exceed whites.”

Yet, at the same time, black Americans remain remarkably excluded from most regions of the nation’s private sphere: They are now more segregated than ever, have astonishingly few intimate friendships with non-blacks, and are the most endogamous group in the nation. The Equality Index shows either stagnation or decline in economic, educational, health, and social justice measurements comparing blacks and whites. This apartness prevents America from achieving true equality, and it has worsened even as blacks’ public integration has progressed apace, a contradiction magnified by the policies of the Bush years. Will things change with Obama’s election?

African Americans have always been critical to any assessment of American equality. As Gunnar Myrdal observed 64 years ago, the condition of blacks posed a terrible dilemma for the nation’s founding creed, namely that America was a land of equality and respect for individual rights. Myrdal would have been gratified by the extraordinary progress America has made in living up to its creed in the public sphere. But he would have been dismayed by the persistence of segregation and apartness in private life. Separate, Myrdal strongly believed, always meant unequal, a belief that undergirded the civil rights movement and the great legislative and legal progress of the 1950s and ‘60s. And it would have shocked and puzzled Myrdal to learn that many black Americans have now embraced the notion that separation and equality are compatible goals.

The paradox began with the founding of the nation: the fact that a government based on the principle of equality nonetheless held more than a sixth of the nation in slavery. The abolition of slavery, however, merely liberated individual slaves from individual masters. It did not topple the culture of slavery and the attitudes held toward blacks as quintessential outsiders. Jim Crow in the South institutionalized the condition of black Americans as a group apart, excluding them from both the public and private spheres of the dominant white society and polity. Even in the North, where Jim Crow laws were largely absent, they were truly invisible, of little or no consequence in public life and culture, and nearly totally segregated from the neighborhoods and daily personal lives of white Americans. To the degree that they served a national function–beyond being exploited in the most demeaning of work–it was to define what constituted whiteness. The positional good that became America’s most precious cultural commodity was largely a negation defined in terms of what one was not: black.

The civil rights movement marked the triumph of early-twentieth-century efforts to overcome black exclusion in the public sphere. And it was, by any reckoning, a revolution. Within less than a generation, the entire institutional infrastructure of Jim Crow was dismantled. Blacks achieved legal equality and soon became a constituent element of the nation’s political life. The rise of a genuine black middle class, including their acceptance in the chief executive roles of some of the nation’s mightiest corporations, is perhaps the most important aspect of this public integration, facilitating other areas of public inclusion, such as the military. This public inclusion is also dramatically reflected in the nation’s popular and elite culture, where the black presence is pronounced and, in some areas, dominant.

However, a closer look at the black middle class reveals problems that point to the other side of our paradox: the persisting apartness of blacks from the nation’s private life. There was substantial absolute growth in black median household income between the 1960s and mid-1990s. But when we closely compare different quintile segments of the black and white income spectrum even over this period, we find only very modest relative improvement in the middle quintiles, no change in the upper quintile and an actual worsening of the relative position of the poorest black group.

This has worsened still during the Bush years, as the already modest narrowing of the racial income gap halted, accompanied by an absolute decline in black income. Thus between 2003 and 2005 white income inched up by only $130 (to $49,554) while that of blacks actually declined by $552, to $30,945. The poverty rate for blacks, having declined during the Clinton years from 32.9 percent in 1992 to 21.2 percent in 2000 (the lowest on record), climbed to 24.5 percent in 2007. During this period the non-Hispanic white rate, in contrast, rose by only 0.4 percent.

The socio-economic fragility of the black middle class and its declining fortunes during the Bush years is further exemplified by two other factors. One is their low net median worth, both in absolute terms and in comparison with whites, a direct result of their centuries of exclusion from the more prosperous levels of the national economy and, in particular, from opportunities for home ownership. In 2000, the median net worth of non-Hispanic whites was $79,400, or 10.5 times that of black householders, estimated at $7,500. This enormous gap began to widen almost immediately after Bush’s election. As of 2002, the median net worth of non-Hispanic whites was $88,000, which was 14.5 times greater than that of African-American householders, which declined to $6,000. The recent housing crisis has made this situation even worse. Recent work indicates that African Americans were between 6 and 34 percent more likely to receive a higher-rate, subprime loan than whites with similar financial profiles, and some estimates depict a net foreclosure loss of 9 percent of all homes purchased or refinanced by blacks in 2005 alone.

This fragile economic base, along with persisting, though declining, labor- market discrimination, residential and marital isolation, and a reversal in educational attainment and high school graduation rates (especially among males), as well as internal cultural problems, account for a second stunning recent development. In 2007, a joint Pew Foundation/Brookings Institution study found that there has recently been massive inter-generational downward mobility from the black middle class. The study reports that ” a majority of black children of middle-income parents fall below their parents in income and economic status.” This turns out to be not an incremental decline: Fully half of all black middle-class children fall to the very bottom of the income ladder.

The paradox of public inclusion and increasing private separation is glaringly revealed in the persistence of large-scale racial residential segregation in America. Analysis of recent Census data by the demographers William Frey and Dowell Myers indicate a black-white dissimilarity index of 58.7–meaning that almost 60 percent of blacks would have to move to achieve a distribution across neighborhoods that reflected their actual proportion of the population–for all metropolitan areas, compared with 42.9 for Asians and 42.2 for Hispanic ethnic groups. Segregation levels vary by region in quite unexpected ways, compounding the paradox. Strangely, they are highest in precisely those regions of the country where blacks have achieved the highest levels of public integration: the liberal metropolitan regions of the Northeast and Midwest. New York City, where blacks have arguably achieved the strongest presence and influence in public life–politically and culturally, in the media, spectator sports, and in leadership by exceptional black executives of several of the city’s, and the world’s, greatest corporations–ranks near the bottom of the nation’s metropolitan areas in levels of residential integration. Chicago, which gave the nation its first black president, remains horribly hyper-segregated as well.

It is important to note that economic differences do not explain this pattern of separation. The black middle class is almost as segregated from white neighborhoods as poor blacks living in the inner cities. Similarly, although inter-marriage rates have been increasing, blacks remain highly endogamous, with over 90 percent of marriages within the group. This is in sharp contrast with other non-whites. Asians and Hispanic out-marriage rates are three times that of blacks: About three in ten of all Asian and Hispanic marriages involve a spouse from another race, usually white, compared with one in 10 black marriages. These differences are even greater among young married couples. UC-Irvine sociologist Jennifer Lee has argued that a new racial divide may be emerging in America: the old black-white division which emphasized whether one is white or not, is being replaced by a new split with the emphasis on whether one is black or not.

Nowhere is the paradox of public inclusion and private separation more stark than in the present situation of black American youth, especially poor males. On the one hand, they are the most segregated segment of the nation’s population and face the most formidable set of socioeconomic problems: drop-out rates of over 40 percent, chronic violence and homicide mortality rates, and staggering rates of incarceration resulting in one in three male African-Americans in their 30s having a prison record. The non-incarcerated face unemployment rates of over 20 percent among those still seeking work. This social catastrophe has grown significantly worse under the Bush Administration’s domestic agenda, with its emphasis on punitive measures and neglect of rehabilitative programs for youth who get into trouble. And yet this same group is the creator of a powerfully vibrant popular culture, one that has had an outsized impact on the broader culture of America and is today perhaps the most potent force in the globalization of American popular influence. White American youth avidly consume the hip-hop cultural productions of black ghetto youth–their music, fashion, poetry, dance, lifestyle, and language–and whites of all classes and age groups idolize their sports heroes.

How do we explain this incredible state of racial affairs? To the degree that black social isolation is influenced by economic inequality, the Bush presidency’s egregious transfer of wealth to the wealthiest Americans has intensified the contradiction. Nonetheless, it has been long in the making. When the civil rights movement began, its goal was clearly aimed at both public and private incorporation. By the late ‘60s and early ‘70s, however, all this was to change, directed as much by black leadership as whites. The white backlash against the early integrationist vision and achievements of the great society program of the ‘60s was channeled by the perverse political genius of Richard Nixon into the wedge policy and Southern political strategy that underpinned the Republican ascendancy that has lasted until this year, when it finally collapsed. One of the great ironies of modern American political history, as sociologist John Skrentny has shown, is the fact that Nixon quietly supported affirmative action and other policies that effectively promoted the black middle class and black public incorporation. At the same time, in what is surely the most wickedly Machiavellian move in American politics, he used the white resentment that these very policies incited to break the coalition of Southern and Northern Democrats that had dominated American politics since the Roosevelt years, weaken unions, and alienate northern white males from the party, a process completed by Ronald Reagan.

Partly independently and partly in reaction to this development, black leadership aided this process in its turn toward identity politics and the rejection of the integrationist vision of the early civil rights movement. The harsh reaction against school integration led many blacks to ask if it was worth the hassle, especially when educational gains from integration were shown to be modest. A more forceful reaction, prompted by black pride and the self-interest of its leadership, led others to ask why blacks needed to sit next to resentful whites in order to learn in the first place. Eventually, a misguided consensus emerged that black control of equally funded schools was preferable to integration, particularly as the emerging black middle class found that it allowed them occupational and income advantages in school districts like Washington, D.C. and Maryland’s Prince George’s County.

To hold that black middle-class residential segregation and persisting endogamy resulted, in part, from racial preference is a controversial position. The “proper” sociological view is that middle-class blacks live in bourgeois ghettos because they are not welcome in white communities and the few who courageously try to live in them induce hostility and tipping-point style white flight. But such conventional wisdom tells only part of the story. The basic problem resides in the very different conceptions held by blacks and whites of what constitute an ideal racial mix. Repeated surveys indicate that whites are comfortable with a racial mix of up to about 75 percent white and 25 percent blacks. Blacks on the other hand, seem most comfortable with a mix of at least 40 percent blacks and 60 percent whites. This will never work. According to tipping point theory, even if the typical white resident is truly comfortable with a 25 percent mix of blacks, white communities will start tipping long before this ratio is achieved due to (mis)expectations about how other whites are likely to behave. This is an elegant mathematical theory, but I have grave doubts about its empirical validity, as do other empirical social scientists. As a result, American race relations in this opening decade of the twenty-first century have stalled, paradoxically, at the half-way point toward full incorporation and equality, and the reasons are sufficiently complex as to prevent easy identification of the next step forward.

Obama’s election is the d‘nouement of America’s great achievement in the public incorporation of its black population, and many whites will no doubt point to it as proof of the end of racial discrimination and inequality in America. Indeed, opponents of affirmative action were already citing Obama’s presidential nomination by the Democratic party as an unassailable reason for its dismantlement.

Black Americans will angrily, and rightly, reject such claims. Persisting racism undoubtedly explains, in part, private exclusion and its consequences. But only partly. An outdated identity politics, abetted by leaders of the grievance industry, has steered black Americans away from what every other previously disadvantaged minority and class has concluded, as well as the leaders of the early phase of the civil rights movement: Namely, that the path to full equality ultimately entails private incorporation into mainstream American society, especially through residential integration, exogamy, and the forging of both weak and strong network ties with the majority. Separation of any and all kinds results in exclusion from critical avenues of social capital–education, community status, and so on–as well as the informal cultural capital that are the real sources of competence and success in America. These vital skills, as French social theorist Pierre Bourdieu showed definitively, are not learned at school. To the contrary, they are major determinants of school success.

Is there a way forward? Without going too far into hero-worship, part of the answer may lie in the president-elect. There is no better role model of the advantages of private inclusion and integration than Barack Obama himself. His success is not, as is often superficially claimed by others, due to his transcendence of race. His life and achievement, rather, is the perfect exemplification of the value of cultural integration. White Americans, including many with persisting racial biases, could look beyond race in judging him precisely because they also saw in him someone who had achieved mastery in both the formal and informal areas of American culture. And he was able to do so in good part because of his socialization into the private domains of both white and black private life: by his white mother and grandparents and by his black wife, friends, co-workers and co-religionists.

Beyond the promised return to more egalitarian social and economic policies, which inevitably benefits black Americans, Obama can further enhance the condition of black Americans, and immeasurably improve race relations in his country, by encouraging blacks and whites to learn from the example of his own life and success. He should, and undoubtedly will, use the bully pulpit of the presidency to encourage the embrace of mainstream cultural practices (in part the creation of blacks, let it not be forgotten) that too large a minority of blacks, especially poor black youth, presently find either too challenging or (mistakenly) too “white.” And, hopefully, he will remain faithful to his own upbringing by actively returning to the early civil rights ideal of integration. Among the ways he can do so are by heavily promoting educational diversity through integrated, high-quality charter and magnet schools; encouraging mixed race adoptions and marriages (of which he is a product); and enhancing residential diversity through the institutionalization of already existing experimental initiatives like the Movement to Opportunity program, which provides social training and housing vouchers to minority householders on condition that they move out of inner-city ghettos. With the Obama presidency, America may, at last, begin to fulfill Martin Luther King, Jr.’s vision of an integrated, “beloved community” whose “ultimate goal is genuine intergroup and interpersonal living.”