Magical realism, in which the beautiful collides with the terrifying and the mundane intersects with the fantastical, is the defining style in Latin American literature. It seems to have a fair hold on the continent’s politics, too. These days, a contentious and at times violent debate is underway from Puerto Vallarta to Puerto Montt over the destiny of a vastly complex, bountiful yet impoverished region of 560 million people. Like a Borges novel, the Latin American electorate resists easy explanation or categorization. Thirteen elections in 2006-2007 failed to answer the simple question: Is Latin America the home of free-market democracy or the bastion of populist autocracy? It is both, of course, and everything in between.

In Forgotten Continent: The Battle for Latin America’s Soul, Michael Reid, the Economist’s editor for the Americas, has amassed an impressive body of statistics, anecdotes, and arguments to help us understand better, if incompletely, the conundrum of Latin America. As a heavy consumer of daily intelligence during my role as a director of the National Security Council in the 1990s, I can appreciate the enormous task the author undertook to piece together nearly two centuries of historical material into a compelling and astute narrative of the roots and prospects of capitalism and self-rule in the Americas. He has done an outstanding job; only a well-seasoned journalist steeped in the drama of the region during these last 25 years could pull off the rich blend of story-telling and rational behind-the-headlines analysis offered here. But despite the wealth of information presented, the book ends up where it began, with more questions than answers about the state of capitalist democracy in a part of the world still largely unknown to outsiders.

But it’s hard to blame him. The reality of the booming Latin America of today is as confounding as the crisis years at the turn of the millennium. What’s different now from the lost years of the 1998–2003 economic downturn, as Reid points out, is a sense of cautious optimism brewing in the hemisphere; according to a new Gallup International poll, Latin Americans were the most optimistic of any regional group in the world about the new year. The same cannot be said, however, about the prospects of improving U.S. relations with the continent, at least while George W. Bush is in the White House. Experts in Washington, quick to lament the harmful missteps of the current administration (especially its first term), are scratching their heads about how the next president should address the real challenges posed by our southern neighbors. But Reid’s history, if unintentionally, makes the course clear: What Washington needs is a raft of new policies that draw on a more dynamic understanding of the hemisphere.

In the early 1990s, the end of the cold war and its proxy battles in Latin America ushered in a new era to the region and its relations with the United States. The habits of democracy were gradually taking root, increasingly free from the damaging, militaristic approach taken by Washington for so many years. As the wars in Central America wound down and General Augusto Pinochet and his ilk were shown the door, the inter-American agenda took on a cooperative tone, exemplified by the 1994 Summit of the Americas in Miami. There, the emerging democracies of the region agreed with Canada and the United States to begin negotiating a Free Trade Area of the Americas; they also created a wide range of consultations and forums to deepen cooperation on judicial issues, defense, education, and energy. Notably, the Americas led the world in fashioning a multilateral response to coups and other threats to democracy, intervening in such places as Paraguay, Peru, Guatemala, and Haiti, mostly to good effect. And, after an acrimonious debate, Washington even moved away from its hated unilateral counter-drug certification process toward a multilateral evaluation mechanism. Compared with the current moment of distrust and outright animosity in some quarters, the Clinton era was a high-water mark for hemispheric relations.

So what happened? Why, after years of relatively good news, has Latin America seemingly turned so strongly against the North? Yes, the economic downturn of the last decade soured Latin Americans on free-market recipes pushed by Washington.

But the answer, in Reid’s eyes, has less to do with changes in U.S. policy per se than with the persistent tension within Latin American history between autocrats and democrats. It’s a long story, as Reid reminds us by devoting several chapters to Latin American political and economic history, from the struggle for independence in 1810, through the rise of the strongman caudillo, to the liberal reformers of the mid-nineteenth and early twentieth centuries and the populist nationalism of the 1920s to 1960s. Hovering throughout is the ever-relevant Simón Bolívar, hero not only to the Spanish colonists but also to the controversial Hugo Chávez, president of Venezuela. For Reid, Bolí­var represents the unity of two archetypes: the caudillo and the modernizing technocrat. An admirer of the United States and Great Britain’s parliamentary monarchy, a reader of Voltaire, Montesquieu, and Locke, and an experienced military leader, Bolívar favored a strong central government, regional integration, and “paternalist authoritarianism” with some modest checks and balances.

In Venezuela, dictators (elected and otherwise) have used and abused Bolívar’s legacy for their own purposes, but none with such great fanfare as Chávez, the current proponent of a “Bolivarian Revolution” for Venezuela and its neighbors. Chávez and his aggressive campaign to revive militarist populism in Latin America, financed by the global surge in oil prices, are what animate Reid (and many others) the most. Forgotten Continent presents a powerful rejection of the Bolivarian cause, not only by reaffirming the varied strands of liberalism woven throughout Latin American history, but also by giving us an upbeat interpretation of the region’s slow but steady progress toward economic growth and democratic consolidation.

Indeed, contrary to Chávez’s attacks on oligarchic capitalism, the news of late is good. The economies of the region have entered their fifth year of expansion, unemployment is down, poverty is declining, inflation has dropped dramatically, exports and trade surpluses are high, and credit ratings have improved. Latin America did not get here by adopting the failed economic policies of Soviet-style socialism or populist nationalism of past military regimes, according to Reid. Rather, it was the result of a painful reckoning with elements of that past by modernizing technocrats in Brazil, Mexico, Bolivia, Chile, and Peru who took the difficult steps needed to get hyperinflation under control, end the debt crisis and discipline fiscal policies.

Now the region is in a much better position to take advantage of the growing global demand for its commodities. But the political hangover of the previous 20-year slump, during which millions of Latin Americans lost their jobs, went hungry, and lost faith in their leaders, is still raging. As seen in recent election results in Ecuador, Bolivia, and Nicaragua as well as in Chávez’s nearly unbroken chain of electoral victories, populist leaders are winning votes by railing against the “neoliberal” policies of the last two decades. Otherwise known as the “Washington Consensus,” the painful economic medicine that many governments adopted to escape the ruin of the 1980s debt crisis has cured one set of ills while creating another. The move, for example, toward privatization of state enterprises, promoted by the International Monetary Fund and its allies in the private sector may have helped bring much–needed cash to government coffers, but it especially has harmed the poor and struggling middle classes who have had to endure quantum leaps in fees for basic services like water and electricity. It has also reinforced long–standing complaints about corruption within Latin America’s notoriously closed elite.

It’s no wonder, then, that some populist politicians are exploiting this hostility toward policies seen (albeit incorrectly so, says Reid) as imposed from the outside. Meanwhile, moderates from the center-left (Chile, Uruguay) and center-right (Mexico, Colombia) are scrambling to increase social services without abandoning the fiscal discipline that has contributed to current government surpluses. Put simply, while the reforms of the 1980s and 1990s laid the groundwork for the economic expansions of the current decade, they also created real pain that revived Latin America’s latent propensity toward demagogic populism. As a result, today both growth and democratic populism are ascendant.

These are, indeed, interesting times for the so-called “forgotten continent”–a pivotal moment, Reid believes, when the path its governments and voters choose now will determine whether the region slips backward to an era of failed populism or forward to a more stable period of free-market democracy. Either way, Latin America will continue to face a rocky road, both economically and politically. Its current positive economic performance is saddled with low productivity, poor education and health services, outdated technology, red tape, inadequate physical infrastructure and weak public institutions. It continues to undertax its citizens and corporations, especially the wealthy, leaving the state too weak and under-resourced to provide the safety net needed to reach the next stage of development. If and when a global recession hits, Latin America’s reliance on commodity exports will certainly make it more difficult to weather the storm.

Things look better on the other main field of Reid’s inquiry, that of democratic consolidation. As Reid explains in a thoughtful chapter on “the stubborn resilience of flawed democracies,” the countries of Latin America are entering a new stage in which their democratic development is qualitatively different than what came before. The pendulum swing between democracy and dictatorship, he asserts, has stopped, as militaries return to their barracks and the revolutionary left takes its campaign to the ballot box; universal, effective suffrage has been achieved, as seen by the growing influence of indigenous, Afro-Latino and women leaders; and decentralization has helped deepen democracy beyond the central capitals. Each of these trends bode well for a future in which good government–one that responds to the needs of the majority while respecting minority rights in accordance with the rule of law–may finally be institutionalized.

There is yet another area of democratization that Reid does not explore (as he admits, for lack of space) but which does deserves special mention: the remarkable progress of Latin judiciaries toward combating impunity for past crimes of human rights and malfeasance. A summary of recent news from the region illustrates the point: Seven former army commanders and an ex-policeman were sentenced to 20 to 25 years in prison for crimes committed during the military dictatorship in Argentina, after legal pardons were annulled in 2003; a retired general and two former sergeants were fined and sentenced to 10 years in prison for killing a leftist couple shortly after the 1973 military coup in Chile; and ex-President of Peru Alberto Fujimori, facing charges of authorizing army death squads to kill leftist sympathizers, was sentenced to six years in prison for sending an aide to steal documents from his spy chief. And these were just one month’s headlines. Even in battle-hardened Colombia, a peace deal with paramilitaries involved in a parallel covert enterprise aimed at helping the government defeat the guerrillas is revealing long-suspected close ties with senior political and military officials. While punishment for their crimes is still to be determined, ex-paramilitaries and their victims in Colombia recently banded together to destroy a huge cache of weapons that was handed over as part of a 2003 peace pact. From the first-ever U.N. truth commission established by the parties to the Salvadoran peace accords in 1992–93 to a steady growth of transitional justice mechanisms throughout the region, a measure of justice is being served in new and surprising ways.

But while there is reason for cautious optimism when it comes to Latin America’s transition to democracy, the region’s democratic governments also face serious challenges. Entrenched poverty and the world’s highest rate of inequality are perhaps the most insidious threats to democratic consolidation. Citizens are losing faith in their leaders’ promises to deliver jobs, health care, roads, and pensions, and they are voting with their feet through street protests or migration. More direct threats include outlandish criminality by drug and other criminal gangs who are overwhelming law enforcement. The inability of many Latin American governments to establish public security is driving a demand for private security services and a greater role for the military in fighting street crime, both dangerous trends that undermine civilian democratic rule. Notably, as traditional political parties weaken, presidents find they cannot get important legislation through fractured, often hostile legislatures. This gridlock tempts presidents to rule by executive decree (e.g., Argentina) or to call for constituent assemblies to rewrite constitutions as a way to break the logjam and consolidate their own hold on power (e.g., Bolivia, Ecuador, and Venezuela). Increasing attacks on journalists, both physical and legal, though not discussed much by Reid, also raise worrying concerns about the ability of Latin American societies to keep their governments in check. In other words, the strongman appeal of Chávez and his ilk is likely to last for a while.

Now it is America’s turn to hold national elections, raising the question of how, if at all, the next administration’s foreign policy will address not only this revival of anti-Americanism but also stable growth and equitable development more broadly. The next president will inherit an enormous deficit in the image of America abroad, especially in Latin America, where opposition to the war in Iraq, U.S. farm subsidies, counter-drug tactics, tacit support for the overthrow of Chávez in 2002, picking sides in internal elections, and immigration policy have stoked long-standing resistance to the dominating influence of the north. Reid, although not in the “blame America for everything” school of so many other observers of Latin America, is right to remind the reader of the pivotal role Washington has played in favoring regime change over indigenous democratic development, damaging U.S. credibility in the region for years. It’s not surprising then that, despite the gains of the 1990s, the iconic ghosts of Che Guevara and Bolí­var will not rest; what seemed to be a new day was really just a temporary loosening of the seemingly eternal tension between pro-U.S., free-market forces and authoritarian hostility to the continent’s northern neighbor. This, then, is the dynamic at the heart of America’s Latin American conundrums, and no U.S. policy toward Latin America will work if it fails to take this into account. Only by improving its image among the average voters–not the bankers or the politicians–of Latin America can the United States regain influence in the region, and profit from its growth. And the region is a critical one, with great potential as a community of stable democratic and prosperous societies, increasingly interlinked with us through trade, immigration, and energy.

What would such a dynamic foreign policy look like? First, the next president must renounce harsh counterterrorist tactics that have not only seriously damaged our ability to protect national security, but severely undermined our image abroad especially with these new, uncertain democracies to whom, in the past, we have preached restraint. As for issues specific to the region, the most important priorities for improving U.S.-Latin American relations–trade and immigration–are some of the toughest to tackle in part because domestic constituencies in the United States are ready and able to block the kinds of changes Latin governments need the most. And yet there are opportunities. A new regional partnership on clean energy, for example, could be a win-win for an energy-anxious north and the next generation of biofuel farmers of the south. The White House will need to mount a serious campaign, however, to reduce subsidies for U.S. farmers in order for Brazil’s biofuels to compete. Getting illegal immigration under control while securing a path to citizenship for the 9.6 million undocumented migrants who are from Latin America is another urgent priority. The odds of negotiating a deal in either area, however, are not good. A more likely outcome of a 2009 agenda, particularly if the Democrats retain control of Congress, would be a sizable increase in development assistance to help democratic governments in the region accelerate improvements in social and economic standards, thereby reducing the flow of migrants.

The low-hanging fruit available to the Clinton Administration for repairing the damage of the cold war is gone. A day of reckoning awaits. Will the next president consider Latin America enough of a priority to make the necessary tradeoffs with entrenched domestic interests and our own foreign policy bureaucracy? This depends in part on the ability of Latinos in the United States to make themselves not only a pivotal bloc in the 2008 elections, but also a more cohesive voice in foreign policy decision-making. Regardless of who wins in 2008, Reid’s cogent and sweeping treatment of Latin America’s place in the world is a must read for anyone, especially those in Washington, interested in unraveling the puzzle of this lonely and enigmatic continent.