J. Edgar Hoover’s bureaucratic longevity as this country’s chief investigator has long been the stuff of Washington legend. Serving from 1924 until his death in 1972, he made himself indispensable to six presidents, who waived retirement rules as they returned his toxic embrace. Only in recent years have the secrets of his extraordinary tenure become widely known. For decades, Hoover orchestrated the FBI’s nationwide data-gathering powers to compile documented accounts of scandalous personal information on members of this country’s political class. Discreetly advising the subjects of these files of what the bureau had found, Hoover might have borrowed a line from Rhett Butler in Gone with the Wind: “Fear not.… Your guilty secret is safe with me!” These messages sufficed, year in and year out, to keep an ever-growing, elite clientele of public figures compliant, the FBI budget safe, and Hoover himself in office—even as the fortunes of other government agencies waxed and waned.

The key to this extraordinary performance was discretion—the ability to take action, or not, without having to answer to any other party. Did Hoover ever make direct threats or demands for a quid pro quo in return for keeping quiet? He probably never had to. The desperation to avoid disclosure of a congressman’s fling with a prostitute, a judge’s shoplifting arrest, or a White House staffer’s homosexual encounter would invariably concentrate the principals’ minds. No one had to explain the likely consequences of crossing an institution sitting on top of such volatile information. The result was a pervasive chilling effect, with every serious player in Washington knowing better than to challenge what the FBI defined as its vital interests.

J. Edgar Hoover’s personal autocracy is long gone. But the forces that he orchestrated still play a vast, if inadequately noted, role in American life. For any public personality, the behind-the-scenes powers of investigative agencies to disclose or withhold critical information—to threaten, to reward, or to defer crucial action—remain formidable. These powers have flourished in the information-rich environment that has burgeoned in the post-Hoover years. In a world where virtually every action leaves its computerized record, the resources of those who harvest and stockpile such data grow apace. Mastery of these capacities makes it possible to shape public events and shift directions of public life. For anyone concerned about the vitality of liberal democracy, these matters should give serious pause.

Examples are not far to seek. This country has just endured a monumentally degrading electoral cycle, marking a new low in the scurrility of campaign communications. Hillary Clinton’s handling of some State Department business through her private e-mail server, and the alleged dangers of revealing or mishandling government secrets, claimed public attention early on. The practices at issue were, at the very least, questionable. But the magnitude of her departure from her predecessors’ standards, and the degree of resulting danger to state interests, were matters on which most Americans had little basis to judge. Hence any signal conveyed at the discretion of FBI Director James Comey—whether to raise public suspicions or to abate them—carried great weight.

Probably we will never know the exact effect on the election’s outcome of Comey’s Hamlet-like reopening of the e-mails issue, followed by his unexplained about-face, any more than the motives behind it. Was he seeking to placate Republicans who might soon have to render judgment on the FBI’s future? Was he attempting to protect the bureau in case the new e-mails did in fact prove relevant to the Clinton investigation? Whatever the answers to such questions, his clumsy interference illustrates the interplay of substance and dramaturgy in shaping the fate of public personalities and events. To paraphrase an early American social scientist: Situations proclaimed real by official investigators are real in their consequences.

The Clinton-Comey episode, however important in itself, matters here as an exemplar of a pattern that has grown so pervasive that its importance easily escapes us. In sequences like these, dramatic disclosures shape the fate of high-profile figures—and in so doing often redirect the course of public events. Supposedly new information comes to light from allegedly authoritative sources to uphold or discredit a key actor on the public stage—cutting short or redirecting a career, casting doubt on what had been seen as sterling character or posing embarrassing questions. Closely observed, such sequences make it apparent that the real actors—perhaps more than the principals—are those with the option to disclose or withhold information, to act or forebear. It is they who stage-manage the dramas that shape crucial reputations, and hence the exercise of power.

The handful of episodes recounted below will be familiar to nearly all readers. My aim in recalling them is not to bring any new information to light—that would hardly be possible for these intensely-reported cases. Instead, I want to call attention to the sweeping role of behind-the-scenes discretion in pivotal episodes like these, and the seemingly arbitrary relations between revealed infractions and the consequences for the parties concerned. In a world where more and more detailed information about nearly everyone becomes available all the time, the powers of those with access to such data grow commensurately.

Within a few weeks of Mrs. Clinton’s upset defeat, Americans learned that President-elect Trump’s short list for secretary of state included none other than David Petraeus. In 2011, General Petraeus had completed a distinguished 37-year Army career, acclaimed particularly by Republican office-holders for the famous “surge” in Iraq that gave cover for the U.S. withdrawal. That summer, President Obama appointed him CIA Director. Many cast him as a future presidential candidate.

But Petraeus stumbled with the public disclosure in November, 2012 of his extramarital affair with his biographer, military writer Paula Broadwell. Details of the timing and discovery of the affair are murky, suggesting that many parties know much more than has been disclosed. The official story is that the Petraeus-Broadwell relationship came to the FBI’s attention through the complaints about anonymous, harassing e-mails from Ms. Broadwell to Jill Kelley—suspected by Broadwell of rivalry for the affections of the former general. The bureau seems initially to have responded to Kelley’s complaints reluctantly and ambivalently, not convinced that any crime had been committed. But prodded by an agent who had assumed the role of Ms. Kelley’s advocate, investigators uncovered inexpertly concealed e-mail evidence of the Petraeus-Broadwell affair—including evidence of the passage of classified information from Mr. Petraeus to Ms. Broadwell.

Confronted by FBI investigators, then-CIA Director Petraeus first denied, then admitted, the illegal transfer of classified information. The FBI, as late as October, 2012, had still not determined to bring charges or to disclose the explosive results of its investigations. At that point, the former general reportedly still expected investigators to keep the scandalous news out of the media, and to keep his grip on his job. It was only when news of the findings was leaked to high Republican officials that someone in the Obama Administration insisted on Petraeus’s resignation.

On November 9, after issuing a high-minded statement acknowledging “extremely poor judgment in engaging in an extramarital affair,” Mr. Petraeus resigned in disgrace. Disgrace, that is, but not too much disgrace. The authorities faced the question of what to do with a top-level wrongdoer whom many regarded as a national hero. The Justice Department could hardly avoid prosecuting Mr. Petraeus for lying to FBI investigators and for transferring secret information to Ms. Broadwell. For such offenses, he could have received both a substantial fine and prison time. Yet the prosecutors chose to charge these offenses as misdemeanors, making a low-ball recommendation of a fine of $40,000 and probation. Judge Davis Keesler upped the fine to $100,000, with two years’ probation and no jail time. Mr. Petraeus’s security clearance, remarkably, remained undisturbed. This may have forestalled a rude end to Mr. Petraeus’s lucrative earning prospects and aspirations to further public office. Since his conviction he has been named a partner at Kohlberg, Kravis, Roberts and Co., a top New York investment firm. He continues to be invited to address audiences on global affairs, and he retains his affiliations with a string of universities, including Harvard. In short, his career has continued to flourish—to the extent of bringing him within hailing distance of being named secretary of state.

Things do not always go this way, however, in the stratospheric reaches of national security politics. Last October 18, retired Marine General and former Vice Chairman of the Joint Chiefs of Staff James E. Cartwright pleaded guilty to felony charges of making false statements to the FBI. The statements in question had to do with conversations with journalists concerning U.S. cyberwarfare against Iran. Once known as “Obama’s favorite general,” Cartwright thus joined the highly dis-favored group of former officials subjected to aggressive prosecution by the Obama Administration for discussing classified information with the press. His sentence was ultimately commuted, but he faced maximum penalties of five years in prison and a $250,000 fine—still markedly less than the 35-year sentence Chelsea Manning received for her disclosures to WikiLeaks, seven of which she’ll have served before President Obama’s commutation takes effect.

These stunning disparities in the authorities’ reactions to similar deeds are nothing new. Consider the cases of two foreign-born, government-employed scientists that dominated national news at the beginning of the new millennium: Wen Ho Lee (born 1939 in Taiwan) and John M. Deutch (born 1938 in Belgium). In 1999, Lee was an obscure nuclear researcher at Los Alamos National Laboratory. Deutch was the immediate former director of the CIA. Both had transported secret materials, to which they had routine access, from their places of work to their home computers. The computer used by Deutch was also found to have accessed pornography sites, a particular security risk. But Lee fit federal investigators’ profiles as a Chinese spy, suspicions apparently based on his Taiwanese origins and his contacts with Chinese scientists.

In December, 1999, prosecutors charged Lee with 59 counts of violating the Federal Atomic Energy Act and the Federal Espionage Act. At their request, he spent nine months in solitary confinement, much of this time in shackles. During his lengthy interrogations, one FBI agent threatened him with execution, citing the cases of Julius and Ethel Rosenberg. Meanwhile, government sources carried out an anonymous campaign of disinformation against Lee. Major national media amplified this propaganda, with The New York Times at one point suggesting that Lee “may be responsible for the most damaging espionage of the post-cold war era.” Asian Americans and others accused the authorities of racism. But, in September of 2000, the government’s overweening case collapsed for lack of evidence. Lee pled guilty to one felony count of mishandling of government data. U.S. District Court Judge James Parker sentenced Lee to one day less than the days he had spent in solitary confinement, denouncing the prosecutors and apologizing to Lee for the “unfair manner in which you were held.” Lee won a $1.6 million settlement from the government and five major news organizations, including ABC News and The New York Times. He now lives quietly in retirement in New Mexico.

Soon after the verdict, President Bill Clinton pronounced himself “quite troubled” by the prosecution.

Deutch, so recently at the pinnacle of the military-intelligence establishment, obviously presented a different problem for prosecution. His guilt in removing classified materials to his own computer was apparent, even flagrant, but, as with David Petraeus, the consequences were muted. Deutch was preparing to plead guilty to misdemeanor charges of keeping classified information on his home computer, but was then pardoned in one of President Clinton’s last official actions. Deutch’s security clearance was withdrawn. But he remains Institute Professor at MIT and a member of the Trilateral Commission; he continues to sit on the boards of major corporations.

Deutch’s soft landing also strikes a sharp contrast to that of former House Speaker Dennis Hastert, known in Congress as “Coach,” for his early career coaching high school wrestling. Hastert assumed the role of Speaker in 1998, ultimately serving until 2007. During this time, he could hardly have been more central to any Washington power diagram. On his retirement from the House, he predictably charted course as a lobbyist, growing wealthy by exploiting a career’s harvest of contacts.

Hastert’s case warrants special attention here precisely because it involved no national security issues, but massive exercise of discretion coupled with intrusive, technologically abetted monitoring of personal information. It demonstrates that even the highest public standing does not always afford protection from harsh consequences, when the authorities are convinced that the latter are justified.

In 2010, Hastert attracted attention from government watchers by making large withdrawals from his private bank account, including fifteen of $50,000 apiece.

Bank officials apparently warned the former speaker of federal requirements to report withdrawals of more than $10,000, after which he continued to make withdrawals of just under that figure. Questioned by the FBI, Hastert also lied at first, citing his distrust of banks. In fact, the withdrawals were for hush money, to prevent disclosure of Hastert’s sexual advances more than 30 years before to one of the boys he was coaching. In time, more former pupils came forward with similar allegations—none of which could be prosecuted because of the statute of limitations. But the government proceeded to prosecute the former speaker for “illegally structuring” withdrawals from his own accounts; he pled guilty in October of 2015. Excoriated by the judge as a “serial child molester,” he is now serving 15 months in federal prison.

So why did Hastert feel the full force of federal prosecution, while Deutch and Petraeus got off nearly unscathed? It’s hard to escape the conclusion that, unlike Petraeus and Deutch, the former speaker simply had nothing left to offer Washington’s movers and shakers that would warrant their putting the brakes on his prosecution.

More to the point: What interests drive federal investigators’ willingness to open inquiries like these in the first place? At least one observer of Petraeus and Broadwell—while Petraeus was still commanding American forces in Afghanistan—noted that the two seemed to make little effort to conceal their more-than-merely-professional relationship. What changed to invalidate that insouciant attitude, and when?

And who promoted the idea that digging into former Speaker Hastert’s outsized withdrawals from his own accounts warranted a full-scale FBI investigation? There was, after all, nothing illegal about the largest, early withdrawals: Such transactions must occur constantly, throughout the country. Hastert ultimately confessed to charges of adjusting the later withdrawals so as to evade federal reporting requirements.

This is a strange and disquieting offense. Guilty or innocent, nearly everyone prefers not to have to account for the fine detail of their lives to government investigators. As Conor Friedersdorf commented in The Atlantic, “If the state can decide that specific, legal behavior will trigger scrutiny…and that any attempt to avoid that scrutiny is illegal…everyone’s privacy and freedom from unjust arrest is undermined.” Hastert was convicted not of sexual abuse, but of evading federal surveillance; why the watchers exercised their discretion to focus on his particular case remains unclear.

The spectacle of grossly unequal consequences for similar wrongdoings is itself corrosive to any democracy. But my deeper concern here has to do with the second- and third-order consequences of such practices. Walter Lippmann was profoundly right: Life in a liberal society presumes some congruence between “the world outside” and “the pictures in our heads.” When ordinary citizens—most of us, most of the time—can never be sure of the true reasons for critical public events, the accountability of the system as a whole is degraded. When it is impossible to know whether discrediting disclosures about public officials represent real news, for example, or simply a change in attitude among those controlling the flow of public information, something crucial is lost.

We do not know who ultimately made the decision, in the autumn of 2012, to publicize former General Petraeus’s extramarital exploits, triggering his resignation as CIA director. Nor do we know how it was determined that Mr. Petraeus, following his guilty plea for disclosing government secrets, would be allowed to retain his security clearance—whereas his predecessor from two decades before, and, for that matter, his paramour Ms. Broadwell, both lost theirs. We do not know whose deliberations led to charging Mr. Petraeus with a misdemeanor rather than a felony. We do not know what triggered the vastly overreaching prosecution of Wen Ho Lee as a spy. We do not understand why the authorities found it reasonable to launch an investigation into Dennis Hastert’s withdrawals from his own accounts. And we certainly do not know why FBI Director James Comey staged his much-decried reopening of the controversies surrounding Mrs. Clilnton’s private e-mail use while secretary of state.

All these blanks in our mental maps reflect a bedrock reality of the world we have come to inhabit—the vast disparities in the information available to those at the top, versus the great public. Those who greeted the rise of the Internet with the slogan “Information wants to be free” may have caught the spirit of a historical moment. But the reality, now that the new age is upon is, is quite the opposite—particularly as regards personal information.

Like J. Edgar Hoover in the bad old days, twenty-first century organizations that compile vast data-systems on people do not go to these lengths without expecting concrete gains for their trouble. From the NSA to the corporate giants of consumer credit reporting, their uses of personal information are highly strategic. Accordingly, we can expect the FBI, the NSA, and other key players in Washington to keep their cards close to their institutional chests regarding what they know, what they will disclose and withhold, and why—at least, until there is action to stop them.

And let us never forget that we all continue to generate more and more such information, willingly or not, simply by living even the most ordinary of existences. From our online purchases and financial operations to our telecommunications, personal movements, and even our interactions with household appliances, we are constantly documenting our own lives, or having them documented for us. The resulting information-rich world affords investigative agencies sweeping opportunities to reach quickly and deeply into the lives of the governed. The accounts above remind us how consequential such surveillance can be.

Fueling the ascendance of surveillance interests, of course, is the preoccupation with terrorism that has engulfed the nation since 9/11. This sense of vulnerability has opened the way for the extraordinary agenda adopted by the NSA—that of watching all Americans, all the time. These efforts at monitoring the entire American population now represent a considerable sector of the economy. The NSA employs an estimated 30 to 40 thousand Americans; its (secret) annual budget was recently estimated at more than $10 billion—greater than the gross national products of some 50 countries. And the NSA is just one part of the national security complex. Roughly four million Americans, about 2.5 percent of our total labor force, have jobs requiring security clearances. Many of these workers, though certainly not all, are engaged in tracking the lives of their fellow Americans.

But are these vast personal-data-dredging operations really cost-effective in protecting us? The NSA struggles even to police its own people. The ease with which Edward Snowden downloaded something close to a complete roadmap of the agency’s surveillance activities, then managed to leave the country before being detected, was supposed to have shaken up internal security within the NSA and its contractors. But only last August, the FBI arrested one Harold Thomas Martin III, a NSA contractor at Fort Meade, Maryland. Investigators believe that Martin brought to his home considerably more secret data than taken by Snowden and that he did so in installments, over a period of nearly 20 years. Apparently Martin moved the data, in the form of computer discs and drives, through NSA security as he left work. It is still not clear whether Martin accomplished this stunning appropriation of highly sensitive data for nefarious purposes, or was simply responding to some idiosyncratic hoarding impulse.

Seeking to excuse the ineffective controls in these situations, former CIA and NSA Director Michael Hayden provided a flight of unconscious irony: “You don’t want to create a Stasi-like atmosphere,” he remarked to The New York Times.

Years of concerted efforts by Hayden and his colleagues have indeed moved the entire country in the direction of East Germany during the days of the Stasi. Americans today cannot be sure what government agencies may be monitoring the multiple flows of data that we participate in every day, or to what effect. To be sure, we retain legal and institutional means of resistance unavailable to the former East Germans. As Wen Ho Lee’s case demonstrates, those targeted by government investigators still have resources to defend themselves and, sometimes, win. But the direction of change is disquieting.

What we know for sure, though, is that discretion-driven command of government surveillance powers has chilled or blunted vital elements of the democratic process. Even those whose lives are above reproach can hardly remain indifferent to the possibility of having all their personal data reviewed by unfriendly federal investigators. The prospect of having one’s life turned upside down, and being required to justify every element of one’s “record,” is enough to breed caution in almost anyone. In the telling phrase of legal scholar Malcolm Feeley, “the process is the punishment” in cases like these.

When people come to view major public events as shadow plays staged by play masters working out of sight, democracy loses. When every public figure has to worry about the potentially unfriendly uses of information resources offering comprehensive views of his or her personal life, free expression and open debate are threatened. Progressives need to resist these trends by insisting on more rigorous accountability of public pronouncements by data-keeping organizations; harsh skepticism of appeals to supposedly informed, anonymous sources; and penetrating questions on the role of discretion in official action.

The founder of the FBI has left us, but his organization still has its headquarters in the J. Edgar Hoover Building. It is time the use of citizens’ information in places like this became considerably more transparent.