A Congressional report explores how AI can compromise personal data, and reveals that federal agencies fall short in addressing privacy-related challenges.
In March 2026, the Government Accountability Office (GAO) released a report exploring the specific ways artificial intelligence (AI) can expose, misuse, or compromise personal data. The report from the Congressional watchdog also examines the guidance issued to federal agencies on AI use and implementation by the Office of Management and Budget (OMB). In its note to congressional offices, GAO concluded that OMB’s guidance, which fails to adequately address 8 out of 10 privacy-related challenges, falls short. The report recommends that the Russell Vought-led office address the identified shortcomings.
The GAO’s concerns are not merely hypothetical in nature, with numerous recent real-world examples that directly map onto the risks specified in the report. It’s a serious warning acknowledging the issues that arise when lousy safety protocols govern AI use, one we should all take seriously considering the Trump administration’s feverish adoption of AI tools across federal agencies, many of which are privy to sensitive personal information.
Here is a quick rundown of some of the major areas of concern:
Invasion of Privacy From Data Aggregation: “AI may combine various pieces of data about a person to make inferences beyond what is explicitly captured in those data (e.g., social scoring), which can invade an individuals’ personal space and solitude by revealing private information (e.g., health-related, financial, location).”
- Similar systems have been implemented in other governments, but were struck down for privacy violations. The Dutch government, for example, had been using a fraud detection system that combined personal data from different sources to detect social security fraud and also predicted the likelihood that someone would commit fraud in the future. The system was exclusively used to monitor poor communities with large percentages on non-Western immigrants. However, a Dutch court decided the system was unlawful because it did not comply with privacy rights under the European Convention of Human Rights. The court also found the system collected too much data and the reasons for collecting the data were not specific enough.
Lack of transparency related to data use: “AI may be used without providing individuals with notice and control over how their data is being used.”
Lack of transparency in AI model algorithmic decision-making: “The workings of AI tools could include decisions based on individual data that one is unaware of.”
The GAO report identified transparency failures in AI data use and decision-making. Idaho managed both at once:
- In 2016, a federal district court struck down an automated budgeting system Idaho had been using to determine benefits for Medicaid recipients. Idaho did not disclose that the system was being used, and when budgets were cut, recipients were left blind sided and received no explanation. Budgets were cut up to 35%, resulting in the loss of tens of thousands of dollars in benefits. When finally compelled to offer an explanation on how the system worked, Idaho’s Department of Health and Welfare officials claimed the automated systems algorithm was a “trade secret” before the court forced the state to turn over its formula to the plaintiff.
Generation of deceptive or inaccurate outputs: “AI may be used to intentionally or unintentionally generate deceptive outputs (e.g., deepfakes) or inaccurate outputs (e.g., hallucinations) that may result in harm towards individuals.”
- In October 2025, the Tech Transparency Project found that 63 scam advertisers spent $49 million to run more than 150,000 deepfake political ads on Facebook and Instagram. The ads featured deepfake video of political figures such as President Trump, Elon Musk, and Bernie Sanders to promote fictitious government benefits specifically targeted at seniors. The ads directed seniors to webpages warning that they need to enroll for benefits soon and collected personal information like names, email addresses, and credit card numbers.
How will AI adoption impact the federal government?
Outside the risks and challenges identified in the GAO report, the private sector offers examples of the negative consequences of using AI without properly evaluating how to integrate it into existing systems. PocketOS, a small software company that builds management tools for car rental companies, learned the hard way when one of the AI agents assisting with software development went rogue and deleted a live customer database, plus the backup data, despite being deployed in a test environment cordoned-off from the company’s live systems. The whole process took nine seconds. When asked to explain itself, the AI agent responded, “’I violated every principle I was given. I guessed instead of verifying. I ran a destructive action without being asked. I didn’t understand what I was doing before doing it.’” That’s a scary thought: the AI itself didn’t understand what it was doing, suggesting this technology may be more outside our control than we realize.
It seems that PocketOS had covered all its bases, yet the AI agent found a way around the parameters set for it. It’s a cautionary tale for businesses rapidly adopting AI and plugging it into systems before reliable guardrails are built. PocketOS is not the first to suffer this fate, nor is it likely to be the last, as AI agents continue to act in unexpected ways.
The upshot of similar AI mishaps occurring inside the federal government will be exponentially more catastrophic than not being able to rent a car. Sensitive information under the Trump administration has already been exposed and misused; introducing AI into the fold without proper safety measures will exacerbate that trend, potentially spilling Social security data, medical records, and personal financial information out into a world that is becoming increasingly capable of exploiting it. As the operation of physical infrastructure and military operations become more digitized, even small AI mistakes will have outsized consequences.
Is the government prepared for the risks of AI?
The GAO report makes it clear that the government has not readied itself to address the risks of introducing AI into the vast federal bureaucracy. The Cybersecurity and Infrastructure Security Agency (CISA), the federal government’s main bulwark against cyberattacks, has been weakened to the point of offering no meaningful defense against the cyberthreats now armed with a widening arsenal of increasingly powerful AI tools.
Yet the signals from the White House indicate that light regulation and widespread adoption of AI are the priority. On June 2, 2026, Trump signed a scaled back version of the executive order he planned to issue two weeks prior after succumbing to industry pressure. The final version established a framework to vet the national security risk of advanced AI models but leaves much to be desired: participation by AI firms is voluntary, and the pre-deployment window has been shortened from 90 days to just 30.
What’s more, the President’s Council of Advisors on Science and Technology (PCAST), Trump’s sounding board for these issues, is full of tech executives and billionaires with a vested interest in pushing AI into all corners of government. Factions within the administration are emerging over AI safety, with usual Trump ally Scott Bessent alarmed over the slow pace of progress on federal policy guidance as the cyber threat landscape becomes more dangerous. Whatever AI fiascos happen in the Trump administration, they will be the result of deliberate policy choices and failures to address the rising concerns of bringing AI into the federal government.
Photo: President Donald J. Trump holds a cabinet meeting in the Cabinet Room Wednesday, May 27, 2026. (Official White House Photo by Daniel Torok)