In July 2026, a security researcher pointed a wire-tap at xAI’s Grok Build coding tool. The finding was specific. The model needed 192 kilobytes to answer the coding question. The tool uploaded 5.10 gigabytes of the developer’s full codebase to xAI’s cloud storage anyway. That’s a 27,800-times gap between what the task required and what left the machine. Disabling the tool’s “Improve the model” toggle didn’t stop it. This isn’t a minor bug in a coding assistant. It’s a clean, dated case study in how AI data leakage actually happens. In fact, it shows why the fix was never going to be a better privacy setting. The only reliable protection is data that never leaves your hands in the first place. That distinction matters far more to a defense manufacturer than to an individual developer.
What Actually Happened: The Anatomy of an AI Data Leakage Incident
A researcher publishing as cereblab ran Grok Build through a network-interception proxy and watched exactly what left the machine. The results were unambiguous. Grok Build packaged an entire tracked Git repository, every committed file plus full commit history, into a single bundle. It then uploaded that bundle to a Google Cloud Storage bucket named grok-code-session-traces, regardless of what the model needed for the task at hand.
To prove the point, the researcher planted a file the tool was explicitly told never to open. A fake secrets file, containing a marked API key and database password, sat nearby. Both surfaced, unredacted, in the uploaded data, recoverable simply by cloning the captured Git bundle. On a 12-gigabyte test repository, the model’s working channel moved 192 kilobytes. The storage upload moved 5.10 gigabytes before the capture was even complete.
xAI’s public response came through informal social media posts rather than a security advisory. Elon Musk pledged to delete previously uploaded data, and the upload behavior stopped a day later. But the fix was a silent server-side flag, not a software update. There was no changelog, no client patch, and the upload code still ships inside the current binary. xAI can turn the upload back on without anyone downloading anything.
None of this proves anyone at xAI read the uploaded code. It proves the code left the developer’s machine and became something xAI’s infrastructure now controls. That, in one sentence, is the entire risk.
A Privacy Toggle Regulates Trust, Not Possession
Every developer who disabled Grok Build’s “Improve the model” setting believed they had opted out. They hadn’t. The setting controls whether xAI trains future models on your data, a policy about future use. It has nothing to do with whether your repository leaves your machine today. After the researcher switched the setting off, the tool’s own status check still reported trace_upload_enabled: true. The codebase kept uploading exactly as before.
Stopping the upload eventually required a second, separate flag: disable_codebase_upload. That’s the architectural distinction worth sitting with. One switch governed how xAI could use data it already had. The other governed whether xAI got the data at all. Only one of them was visible to the user, and it was the wrong one.
| "Improve the model" toggle | disable_codebase_upload flag |
|---|---|
| Governs training consent | Governs data transmission |
| Visible to the user | Not surfaced to the user |
| Turning it off: no effect on upload | Turning it off: upload actually stops |
This is why a privacy toggle can never substitute for actual governance. A toggle is a promise about behavior. And promises are revisable — by a policy change, an acquisition, a compromised account, or, as here, a flag nobody documented. Possession isn’t revisable in the same way. If your repository, your telemetry, or your operational data never physically reaches a third party, that’s the end of the story. There’s no setting anyone can flip to change it later. The only governance model that survives a vendor’s next decision is one where the vendor never had the data to decide about.
The Shadow AI Problem: Every Ungoverned Tool Is Somewhere Your Data Now Lives
Grok Build happens to be a coding assistant. But the AI data leakage mechanism it exposed isn’t unique to coding tools. Any AI system that touches your environment is a potential destination for whatever data it’s given access to. That includes a copilot, an analytics assistant, or a maintenance chatbot, not just a coding tool. Security teams have a name for the resulting blind spot: Shadow AI. It’s the growing set of AI tools operating inside an organization without formal review. Often, a team adopted one simply to solve an immediate problem faster than procurement could weigh in.
For a defense manufacturer, the data at stake isn’t a codebase. It’s programmable logic controller (PLC) configurations, MES and SCADA system credentials, VPN settings, and telemetry from classified assets. Handing any of that to an ungoverned AI tool doesn’t just risk a leak. It means that data now permanently exists somewhere outside your security boundary, subject to whatever that vendor’s infrastructure does next.
There’s a detail in the Grok Build case worth remembering here. The upload captured full Git commit history, not just current files. A secret committed months ago and deleted from the working tree can still ride along in that history. The equivalent in an industrial setting is a configuration file, a script, or a log with months of accumulated context. That’s exactly the kind of file a plant-floor AI tool might read once, and then, without anyone intending it, keep permanently.
What Real Data Retention Requires for Defense Manufacturers
The alternative to trusting a vendor’s toggle is straightforward, if less talked about. Deploy so that sensitive data never has a reason to leave your boundary in the first place. That’s the actual substance behind terms like on-premises deployment, air-gapped networks, and GovCloud. They’re not compliance checkboxes. They’re architecture that removes the decision from any third party’s hands.
Thinaer Capture Layer
Customer-Owned IAM
Customer's Choice
This is the model Thinaer builds around as the Physical AI capture layer for defense and manufacturing operations. Sensor and telemetry data captured on the plant floor stays inside the customer’s own identity and access management system, or IAM. IAM is the layer that controls who and what can touch the data. That might mean an on-premises data center, a fully air-gapped network with no external connectivity, or a customer-controlled GovCloud environment. Thinaer holds a patent pending for deployment in classified environments. It’s HERO ZERO certified for ordnance-adjacent facilities, and DISA approved for secure government use. None of that is a promise about how Thinaer will behave later. It’s a description of where the data already sits.
An AI governance framework built on vendor assurances is really a framework for managing risk you’ve already accepted. An AI governance framework built on retention starts from a different premise. The AI model might be a locally deployed LLM, a cloud analytics tool the customer trusts, or something built in-house. Whichever it is, it only ever receives the data the customer chooses to route to it. That routing happens from inside a boundary the customer controls throughout, and the data arrives already structured and AI-ready. Thinaer doesn’t build that AI model. It makes sure the data reaching it never had to leave home to get there.
Retention Is the Only Governance That Doesn’t Depend on Trust
xAI’s fix for Grok Build was a flag nobody could see, applied to a mechanism nobody had documented. The tool’s upload code still ships in every download. That isn’t a criticism unique to one vendor. It’s what happens whenever a setting decides who controls your data, instead of you.
The models organizations want, for coding, for maintenance, for operational analytics, are already here and improving quickly. What determines whether adopting them creates AI data leakage risk isn’t which vendor you pick. It’s whether the data ever had to leave your hands to reach the model in the first place.
Thinaer’s Sonar platform gives operations teams real-time visibility from day one. It runs on a capture layer that deploys inside the customer’s own boundary — on-premises, air-gapped, or GovCloud. The data never needs to travel to be useful. See how that works for your environment.