The Week AI Tried to Put Our Customer Emails on the Public Internet

This is the third in my running series on what it is actually like to build a small business with AI in the loop. The first was about Google indexing. The second was about forecasting. This one is the most important of the three, because it is the first time the AI's first instinct would have created a genuine security and privacy risk, and it did it without a word of warning.

What I set out to do

We have a marketing site for Tutaris, one of the products I am building. It has a newsletter signup form. Someone types in their email, it lands in our database, done. Nothing exotic.

All I wanted was a way to glance at it roughly once a week and see whether anyone new had signed up. That is the entire brief. A number, maybe a list of new addresses, once a week, for my eyes only.

What it built

The AI came back with a plan that, on the surface, sounds completely reasonable. Build a public, internet-facing API endpoint that returns the list of subscriber email addresses. Protect it with a secret token. Then set up a scheduled cloud agent to call that endpoint every week and report back.

It is the kind of answer that reads as clever. It is "productised." It has an API, a token, a scheduler, the whole shape of a proper system. If you were skim-reading, you would nod along.

Where the AI got it wrong (this part matters)

Read that proposal again with your security hat on.

To check a number once a week, the AI was going to take every customer's personal email address and publish a way to download the whole lot over the public internet. The only thing standing between the open web and our entire subscriber list was a single shared secret token.

Tokens leak. They end up in logs. They get pasted into a chat, committed to a repo, cached by a proxy, or sat in an environment variable on a box that later gets compromised. The moment that one string gets out, anyone on the planet can pull down every email address we hold. And the endpoint would just sit there, exposed, twenty-four hours a day, for the sake of a check I run for about ten seconds a week.

That is the trade the AI quietly proposed: turn a private, internal, occasional task into a permanent, public attack surface holding personal data. Under UK GDPR those email addresses are personal data we are responsible for. "We leaked the subscriber list because a token escaped" is not a sentence I ever want to say.

Here is the bit that actually unsettled me. The AI did not flag any of this. It did not say "note, this exposes personal data publicly." It did not offer a safer alternative. It presented the risky design as the answer, in the same confident tone it uses for everything. The risk was mine to spot or mine to miss.

What the right answer was

With twenty years of doing this, the better approach was obvious the moment I read the proposal. Do not build an endpoint at all.

I already have credentials on my own machine that can talk to the database. So the check runs locally: query the database directly, count the new signups, show me the list, done. Nothing is published. No new endpoint exists. No token can leak because there is no token. The personal data never leaves the controlled environment it already lives in.

It is simpler than what the AI proposed, it is safer by a mile, and it took minutes to set up. And the telling part: the moment I told the AI to build it that way, it did, instantly and without complaint. It was perfectly capable of the safe design. It just did not reach for it on its own.

Why this keeps happening

If you have read the other posts in this series you will spot the pattern. The AI optimises for "this works," not "this is the safe, proportionate way to do it." Asked to make subscribers reachable on a schedule, it found a design that technically delivers, and stopped there. Proportionality, blast radius, what happens when the token leaks, whether the task even needs to be on the internet at all: none of that is in "does it work."

It is the same shape as the forecasting mistake (reach for the plausible default) and the indexing one (state it confidently, check later). Here the default just happened to be one that put customer data on the open web.

What I would tell you

• AI defaults to the more complex, more exposed solution. Endpoints, tokens, schedulers, cloud agents. If a task can be done quietly and locally, the AI will often still reach for the networked, "productised" version. Ask yourself whether the thing needs to be on the internet at all. Most internal checks do not.
• It will introduce security and privacy risk without warning you. This is the one to burn into your memory. The AI did not say "heads up, this exposes personal data." It just built it. Assume it will not flag the risk, and make checking for it your job, not its.
• Anything touching personal data or public infrastructure needs a human review. Customer emails, anything covered by GDPR, anything that opens a new public surface. That is exactly the category where "it works" and "it is safe to ship" are completely different questions.
• Once you know the safe design, the AI will build it perfectly. The judgement about what to build has to be yours. The AI is brilliant at executing the right plan. It is not reliable at choosing it, especially when the wrong plan is the more impressive-looking one.

AI did not make me slower here. It made me faster, including faster at the safe version once I had specified it. But the gap between its first answer and the right answer was the difference between a private ten-second check and a standing data breach waiting to happen. That gap is the job. It is why experience still matters, and it is not going away.

If you are building something that touches customer data and you want someone who has done this for twenty years to look over what your tools are producing, get in touch. The quote generator is the easiest way to get a feel for how we work, and you can see what we have built over on the work page.