I Used Claude to Find Out Why Google Was Ignoring Our Website

This is the first in a series where I write up, honestly, what it is actually like to build and run a business with AI in the loop. Not the hype, not the doom. What works, what breaks, and what I'd tell you if you were doing the same thing. I'm starting with one that had been quietly driving me mad for months.

The problem

Our website was barely getting indexed by Google. Page after page just would not show up. And it wasn't for want of trying. Around April, for a solid week or two, I resubmitted the sitemap every single day. I went into Search Console and requested indexing on the pages, by hand, one at a time, every day. I did everything the internet tells you to do.

Nothing moved. It genuinely started to feel like Google had decided it didn't want my site. That is a horrible feeling when you do this for a living.

What I did differently this time

Instead of guessing and re-reading the same five SEO blog posts, I sat down with Claude and told it to actually look at the data rather than theorise about it.

Within a few minutes it had pulled our real numbers out of Google Search Console and Google Analytics, and then it did the thing I would never have had the patience to do by hand: it ran every single URL in our sitemap through Google's URL Inspection tool and reported back the verdict on each one.

The result was brutal and clarifying. Of 50 pages, 13 were indexed. The other 37 came back as "URL is unknown to Google" — meaning Google had never even crawled them. Not "crawled and rejected." Never looked at. Ever.

The answer I didn't want to hear

Here's the part that reframed everything. The pages weren't broken. Claude checked: they returned a normal 200, they said "index, follow," nothing was blocking them, the sitemap was valid. It wasn't a penalty and it wasn't a bug.

It was crawl budget. We're a young site without many links pointing to it, and Google rations how much it bothers crawling sites like that. It had read our sitemap once, crawled a handful of pages, and never come back.

Which meant the uncomfortable truth: all that daily resubmitting and requesting indexing had been pulling a lever that wasn't connected to anything. Manual "please index me" requests are a polite nudge, not a command. They can't override Google's decision that your domain isn't yet worth the attention. I wasn't doing it wrong. I was doing the one thing that was never going to work, very diligently, for two weeks.

The actual fix is authority — real links from other sites — which tells Google the site is worth crawling. So we added a few. That's the lever I'd been missing the whole time.

Where the AI got it wrong (this part matters)

I'm not going to pretend Claude just handed me a perfect answer. The reason these posts are worth writing is the friction, so here's the honest version.

It suggested something I flat-out disagreed with. At one point it recommended adding "noindex" tags to some of my lower-value pages to concentrate crawl budget. I don't like that approach and I told it so. It dropped it immediately and didn't try to sneak it back in, but if I'd been less opinionated I might have let it talk me into something I'd regret.

It over-egged a minor issue. It flagged a technical thing about one of our redirects (a "307 vs 308", if you care) as a fix worth doing. When I pushed on whether it actually mattered, it backed down and admitted it was basically cosmetic and not connected to my problem at all. It was right to flag it, but it had presented a footnote with the same confidence as the headline.

It briefly invented a problem. At one point it announced I had a broken link on the site. A moment later it worked out that its own search had matched a fragment of an image filename, and the link was completely fine. It caught its own mistake, but it made the claim first and checked second.

The pattern, if you build with these tools enough, is consistent: AI is fast, tireless, and confidently wrong often enough that you cannot switch your brain off. The judgement has to stay yours.

What I'd tell you

• Get the data before you theorise. The single biggest win was replacing "I think Google hates my site" with "37 specific pages have never been crawled." I'd been arguing from feelings for months. The facts took ten minutes.
• Indexing is about authority, not effort. Resubmitting harder does nothing. Earning a few real links does. If you're stuck where I was, that's where to put your energy.
• Treat AI like a brilliant, fast, slightly overconfident junior. It will do in minutes what would take you a day, and it will occasionally tell you something wrong with a completely straight face. Both things are true at once. Use it for the speed, keep the final call for yourself.

I'll report back in a few weeks on whether the links actually shifted the indexing, because that's the honest way to write these. If they don't, I'll tell you that too.

If you're fighting something similar with your own site and want a second pair of eyes on it, get in touch — and if you want a rough idea of what working with us looks like, the quote generator is a good place to start.