Consider Google’s incentive to index your site and prepare for the next chapter of the internet

Jono says: “My number one tip is a scary one. It’s to start worrying about what happens when Google has no incentive to crawl, index, or rank your site.”

Are you in trouble if your whole content strategy and your entire business model are dependent upon getting indexed? Are you saying that SEO cannot be your marketing strategy?

“It is definitely something you should think about. Almost everybody, to some degree, relies on some form of content marketing – whether we call it that or not. We do marketing keyword research, we try and spot opportunities, and then we build landing pages, we write articles, or we have a blog. The objective of all of those things is to try and boost our rankings, get traffic, and convert visitors. However, that whole ecosystem is built on the assumption that people are typing things into a search box and we can access search volume metrics for that.

Increasingly, that isn’t true when you look at Search Generative Experience, AI-generated content, Google Discover, and the ‘ambient search revolution’, where it’s not as simple as: ‘type the thing, get the results.’

Also, we still assume that Google is incentivised to spend their resources trying to discover our content, index it, evaluate it, rank it, and return it in some kind of list. That’s not in their interest anymore. Increasingly, for many of the things users are searching for, they don’t need to be evaluating our content and serving it. They can synthesise their own results, using SGE or something similar. They understand the inherent problems well enough to be able to solve them without relying on our content.

We’ve seen the beginning of this for a while now, with things like zero-click searches. The balance of power has changed. This is the conclusion of the whole first chapter of the internet. It is no longer necessarily the case that Google is relying on websites as sources for answers. If we’re still plodding along going, ‘I will research my keywords, I will rank my landing page, I will commission my article’, all of the dominoes that would have followed from that – around indexing, crawling, ranking, etc. – no longer exist.

We need to rethink what content marketing is, what content strategy is, and what our SEO is. It’s probably not about trying to write pages to rank for keywords; it’s probably more about brand discovery and preference, and trying to convert people in other channels when you’ve built those relationships.”

How do you incentivise Google to want to rank your content? What does content that highlights your brand and adds value look like?

“This is the heartbeat. I don’t think anybody really knows, but there are some areas you can go to. You can definitely try and work on bringing unique value to a space. Do something that no one has done before. Go deeper, go better, commission the research, and employ the experts. None of this is easy or cheap. In some spaces, it might not even be possible, but this is one of the options.

Also, don’t be too unique and too out there, because misalignment from established norms might mean that you do worse. You can definitely be polarising; I think that’s interesting.

For a long time, it’s been trendy to say that brands need to become publishers. We misconstrued that to mean we should have a blog or commission articles. Actually, when you look at real publishers and real editorial outlets and newspapers, they do research, they are in the field, they form opinions, they have editorial policies, etc. If we do those things as brands, then we might get in front of people. Perhaps not when they’re typing ‘renew car insurance’, but when they’re on their journeys, searching for other things, or solving other problems. We start to build brand preference and association there.

It’s cliché and it’s tedious, but the answer keeps coming back to: be the kind of brand that users expect to see, consume information from, and trust. It does come down to EAT and demonstrating substantial experience.

Tactically, you can cheat a bit. You can compete in other verticals, where it’s easier at the moment. You can double down on e-commerce, video, audio, and other areas where there’s less competition and it’s going to be more expensive for Google to synthesise content. That might buy you another year or two, but that’s not the long game. It’s all about brand.”

If Google decides not to index your content, do you give up on that URL or is it possible to get them to change their mind?

“Anecdotally, I’ve seen more and more people grumbling on Twitter/X or Threads that, ‘I’ve got good content. I’ve got some links. Google isn’t indexing it. Why not? Is it some kind of mysterious technical problem?’

It’s probably not. It’s just that, even if the content is good, it’s probably not valuable, unique, or distinct enough for Google to want to expend resources putting it in their index. I always come back to the example of lasagna recipes. What is Google’s incentive to index the lasagna recipe that you publish on your blog? They have a hundred trillion of them. Even if somebody searches for a weird lasagna recipe that doesn’t exist, they can synthesise it on the fly.

If you’ve got pages that aren’t getting indexed, and you think that they’re good and that you’ve done SEO best practices, maybe you’re not fundamentally solving a problem that either the users or Google has. If the objective is to feed the beast, it doesn’t matter how much more work you put into that page if you’re trying to feed it content that it doesn’t need.

Maybe delete them. Less is more. Find the stuff where you can make an impact and make it 10 times longer, 10 times better, or even 10 times shorter if that’s more valuable. Add video and employ experts. Double down on what Google can’t easily synthesise out of thin air. If that’s the kind of content you’re writing, you’re probably not adding value to the web. Your pages have to truly add value to the corpus of the internet, in the same way that you might if you’re researching a paper.”

Is there a way to easily measure how valuable, unique, and distinctive your content is?

“Not directly, which is where it gets tricky. It’s difficult to build a business case for this and get buy-in when you could just show a list of keywords, search volumes, and CPCs. If you do the big shift – you stop writing pages that try to rank for keywords and, instead, start writing pages that try to solve specific user problems without going to sell or convert directly – you start prioritising the needs of the audience rather than just your customers.

Yes, you can’t monetise it or quantify the impact on the bottom line, but you could do things like measure brand recall and saliency. You could do traditional surveying mechanisms that say, ‘Which of these logos do you recognise?’ You could put voting widgets on the bottom of your articles and resources that say, ‘Was this page useful?’ You can monitor upticks in that time and vaguely correlate that to engagement, then vaguely correlate engagement to revenue.

It’s going to be much harder than a simplistic machine where you get a number for ranking, a number for visitors in, and a number for visitors out. That’s never been how this works, though. Pretending that this is a direct response ecosystem has been a big con. SEO has only ever accidentally been a channel that converts. It’s so much more powerful as a channel that introduces brands and helps people to trust and become familiar with them. From there, you convert in your other channels when you’re solving people’s problems, they’re recommending you, and they recognise your logo when they see you in the supermarket. This is where SEO is far more powerful than other channels, but we’ve become so focused on the bits that are measurable.

Maybe measurement isn’t what we should be optimizing for. Maybe it’s audience success and recognition. You can measure some of that, it’s just harder.”

Is it still useful to look at what websites are ranking number one for your target keyword phrase, looking at the style and type of content it is, and taking lessons from that?

“I think there are two aspects to this. One is that looking at Google is one of the best pieces of research you can do, and the SEO industry seems to have only recently rediscovered this. Google has the best understanding of what the intent might have been and what kinds of content the user might want. In some cases that will very obviously be showing the cheapest product and the fastest way you can get it. In some cases, it might be quite nuanced. It might be a couple of videos, an informational resource, and other bits.

All of that is the product of what the SEO teams at those companies are doing. If they’re all stuck in yesteryear’s thinking, and they’re all trying to compete on keywords, then maybe you end up aping their bad behaviours.

I think a bit of both is sensible. Look at Google search results to identify the types of things that Google does a good job of understanding, in terms of what people want, and then do better than the people in that space. That has always been SEO; ever since the first days. Look at what’s ranking in front of you and do a better job than them.”

You have spoken about the concept of operating in a space that’s solved. What do you mean by that and what do you do if that’s the case?

“I love this concept, and the lasagna recipe is a really good example of this. The lasagna, at a conceptual level, is a solved problem. Google comprehensively understands the concept of ‘lasagna’. It understands what it is, it understands all the things that might be in it, how you might cook it, where you might eat it, etc. You are going to struggle to create a piece of content that adds unique value to that space.

Google has such a comprehensive understanding of the concept that, even if you did, they don’t need it. They can synthesise new information based on what they’ve already got. If you want a recipe for a blueberry lasagna, even if that doesn’t exist, Google can take a strong enough punt at it with generative processes and a bit of AI, and they can create that on the fly. There is no content strategy where you can write an article, page, or recipe and rank on ‘blueberry lasagna’.

There are lots of spaces like this. More practically, every car insurance website and aggregator on the web has a page about how to prevent your pipes from freezing in winter. They’ve done a bunch of keyword research, they’ve seen that this is the kind of thing that people search for, they understand it’s loosely in their sector, and they write a 500-word article that says the same thing as every other page. This is a solved problem. There is enough content out there that touches enough of the angles that Google, having ingested all of it, can solve users’ problems and synthesise answers in spaces that it hasn’t encountered before.

There is no way into these spaces – and there are going to be more and more of these. Anywhere where there is either a simple binary answer (which is the case with zero-click searches today) or, more increasingly, spaces where there’s enough information on all the different permutations that it doesn’t make sense to add another one.

This goes right back to the beginning. If your content strategy is to spot keywords, write pages, get ranked, and get traffic, all of that stops when Google has no incentive to consume your pipe tips or your lasagna recipe. It’s a solved problem.”

How do you limit the danger of Google not indexing your content? Are there other channels that you can work with?

“There are definitely other channels. You could go all in on TikTok, or Threads, or X, or whatever we’re doing today. However, a more diverse content strategy that does some kind of hub and spoke thing is probably not the craziest idea, to spread that gamble a bit.

SEO is going to be harder and harder to do in the way we’ve done it. There will always be a need to have content about the services you offer and the things you do. Increasingly, though, the kinds of ‘hearts and minds’ content might make more sense on Instagram. Maybe you just do the hard sales stuff via page search.

SEO looks less and less like a channel that can and should convert. You can’t attract and compete with that type of audience, at least not reliably. It’s not in Google’s interest to do that. Wherever there’s a clear route to monetisation and transaction, Google is going to try and do that in the SERPs rather than funnelling to your site. We see this happening in sector after sector. They take click-to-buy, flights, hotels, jobs, etc., and they just escalate it out.

That’s going to happen pretty much everywhere. It’s a better user experience and it makes more sense strategically. Whilst I love SEO, and I think it is truly the best way to attract, influence, and engage an audience on any axis, it’s probably not where you want to be piling all of the money that you’re expecting to turn into more customers. You’re going to need a more diverse strategy.”

If an SEO is struggling for time, what should they stop doing right now so they can spend more time doing what you suggest in 2024?

“I’m going to get some hate for this but I’m going to say Digital PR, as an entire category. Let’s go to war. If you think that you need more links or better links to get better rankings, then you’re operating way too far downstream. You probably have brand, quality, product-market fit, budgeting, or political problems. Those are the reasons that you think you need the links.

Maybe it’s hard to fix those, or maybe you can’t fix them, but piling money again and again on campaign after campaign to keep your site yo-yo-ing in the search results doesn’t feel like a smart long-term bet. If you do this right and you think that you need PR to gain recognition and build brands, etc., then that’s great, but do PR. Don’t do PR to get links from diverse domains. Embrace it and do proper PR. Some of the best SEO strategies I’ve seen are people running billboards and TV adverts because it changes how people search, what they type into Google, and which brands they recognise. Think about SEO as a more holistic thing. Let’s downplay the focus on Digital PR as a magic solution.

I have seen digital PR done very well and I have seen it used very effectively. Tactically, I think that it is an appealing sugar rush that people turn to instead of doing the hard work of earning rankings, brand recognition, etc. They burn a lot more money renting that awareness over time than they would by positioning themselves in a place where they would earn it themselves. It’s not inherently evil, it’s just lazy and it’s not a long play.”

Jono Alderson is an independent Technical SEO Consultant, and you can find him over at JonoAlderson.com.