If we ask someone with no knowledge of SEO (Search Engine Optimisation), to try and guess which of two sites rank better in comparison, they will probably do worse than a coin flip.

But even if you are an expert Professional, you get sometimes very puzzling results from SEO Testing. That’s what happened with the experiments you will find in this article and much can be learnt from them.

Will Critchlow, CEO of Distilled refers how much they’ve been investing in performing an intensive Split SEO testing, applying the most common pieces of advice provided by most SEO Professionals to specific web pages. These were split into two groups: one group was altered to include one of the most common SEO recommendations and the other was left unchanged.

The most common SEO recommendations they checked were:

  • Common title tag recommendation
  • Adding structured data
  • Using javascript to show content
  • Removing “SEO Text”

The result of their SEO testing was that most of “optimised” pages lost dramatically in ranking as a result of those changes.

Is SEO harmful to a site’s ranking, then? Not at all. SEO can actually give much better traffic results, but it must be done very carefully. (tweet this)

One of the very common SEO recommendations they tested was to change a page’s title tag to include a required keyword. The result was surprising: in the changed control group, the page’s rank dropped. This could be:

  1. a consequence of that title becoming too artificial, which can easily happen, if the title gets changed to include the keyword, without enough attention. If a page’s title sounds weird, people won’t trust and click to open it.
  2. Also, sometimes changing the title doesn’t disambiguate it. The users may expect a different content when they open the article. If they find that it’s not relevant to their search intentions, they will close it very quickly. Bounce rate will increase, and we will get again dramatic losses in ranking.

The conclusion is that it’s really hard to write better title tags. You need to be very careful! (tweet this)

Many more cases have been studied by them with Split SEO Testing: you can read on the Distilled site the ones performed for structured datainternal linking and meta descriptiontitle and header tags, testing JavaScript for SEO and find more details in this presentation.

Will Critchlow explained how the Split SEO testing can change consulting, producing more subtle impacts on the recommendations. He realised he can now recommend small tweaks and see the benefits compound, test wild hypothesis with unknown upsides, try things that might have a downside (more focused targeting, less copy, etc.) and that’s even before getting the actual benefits of testing click-through rate and the pretty charts to show to the boss, highlighting the impact of your work!

  1. Summing up, the lesson to be learnt from here is: “Everybody needs to make more testing!”.

Larry Kim, instead, winner of Search Marketer of the Year Awards from Search Engine Land, the US Search Awards and PPC Hero, Founder of WordStream and CEO of MobileMonkey, in the same “Super League: 10 SEO Experiments That Will Blow Your Mind” SEM Rush Webinar, listed some assumptions he tried to prove with his SEO testing, such as:

  • Does click-through-rate (CTR) impact the websites’ rankings?” Not surprisingly, the answer was positive: CTR boosts are a very valuable effect, that can be summed up as: Higher CTR = More Clicks; Higher CTR = better Ranking = More Clicks.
  • The other question he tried to verify was: “Do Engagement rates impact rankings?”. The positive answer was not surprising either: we know that Google measures dwell time from the SERPs and this resulted in correlations such as the bounce rate is too high = the site is less likely to occupy the top positions; the time on site is not enough = the website isn’t good enough for the first 6 positions.
  • “How are featured Snippets selected?” There many different theories on this topic, some of which claim that snippets are based on classic ranking factors alone. Instead, Larry Kim verified that a third of the snippets he analysed were taken from positions from 5 to over 71 of organic ranking! Actually, having Snip-able content helps, but this is not the whole story. The answer to the question was that the selected sites have ridiculously high CTR & ridiculously hight time on site.
  • He tried to understand then “what’s the REAL Relationship between social shares and organic rankings?”. For many years it has been denied that Google uses Twitter or Facebook Social signals to rank pages. Instead, some correlations do occur. Their reasons can be in found in the machine learning process: Higher Social Media Engagement Rates = Higher CTR (& vice versa). When some content is shared on Facebook, it can get better CTR and engagement rate, ending up in a higher likelihood to become a featured snippet.
  • What’s the relationship between engagement rates and conversion rates” was the last question he tried to answer. The answer was: Higher CTRs = Higher Conversion rates.


  • CTR Impacts Raking. You will have to beat the expected CTR to get a given position and average CTRs keep going upper and upper!
  • Dwell Time impacts ranking. It validates the CTR and the featured snippet eligibility, overriding all other signals!
  • Other systems reward high engagement, too, like the Facebook newsfeed, PPC and CRO.

The key takeaway from this kind of SEO Testing is that: “the new unicorn of the ranking factors is the user engagement!“. Actually, Larry Kim renamed his own special theory as “unicorn marketing”, and we find it so interesting that we promise we will dedicate entirely a new article to it.

The recapitulatory lesson that can be deducted from this collection of SEO Testing is that: “When it comes to Data vs Opinions, in SEO, Data wins the argument!”

What do you think? Have you ever experienced this kind of situations while working on your site? Please, let us know in the comments on the post here and on Twitter, Facebook or LinkedIn: this will help us all grow from each other’s experience and know-how!