SEO testing 101 — Will Critchlow // SearchPilot

Will Critchlow, CEO of SearchPilot, discusses a full series on SEO testing. With AdWords and conversion rate tests, you can test two different landing pages by sending half your users to one and half of the users to another. That approach is not applicable to SEO, but you still have to be able to prove its value and impact on revenue, and that’s where SEO testing comes in. Today, Will talks about SEO testing fundamentals.

Show Notes

  • 02:58
    Challenges with SEO testing
    You can't have multiple experiences shown to Googlebot for the same page. So, SearchPilot runs tests by applying changes to a subset of pages and keeping a subset of the pages unchanged and looks at performance to determine the winner.
  • 06:32
    Determining whether a websites site sections are suitable for SEO testing
    Ideally, the group of pages should have hundreds to thousands of pages to test statistical significance. From there, youre trying to select a controlled and variant set of pages that are statistically similar to each other in terms of seasonality, variance, and traffic.
  • 10:00
    Data sources used to determine test and control pages
    The focus is mainly on analytics data, organic traffic, and statistical properties like seasonality, variance, average levels, and conversions where possible. Essentially, the bigger the website, the smaller an impact can successfully be detected with statistical confidence.
  • 12:50
    Looking at organic traffic for site sections vs a sample set of keywords
    Unlike click through rate and rank position data, organic traffic is the only source that captures everything about those pages. Organic traffic also gets you closer to proving business value and positive business impact.
  • 15:30
    Building business cases from SEO testing
    SEO testing allows you to determine winning strategies and avoid the losing ones. Youre also able to improve efficiency by avoiding strategies with no detectable positive impact and only deploy winning strategies to get ahead of competitors.
  • 18:15
    Changes within SEO that fit into the SEO testing framework
    The simplest things to test are HTML changes and shifting the layout of whole pages. Internal linking tests are a lot more complicated and require the setup of multiple measurement sections.
  • 20:19
    Using SEO testing to improve website migration
    SeachPilot has worked with companies where SEO testing has helped the company to decide between two different versions. Its also been used to improve winning versions and mitigate SEO issues before deployment.


  • "If you're running AdWords tests, you can test two different landing pages by sending half your users to one and half of the users to another. That kind of approach doesn't work in SEO." -Will Critchlow, CEO, SearchPilot

  • "Not every website is configured for SEO. You need scalable site sections." -Will Critchlow, CEO, SearchPilot

  • "A site section is suitable for SEO testing if as a collective, it gets a thousand organic sessions a day. So, 30,000 a month and at least dozens of pages and ideally hundreds if not thousands of pages." -Will Critchlow, CEO, SearchPilot

  • "The bigger the website, the smaller an impact you can successfully detect with statistical confidence. If you are on a smaller website with less traffic, only the largest effects will be discoverable." -Will Critchlow, CEO, SearchPilot

  • "For the typical website, less than 50% coverage of all traffic is actually covered in impressions and click through rate data." -Will Critchlow, CEO, SearchPilot

  • "If you can get to a reasonable level of confidence, avoid the losers, roll out as many winners as possible, then its speed that is the commercial differentiator and that gets ahead of the competition." -Will Critchlow, CEO, SearchPilot

Up Next: