AISEOCourse.academy Module 8

Module 08 · Lesson 1 of 4

Running SEO Experiments You Can Trust

About 12 minutesPrerequisite: Module 7
After this lesson you canwrite a testable SEO hypothesis, change one thing at a time, choose a fair comparison and a sensible window, and document the result so it becomes knowledge.

Most "SEO tests" are just changes with a story attached: something was edited, something moved, the edit got the credit. A trustworthy experiment is smaller and stricter: one predicted change, one variable changed, a fair basis for comparison, a window long enough to matter, and a written verdict either way.

The five parts of a real test

  1. A written hypothesis: "If we rewrite this page's answer passage to answer the query in the first 40 words, citation presence for query X will increase within six weeks." Prediction, mechanism, metric, window: all four stated before you touch anything.
  2. One variable: the passage rewrite, and nothing else that month on those pages. Ship a passage rewrite, a title change and an internal-link push together and you have learned nothing, whatever moved.
  3. A fair comparison: either before-and-after on the same pages against your own baseline (with the honest caveat that the world moved too), or a small set of comparable pages where only some receive the change.
  4. A window matched to the metric: classic rankings and traffic need weeks; AI answer presence needs repeated runs across weeks because of variability (lesson 2.4). A three-day window proves nothing about either.
  5. A written verdict: supported, contradicted or inconclusive, with the numbers. Inconclusive is a legitimate result and often the most common one; recording it stops you re-running the same folklore next year.
DefinitionAn SEO experiment is a pre-registered prediction with one changed variable and a fair comparison. Without the pre-registered prediction, every outcome gets rationalised after the fact.

What SEO can and cannot prove

Be honest about the physics. You rarely get laboratory control: competitors act, platforms update, seasonality shifts, and single pages are small samples. So treat experiments as evidence accumulators rather than verdict machines. One clean test nudges your confidence; three tests pointing the same way across different pages and quarters is knowledge you can build on. That standard is reachable by any site with a baseline (lesson 7.1) and discipline, and it puts you ahead of most of the industry, which ships changes and keeps no record at all.

Worked example: one experiment, documented (illustrative)
  1. Hypothesis: rewriting the pricing page's opening into a direct answer passage increases citation presence for the two starred pricing queries within six weeks.
  2. Variable: the passage rewrite only. Titles, links and everything else untouched on those pages.
  3. Comparison: four weeks of baseline before, six weeks after, weekly runs logged throughout.
  4. Result: cited in 1 of 8 baseline runs, 6 of 12 post-change runs; competitor stable on the same queries (their numbers act as the world-moved control).
  5. Verdict: supported, modest confidence, single page. Next test: same pattern on a second pricing-adjacent page to see if it repeats.
Workbench 8.1
  1. Write one hypothesis in the four-part format for the opportunity card you are executing from Module 6.
  2. Check it changes exactly one variable. If not, split it into two experiments.
  3. Confirm the baseline exists for the target pages and queries (four weeks minimum) and fix the end date now.
  4. Create the experiment record: hypothesis, variable, comparison, window, and an empty verdict row that will be filled honestly.
Self-check
Why write the hypothesis before making the change?
Because after the fact, any movement can be rationalised as success. A pre-registered prediction with a metric and a window is what turns a change into an experiment.
You shipped three improvements at once and traffic rose. What did you learn?
Nothing attributable. One variable per experiment is the rule; bundled changes produce anecdotes, not knowledge, no matter how good the outcome feels.
Is "inconclusive" a failed experiment?
No. It is a recorded result that stops you repeating the same untested idea later. Failed documentation would be leaving the outcome unwritten.
Key principleOne prediction, one variable, one fair comparison, one written verdict: that is the whole experimental method.

Sources used in this lesson
Google Search Central: AI features and your website
Google Search Central: helpful, reliable, people-first content

Previous 7.4 Tying Visibility to Conversions