Estimating Statistical Significance in PPC Ad Copy Tests

Is there an easy way to determine the statistical winner of a ppc ad test without using crazy formulas or testing tools?

To estimate how many impressions are needed per ad to end a CTR test (or clicks are needed to end the test based on conversion rate), subtract the rate in one ad from the other, and square the answer. Divide 1 by this number.

Impressions needed per ad = 1/(CTR1 – CTR2)2

This is the most general way to estimate statistical significance, and often is more conservative than you need. If you’re looking at two ads that perform pretty similar, this will return high impression requirements. To get a more exact estimate, use the same equation, but replace the “1” with a more appropriate number from the table below. I’ve used impressions and CTR for metrics, but the equations are the same if you substitute clicks and conversion rate:

Impressions needed per ad = x/(CTR1 – CTR2)2

If the CTR is approximately: Then x =
25% 1
20% 0.85
15% 0.7
10% 0.5
5% 0.3
4% 0.25
3% 0.22
2% 0.18
1% 0.14

*For conversion rate testing, replace Impressions above with Clicks, and CTR with conversion rate

If each ad has more impressions than the required number, you can confidently delete the poor performing ad.

To properly determine if you can end a test with confidence, however, you will need to use a probability distribution like what’s used in the chi squared test. A search for “split test calculator” comes back with several online tools, but the estimate above should cut down on the number of tests you need to enter into the tool.

Point It About the author
  • Thanks for the simple break down. It is very hard to find a quick simple CTR breakdown that explains it so thoroughly for a presentation to an executive.

    May 3, 2012 at 2:25 pm

Leave a Comment: