Ask HN: statistical significance on group of Adwords keywords
I am trying to optimize my adwords campaign, and running into the 'statistical significance' question.
I read this great post from Paras Chopra here: http://visualwebsiteoptimizer.com/split-testing-blog/ab-testing-significance-calculator-spreadsheet-in-excel/
Though in my case I started my campaign with many keywords (about 400 keywords), and I now have the following:
Keyword group A (10 keywords - the "best"): 47 clicks, 16 conversions
Keyword group B (390 keywords - the rest): 244 clicks, 18 conversions
If I apply these data into the A/B Testing Significance calculator, I obtain a 99% confidence that A is better than B.
But does it really work with a group of keywords, or is this calculation meant to be used only for strict "singular" test cases? You can use it this way. But what does the data tell? Group A is definitely performing better than group B or better it is not a random coincidence that group A is better performing.