Signal vs. Noise |
Behind the scenes: A/B testing part 2: How we test|
A few weeks ago, we shared some of what we’ve been testing with the Highrise marketing page. We’ve continued to test different concepts for that page and we’ll be sharing some of the results from those tests in the next few weeks, but before we do that, I wanted to share some of how we approach and implement A/B tests like this.
Deciding what to test
Our ideas for what to test come from everywhere: from reading industry blogs (some examples: Visual Website Optimizer, ABtests.com), a landing page someone saw, an ad in the newspaper (our long form experiments were inspired in part by the classic “Amish heater” ads you frequency see in the newspaper), etc. Everyone brings ideas to the table, and we have a rough running list of ideas – big and small – to test.
My general goal is to have at least one, and preferably several A/B tests running at any given time across one or more of our marketing sites. There’s no “perfect” when it comes to marketing sites, and the only way you learn about what works and doesn’t work is to continuously test.
We might be simultaneously testing a different landing page, the order of plans on the plan selection page, and wording on a signup form simultaneously. These tests aren’t always big changes, and may only be exposed to a small portion of traffic, but any time you aren’t testing is an opportunity you’re wasting. People have been testing multiple ‘layers’ in their sites and applications like this for a long time, but Google has really popularized this lately (some great reading on their infrastructure is available here).
Implementing the tests
We primarily use two services and some homegrown glue to run our A/B tests. Essentially, our “tech stack” for running A/B tests goes like this:
- We set up the test using Optimizely, which makes it incredibly easy for anyone to set up tests – it doesn’t take any knowledge of HTML or CSS to change the headline on a page, for example. At the same time, it’s powerful enough for almost anything you could want to do (it’s using jQuery underneath, so you’re only limited by the power of the selector), and for wholesale rewrites of a page we can deploy an alternate version and redirect to that page. There are plenty of alternatives to Optimizely as well – Visual Website Optimizer, Google Website Optimizer, etc. – but we’ve been quite happy with Optimizely.
- Finally, we do set up some click and conversion goals in Optimizely itself. This primarily serves as a validation—visitor tracking is not an exact science, and so I like to verify that the results we tabulate from our Clicky tracking are at least similar to what Optimizely measures directly.
Evaluating the results
Once we start a test, our Campfire bot ‘tally’ takes center stage to help us evaluate the test.
We’ve set up tally to respond to a phrase like “tally abtest highrise landing page round 5” with two sets of information:
- The “conversion funnel” for each variation—what portion of visitors reached the plan selection page, reached the signup form, and completed signup. For each variation, we compare these metrics to the original for statistical significance. In addition, tally estimates the required sample size to detect a 10% difference in performance, and we let the experiment run to that point (for a nice explanation of why you should let tests run based on a sample size as opposed to stopping when you think you’ve hit a significant result, see here).
- The profile of each variation’s “cohort” that has completed signup. This includes the portion of signups that were for paying plans, the average price of those plans, and the net monthly value of a visitor to any given variation’s landing page (we also have a web-based interface to let us dig deeper into these cohorts’ retention and usage profiles). These numbers are important—we’d rather have lower overall signups if it means we’re getting a higher value signup.
Tally sits in a few of our Campfire rooms, and anyone at 37signals can check on the results of any test that’s going on or recently finished anytime in just a few seconds.
Once a test has finished, we don’t just sit back and bask in our higher conversion rates or increased average signup value—we try to infer what worked and what didn’t work, design a new test, and get back to experimenting and learning.
|(Published: Tue, 02 Aug 2011 19:22:00 +0000)|
The Apple Blog |
CableKeeps keep charger and cable together|
|Apple's MagSafe power bricks come with integrated flip-up tabs that let you coil the attached power cable for easier packing and storage. Chargers for iOS devices afford no such luxury. Luckily, a new Kickstarter project called CableKeeps hopes to remedy that situation.|
|(Published: Fri, 12 Aug 2011 21:40:18 +0000)|
Health And Beauty Tips From Pure Soft And Sumptuous |
Commercial Cookware Improves Cooking|
Commercial cookware has opened up a new world of cooking to me. As some of you know, I love cooking natural and healthy and the biggest improvement to my cooking happened when I tossed out the old cheap cookware and replaced it with chef cookware. The difference in the quality of chef cookware is amazing.
Cheap cookware doesn't disperse heat evenly. That is THE biggest problem. If you're not cooking evenly, you're going to have poor quality results. Also, I didn't know that some food is reactive to certain metals. For example, you can't cook tomatoes in aluminum cookware because the acid mixed with aluminum will give the food a metallic taste.
My first piece of commercial cookware was a simple All-Clad stainless steel skillet. I thought I was spending way to much money for a piece like this, but it turns out that it was worth it. All-Clad uses a copper or aluminum core in it's stainless steel cookware to give the even heat dispersion chef's want and the durability of stainless steel. This type of commercial cookware is built to last a lifetime. No longer do I have a shakey handle! I can cook an omelet and slide it off onto the plate without worrying about the handle coming off!
Check out these great tips about choosing chef cookware.
| Leave a comment »
|(Published: Mon, 07 Jun 2010 16:36:00 -0700)|
Scrubs Apparel |
SCRUBTASTIC – The greatest Scrub Store In Las Vegas|
|The greatest Scrubs store in Las Vegas. Our Scrubs are Scrubtastic !!! Visit us on the web www.scrubtastic.com
Sassy Scrubs, ABC Family and Disney Partner to Set a New World Record | TheMouseForLess Disney News|
|(Published: Tue, 21 Dec 2010 19:19:01 +0000)|
easy credit card |
Best credit card for military|
Best credit card for military
Best military credit cards.Choose and apply online. Here you can find the best credit card for military, including rewards credit cards, American Express, Visa and Mastercard credit cards. Military credit cards guaranteed approval are also available, and you can apply online, even if you have bad credit. The best military credit cards for bad credit with guaranteed approval: The best military rewards card: Better than a military gold card it’s a military American Express credit card:
|(Published: Tue, 09 Feb 2016 19:14:13 +0000)|
Pineapple Table Lamp |
Florida’s Space Coast|
|Indian Harbour Beach Vacation Rentals, Homes, Condos, Villas and Beach Rentals Indian Harbour Beach is part of the Titusville-Palm Bay metropolitan area, located in Brevard County overlooking the Indian River and the Atlantic Coastline. This community was built in the 1960s as housing for the missile workers at Cape Canaveral and Patrick Air Force Base.|
|(Published: Tue, 29 Jun 2010 05:38:13 +0000)|
Cappuccino Cups |
Konitz K103 Coffee Bar Cappuccino Cups and Saucers Set of 4|
|Are you looking for cappuccino cups? There are many styles of cups and saucers sets available. Designed especially for a cappuccino, and double espresso or filter coffee, these cups are for those warm drinks enjoyed in the morning or after meals. The open rims of the cups keep the aroma mild and allow the taste|
|(Published: Tue, 30 Mar 2010 20:55:23 +0000)|
Best Free iPad Apps |
6 Free iPad Apps You Need|
|6 Free iPad Apps You Need
It is hardly surprising that the lucky people who managed to get hold of an iPad Saturday immediately started to download applications for the device – and many free ipad apps. The following Monday, Apple announced that the owners iPad downloaded over one million applications from its AppStore, the first|
|(Published: Mon, 17 May 2010 14:05:03 +0000)|
War Machine Costume Outfits |
Iron Man War Machine Costumes and Outfits|
|With the growth of the Iron Man series of feature films, there has developed a great interest in the various Iron Man characters as costume themes. One of the most popular costume is probably the Iron Man War Machine Outfit. It is ideal for those wishing to show their macho credentials. It fits in extremely|
|(Published: Thu, 24 Jun 2010 19:43:01 +0000)|
Buy Natural Gemstones Online |
|The birthstone for the month of November, Topaz chemically is a compound of fluorine aluminum silicate. This very gemstone has been present in the midst of humanity for quite some time and its roots can be traced way back into several pre historic era’s. When talking about topaz it is seen that this very gemstone|
|(Published: Wed, 04 Nov 2009 06:30:07 +0000)|