A/B Test Case Study – Hawaii Travel Agency Firstwise (Simple Multivariate Test)

2014年11月13日 | Evan Burkosky


This simple test increased contact form conversions 132.1%!

Even a complete beginner to Optimizely can easily repeat this same test.


4 pattern multivariate test (including original) of contact form button.

This test was carried out  on the website of Hawaii Travel Agency “FirstWise”. The FirstWise website is operated by Seven Four Seven Enterprises, which provides local arrangements and support for visitors to Hawaii. Although they handle all aspects of the travel experience, they don’t focus on discount travel but rather 4 star and higher hotels and business class tours. This site focuses on conversions by telephone or through the contact form.

The number of inquiries have been increasing, and FirstWise wanted to optimize their contact form conversions. For this test we analyzed which pages get the most traffic, then decided to test the button that takes users from the tour details page to the contact form.

This button was still labeled “Contact us by email” (English translation of original Japanese), even though it no longer opened an email client, but linked to an online contact form instead. The wording of this button had remained unchanged since the time that email clients were used instead of an on-site form.

In order to fix this mismatch of wording and button action, new text was required. But what text would make the most sense to users and inspire them to click to visit the contact form? A multivariate test was setup with the following 4 variations (original text and 3 variations) to help answer this question.

00 01 02 03


Original text: Contact us by email
Variation 1: Contact form
Variation 2: Inquire about seat availability
Variation 3: Inquire about seat availability (free)

Variation 1 makes it clear that the button goes straight to the contact form.
Variation 2 provides a direct call to action and reason for using the contact form.
Variation 3 provides a direct call to action, reason for contacting, and allays fears by explaining that the service is free to use.

The unique feature of this test is its extreme ease of set up. The only thing that needs to be done in Optimizely is to change the text in the relevant place. No knowledge of CSS or Javascript code is required so anyone can run this test. Even including setup of delivery and conversion settings it takes less than 30 minutes to complete the process.


Result: Contact form conversions increased 132.1%!

The result of this test is that all 3 variations outperformed the original. The original contact form conversion rate (rate of completed contact form inquiries) was 4.26%. The best performing variation was number 2, with a conversion rate of 5.62%. This change from 4.26% to 5.62% meant an increase in contact form completions of 132.1%!

These results tell us that users are more interested in seat availability  than general inquiries about tour information. By pinpointing the message and giving users a clear call to action, we were able to see where their interest lies and to determine that the correct text drives more contact form conversions.

Additionally, the “Inquire about seat availability (free)” text had less of a result on contact form conversions than anticipated. It depends on the product, but we can infer that in this case users were more interested in getting directly to their desired information than in knowing that it’s free to check availability. In case of FirstWise, the reason why there were fewer users attracted to the “free” text could be because they are introducing high level luxury hotels.

Based on these results we can see that multivariate testing had a very positive impact, and we plan to undertake further tests throughout the site based on these findings.

Create a hypothesis based on user behavior and search for the best sentence to drive conversions.

Multivariate and A/B testing with Optimizely allows for a great range of testing, with the potential to change various design elements and try different formats, but it’s also possible to get great results just by changing the message of site text. In addition to the changes to button text which drove conversions in this particular test, it’s also possible to add text next to action buttons that explains service merits, helps reassure users about site credibility, and helps drive a variety of user actions.

When thinking about how easy it is to make changes to text, and how effective these changes can be, it’s easy to see the cost performance benefits of using Optimizely for this kind of testing. If you are unsure of what type of multivariate or A/B tests would provide the best benefits, it can be very helpful to take a look at what type of actions site users are taking, and how text can help drive these actions along more effectively

Click here to visit the FirstWays website (Japanese)

Related service

Easy A/B testing with Optimizely Support Package | e-Agency


Evan Burkosky