What is Video A/B Testing? How to Optimize Marketing Videos

As a digital marketer, you’ve likely started A/B testing (or split testing) various video ads and social media videos to see what clicks with your audience. But why stop there? You can do the same thing with the videos on your website! That’s where Wistia’s video A/B testing feature comes into play.

In this guide, we’ll explain what video A/B testing is and how to use Wistia to A/B test the videos embedded on your site.

What is video A/B testing?

Video A/B testing is putting two different versions of a video head-to-head, with half of your audience seeing one version and the other half seeing the other. During the test, you can compare the performance metrics to figure out which version resonates best with your audience.

Why should I test my videos?

A/B testing is an excellent practice to have in your video marketing strategy. It can help you quickly figure out what might be working and what isn’t with your site videos. With the data from the test, you can quickly iterate and optimize your videos for the metrics you want to hit.

What should I test between my videos?

When A/B testing videos, you might test one of the following customizations for a video:

• Video thumbnails (including static vs. moving)
• Player color customizations
• Lead generation forms or Calls to Action (CTAs)

You can also test two versions of a video that have different:

• Video lengths
• Background music
• Titles or descriptions
• Edits: trims, cuts, transitions, etc.

Or you can get experimental and test two entirely different videos! Hey, it’s your test. Try it out!

Which metric should I measure in the split test?

When comparing two videos, you will primarily be looking at the following video metrics:

• Plays
• Engagement
• Conversions (or timeline actions)

Depending on the goal of your video, one of these metrics might be more valuable than the other. Keep that in mind when choosing a winner!

Now that you know what you can split test for your videos and what to measure for success, let’s set up your first A/B test in Wistia.

How to split test videos in Wistia

If you want to test and optimize the videos on your website, Wistia is your best bet. Our platform makes it super easy to get started with video A/B testing. Here’s how:

1. Prepare your videos

Before starting an A/B test, you need to prepare the “A” and the “B” videos to test against each other.

Let’s say you’re testing the same video but with different customizations (thumbnail, player, CTA, etc.). You can make a copy of the original video and then customize the copied media to create your second version for the test.

We recommend adjusting the title of the videos to reflect what you’re testing. You can do something as simple as adding an A or a B to your title, or you can get more specific.

For example, if you’re testing the player color, you might include it in the title to help you quickly identify that version. (ie. “Video Title - Blue Player” and “Video Title - Green Player”).

2. Set up your test

Now that your two videos are ready, the A/B test is ready to run. In your Wistia account, hover over Analytics in the top right menu and then select A/B Tests.

Select the “Choose video” button to open your content library and pick your control video on the left and the video you want to test it against on the right.

3. Start your test

Now it’s time to start running your video split test!

All you have to do is press the “Start A/B test” button.

4. Embed your test

With the test started, the only thing left to do is to put the videos in front of your audience.

Press the “Test Actions” button and select “Embed and Share” from the dropdown.

From the share menu, choose your embed method, customize the embed options, and copy the embed code with the blue “Copy" button.

And then go paste the code onto your site! You can embed it on your homepage, a product page, a blog post, or even a paid landing page.

5. Measure the test results

So your video A/B test is up and running on your site. Great! Now it’s time to track its performance as the views start coming in.

It may take some time to see which video might be performing better. You also want to remind yourself which metric is the most important for your test: plays, engagement, or conversions.

6. End the test and declare a winner

Once you feel that your A/B test has received a significant amount of data to determine a winner, you can end it.

Just press the “End A/B test” button. Below each video, a button will appear where you can select the test winner based on your video strategy’s goals.

Remember, performance metrics alone don’t determine a winner. So choose the winning video that performed the best on the metric most important to you (plays, engagement, or conversions).

Where we’ve split-tested videos on our site

Here at Wistia, we’re all about optimizing our videos. Better videos lead to more plays, which leads to more engagement with our brand, which leads to more viewers converting to paying customers. Here are some knockout examples (let the best videos win!):

Homepage videos

We tested the “Welcome to Wistia” video on our homepage. Our goal was to increase the play rate of the featured video by trying out a video thumbnail against the static thumbnail. The video thumbnail won!

Blog videos

Next up: two identical videos in a blog article about iPhone video settings. We tested a blue player vs. a green video player to see which color would increase the play rate. So far, it’s looking pretty good for blue!

Long-form videos

Finally, we’re testing variations of the 2023 State of Video Report video (clocking in around 11 minutes). One version has a fun bumper video, and the other has no introduction and goes right into the content. We wanted to see if this mystical statistical wizard of video marketing could get audiences excited and engage with more of the long-form video. So far, it’s pretty even, but there is a slight edge for the mystical man!

Optimize your videos with A/B testing

Like anything in digital marketing: test, test, test! Testing is the only way to find out what resonates with your audience. A simple thumbnail could drastically increase the play rate of a video.

The data and insights you can get from a video split test can help you make actionable decisions for your current video library — and inform your future video marketing strategy. With Wistia’s Pro Plan and above, you can leverage A/B testing and let two videos in your account compete for the winning title.