Marketing the TNW way #15: How TNW uses Google Optimize 360
In this blog series, we shed some light on our marketing approach at The Next Web through Web analytics, Search Engine Optimization (SEO), Conversion Rate Optimization (CRO), social media, and more. This time we talk about how we started using Googles Optimize 360 and what our first insights into the tool are.
Improving our reader experience is of great importance to us, that’s why over the last two years we intensified our testing program.
While our testing process keeps us sharp, it also makes us less dependent on tools. The geeks we are, though, makes it fun for us to keep trying out new tools that come to the market. So when Google Optimize 360 was quietly announced we reached out directly to Google to see if we could get a sneak peek and become part of their BETA testing program.
So far, it’s been fun. So here’s some insights into how we use Google Optimize 360 at TNW:
The first thing you need to handle is integrating it within your own site.
The beauty of the 360 Marketing Suite from Google is that it’s integrated (almost) seamlessly with each other. With other vendors/tools, you have to hardcode or create your own tags for it in Google Tag Manager. Maybe not a huge deal for smaller companies, but definitely for larger enterprises.
So, Google Optimize 360 has its own featured tags to make it easy to integrate. Assuming you’re already using Google Tag Manager… But why wouldn’t you?!
Native Google Analytics 360 Integration
Hands down, the best feature of the Google Optimize 360 is its integration with Google Analytics.
When testing, your analysis usually takes the longest (save for waiting for your test to become significant, which is something you can’t do much about).
As with most tools, you have to manually set up goals or integrate their own analytics tracking – which is quite annoying if you just want to get started with testing and/or you already have a great set up in Google Analytics. Which is the case for us – our testing tools should always easily work with Google Analytics.
[conf-ad-unit]
So time to analyze!
First want to know if your test reached the right significance level for your primary objective. If it did, then you want to dive in deeper to find out what the impact is on secondary objectives, and if these metrics also hit significance within the test.
All in all, this would usually cause you to set up, create and generate a ton of reports. In our case, specifically, we’d drill down our A/B tests on geo locations, user types (new and returning), as well as traffic sources to find out how certain user segments will react to a certain test.
Google Optimize 360 makes both way easier. It provides the right metrics to see if your significance is good enough (they’re using a Bayesian calculation for this).
Segmentation and targeting: All the users!
Second best opportunity, and probably even more for companies that like to test certain groups of users. As a publisher, we unfortunately know a bit less about the intention of our users.
That said, the new targeting options are about the same as most other vendors. But the fact that it’s a Google tool will allow you to use the audiences that you already have in Google Analytics/Adwords and use them to target a certain test.
While we haven’t used it in our case yet, we probably will soon as it makes it easier to test certain impacts for a specific traffic source.
Testing: How it actually works
Since implementation, we’ve run about 15 tests including MVT tests and A/B tests.
How, exactly? Let’s look at an easy test: Changing the CTA on the in-content block at the beginning of this (and all) blog post.
We tested the three different CTAs there. Changing the text in the different variants was obviously very easy with the WYSIWYG editor (a feature we don’t usually employ, it’s faster and better quality to write the code ourselves). But in under three minutes, we could create different variants.
As the goals are already part of our Google Analytics setup (tracking in-content block clicks) we could set-up the primary objective and chose the secondary ones from the drop down menu. Adding secondary objectives is important as you’re able to not only make sure your primary metric is moving, but you can also take into account everything else that is
Leaving us with the last step: QA to make sure it’s working in both preview mode as well as the live environment. A quick check in this case, as the code for changing the text itself can’t really be messed up that much. Ready to have it run!
Within minutes, realtime analytics provides insights into how many users are currently in the different variations of the tests. After a couple of hours, you’re provided with insights on the objectives that you chose.
Your test is working, time to wait for the runtime of our test. In our case seven or 14 days. After 14 days, when it reaches significance, it well tell us for if there is a clear leader found or not. Luckily these variants had such an impact on the test that it found a clear leader. Time to analyse!
It’ll show a clear overview what the results were for the primary objective, and the drop downs give you the opportunity to drill down the results even more making sure your secondary objectives weren’t hurt.
So, what to improve?
We’ve mentioned it before in previous blog posts – the way Google products work together makes us love them. But obviously there are some things that we found that could be improved.
The Optimize team already knows these, but we wanted to mention them as some of them are good to be aware of for now and could also smooth out your process:
Pausing tests: When you pause a running test and you want to make a change… well, that’s hard as you need to copy the existing test. The upside, though, is that it’ll keep you aware when you start a certain test. It’ll be QA’ed properly to make sure it keeps working and that your intention with it was right.
Need for more goals: It’s required to use a goal or one of the (basic) Site Usage, Adsense or E-commerce metrics to validate if a test is winning or not meaning you might run out of goals quite quickly. If you speed up using Optimize 360, you can run quite a lot of new tests with new ideas and goals. Currently, we’ve run out of goals for our primary view creating the need for a secondary (non backup) view to accommodate this.
Inconclusive results: While the Bayesian method that Google uses requests you run the test for at least two weeks, the impact of your test still hasn’t reached significance within that short time period. It would be great if Google would make that more clear. If you only have a 0.1 percentile difference after two weeks, it’s unlikely it’ll reach your goal by running any longer (at least in our case). Stating the test as ‘inconclusive’ in this case would be a great improvement.
Holdback tests: This is something we’ve been wanting to do internally for a long time. After a test produces a winner, keep it running one to two percent of your traffic for a specified amount of time to see if the winner is still a winner and what its effect is over time.
Two weeks runtime: The Bayesian method they’re using for the analysis of significance is one that generates more value over testing over a longer period. Our belief is that a single method is not best. That’s why it would be great if you could either have a way of shortening your tests or picking another analysis method.
But in the end they’re really #firstworldproblems, and the tool itself already makes our life easier. Working with the Google Optimize team will probably help improve our process and their product.
So… are you looking forward to use Google Optimize – either free or paid? Leave a comment to let us know what you’re planning to test.
