We use the Optimizely Web Experimentation software to facilitate our A/B testing on the front end of our website. This allows us to freely be creative with different front end user designs, and use data to back up any results to consistently improve the user experience across our website.
Pros
It offers the ability to work with many metrics, enabling us to receive the data and knowledge we really want from each individual test
It allows seeing full versions of each test before releasing, making it easy to review
It is a straightforward and easy to use, with a very logic, step by step process
Cons
The UI part of the software can sometimes load slowly
Although the step by step is logical, it can be long winded and take a while to set-up each individual experiment
Would benefit from the addition of dashboards for easier viewable data
Likelihood to Recommend
Its useful in allowing us to test front end changes, on both smaller and larger scales without the need for full developer resource. It means that we can test changes that we want to make prior to adding them into a dev queue which can be time consuming and we can be confident in the data that backs up the changes that do make it into the dev queue, therefore improving the UX.
VU
Verified User
Administrative Assistant in Marketing (Retail company, 1001-5000 employees)
A/B testing of various web experiments on our semi-static published site and our dynamic, complex app focuses on UI changes within a page; in my opinion, the optimized is poorly suited for true feature flag functionality.
Pros
A/B testing on simple UI changes within one page.
Cons
Feature flag management of A/B testing of multi-page web flows.
Integration with Segment is extremely poor, especially when integrating front-end and back-end signals.
Integration with Segment is extremely poor when tying into the web; many excuses are given by Optimizely's advanced tech teams.
Likelihood to Recommend
A/B testing of a simple UX change or wording change on a single page, where there can be many entry and exit points; crude feature flag support for feature testing if you're willing to write significant, complex, and difficult-to-debug and maintain code unique to Optimizely alongside all your 90+ other SaaS tools.
VU
Verified User
C-Level Executive in Engineering (Retail company, 201-500 employees)
We use Optimizely Web Experimentation for content testing (images and/or copy), recommendation algorithm tests for product carousels, and feature testing (net new features that we develop and want to validate & calculate revenue incrementality before launching). With the first two: we want to test along the customer journey to understand what pieces of content or product assortments provide the best benefit to her on site. We also use audience segmenting to be able to target the right users (coming from X marketing channel, mobile-only users, or logged-in users) to be able to refine our testing.
Pros
Extensions - allows the company to tailor the tool to unlock certain capabilities
Developer-friendly, with GA and other integrations
Intuitive results page that can be shared out
Cons
Hard to use extensions without some knowledge of HTML/CSS
Inconsistent user counts between GA & Optimizely when it comes to calculating MAUs, hard to be able to forecast budget and overages when our source of truth (GA) differs from our testing platform (Optimizely)
Impressions model doesn't support scaling personalization experiences - hard to run a serious data driven testing program when you have to cut tests short before 14 days (ideal length to get to any statistical read) in order to save on impressions
Likelihood to Recommend
Optimizely Web Experimentation is well suited for a team that tests across multiple facets of testing – full tech developed features, pieces of content or recommendations, landing page optimization. However - if a team only does limited testing, and only in the content testing area, other tools may integrate better IF they are not using Optimizely's CMS (and if they have a CMS that easily supports testing - like Adobe AEM x Adobe Target). However - as testing scales - Optimizely Web Experimentation will start to make more and more sense.
VU
Verified User
Manager in Product Management (Retail company, 1001-5000 employees)
We use Optimizely Web Experimentation to leverage all on-site experimentation and personalization across multiple brands. We use it to improve conversion and engagement metrics for targeted optimizations and strategic efforts. Additionally, we use Optimizely to answer user experience questions and mitigate risk for major structural changes on our sites and across all of our brands.
Pros
Traffic splitting and audience targeting.
Statistical validation.
User friendly UX.
Cons
Bulk archive ability.
Copying experiments between projects.
Likelihood to Recommend
Well suited for a company with a B2C approach, with enough on-site traffic to reach statistical relevance. Not as appropriate for a B2B company or an industry with excessively long sales cycles or lead times.
We use Optimizely for A/B tests and for any urgent fixes that are required on the site when we don't want to wait for our normal CMS content push which is usually overnight. We can easily make front-end changes with Optimizely that update within minutes so if there is an incorrect image, spelling error, etc. we can hide it. We also use Optimizely to push changes at 100% until we are able to build them into our website, for example, keeping a test winner turned on while we wait for dev resources.
Pros
Quickly update your website
Fairly easy to use
Cons
Setting up custom metrics can be confusing
Only one way to track statistical significance of a test through the platform
Likelihood to Recommend
It's well suited for A/B testing, pushing experiment winners to 100%, and last-minute adjustments on the website when your CMS isn't able to make changes ASAP
VU
Verified User
Manager in Marketing (Retail company, 10,001+ employees)
I use Optimizely for all of our AB tests. This helps us validate new features and minor styling changes and gives us insight into our roadmap. It even allows us to make changes to the site without requiring developer support. We make sure everything is AB tested before launching, and Optimizely is a key part of making that happen.
Pros
AB Testing.
Data Analysis.
Web Editor.
Cons
User Interface.
Multi-project management.
Dynamic Website Set-up.
Likelihood to Recommend
Optimizely is well suited for on-site AB testing and experimentation. If you want to see how a change will impact your metrics, Optimizely can make that happen. It is not useful for feature flagging or product rollout and can be a bit tricky when trying to AB test a net new feature.
VU
Verified User
Director in Marketing (Retail company, 201-500 employees)
We used optimizely at the start of the pandemic to help reduce the amount of time a customer needed to send inside a store. This helped us hone in on the correct form layout and other aspects that were weren’t prepared for. Optimizepy came through in clutch time for us as we really moved at a rapid pace
Pros
A/B Testing
Data reporting
Ease of use
Cons
Data collection
A/B Testing
Likelihood to Recommend
I found running an test and personalizations really fit within the wheel house of the product. I think there are areas to improve especially with session replays and sessions heat maps. Overall the product has grown since the beginning and continues to take customer feedback and grow the product to fit more needs.
VU
Verified User
Analyst in Marketing (Retail company, 501-1000 employees)
At Northern Tool + Equipment the design and analytics teams primarily use Optimizely to test design, usability and promotional changes to see what impacts our users in a meaningful way. We run experiments to determine whether or not a new feature is worth building, an existing feature or piece of content is useful or clutters the page and should be removed, and also what promotions and marketing messages resonate.
Pros
Quickly and easily set up new A/B tests. Some tests can be completed in only a few minutes.
A robust results engine. It automatically creates segments based off URL parameters and other common criteria (new vs returning) in addition to predefined criteria you set up.
The support of the Optimizely implementation team and customer care is fantastic. Optimizely easily ranks as one of our best third party software vendors. They provide not just knowledge of how to use the tool, but how to build a culture around testing and experimentation within your company.
Acts as a great lightweight content management system for making temporary changes. We use Optimizely to make changes for 100% of traffic for controlling scheduled promotions and also bug fix patches while waiting for permanent fixes.
Cons
Geo-targeting for tests does not allow you to use ZIP Codes.
URL targeting does not have a mass upload option for lists of unrelated URLs. It does have substring and regular expression matches that cover most other needs.
The preview feature rarely works as expected. We run tests on a dedicated testing environment to preview and do QA.
Likelihood to Recommend
Optimizely is a great tool to get you up and running with A/B testing quickly without sacrificing quality. The WYSIWYG editor mode is passable for someone without html and javascript knowledge, but to get the most out of the tool you'll definitely want to have a developer that is fluent in javascript. As of a few months ago Optimizely now offers server side (or full stack) testing in addition to their client side testing. This is an additional feature you have to pay for, but if you have the need and teams to support server side testing it allows you to test a lot of different things you won't get out of client side testing.
We are currently using Optimizely for both multivariate and A/B testing our websites, and we have received some pretty big incremental wins. Optimizely is easy to use and affordable. I like that there are several tools to help cross-browser check the edits you make on the interface. There are more advanced options available, but for most companies, Optimizely should certainly be enough. (Especially since Google does not offer multivariate testing these days).
Pros
Multivariate Testing: Run many variation tests
Easy to use: Can design changes within the interface and after the first install, no additional code is needed
Cross-Browser Testing: Allows you to test the changes in many browsers
Cons
Advanced options: I have had some issues configuring very advanced tests
Likelihood to Recommend
Website optimization/testing is critical for success. It's a great place to get big wins. I find Google Experiment harder to use and less effective. For companies that need more than Google provides, Optimizely is a great choice. On the other hand, Optimizley is also not ideal for advance testing. It is the ideal middle solution, which is where most companies are (or should be).
