We primarily use this tool to run AB tests on our website, answer our own hypotheses about what messaging and elements will drive growth and speak to the right audience, and measure those tests as well.
Pros
AB testing capability.
Ability to conduct tests inside of Optimizely (vs. pure redirect testing).
Reporting on test success.
Cons
We have had some reporting inconsistencies between your platform and our BI tools.
Some of the UX elements are confusing to understand.
It is extremely expensive.
Likelihood to Recommend
It’s definitely well suited to a high-volume SaaS product that serves consumers or prosumers. I don’t recommend it for smaller-volume tests or upmarket B2B because test results are too expensive to make a difference.
VU
Verified User
Technician in Marketing (Computer Software company, 51-200 employees)
We use Optimizely Feature Experimentation to quickly evaluate if UX or UI changes to our application or features will be worth putting through our full development process and we also use it to do quick patches while we are waiting for our development team to implement them if they require immediate attention.
Pros
Isolating target audiences by very specific criteria
Rapid releasing to production
Integrating with the base platform
Cons
Testing and re-releasing requires the clearing of cache
There should be a way to remove yourself from a variation
There is no reset for variations on the main experiment, it is only available for templates
The editor preview does not work anymore
The preview overlay should be smaller or hideable
You should be able to see what audience or targeting criteria are causing your experiment to not show
There should be better integration out of the box for SPA including monitoring page changes
Likelihood to Recommend
It is very suited to quick changes in the UI or very basic feature tests. As soon as you start relying on other experiments running or backend changes it becomes a lot more complicated to run. It is also not fantastic for UX or flow changes to your application. With enough tinkering you can redirect users in the manner you want, but it is not well suited for that.
VU
Verified User
Executive in Product Management (Internet company, 51-200 employees)
We primarily use FeatureX for all experiments because it allows us to test sophisticated product-led features related to user journeys and user experiences. Additionally, we rely on Targeted Delivery for soft-launching new features. In certain simpler scenarios where WebX could be applicable, we still opt for FeatureX as WebX often causes a noticeable flicker, which can impact the user experience.
Pros
Collects and processes events
Buckets users into variations
Calculates stat sig
Cons
Ability to force variation for Targeted Delivery to demo stakeholders
Better integration with GA4
Easier scheduling (no need in duplicating rules enablement)
Likelihood to Recommend
In my opinion, Feature Experimentation should be used always to improve a product, always adding new variations based on previous learning.I think the only case when Feature Experimentation is less appropriate, when there is a small design test below the fold (which doesn't cause a flicker).
VU
Verified User
Manager in Product Management (Media Production company, 5001-10,000 employees)
At our organization, we've integrated Optimizely Feature Experimentation into our development and release processes to address several key business challenges, including employing it for feature flags for controlled rollouts of new functionalities, and for A/B testing which has become integral to our decision-making process. We conduct gradual rollouts and experiments across our application and website to optimize user experiences and drive key performance indicators and to reduce rollout risks before doing a full general availability rollout.
Pros
Its robust feature flagging system, which allows our development teams to gradually roll out new features to specific user segments, enable or disable features without deploying new code, and test features in production environments with minimal risk
For example, we would release a new checkout process to 5% of users, monitor its performance, and gradually increase the rollout percentage based on real-time data
Server side performance optimization is really important at our scale, as it ensures improved page load times by avoiding client-side rendering of experiments and the elimination of visual flickering often associated with client-side testing
Precise targeting and segmentation, including the use of custom attributes for granular audience segmentation, ability to target users based on behavior, demographics, or technographics dimensions and creating nested audience definitions for sophisticated experiments that imply potentially overlapping audiences
Cons
The code editor for setting up complex audiences can be challenging to use, especially for team members without strong technical skills. A more intuitive interface for creating sophisticated audience segments would be beneficial
A one-click rollback option or a more streamlined process for quickly reverting to previous versions would enhance our risk mitigation agility and reduce risk in production environments
Although Optimizely has an expansive list of integrations options, we could benefit from an even wider range of native integrations with popular development and analytics tools to be able to analyze our experiment data faster
Likelihood to Recommend
Based on my experience with Optimizely Feature Experimentation, I can highlight several scenarios where it excels and a few where it may be less suitable.
Well-suited scenarios: - Multi-Channel product launches - Complex A/B testing and feature flag management - Gradual rollout and risk mitigation
Less suited scenarios: - Simple A/B tests (their Web Experimentation product is probably better for that) - Non-technical team usage -
[...] is in the process of migrating Optimizely Full Stack (OFS) to Optimizely Feature Experimentation in the next few months. Optimizely Feature Experimentation's backward capabilities allow OFS to migrate to Optimizely Feature Experimentation safely without a code change.
We have instrumented Optimizely Feature Experimentation on UI, and Backend.
Optimizely Feature Experimentation is used in backend services to automate pricing test capabilities and other features. Optimizely Feature Experimentation is used for many UI experiments, such as enabling features on UI. We plan to address more Optimizely Feature Experimentation's feature variable capability to eliminate experiment variation configs.
Pros
Environment based experiment control
Unified interface for experimentation
Backward compatible sdk interface with OFS
Easy migration from OFS experiments to Optimizely Feature Experimentation experiments
Good and descriptive developer documentation
Cons
Running Experiments Exclusively with user profile service. If one experiment is already allocated with the user profile, another experiment in the exclusion group should not trigger the experiment.
Ability to reevaluate audience condition with user profile service, in case audience condition do not match remove override from user profile override. Differentiate between qe override and user profile override.
Ability to connect multiple IDs to achieve consistent experimentation across devices.
Any way to achieve optimizely web capability using Optimizely Feature Experimentation.
Likelihood to Recommend
Following scenarios, Optimizely Feature Experimentation is well-suited Experimentation tests require results to be measured, like pricing tests and UI tests. feature flags and variables can be used as dynamic configs. etc...
Following scenarios, Optimizely Feature Experimentation is not well-suited Experimentation without code changes, such as CSS changes or text changes on the page, supported by Optimizely Web, is not well suited for Optimizely Feature Experimentation. If this functionality is also supported by Optimizely Feature Experimentation, then it can be really helpful.
VU
Verified User
Engineer in Engineering (Computer Software company, 1001-5000 employees)
We use Optimizely Feature Experimentation for managing our releases, A/B testing of new features, and measuring the impact of our releases. It's used by our entire product organization for feature flagging too.
Pros
Integrated roll-out and A/B testing ability
Ability to personalize roll-outs based on other user traits
Measure the impact of releases
Cons
Better integration with web experimentation product
Integrate with Segment and other data sources for easier feature release measurement
More granular audience building / available traits
Likelihood to Recommend
Well suited to teams who are shipping fast but wish to control roll outs to certain audiences, or to test the impact of releases for safety.
VU
Verified User
Executive in Marketing (Internet company, 501-1000 employees)
We use Optimizely Feature Experimentation for feature flagging services and ultimately to hide or show upcoming feature implementations based on the values these flags return. We also built out audiences and experiments to finely tune how we want to serve these features. We then look at the results and make business decisions on these.
Pros
Feature flagging
Segmenting
Quickly updating
Providing metrics
Cons
Initial setup
Onboarding
Experimentation
Likelihood to Recommend
When delivering incremental features without slowing down the process, Optimizely Feature Experimentation can shine. However, for targeting more sensitive information, it might not be the best solution because all of the information is sent down to the client and evaluated there.
VU
Verified User
Engineer in Information Technology (Music company, 501-1000 employees)
We use Optimizely Feature Experimentation to test and optimize user experiences on our digital platforms. It addresses issues like user engagement and conversion rates. By A/B testing different features, we gain insights into user preferences, enabling us to make data-driven decisions to enhance our website's performance.
Pros
a/b testing
multivariate testing
Feature Flagging
Cons
Learning Curve
Limited Customization in Reporting
Likelihood to Recommend
As a digital marketing manager, I've found Optimizely extremely effective for refining web interfaces, testing marketing messages, and optimizing user journeys on large e-commerce platforms. Its strengths shine in environments with high user traffic. However, for smaller websites or those with limited traffic, it's less appropriate, as substantial user data is essential to achieve statistically reliable results and insights. This tool is best leveraged in scenarios where user engagement can be measured at scale.
VU
Verified User
General Manager in Marketing (Computer Software company, 51-200 employees)
We are using Optimizely Feature Experimentation to help us with A/B testing new changes. We have used it for layout tests to test if user conversion is increased by changes and we have also used it for new features to see if general user engagement with the website increases.
Pros
Testing new layouts for conversion
Testing new features for user interaction
Tracking user engagement
Cons
Better documentation. It's very fragmented and hard to navigate.
More opinionated API. Multiple ways to get the same functionality, but with various degrees of documentation. For example, do you activate an experiment or do you use the user decision?
More examples for common patterns would be nice. For example, generation and storage of a user ID in a cookie.
Likelihood to Recommend
It is not needed when using a platform like Adobe Experience that already has things like A/B testing in addition to a comprehensive analytics data collecting.
