The Value of Experiments

Part Two

This continues the in-depth exploration of the role of experiments in product engineering. In part one, we discussed the primary objectives of experimentation and ways to properly convey the value of experiments to stakeholders.

I’m once again delighted to have input from Ellen Chisa of Dark Inc, Daniella Patrick from Accenture, Product Management Consultant Jordan Bergtraum, and Nis Frome from Alpha.

Now let's tackle what exact benefits experiments have for both product design and end-users.

What can experiments do for product design?

Remember that the most common types of experiments fall into these categories.

Ellen believes that these stats may be very accurate for companies that are looking to hire UX experimentation firms. Companies that aren’t hiring these consultants may have a different and more neutral perspective.

These startups or smaller companies don’t view experimentation as product concept and user discovery, they rather view it as usability and A/B testing. Ellen believes that the industry ought to agree with these stats, but alas! It doesn’t.

So many of these smaller companies start with usability or UX refinement, simply to make sure that their screens are working properly.

Nis agrees but adds that this quote can apply to almost any role rather than just the user experience design. Any company that isn’t reducing risks, improving productivity, and reducing costs is setting itself up for failure.

Personally, I tend to focus more on engineering costs because of my computer science background. Someone with a different background—finance or marketing—will most likely focus more on other areas. I always consider how my product or idea affects my developers.

Of course, I am very considerate of my users as well, but my developers and whether they have the right tools at their disposal tend to come first.

The vital role of marketing

After interviewing over 40 product managers about their experiences with failed products, Nis knows that engineering challenges or costs are rarely the main problem. The main reason behind their failures was the failed relationship between them and their marketing teams.

Jordan agrees with Nis and emphasizes the importance of having a cordial relationship with your marketing team. There was even a time when his team came up with an idea which they and the users liked. They had to scrap the idea, however, because their marketing team didn’t fully understand and there is absolutely no point in making a product which your marketing can’t sell.

Jordan believes that your marketing message must be able to accomplish one of the following three things:

• Make your clients money
• Save your clients money, or
• Make your clients’ lives better.

If you are unable to achieve at least one of the above with your marketing message, your product will definitely struggle.

Testing Concepts: Startups vs Enterprises

For start-ups, the product or idea goes from concept, passes through the launch/iteration phase before ending up at validated learning. For enterprises, however, it goes from concept, goes through some experiment and ends up in validated learning. So basically, the difference between them is the timeline in between that links ‘concept’ and ‘validated learning together.’

Daniella agrees with these timelines and thinks there is an important difference between the terms ‘experiment’ and ‘launch.’ When a company decides to experiment, it seems like a low-risk endeavor and people are pretty calm about it.

However, when a company decides to launch a product, it is seen to be much more complicated and stressful and can make people really worried.

Ellen, on the other hand, embraces the startup launch process as a way to energize everyone on the team. She’ll often use the term ‘launch’ just to get people excited. Simply the concept and thought of launching has the capability to get people riled up and this energy often boosts performance across the team.

Jordan thinks the trick is to find a way to express the customers’ needs in your pitch. Believe me, it is much harder for them to deny their customers than it is to say no to you.

Experimenting with pricing

My view on pricing is that it is the crudest and most subtle signal you can send to your customers. I believe that customers really don’t think about how they react to this signal, rather, they feel it. If you want to truly understand your customers and decide on what pricing strategy to use, then you are going to need to carry out experiments. Having an idea of what your users are ready and willing to pay for the product would assist you in determining the perfect price.

Here’s just one example of a pricing experiment. In this experiment, one can observe what goes on when you try purchasing a software product and come across multiple pricing options. What the company is doing is seeing what people are willing to pay for what they want. For instance, some people would opt to go for a cheaper option with less and inferior features while others would go for the more costly option with more top-notch features. These results are what the company will use to determine what price to slap on products they make in the future.

Jordan adds that everybody in a team should be involved in the pricing discussion and process. He sees pricing is a perfect example of experimenting as you can conclude your price based on users, time, data volume, transaction volume, etc. What is important is that you are able to use this price experimentation to find a pricing model that is directly based on value.

Ellen also thinks that if you’re a traditional marketer, then product and pricing should be equally important to you. This is especially important to know as most people, especially those from the engineering side tend to underestimate the importance of pricing and merely assume the price of a product solely based on costs.

Jordan has seen that the pricing experimentation method used depends on the company. You can decide to pick a random figure and see how your customers respond to it. What you ought to know is that sometimes when there isn’t enough data to develop a good pricing model, pricing can be subjective.

What will users gain from experiments?

Ellen spends about 50% of her time talking to customers. In addition to this, she casually and simply talked to people about the industry. In fact, on the day we had this discussion, Ellen was able to engage about four people in casual conversation and talked to them about her product. Her company has a director of analytics who is responsible for talking to every stakeholder group in addition to a user research coordinator who runs a major user initiative about every two weeks.

Daniella spends about 25% and 50% of her time talking to customers depending on what phase or stage the product or the experiment is in. To her, the biggest challenge her team often encounters is convincing stakeholders that their product is worth investing time and money in, without having enough user feedback.

Her company tests their products with different types of users within the company. These users differ in rank, geography, and position. Some will be internal while others will be client-facing. This way, they are able to know the true value of the product across a wide network of users. These tests can also help them discover niche groups of users that weren’t considered before.

Jordan adds that it is important to “have a level of vagueness.” That means being able to say that certain things performed better than others without feeling the need to state the statistical numbers. To do this, you’ll have to think more along the lines of business significance rather than statistical significance. For instance, there is no point talking about a 4% increase in certain metrics even though, technically, it is an increase, if it really is insignificant to your business.

Ellen states that A/B testing ought to be used for further optimization, rather than initial data gathering. Startups shouldn’t really emulate larger companies by running A/B tests early on to try and get minimal effects. They should consider their more limited time and resources and focus more on the big picture.

Sure, they can carry out the A/B testing but it should be done later on in the development process when they already have a solid product and a significant user audience.

Jordan thinks that if you’re able to present hard data to your stakeholders, you are on the right track. “Most CEOs can’t argue with facts.”

Daniella emphasizes that it is also important to consider each individual stakeholder. You need to understand that not everyone will be all too convinced with your presentation. That’s fine. What you need to do is focus on those you know are being convinced.

It is important to know what appeals to each individual stakeholder. Do they get swayed by an emotional presentation? Do they prefer charts with data?

These are the things you ought to know in order to know how you are pitching your findings to them. Sometimes all you might need to do is make a one hour presentation, while other times, you might require eight separate one-hour meetings if you want to convince each and every one of your stakeholders.

Ellen believes that good storytelling is key to selling your product. She says that if a product is still a concept, you need to create a story to help you sell the product and the users to the stakeholders. What you want to do is make your stakeholders empathize with your hypothetical user.

If your product has passed the concept stage and is further along the development process, you might need to talk about the potential changes you are making to your product and show them how it will bring more users to the company.

Nis has seen a company that spend almost twice as much money creating videos to present test results they did after running some experiments. “This is just the way the corporate world works.” Of course, data is really important but if you want to sell your product to people, you need to have a solid presentation. You need to bring the users’ voice to life for the stakeholders.

No matter how good we are at predicting and speculating, experiments can not only solidify your predictions but can unveil totally new perspectives as well.