What I learnt from an experiment we had to park

Product Pill #4

It’s been a while since the last edition.

Since last time, there’s a bunch of lessons I have learnt, which I would love to share with those interested in building products. 

In today’s edition, I will discuss the lessons I learnt from an experiment idea we ended up deciding to park.

It’s never easy to let an idea that you’ve been working on go, but there are times when its necessary for the team to move in the right direction.

These are the 4 key lessons I’ve learnt from this experience, which I’ll dive into in more detail:

1. Use reach as an early proxy when deciding if an idea is worth pursuing 

2. Zoom out and don’t lose focus of the holistic user journey

3. Involve stakeholders early to ensure alignment on the problem you’re solving

4. Don’t rush to develop designs and high-fidelity prototypes without brainstorming alternative solutions

1. Use reach as an early proxy when deciding if an idea is worth pursuing 

Many of you may already be familiar with the RICE prioritisation model. RICE is a common acronym in the product world used to evaluate a project based on the reach, impact, confidence and effort associated with the project. Intercom has a great explanation of the RICE prioritisation model if you want to learn more about it here

RICE Prioritisation Model

It can often take some investigation and digging into the data (results of previous projects, funnel metrics data etc.) to determine the reach, impact and effort of a project. The data exploration work required can easily span over a few days to a week. Before digging deep into the data, use reach as a proxy to decide whether an idea is even worth entertaining further.

Think about how many eyeballs will be on the feature/experiment. What percentage of all users will see the feature/experiment you’re considering over a given time period? If the idea isn’t going to be seen by many users, chances are it’s going to be deprioritised, unless it has an extremely high impact and low effort, which is going to be pretty rare.

With this in mind, it makes sense to start at the top of the user funnel when brainstorming ideas to solve a problem. The top of the funnel will have the most eyeballs and greatest reach potential.

The idea we ended up having to park was an email communications experiment, with the aim of getting the user back to the platform using relevant messaging. We realised that sending an email wasn’t necessarily the best way of solving the problem, because it meant we were instantly limiting the amount of eyeballs on the idea from ~100% of users to ~20-30% of users who would likely open the email.

Using reach as an early proxy would have helped us refocus towards higher impact solutions.

2. Zoom out and don’t lose focus of the holistic user journey

Nothing exists in a vacuum in product. Regardless of whether its a new feature or an experiment, it’s always going to be part of a broader user journey.

Understand how the idea you’re proposing will impact the holistic user journey when you zoom out so you’re across any potential risks early on. Highlight the risks and make them clear to any stakeholders involved in the decision-making process so everyone understands the tradeoffs you’re making.

In reference to the experiment idea we had to scrap, it was a helpful exercise to take a step back and map out the current user journey vs the new user journey with the proposed idea.

Once we could see the bigger picture, we started to see some of the risks of including an additional email in the the user journey.

3. Involve stakeholders early to ensure alignment on the problem you’re solving

You should be getting buy-in from stakeholders at the problem discovery stage instead of starting to involve stakeholders in the solution discovery stage.

If key stakeholders aren’t in agreement that the problem you’re trying to solve is worthwhile focusing on, then you will likely face some form of pushback later down the track.

Sometimes sharing documentation on the problem may not be enough to get everyone on the same page. As old fashioned as it sounds, it helps to get everyone in the same room to get alignment on the problem. An hour or even 30 minutes of everyone’s time could potentially save hours going back and forth on the solution itself. If stakeholders begin to question the objective of the idea or experiment, it could be an indication that you’re not aligned on the problem.

Involving stakeholders during the solution discovery stage is also important and shouldn’t be overlooked. Once you have buy-in on the problem, its helpful to have multiple perspectives at the table when brainstorming solutions.

4. Don’t rush to develop designs and high-fidelity prototypes without brainstorming alternative solutions

As soon as you have developed high-fidelity designs for an idea, you will tend to bias that solution over other potential ideas because you become more familiar with the solution.

If you’ve already committed effort to creating high-fidelity designs, you may also become less inclined to consider alternative solutions and take feedback onboard.

Rushing to create high-fidelity designs without considering all the different ways of solving a problem can limit your creativity and narrow your perspective. Stick with low-fidelity prototypes until you’ve discussed the solution alternatives with key stakeholders and settled on the best approach.

Low fidelity prototypes help you stay focused on the problem

Even when you are clear on the solution, keep designs low-mid fidelity before you’ve got a clear understanding of the technical requirements and development effort to build the feature/experiment. You might end up having to change the design of a feature/experiment due to technical constraints, in which case you will end up wasting design resources.

Thanks for reading

If you’ve got this far, thanks for reading.

I would appreciate if you share this article around if you know someone who would find it helpful.

See you in the next edition!