A Guide To Validating Product Ideas With Quick And Simple Experiments

Advertisement

You probably know by now that you should speak with customers and test your idea before building a product. What you probably don’t know is that you might be making some of the most common mistakes when running your experiments.

Mistakes include testing the wrong aspect of your business, asking the wrong questions and neglecting to define a criterion for success. This article is your guide to designing quick, effective, low-cost experiments.

A Product With No Users After 180 Days

Four years ago, I had a great idea. What if there was a more visual way to learn about our world? Instead of typing search queries into a text field, what if we could share visual queries with our social network and get information about what we’re looking at? It could change the way people search! I went out, found my technical cofounder, and we started building a photo Q&A app. Being a designer, I naturally thought that the branding and user experience would make the product successful.

Six months later, after rigorous usability testing and refinement of the experience, we launched. Everyone flocked to the app and it blew up overnight. Just kidding. No one cared. It was that devastating moment after a great unveiling when all you hear are crickets chirping.

Confused and frustrated, I went back to the drawing board to determine why this was. My cofounder and I parted ways, and, left without technical expertise, I decided to step out and do some research by interviewing potential users of the app. After a few interviews, the root cause of the failed launched finally dawned on me. My beautifully designed solution did not solve a real human need. It took five days of interviews before I finally accepted this truth and slowly let go.

The good news for you is that you don’t need to go through the same pain and waste of time. I recently started working on another startup idea. This time, I followed a structured process to identify key risks and integrate customer feedback early on.

A Product With 16 Paying Customers After 24 Hours

I work with many entrepreneurs to help them build their companies, and they always ask me for feedback on the user experience of their Web and mobile apps. They express frustration with finding UX talent and want some quick advice on their products in the meantime. This happens so frequently that I decided to learn more about their difficulties and see what might solve their problems.

I specified what I was trying to learn by forming a hypothesis:

“Bootstrapped startup founders have trouble getting UX feedback because they have no reliable sources to turn to.”

To test this, I set a minimum criterion for what I would accept as validation to further explore this opportunity. I had enough confidence that this was a big problem to set a criterion as high as 6 out of 10. This means that 6 out of the 10 people I interviewed needed to indicate that this was a big enough problem in order for me to proceed.

By stating my beliefs up front, I held myself accountable to the results and reduced the influence of any retroactive biases. I knew that if these entrepreneurs already had reliable sources to turn to for UX feedback and that they were happy with them, then generating demand for an alternative solution would be much harder.

Design of my first experiment on the Experiment Board.1
Design of my first experiment on the Experiment Board162. (Large preview3)

In three hours of interviews, I was able to validate the pain point and the need for a better alternative. (You can watch a video walkthrough of my findings.) My next step was to test the riskiest assumption related to the solution that would solve this problem. Would they pay for an online service to get feedback from UX designers?

Instead of building a functioning marketplace with designer portfolios and payment functionality, or even wireframing anything, I simply set up a landing page with a price tag in the call to action to test whether visitors were willing to pay. This is called a pitch experiment. You can use QuickMVP4 to set up this kind of test.

Test if customers will pay for the service.5
Test if customers will pay for the service. (Large preview6)

Behind the landing page, I attached a form to gather information on what they needed help with. Within a few hours, 10 people had paid for the service, asking for UX feedback on their websites. Having validated the demand, I needed to fulfill my promise by delivering the service to the people who had paid.

Did not build any functionality; just a form to collect information.7
Did not build any functionality; just a form to collect information. (Large preview8)

Because this market is two-sided — with entrepreneurs and designers — I tested the demand side first to see whether the solution provided enough value to elicit payment. Then, I tested the supply side to learn what kind of work designers were looking for and whether they were willing to consult virtually.

Test the second side of the market: the supply side.9
Test the second side of the market: the supply side. (Large preview10)

To my surprise, the UX designers I spoke with had established clientele and were very picky about new clients, let alone wanting to consult with random startups online. But they all mentioned that they had been eager to take on any work when they were first starting out and looking for clients. So, I switched my focus to UX designers who are not yet established and are open to honing their skills by giving feedback online.

Armed with these insights, I iterated on my landing page to accommodate both sides of the market and proceeded to fulfill the demands of the customers I had accumulated. No wireframes. No code. Just a landing page and two forms!

A landing page that tests both sides of the market, simultaneously.11
A landing page that tests both sides of the market, simultaneously. (Large preview12)

Interest in this service can also be measured by their willingness to fill out the form. 13
Interest in this service can also be measured by their willingness to fill out the form. (Large preview14)

To simulate the back-end functionality, I emailed the requests to the UX designers, who would then respond with their feedback, which I would email back to the startup founders. Each transaction would take five minutes, and I did this over and over again with each customer until I could no longer handle the demand.

Do things that don’t scale in order to acquire your earliest customers and to identify business risks that you might overlook when implementing the technical aspects. This is called a concierge experiment. Once the manual labor has scaled to its limit, then write the code to open the bottleneck. Through this process, I was able to collect feedback on use cases, user expectations and ideas for improvements. This focused approach allowed for more informed iterations in a shorter span of time, without getting lost in wireframing much of the application up front.

Today, BetterUX.it15 is a service through which startup founders connect with UX designers for feedback on their websites and apps, and designers get paid for their expertise and feedback.

How To Create A Product That People Want

What did I do differently? The structured process of testing my assumptions saved me time and confirmed that each part of my work was actually creating value for end users. Below is a breakdown of the steps I took, so that you can do the same.

Should You Build Your Idea?

My first mistake with my first startup was assuming that others had the same problem that I experienced. This is a common assumption that many gloss over. Build products that scratch your own itch, right? But many entrepreneurs realize too late that the problems they’re trying to solve are not painful enough to sustain a business.

As product people, we often have many ideas bubbling in our heads. Before getting too excited, test them and decide which one is the most viable to pursue.

Design An Effective Experiment

To get started, break down your idea into testable elements. An effective experiment must clearly define these four elements:

  1. hypothesis,
  2. riskiest assumption,
  3. method,
  4. minimum criterion for success.

At Lean Startup Machine, we’ve created a tool called an Experiment Board162, which enables us to easily turn crazy ideas into effective experiments in a few minutes. As you go along, use the board as a framework to design your experiment and track progress. Refer to the templates provided on the board to quickly formulate your hypothesis, riskiest assumption, method and success criterion. You can also watch my video tutorial for more information on designing effective experiments.

Construct a Hypothesis

Every experiment starts with a hypothesis. Start by forming a customer-problem hypothesis. Once it is validated, you can go on to form a problem-solution hypothesis.

  1. Define your customer.
    Which customer segment experiences the most pain? They are your early adopters, and you should target them first. These are the people who have the problem you’re solving for, and they know they have the problem and are trying to solve it themselves and are dying for a better way! Most people have trouble identifying these customers. If you do, too, then just segment your potential customer base by level of pain and differentiating characteristics, such as lifestyle and environmental factors. Being specific will reduce the time it takes to run through experiment cycles; once you’ve tested against that one segment and found that the problem doesn’t resonate with them, you can quickly pivot to test another customer segment. In the long run, having a clear idea of who you’re building for will help you maintain a laser focus on what to prioritize and what to dismiss as noise.
  2. Define the problem.
    What problem do you believe you are solving for? Phrase it from your customer’s perspective. Too often, people phrase this from the perspective of their own lofty vision (“The Web needs to be more human”) or from a business point of view (“Customers don’t use our service enough”). Also, avoid being too broad (“People don’t recycle”). These mistakes will make your hypothesis hard to test with a specific customer, and you’ll find yourself testing for a sentiment or an opinion in interviews, rather than a solvable problem. If you have trouble, phrase the problem as if your friend was describing it you.
  3. Form a hypothesis.
    Brainstorm on a few customers and problems to consider all the possibilities. Then, combine the customer and problem that you want to focus on with this sentence: “I believe customer x has a problem achieving goal y.” You have just formed a testable hypothesis!

Identify Your Riskiest Assumption

Now that you have formed a customer-problem hypothesis, poke some holes and extract the riskiest assumption to be tested. Start by brainstorming on a few core assumptions. These are the assumptions that are central to the viability of your hypothesis or business. Think of an assumption as the behavior, mentality or action that needs to be true in order to validate the hypothesis.

Ask your team members, boss or friends to suggest any assumptions that you may have overlooked. After listing a few, identify the riskiest one. This is the core assumption that you are most uncertain about and have the least amount of data on. Testing the riskiest assumption first will speed up the experiment cycle. If the riskiest assumption is invalidated, then the hypothesis will be invalid, and you will have saved your company from going down the wrong path.

Choose a Method

After identifying the most critical aspect of your idea to test, determine how to test it. You could conduct three kinds of experiments before getting into wireframes. It’s best to start by gathering information firsthand through exploration. But you could choose a different method depending on your level of certainty and the data.

  1. Exploration
    Conduct qualitative interviews17 to verify and deepen your understanding of the problem. Even though you experience the problem yourself, you don’t know how big it is or who else has it. By conducting exploratory interviews first, you might realize that the opportunity isn’t as big as you had thought or that a bigger problem could be solved instead.
  2. Pitch
    Make sure the solution would actually provide value by selling the concept to customers before building the product. This will measure their level of determination to solve the problem for themselves. A potential customer not taking a certain action to use your service, like paying a small deposit or submitting an email address, indicates that the problem is not painful enough or that you haven’t found the right solution.
  3. Concierge
    Personally deliver the service to customers to test how satisfied they are with your solution. Did your value proposition meet their expectations? What was useful for them? What could have been done better? How likely are they to return or recommend the service to a friend? These are all insights you can discover in this step.

Set a Minimum Criterion for Success

Before running the experiment, decide up front what result will constitute success and what result will constitute failure. The minimum criterion for success is the weakest outcome you will accept to continue allocating resources and pursuing the solution. Factors like budget, opportunity cost, size of market, level of demand and business metrics all play into it.

The criterion is usually expressed as a fraction:

“I expect x number of people out of the y number of people in the experiment to exhibit behavior z.”

I like to set the criterion according to how big I think the problem is for that customer segment, and then determine how much revenue it would have to generate in order for me to keep working on it. At this point, statistical significance is not important; if your target customer segment is very specific, then testing with 10 of them is enough to start seeing a pattern.

Once you have validated the hypothesis with a small sample of the customer segment, then you can scale up the experiments to test with larger sample sizes or with other segments.

Run the Experiment

Once you have defined these elements, you are ready to run the experiment! Have team members look at your Experiment Board and confirm whether they agree with what you’re testing. This will hold you and the team accountable to the results, so that there are no subjective arguments afterwards.

Analyze the Results and Decide on Next Steps

After gathering data from your target customers, document the results and your learning on the Experiment Board. Did the results meet your criterion for success? If so, then your hypothesis was valid, and you can move forward to test the next risk with the product. If not, then you need to form a new hypothesis based on your learning to get closer to something that holds true. Track your progress over time on the Experiment Board to get a holistic picture of your validated learning and to continually make informed decisions.

Test and repeat. You’re on your way to creating a great product that people want!

(al, il)

Footnotes

  1. 1 http://www.smashingmagazine.com/wp-content/uploads/2014/04/1-Experiment-Board-large-preview-opt.jpg
  2. 2 http://www.javelin.com/experiment-board.html
  3. 3 http://www.smashingmagazine.com/wp-content/uploads/2014/04/1-Experiment-Board-large-preview-opt.jpg
  4. 4 http://quickmvp.com/
  5. 5 http://www.smashingmagazine.com/wp-content/uploads/2014/04/2-betterUX-large-preview-opt.jpg
  6. 6 http://www.smashingmagazine.com/wp-content/uploads/2014/04/2-betterUX-large-preview-opt.jpg
  7. 7 http://www.smashingmagazine.com/wp-content/uploads/2014/04/3-betterUX-form-large-preview-opt.jpg
  8. 8 http://www.smashingmagazine.com/wp-content/uploads/2014/04/3-betterUX-form-large-preview-opt.jpg
  9. 9 http://www.smashingmagazine.com/wp-content/uploads/2014/04/4-Experiment-Board-large-preview-opt.jpg
  10. 10 http://www.smashingmagazine.com/wp-content/uploads/2014/04/4-Experiment-Board-large-preview-opt.jpg
  11. 11 http://www.smashingmagazine.com/wp-content/uploads/2014/04/5-betterUX-large-preview-opt.jpg
  12. 12 http://www.smashingmagazine.com/wp-content/uploads/2014/04/5-betterUX-large-preview-opt.jpg
  13. 13 http://www.smashingmagazine.com/wp-content/uploads/2014/04/6-betterUX-form-large-preview-opt.jpg
  14. 14 http://www.smashingmagazine.com/wp-content/uploads/2014/04/6-betterUX-form-large-preview-opt.jpg
  15. 15 http://betterux.it/
  16. 16 http://www.javelin.com/experiment-board.html
  17. 17 http://uxceo.com/post/80877539095/quick-tips-for-effective-customer-interviews

↑ Back to topShare on Twitter

Grace is the Co-Founder of Javelin, enterprise software for product teams to test the viability of new ideas. She has trained executives from GE, ESPN, and American Express, in the process of rapid experimentation for product validation and is a frequent speaker on Lean UX. As a designer and entrepreneur, she is constantly exploring the intersection of creative production and impactful creation. She writes about her experience at uxceo.com.

Advertising
  1. 1

    “Within a few hours, 10 people had paid for the service, asking for UX feedback on their websites.”
    The thing here is, you already had a base of people for your site. Some nobody with no contacts isn’t going to get 10 people, yet alone 1 person.

    How do you test an MVP if you are a complete unknown in the field, and have zero contacts? I think those are the more valuable experiments for the majority of people.

    5
    • 2

      It’s a good question but I think it missed a simple logic. If you can’t find people to test your MVP with, how are you going to start a successful business with the same customers?

      Being able to test your MVP is a test in and of itself, if you are the right person to start this business or not.

      3
      • 3

        Apart from the obvious reasons, why are you not doing this yourself? I understand that some start-ups are targeting the start-up community and you see a little chance to do something that way rather than struggle yourself.

        But I imagine that I’m not the only person who is suspicious of why people claiming to have so many great methods are not doing this themselves.

        I have noticed that each thing like this always has a little history of how you struggled too, then came up with a nice method that you want to share with others that I imagine you know full well has a lot missing.

        If it’s so good, why are you not running something great?

        0
    • 4

      I agree Matt. This solves about half the equation for product testing, but there is a huge gaping hole in the “how do you get those first 10 people to sign up – WITHOUT accidentally over-promoting.”

      Its one thing to test a market this way, but the MAIN test is “can I get 500 people to even see this to get those 10 signups?” You still need to have effective marketing to get the concept out there to begin with. You might have the best idea ever and get 100 signups in 24 hours – but not if people don’t know it exists, leading to a false test of your hypothesis where the problem is the advertising – not the idea.

      I could also see the scenario where, due to accidentally effective advertising your concierge service is overwhelmed. Granted, this is a nice problem to have but at the same time you don’t want to strangle your idea in the crib because you can’t perform the service. The last thing you want is to have a killer idea and then disappoint a lot of people while they wait 3-4 months to get the actual product out there.

      It’s definitely an interesting process and I like the scientific method being applied this way, but its not a complete tool.

      0
    • 6

      That’s a good point Matt. An easy way to find your customers is to think of where they might hang out – meetup groups, online forums, LinkedIn groups, shopping mall etc. I believe there is no such thing as “no contact in the industry” anymore; put your social network to use or search for interest groups or conferences where your potential customers congregate.

      1
  2. 7

    For me, the defining moment when building my site was discovering Criticue. It uses a tic-for-tac system where you rate websites and they rate yours. I found it to be very enlightening, where the feedback was incredibly insightful and included things that I never would have thought of. Since then our engagement has been much higher and I now use it for almost everything that I make. Testing along the way solves major hiccups later on.

    0
  3. 8

    I wonder what is the metaphor for: ‘I have an idea, I should make it, it is going to be big!’, because that seems what the article is implying at the first part of the post.

    I have always tried to depend on my long-term experience of my field, and of course, the opinion of the people I’ve worked with alongside that experience.

    - Shawn

    0
    • 9

      Experience is great. Though it is easy to fall into the “knowing trap” and operate in the familiar way even when circumstances have changed.

      0
  4. 10

    Thanks for this Grace! I’m excited to give this a try!

    0
  5. 12

    Thanks for sharing your stories and advice. And thanks for the tips on how to approach creating a startup.

    0
  6. 13

    Inspiring and nicely explained.
    Do you use this methodology to decide about product features, or only for a product need?

    0
  7. 15

    Thanks for this post Grace, I found it very useful, it’s clearer for me now that I’m running experiments and I’m facing with some challenges along the way on how to do this or that, this post gives me a north to follow…

    :)

    Elco

    0
  8. 16

    Cool article, I didn’t know about betterUX! Thanks Grace!
    Also, I have developed a tool to validate Ideas and get the first customer. If you would like to have a preview, let me know!

    0
  9. 17

    Excellent article Grace!
    Great value for newly and experienced entrepreneurs.

    0
  10. 18

    Colin Hutchinson

    June 25, 2014 4:34 am

    The link on “watch my video tutorial” is malformed (Chrome OS X)

    0

Leave a Comment

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or else it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

↑ Back to top