19a. Product Testing

Let’s talk about testing.

You: “Wait, testing? We’re talking about testing?! Haven’t we been talking about testing for days with all this build-measure-learn prototype talk?! More testing???!!!”

OK, OK. Take a deep breath. Calm down.

Yes, prototyping and experimenting with your product is a form of testing . . . but testing and prototyping are not the same things, and it’s important to understand the distinctions.

We’re going to cover that distinction as well as other topics like

• Alpha testing
• Beta testing
• Market testing

Without further ado, . . . here is testing, distilled.

Prototyping versus Testing — They Ain’t The Same

To elaborate on this, testing and prototyping are indeed different. Testing is more about refinement than it is about confirming a hypothesis or understanding product-market fit.

Prototyping — testing a hypothesis; making decisions; designing to meet customer needs; what should we build?

Testing — ensuring your product works as expected; refining; debugging; testing under various conditions; preparing for launch.

The distinction between prototyping and testing is perhaps a little subtle, but there is a distinction nonetheless.

Prototyping questions

• Is this product meeting customer needs?
• Should we incorporate this feature?
• Are we delivering value to the customer?

Testing questions

• Is the product functioning as expected under various conditions?
• Is the product reliable and robustly built?
• Is the product not bursting into flames?

Why Testing Is Important

Testing is your last chance to make sure your product isn’t a hot mess. Product-market ft matters, but so does quality.

There should come a time when you stop altering the product and instead focus on quality and robustness. In other words, at some point, you should stop exploring and start refining.

This step can be difficult for product developers.

Product developers don’t like to stop tweaking. They don’t like to stop improving. A good product developer is always learning and always considering how to modify and improve a product.

It’s very tempting to watch people test your product and then try to implement changes and improvements.

Also, there are no “testing police” who will come and arrest you if you try to implement new features after conducting quality testing. Technically, you could be changing things up to the very last possible minute.

But that’s not a great idea.

If you keep making changes up to the very last minute, chances are you’re going to launch something that’s going to fail in some manner.

If you’re building an app that reminds you of when to walk your dog, maybe it’s no big deal if the product fails. If you’re building a surgical tool or a rocket, you might want that thing to work.

Quality matters, which means testing matters.

Source

19b. Testing 1–2–3 (Testing Basics)

Let’s start with some testing basics.

Testing is done in an attempt to assure quality. If you work in NPD, you’ll become very familiar with QA or Quality Assurance.

The QA team’s job is to assure quality. If you don’t have a QA team in your organization, then you are the QA team.

People who have this picture hanging in their office have a 100% chance of not being the coolest person in the office.

Quality assurance is not to be confused with quality control, which might be a more familiar term for some of you. Quality control inspects for quality in products that are already in production. Quality assurance attempts to improve quality before a product launch.

Being Deliberate

Like anything else in product development (and like prototyping in particular), testing should be done deliberately during the product development process.

There should be a clear strategy and objective. You should know what you are trying to learn.

Yes, it’s fun just to take a product concept and break it, but unless you do it rigorously and scientifically, the results are not going to be trustworthy and informative.

Don’t just be like, “Let’s break this thing.”

Ivan Drago, the ultimate QA Manager.

When running testing, ask yourself, ‘What are we trying to learn?’

• Does the product function under normal working conditions?
• Does the product function in extreme conditions?
• Does the product pass accelerated lifecycle use testing?
• Does the product pass drop testing, vibration testing, or whatever another extreme environmental testing it needs to withstand?
• Does the product pass whatever certifications are required of it, be it environmental or other?
• Can your product handle scale — be it scaled manufacturing or scaled loading of users?
• Does the product integrate with other products as expected?

These questions are important, but perhaps I jumped over the more important, fundamental question: Should we even be doing any testing, or should we just launch?

Spoiler alert — I can’t tell you if you should be testing or not. It really depends on the situation, the product, and the market.

(Unless you’re working on a product that could kill someone. If that’s the situation, then yes, by all means, you should test your product extensively.)

Otherwise, notwithstanding governmental or industry-specific regulations, it will be the product development team’s responsibility to establish how much testing is sufficient.

When should you test?

In a perfect world, you would get a dress-rehearsal for everything. But, the world ain’t perfect.

The two biggest constraints to testing are probably time and money. And since time is money, well, the biggest constraint is money.

Testing takes time. Even worse, if you think testing takes time, that ain’t even the half of it; reacting to test results takes time.

There ain’t nothing quite like being close to a deadline and learning that the best solution you have come up with doesn’t really work. It’s a very special kind of panic.

Source

And it’s not just the time to run the test — it can be very costly to react to testing results. If testing happens toward the end of the NPD process, you’ve likely already invested heavily in your current product concept, perhaps in making machinery, tooling, or hours of engineering time building code.

Changing those things is not trivial. These changes are why very often the reaction to negative test results is the same: rerun the test, that can’t be right!

Some Testing Terminology

EVT — Engineering Validation Testing — Functional testing of the product to confirm that the performance meets expectations. These results could include things like how fast a computer starts-up, how reliably an app runs, or how well a car performs in a simulated crash.

DVT — Design Validation Testing — Testing things like cosmetic wear, overall durability (think phone drop test), trying to answer the question of whether the product will hold-up in the real-world environment.

PVT — Production Validation Testing — If making a hard good, physical product, this is actually running a small batch or “pilot run” in a production setting to confirm consistency from part to part. If making software, this might be testing scalability (whether your product can function if used by a large number of people).

False-positive — This result is when you get a signal that something is wrong, but it actually isn’t.

False-negative — This result is when you get a signal that something is OK, but it’s actually F’d.

19c. Alpha and Beta Testing

Let’s distill these two concepts and then talk about how/when/why to do them.

Alpha Testing — Testing your product internally.

Beta Testing — Testing your product, externally. Real users testing in a real environment.

That’s about it. That’s the difference. Boom . . . distilled.

Alpha testing

This test is internal, so one positive is that there is a reduced risk of publicly embarrassing yourself or harming your brand.

Alpha Testing should be done deliberately. Carefully establish the parameters of the test and choose clear metrics to measure. This measurement is not that different from the build-measure-learn. There should be a rigorous process to examine the results and glean insights from qualitative and quantitative information.

Beta Testing

A beta test is done with several customers and is meant to validate product functionality.

It is meant to identify bugs, defects, glitches, or other substandard qualities so that these can be remedied before launch.

Source

You might think that you can test the product internally to learn what you need to know. You would be wrong. There is simply no replicating the actual user and market environment inside a laboratory, studio, garage, or wherever the hell you may be developing your product.

“End-use is indeed complex, and there is no way it can be simulated in laboratories, where use is isolated from user mistakes, the competitive trashing of the concept, and objectives by those in the user firm or family whose work or life is disrupted by the change.” New Products Management, Chapter 16.

Other Types of Tests

Yes, there are other types of tests — many of them. There are internal beta tests. “Isn’t that an alpha test?” Sort of, but it typically involves the completed product, rather than just sub-elements of the product. If you work in Silicon Valley or a startup, you might hear the term ‘dogfooding’ (it’s an internal beta test) or ‘are the dogs eating the dog food?’ If that happens, just keep walking.

Beta Testing — When

Stupid answer: beta testing is done before launch and after prototyping.

In the real world, that answer isn’t all that helpful.

A more practical approach to schedule Beta testing is to work backward from launch. Consider how long it will take to conduct the testing, to organize and process the results, and then . . . this part is important . . . how much time it might take to react to the results. Obviously, the amount of time needed for these steps varies depending on product complexity.

The good thing about this approach is that it generally can be applied to all industries and product types. The negative is that you’ll see just how much time a thorough beta test can take. You’ll likely be trading-off design time. NPD always entails trade-offs between quality, scope, time, and money.

Beta Testing — How and Who

Ahhh, this is good stuff . . . how to run a proper beta test and who to select for testing.

Ultimately these two factors — the how and why — will influence the effectiveness of beta testing more than anything. In other words, proceed carefully.

19d. Beta Testing — How and Who

Answering the “who” question requires revisiting some of the stuff we talked about in the chapter on marketing. Things like demographics and market segments.

Without question, your beta test should be targeted toward the intended consumer of your product. The key idea here is that you are trying to get people to use the product as it will be used in the market. Nobody is better suited to behave like your customers than your actual customers.

That said, testing is about finding bugs and controlling quality. Contrast that with prototyping where you’re really trying to design the product to meet customer needs better. To some extent, you just need some warm bodies for testing. Enthusiastic, warm bodies who can communicate well. A mute zombie would be the opposite of what you want. Don’t use mute zombies for beta tests.

Poor candidates for beta testing because they can’t effectively verbalize constructive feedback

There are actually a lot of websites dedicated to helping you find people to conduct beta tests. Google “websites for finding beta testers” and off you go.

How to Beta Test

First of all, a disclaimer: I’m hesitant to list specific testing tools because they will become obsolete pretty quickly. There are many platforms and software for creating digital beta tests. Again, use “The Google” or whatever search engine you prefer to find these tools.

Here are the basics on how to run a beta test, distilled as simply as can be. (Yes, there are more questions here than answers.)

• Set goals. Yes, improving quality is a goal. Anything else?
• Find your tools. You’ll typically need to select specific apps or software to communicate with testers, to distribute the product, and to gather feedback.
• Project management. Establish the timeline. Coordinate with the team.
• Measure. Collect information in an organized manner.
• Respond deliberately. Carefully consider what actions must be taken based on results.

Other questions you may want to consider:

• Should the test be supervised or unsupervised?
• How much should we inform the users about the product prior to the test?
• How will we capture and measure results?
• Did the product test in one form? Variants? Against competitors? A/B Testing?

19e. Market Testing and Testing Summary

Alpha and Beta testing are pretty technical in nature. They are generally in the jurisdiction of engineering, R&D, or whatever you call the technical side of the NPD team.

“What about marketing and sales? Shouldn’t those assholes do some testing?!” — A technical person who just completed a Beta test and has to pull an all-nighter to fix a bunch of issues.

Yes, they probably should.

Let’s talk about Market Testing. First, some definitions:

Product Use Test — A super generic catch-all term, probably what you’re thinking of, when a product is tested in the field or with actual users, and more of the technical testing.

Market Acceptance Testing — More specific way of describing testing that is meant to test if a certain group of customers accepts a product; marketing/sales testing.

Simulated Test Market — Using a small group of customers to represent a market and testing your product with them, generally focusing on Marketing-type learning

Test Market — When you actually launch your product in a small, controlled market, with the general aim of learning about Marketing-type stuff.

Market Testing Distilled

Market testing is a really broad concept. In the most general meaning, a market test is an experiment conducted before the actual launch of the product where you learn things about the market.

Contrast that with a beta test where you are specifically trying to refine the product. The market test isn’t as focused on the product — it’s focused on the market and marketing.

In other words, a market test is specifically meant to help you refine the marketing mix: price, promotion, product, and placement. (That’s right, the four P’s — hopefully you knew them!)

Here are some of the questions you might answer through market testing:

• What features and benefits should we be messaging in advertising?
• How do we answer the tough questions from potential customers?
• How will competitors try to discredit this product, and how will we respond?
• What price should we sell this thing for?
• Do we need to augment the product somehow (warranty, service, and support)?
• Should we adjust our forecast due to new insights about demand?
• Should we be selling this product to a different target?

The mentality in a market test is basically this: ‘We know we can’t change the product much at all at this point, but we can change how we talk about it, augment it, and “dress it up” in the market.’

It’s probably intimidating to think about running an entire “Market Test,” but market tests can come in all shapes and sizes.

Just pitching your product to a friend, a spouse, a barista, a stranger in an elevator, or the DoorDash guy as he hands you your Ramen still sorta/kinda qualifies as doing a market test.

Of course, market testing can be huge, as well. It can mean launching the entire product into a test market, such as a particular city or region. That’s obviously a big and complicated project, and there’s no good way to distill it here. Suffice to say that if you’re involved in executing a test market launch, please do your homework!

Finally, Testing In Summary

• To test or not: it depends on cost, time, and ability to react with product changes. Like most things in new product development, it depends.
• Testing is like prototyping in that it’s about learning.
• Testing is unlike prototyping in that its purpose is refinement and debugging rather than gauging if customer needs are being met.
• Alpha Testing: internal (inside the company).
• Beta testing: external with actual target users (outside the company).
• Market testing: learning more about the four P’s before launch.

Best Resources for Testing

New Products Management — This is a textbook that was written at least 20 years ago, but it covers testing much more extensively than many other books written more recently.

Winning at New Products — There are 10–20 solid pages on testing here. I like it for a quick read, but it doesn’t dive as deep as New Products Management.

6 Key Steps to Follow For Beta Testing Your Products — Hacker Noon