Discovery / Shaping's Ability: Qualitative Research

Great Product starts with a deep understanding of our clients/partners and their goals.

Qualitative methods such as interviews, ethnographic field studies, and (to some degree) usability tests are often more exploratory and seek to get a more in-depth understanding of the individual users’ or user group’s experiences, motivations, and everyday lives. In qualitative methods, the researcher accepts that they are (to some degree) a co-creator of research results, so the results are not completely objective and reproducible. The results of qualitative research are usually not expressed numerically, but rather as themes or categories that have occurred during the research.

The tools of the trade for those wielding the power of qualitative research include, but are not limited to:

• Interview Snapshot
• Survey tools

There are differing goals to qualitative research. This fact makes this being a single Ability tough. Let's explore why...

Different goals

First, based on what you are trying to know, there are different tactics to be used... just in qualitative research. This is an over-simplification, but the matrix I often use is: Generative vs Evaluative & Opportunity vs Solution. I call this the Product Value Learning Quadrants.

1. Generative => Opportunity
   • This is the entrepreneurial quadrant. Here we are trying to learn what desires we can meet. The result of this qualitative analysis is entire product lines or net-new plans.
   • Usually this is done by Product executives (Betting Table or Product Leadership). However, Product Marketing and the discovery-side of Product Management have direct responsibility as well. I'll call this group Product Discovery Strategist (PDS). PDS is often utilized here because the goal is to understand things like TAM, ARPU, CAC, TCO, and LTV (I'm using acronyms on purpose... if you aren't familiar with these terms that is totally OK, but it also means you aren't doing deliberate "trunk-level" Generative => Opportunity discovery. You may however be building an oppty tree and doing more "leaf-level" Generative => Opportunity discovery. Also, it is amazing, and a goal of mine (AR3) to have the full Product Trio (P3) [Product Management, Product Design, and Product Technical Leadership] be heavily involved with product strategy. It is just not the primary goal of P3.
2. Evaluative => Opportunity
   • This is when we have an opportunity we think is worth investment and we are looking to validate/invalidate it.
   • For "trunk-level" Oppties, evaluating the potential trunk-level-opportunities aka annual+ objectives, is still in the realm of PDS, but this is a failure if P3 isn't involved at this point. For "brand-and-leaf-level" Oppties aka quarterly objectives, that is the primary duty of the P3.
3. Generative => Solution
   • We have a strategy aka trunk-level-oppty. It is connected to a branch-or-leaf-level-oppty. Now it is time to ask "how might we achieve the objective". This is the diverge-converge, the input into ShapeUp, the assumption-testing, the seeding of the solutions we want to experiment with. Some call this "solution discovery". What are we going to build that gives us the best chance at: Continuously Delivering Value that Clients/Partners Love – and Works for the Business.
   • This should be exclusively in the hands of P3. PDS, specifically product executives should give feedback and may call a shot or two, but this is the realm of P3... primary reason for the existence of P3.
4. Evaluative => Solution
   • Same as above, but it is the actual testing, the actual results, the actual iteration and experimentation, the actual hypotheses, the actual answers.
   • Also, purely a P3 function.

Ultimate goal: Customer Knowledge

"Knowing the customer" is the goal. Qualitative research is an Ability, but only one of the Abilities that lead to being able to say "Yes" to "I know my customer".

Second, customer has so many meaning. Are we talking about the market, the buyer, the primary user, the secondary and tertiary users, the beneficiaries, both sides of a true or semi marketplace (in hiring there are the manager that needs a hire and job seekers that need jobs; in a market-based distribution channel there is the "mothership"/partner/franchisor and then there are the "children"/network members/franchisees; in employment there is the entire hierarchy from CEO/Owner to managers to employees). "Knowing the customer" is not a thing that is ever true... it is a constant quest. Not only because your "customer" means so many different things, but also because all of them are constantly changing based on time and their context.

Other ways to attain Customer Knowledge

Third, there are many ways to attain Customer Knowledge. There has to be... take the 4 goals (which is really more like 6 when you take into account trunk vs branch-and-leaf level oppties) and multiply them by the groupings of "customers", and you get an overwhelming amount of things we have to hold in our heads as we decide what is best to do. For some of these we want true qualitative... these are conversations with customers. Then you have quantitative-qualitative stuff where we ask questions in surveys like NPS. Then you have the quantitative stuff like actual experiments, results, etc. Then you have indirect customer knowledge from customer-facing teams... which is super valuable, but does not take into account the things discussed in the Qualitative and Quantitive tactics the best Product teams MUST use, so tho it is super valuable it is dangerous if over-depended-on.

So, customer knowledge cannot be an Ability itself can it? Gah. Tough question.

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to a squad when wielding this ability, and therefore I would put them in situations where they can employ this ability with only a small amount of guidance

• Can conduct user interviews following a script.
• Can prep qualitative notes for analysis

• Can synthesize learning such that the results have a minor influence on one of the four Product Value Learning Quadrants; Evaluative => Solution

• When evaluating if you "know our customers"... the answer is "I know who they are and a bit about them. I can speak to how they use our solutions. But I can't speak to their jobs-to-be-done"

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to a squad when wielding this ability, and therefore I would put them in situations where they can employ this ability, with no assistance as well as being a trusted active or passive mentor to others

If folks need to know about the general sentiment of clients, you are known to be a reliable source (if you are at this Milestone). You've conducted dozens of interviews, and have given synopses that have led to better Opportunity discovery and prioritization as well as solution shaping decisions.

• Can create a user interview script to deliver on product discovery and testing goals
• Can work with Product Trio to execute different kinds of qualitative research
• Execute a user interview without introducing bias
• Can analyze qualitative data with help
• Effectively presents research findings to the squad, communicating the research outcomes
• Has read and is actively experimenting with or has implemented something from one of the best books on this topic including but not limited to The Mom Test, Inspired, and/or Lean Startup.

• Understands and can use the appropriate tools to navigate the fact that those who aren't students of the ways of Product discovery often speak in terms of solutions. At Milestone 2 you know this is not a path to alignment, learning, or mutual success. At Milestone 2, about half of the conversations that need feature-request-to-actionable-insight translation – you can tease out the observations, the feelings they had, their unmet need and/or desired outcome.

• Can synthesize learning such that the results have a major influence on one of the Product Value Learning Quadrant; Evaluative => Solution
• Can synthesize learning such that the results have a minor influence on one of the Product Value Learning Quadrants; Generative => Solution
• Can synthesize learning such that the results have a minor influence on one of the Product Value Learning Quadrants; Evaluative => Opportunity (branch-and-leaf-level)

• When evaluating if you "know our customers"... the answer is "I know who they are well. I can speak to how they use our solutions and what results they are looking for. I'm becoming aware of their jobs-to-be-done, but I can't speak to their jobs-to-be-done with confidence."

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to multiple squads when wielding this ability, and therefore I would put them in situations where they can employ this ability as well as being considered an expert within this discipline

You are known to be a client and partner expert. PSMs and HX'ers regard you as someone who understand clients / partners on an equal or even deeper level than the client-facing teams. Depth and breadth. Squads and the Betting Table alike adore your Interview Snapshots (see Teresa Torres' Continuous Discovery Habits). They are the stuff of legend. Used as evidence to make small value enhancements all of the way up to multi-quarter big bets. Insight to Opportunity excavators consult you as they shape the opportunity landscape.

• Can select the best product research method for research questions to be answered, including strategic product discovery as well as tactical evaluation
• Can create a user testing script that reduces bias, delivers on product questions, and research goals.
• Can lead qualitative data analysis, selecting the best method for the level of analysis needed
• Effectively presents research findings to the product team, communicating the research outcomes
• Can recruit for multiple studies at one time, coordinating with CX on strategy
• Can right-size research strategy based on risk-profile of questions to be answered

• Understands and can use the appropriate tools to navigate the fact that those who aren't students of the ways of Product discovery often speak in terms of solutions. At Milestone 3 you are teaching others that this is not a path to alignment, learning, or mutual success. At Milestone 3, most of the conversations that need feature-request-to-actionable-insight translation – you can tease out the observations, the feelings they had, their unmet need and/or desired outcome. Folks come to you when there is a particularly hairy feature requests that need to be excavated into actionable insights.

• Can synthesize learning such that the results have a major influence on one of the Product Value Learning Quadrant; Evaluative => Solution
• Can synthesize learning such that the results have a major influence on one of the Product Value Learning Quadrants; Generative => Solution
• Can synthesize learning such that the results have a major influence on one of the Product Value Learning Quadrants; Evaluative => Opportunity (branch-and-leaf-level)
• Can synthesize learning such that the results have a major influence on one of the Product Value Learning Quadrants; Generative => Opportunity (branch-and-leaf-level)
• Can synthesize learning such that the results have a minor influence on one of the Product Value Learning Quadrants; Evaluative => Opportunity (trunk-level)
• Can synthesize learning such that the results have a minor influence on one of the Product Value Learning Quadrants; Generative => Opportunity (trunk-level)

• When evaluating if you "know our customers"... the answer is "I know who they are well. I can speak to how they use our solutions and what results they are looking for at an expert level. I'm aware of their jobs-to-be-done and can speak to their jobs-to-be-done with confidence."
• I'm desired to be "in the room" when we are talking about a new product line or plan.

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to a squad when wielding this ability, and therefore I would put them in situations where they can not only employ this ability but where they set the tone for this at the company level

You are the one that sets the tone for all qualitative research done at CareerPlug. You are likely responsible for crafting the details of Milestone's 1-3.

• Be a thought leader within the company of how to determine answers to product questions (both generative and evaluative)
• Can create a research plan for product discovery, partnering with product managers and designers.
• Effectively presents research findings to the executive team, communicating the research strategy, outcomes, and strategic impact on the product
• Can analyze qualitative data using coding via grounded theory, framework, discourse or content analysis.

• Teaches others how to synthesize learning such that the results have a major influence on all of the Product Value Learning Quadrants and their branch-vs-trunk variants

• When evaluating if you "know our customers"... the answer is "I know who they are well. I can speak to how they use our solutions and what results they are looking for at an expert level. I'm actively studying their jobs-to-be-done and can speak to the one that resonate and don't resonate with confidence."
• I'm required to be "in the room" when we are talking about a new product line or plan.

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to not just internal teams but the community/industry in general when wielding this ability, and they are recognized by the community/industry as an expert

• You're a community/industry leader in defining qualitative research within product design, seeking to improve it.
• You can conduct qualitative research and analysis to academic publishing standards.