Product Quality's Ability: (QA) Risk Assessment

Definition

The Risk Assessment super-power amplifies all of the powers that have preceded it, helping a tester make the most effective coverage decisions possible (along vectors of breadth and depth) at any given time and provide the best context to stakeholders about the issues they uncover.

Why

Given the testing space is theoretically infinite while time/resources are finite, testers must make risk-informed coverage decisions to find the bugs that matter most in context, not all bugs. Given the formula for Risk equals the “Likelihood of encountering the problem times Impact of the problem,” things with higher likelihood and higher impact are important to find fastest and avoid shipping. The lower likelihood and lower impact issue might be things we ship now and fix quickly. Very low likelihood or very low impact bugs are things we might never fix, so never finding them can be an acceptable outcome.

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to a squad when wielding this ability, and therefore I would put them in situations where they can employ this ability with only a small amount of guidance

Client-facing functional risk

Thinking about how clients use the system and things they’ll do/try/assume is critical to being able to identify high likelihood and high impact scenarios.

Measurement: Most bugs reported are issues that either need to be fixed prior to release or quickly after release. Bugs reported from Production by users and other stakeholders are lower-likelihood and lower-impact, and can generally be considered “acceptable” missed bugs for which there isn’t usually high urgency to fix.

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to a squad when wielding this ability, and therefore I would put them in situations where they can employ this ability, with no assistance as well as being a trusted active or passive mentor to others

Client-facing parafunctional risk (usability, learnability, discoverability, performance)

“Parafunctional” risk is the risk around “-ilities” (like usability, learnability, understandability, discoverability) and performance or perceived security (as opposed to actual physical security, network security, authentication/authorization, etc.). Identifying risk in these areas means a tester helps the team deliver not just functional software but better software that’s easier and more enjoyable for end-users to start and keep using; concerns are escalated to test management and/or product management and/or UX management.

Measurement: Stakeholders report that both design and build are positively impacted by tester questions and observations about parafunctional aspects of the software.

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to multiple squads when wielding this ability, and therefore I would put them in situations where they can employ this ability as well as being considered an expert within this discipline

Feasibility risk

Before features are developed at Lessonly, Product Management will socialize plans and business needs in advance. This represents an opportunity for Test to weigh in on feasibility risk, including things like complexity, testability, scalability, demo-ability, trainability (internal or external), difficulty to implement, etc..

Measurement: Stakeholders report that both design and build are positively impacted by tester questions and observations about feasibility aspects of the software; concerns are escalated to test management and/or product management and/or engineering management.

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to a squad when wielding this ability, and therefore I would put them in situations where they can not only employ this ability but where they set the tone for this at the company level

Lessonly risk (company image, legal exposure)

Not all risk is client-facing. Sometimes a feature, architectural decision, or business direction will represent a risk to Lessonly: our image, alignment with our mission/vision/values, or even legal exposure.

Measurement: Stakeholders report that both design and build are positively influenced by tester input in this area; concerns are escalated to test management and/or product management.

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to not just internal teams but the community/industry in general when wielding this ability, and they are recognized by the community/industry as an expert

Code risk (risk and coverage based on the area in the codebase)

Not all risk is client-facing. Sometimes a feature, architectural decision, or business direction will represent a risk to Lessonly (our image,

Measurement: Engineers report that PR reviews or issue research by Test result in the identification of code that’s functional but fails to conform to Lessonly guidelines or general good coding practices; concerns are escalated to test management and/or engineering management.