Product Quality's Ability: Technical Investigation

Definition

Software testing is a technical investigation into a system designed to reveal quality-related information to stakeholders to help them make decisions. A good technical investigator is persistent, can isolate issues, and adapt their thinking from the broad and abstract to the highly specific.

Why

Developing and refining the ability to conduct technical investigations, beyond conformance to requirements, is at the heart of good software testing.

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to a squad when wielding this ability, and therefore I would put them in situations where they can employ this ability with only a small amount of guidance

UI-exposed errors/irregularities in behavior

This first milestone is table-stakes for a tester: can you manipulate a user interface and uncover issue?

Measurement: Consistently uncovers and reports issues with user-facing elements in a system, e.g. page layout, basic data input/retrieval, issues with workflows, and compatibility issues across different platforms (e.g. browsers, operating systems, mobile device form factors). In addition to identifying issues in delivered code, the tester should also consistently raise questions during the planning/design/build phase that anticipate issues (e.g. mobile form factors, cross-browser compatibility, how errors are surfaced, etc.).

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to a squad when wielding this ability, and therefore I would put them in situations where they can employ this ability, with no assistance as well as being a trusted active or passive mentor to others

Network/console errors

This second milestone is the next step for a tester: using the browser’s native developer tools to better understand errors or issues experienced in the UI.

Measurement: Issue reports consistently describe issues with user-facing elements in a system by including details about the network calls, responses, and javascript errors. In addition to identifying issues in delivered code, the tester should also consistently raise questions during the planning/design/build phase that anticipate issues (e.g. lost connectivity handling, request timeouts, errors on specific method calls, etc.).

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to multiple squads when wielding this ability, and therefore I would put them in situations where they can employ this ability as well as being considered an expert within this discipline

API-level issues

The third milestone involves being able to test at the API level, directly querying the backend system with calls designed to examine behaviors and data responses.

Measurement: Issue reports accurately differentiate between UI-facing issues where the root cause of issues with user-facing elements in a system are due to the API requests/responses the data requested and received) as opposed to an issue with the UI itself. In addition to identifying issues in delivered code, the tester should also consistently raise questions during the planning/design/build phase that anticipate issues (e.g. number or sequence of API requests, failure modes, etc.).

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to a squad when wielding this ability, and therefore I would put them in situations where they can not only employ this ability but where they set the tone for this at the company level

Database-level issues

The fourth milestone is to understand relational database structures and how they affect data added, modified, and retrieved from the system, and how they might impact performance. Basic querying ability and understanding of ERDs are key skills.

Measurement: Issue reports accurately identify those scenarios where the root cause of issues with API requests or performance are related to database issues, as well as database-enforced constraints (eg. uniqueness, compound keys, data types, etc.). In addition to identifying issues in delivered code, the tester should also consistently raise questions during the planning/design/build phase that anticipate issues (e.g. database calls that could have performance issues due to table size, number of joins, etc.).

I have observed this person showing a consistent, comfortable, continuous, and clear positive impact to not just internal teams but the community/industry in general when wielding this ability, and they are recognized by the community/industry as an expert

Code-level issues

The fifth milestone requires software development skills, applied to identifying not just error conditions but the root cause (faulty code) on the front-end system (e.g. HTML/CSS, JS, React/Redux, Ruby/Rails) and/or backend system (e.g. Node, Ruby gems, libraries, Heroku/AWS configuration, API, database queries, architecture)

Measurement: Issue reports consistently identify code root causes by line or method (as opposed to mental models of the code), including in PR reviews. In addition to identifying issues in delivered code, the tester should also consistently raise questions during the planning/design/build phase that anticipate issues (e.g. identifying anti-patterns, calling out dependency issues, etc.).