9 different types of Scrum metrics
Discover more from Winning Strategy
9 different types of Scrum metrics
Metrics to measure scrum team's performance
👋 Hello,
I’m Vibhor, and welcome to my weekly newsletter, the “Winning Strategy.” Each week I explore one question from you about agile, product, roles, processes, frameworks, career growth, working with humans and anything else that’s stressing you at your office. Send me your questions here, and in return, I’ll offer actionable, down-to-earth, and simple advice.
Q: How do you measure team productivity? Burndown charts and velocity are not reliable metrics. What other metrics would you suggest?
Thanks for the question!
This was the question that troubled me the most when I worked as a Scrum Master in large legacy organizations. Over time with the help of my colleagues, mentors and google, I collected 9 different metrics that, when used collectively as a system, provide a near-accurate measure of a scrum team’s productivity per sprint.
This can also be used to compare the performance of different scrum teams. The image below lists these 9+1 different metrics.
This post will be a live document. I’ll add to it as I learn new and effective tactics.
9 metrics to measure scrum team’s productivity
Team Capacity Factor - Team’s availability for the upcoming sprint
Team’s Work Capacity - Total number of story points “committed” during the Sprint
Focus Factor - How much of the work capacity is being used
Adopted Work (%) - Amount of scope creep pushed
Found Work (%) - Amount of scope creep pulled
Estimation Accuracy - The team's ability to correctly estimate the complexity of a piece of work during sprint planning
Forecast Accuracy - The team's ability to correctly estimate the amount of work they can complete during a sprint
Targeted Value Increment - Measure of a Scrum Team's improvement in their ability to deliver value
Win/Lose Record for Sprint - Helps the team understand how successful they are at meeting their commitments.
Let’s get into details
Prerequisite - Team’s Rolling Average Velocity
Before we get into the details of calculating and using the above metrics, you need your team’s “Rolling Average Velocity.” It is your team’s average velocity in the last 4-5 sprints. If your team is new and doesn’t have any historical data available to calculate velocity, I would recommend watching the below video. It will guide you on how to calculate a new team’s “initial” velocity.
Once you have your team’s “rolling average velocity” or “initial velocity,” you can dive into the team’s performance metrics below.
For the sake of education, let’s imagine that your team’s rolling average velocity is 26.5
1. Team Capacity Factor
Definition:
It is the availability of your team for the upcoming sprint. Calculated as a fraction of the team’s baseline capacity (ideal case scenario).
Example Scenario:
Let’s say your scrum team consists of 5 members (not including SM & PO), who work together in 2-week sprints.
Baseline Capacity = (number of working days in the Sprint) x (number of team members) x (available hours per day)
Example Baseline Capacity = 10 x 5 x 8 = 400 hours
Note: If some of your team members are regularly only available for less than 40 hours per week, then the sum total of individual regular hours available will become the baseline.
Team Capacity = (TeamMember#1 hours for 2 weeks) + (TeamMember#2 hours for 2 weeks) + (TeamMember#3 hours for 2 weeks) + (TeamMember#4 hours for 2 weeks) + (TeamMember#5 hours for 2 weeks)
Example Team Capacity = 45 + 65 + 80 + 80 + 25 = 295 hours
Team Capacity Factor = (Team Capacity)/(Baseline Capacity)
Example Team Capacity Factor = 295/400 = 0.74
You must calculate your team’s capacity factor before the team starts the new sprint. It should be based on a realistic assessment of the team's ability to deliver the work.
Don’t forget to take into account any leave, vacations, and things like meetings, planning sessions, and other non-work-related activities that may affect the team's availability during the Sprint.
2. Team’s Work Capacity
Definition:
The team’s work capacity is the total amount of work in story points that the team commits to for the upcoming Sprint. It helps the team understand how much work they can handle in the next sprint. Whether they are over-committing or under-committing. It also helps in identifying areas where the team can improve its efficiency.
Taking too much work can lead to burnout and decreased productivity. Taking too little can lead to underutilization of the total capacity available.
How to calculate:
To calculate your team’s work capacity, you need to estimate the amount of work (in story points) that can be completed during the next sprint. Use your team’s past performance, their knowledge of the work and other factors that might affect their ability to complete the work.
Here’s a simplified formula:
Work Capacity = (Total number of story points the team feels it can commit to) x (team capacity factor)
Example: Your Product Owner and the team together select the user stories that are important to deliver the sprint goal. Let’s say the total number of story points for the selected user stories is 45. Then,
Work Capacity = 45 x .74 = 33.3
This means your team can accept between 30 to 35 story points for the next sprint, taking the team’s availability into account. Where 0.74 is the team capacity factor (calculated above).
Best practices to use work capacity:
Review and adjust every week: Calculate work capacity before starting each sprint.
Communicate changes to stakeholders and keep them aligned for support.
Identify problems: Observe the fluctuation (increase or decrease) every sprint to pinpoint potential bottlenecks, impediments and over-commitment or underutilization of the team’s availability.
Note
The team’s average rolling velocity is 26.5 (shown in the prerequisite section above), less than the team’s work capacity of 33.3.
The question now arises, why is the team accepting more than its velocity?
This is completely normal as opposed to the common understanding.
Typically most Scrum Masters force the team to accept work equal to or below the average velocity. This, in most circumstances, proves to be the wrong approach as it underestimates the team’s ability to improve (the Focus Factor) as described below.
A better practice is to let the Product Owner and team select what is needed (realistically) to achieve the sprint goal.
3. Focus Factor
Definition:
Focus Factor measures how much of its available work capacity the team is using and whether the team is focusing on the most important work.
Focus Factor = (team’s rolling average velocity) / (team’s work capacity)
Why is it important?
It helps the team identify the areas where it can improve its productivity and efficiency.
Example: 26.5/33.5 = 0.794
The focus factor for a healthy team is usually around 0.8 (give or take a few decimals).
If the focus factor is much less (eg. .70), it indicates that the team is over-committing or not using the full team’s capacity.
If the focus factor is too high (e.g. >1.2), it usually means the team is under-committing or ignoring things they shouldn’t, like technical debt.
By monitoring your team’s Focus Factor over time, you can identify trends. This can help you make adjustments to the process (or whatever else) to improve your team’s performance.
How to use Focus Factor?
Use it as a continuous improvement tool
To set realistic expectations for the team. Asking the team to keep their focus factor around 0.8 can help them set realistic expectations
Regularly reviewing Focus Factor can help your team to stay on track and make adjustments as needed
4. % Adopted Work
Definition:
It measures the scope creep that happens in the middle of the Sprint. It makes the team understand how much work they are bringing in from future sprints, helping them identify planning or estimation issues.
% Adopted Work = (no. of adopted story points) / (original story points committed for the sprint) x 100
Why do we need it?
It helps the team measure the amount of scope creep that happens during a particular sprint. If Adopted work is > 20%, it indicates that the planning process must be improved or the team is unable to stick to their plan due to external pressure.
How to use % Adopted Work?
Track it over time to identify trends and make adjustments to your sprint planning as needed. For example, asking the team to spend more time on backlog refinement or improving communication with the Product Owner.
Share it with the stakeholders to make them understand the impact of scope creep.
5. % Found Work
Definition:
It measures the scope creep that happens due to unexpected but unavoidable work.
% Found Work = (no. of found story points) / (original story points committed for the sprint) x 100
Why do we need it?
It helps the team understand the amount of time that is being spent on unexpected work and how this unexpected work may affect the delivery of their planned work.
It also highlights that the team needs to improve their user story-splitting exercise.
Best Practices:
Keep track of all the unplanned work
Analyze the reason behind the found work
Review story splitting exercise
6. Estimation Accuracy
Definition:
Estimation accuracy reflects how accurately the team can estimate the amount of work they can complete during the sprint.
Estimation Accuracy = 1 - ∑(Estimate deltas) / (original total story points committed for the sprint) x 100
What are estimated deltas?
Suppose your team estimates a user story A to be 5 story points. But while working on it during the sprint, the team discovers it is more like an 8-pointer.
Then estimate delta for user story A is:
8 - 5 = 3
The estimate delta for user story A will remain the same if the original estimate is 8 and later found to be 5.
To calculate the accuracy of estimation, add all the estimate deltas for all user stories and use the formula above.
Why do we need it?
Helps the team identify areas and ways to improve user story estimation.
Best Practices:
Ensure that the team has a good understanding of the complexity of the work involved
Break down work items into smaller, more manageable user stories
Should remain close to 80% in healthy teams
If over 90%, the team is spending more time than needed on estimation
if less than 70%, it usually means that the team is experiencing a lot of external pressure, user stories are not well understood, or PO is not available
It may also mean that some members of the team are hoarding knowledge
7. Forecast Accuracy
Definition:
Forecast accuracy reflects how accurately the team can forecast the amount of work they can complete during the sprint.
Estimation Accuracy = (original total story points committed for the sprint) / [ (original total story points committed for the sprint) + (adopted story points) + (found story points)]
Why do we need it?
Helps the team identify how reliable their forecasts are.
Best Practices:
Ensure that the team has a good understanding of the complexity of the work involved
Break down work items into smaller, more manageable user stories
Healthy team’s forecast accuracy is around 80% (again)
100% accuracy means that Goodhart’s law is in effect. The team is targeting velocity
Below 75% means the team needs protection from forced work
8. Targeted Value Increment
Definition:
Targeted value increment is used to measure the improvement in team velocity over time. Whether increases or decreases, the goal is to measure the team’s ability to deliver more value over time.
Targeted Value Increment = (current sprint’s velocity) / (rolling average velocity) x 100
Best Practices:
Make sure your team uses a consistent process to estimate user stories
Keep historical data handy to identify trends
Analyze the root cause of the change
Always use TVI in conjunction with other metrics
9. Win/Lose Record
Definition:
Win/Lose record helps the team measure its ability to meet the sprint goal. It helps track if the team is able to deliver the planned work and manage uncertainty or unplanned work.
Win = 80% of committed work is accepted by the Product Owner, and the combined surprise work (found + adopted) remains at a level of 20% or less of the original forecast
Lose = Less than 75% of work is accepted
Progressing = Between 80% and 75% accepted
Best Practices:
Use a consistent definition of win/lose
Analyze the root cause of losses
Use it in conjunction with other metrics
Focus on continuous improvement
This is it 🙏
This post contains a great deal of information, so don't worry about attempting to remember it all.
Consider it more of a reference guide or a source of inspiration for when you're generating ideas. And when you do come up with brilliant ideas, hit me up!
I am always open to suggestions and feedback. Pls, let me know if I missed something or if something seems odd.
🌟Also, if you share parts of the framework on social media, I request you to please use proper credits. I won’t be as frequent in sharing this kind of content if I have to track copies of it floating around on social media.🌟
Your Questions!
If you want me to answer your questions in this newsletter, please send them my way using this link - Send me your questions.
Have a fulfilling and productive week 🙏
📌 Things I loved this Week
📚 Book - I am still listening to this amazing book on Audible - Sapiens by Yuval Noah Harari. It's a brief history of humankind and honestly, it's fascinating. I used to think history was boring, but I was wrong! It's actually super interesting to learn about how civilizations rose and fell, and how it all shaped our world today. Every chapter makes me feel smarter and more cultured. Highly recommend giving it a listen!
🎙 Podcast - "The Knowledge Project" by Neil Pasricha. Neil is an author with a bunch of bestsellers under his belt, including The Happiness Equation. In this chat, he talks about things like how doing little things can totally change how you feel, where your confidence comes from, and some cool habits you can use to kick anxiety out.
✍️ Quote of the Week
"Culture tends to argue that it forbids only that which is unnatural. But from a biological perspective, nothing is unnatural. Whatever is possible is by definition also natural. A truly unnatural behavior, one that goes against the laws of nature, simply cannot exist, so it would need no prohibition."
- Yuval Noah Harari, "Sapiens"
“I share things I wish I knew in the starting years of my career in the corporate world"
Vibhor Chandel