Overview

This Jira gadget aggregates data across a project for defined categories of defects. The user can specify pass/fail criteria as part of the gadget settings to establish a threshold at which certain categories will provide a “fail” output, indicating that a category’s pass/fail threshold has not been met. This allows Jira users to keep track of particular issue categories on an individual basis in order to see which categories’ quality requirements are met. The gadget can also score across all categories to display an overall pass/fail score. The ultimate goal is to provide teams with a quick overview on how they are performing on specific issue categories, visually alerting them to a critical situation when expected quality thresholds are exceeded.

Background on the Calculations

Each defect category (type) is assigned a weight. These weights add up to 100%.

Each defect category gets a maximum category score based on that weight (e.g. if weight is 5%, the maximum score for that category is 5).

Each defect severity is assigned a weight. These weights can be chosen arbitrarily and indicate how large a deduction is made from the category’s score for each defect within the category.

The actual deduction depends on the category weight:

Category weight x Severity weight = score deduction per defect within the category

Which is then multiplied by the number of defects of this severity within the particular category to arrive at the score deduction for this category and severity:

Category weight x Severity weight x defect count = score deduction in the category

This is done for each severity level to determine the deductions for each severity and arrive at the final category score.

Example 1:

Category weight = 5%
Severity weight = 2 (i.e. a low severity)

This means a deduction of 0.1 will be made from the maximum possible score of that category for each defect of this type/severity. (0.05 x 2 = 0.1)

If there are, let’s say, 9 defects of this type and severity level, then the deduction would be 0.9 (0.05 x 2 x 9), which in our example would bring the category score from 5 down to 4.1.

Example 2:

Category weight = 7%
Severity weight = 15 (i.e. a high severity)

This means a deduction of 1.05 points will be made from the maximum possible score of that category for each defect of this type/severity. (0.07 x 15 = 1.05)

If there are, let’s say, 3 defects of this type and severity level, then the deduction would be 3.15 (0.07 x 15 x 3), which in our example would bring the category score from 7 down to 3.85.

Pass/Fail criteria

There are two ways a category can be considered “Passed” or “Failed”:

1.  The global threshold is set by default to 80%. Any category for which the score drops below this threshold will “fail“.

From the examples above:

• Example 1 category would pass (score of 4.1 out of max possible 5 would be 82% – i.e. above the threshold).
• Example 2 category would fail (score of 3.85 out of max possible 7 would be 55% – i.e. below the threshold).

2. A separate set of pass criteria can be established per severity. This takes into account the number of defects per category compared to the total number of defects within the filter. A category is considered “failed” if it contains more than a certain percentage of defects of a certain severity out of the grand total. This has an overriding effect on the score threshold calculation mentioned above.

Many users may choose to set the “Blocker” severity allowance to “0”, which means that any category that has any blocker defects will be considered “failed”, regardless of its actual score.
The “critical” defects level could be set to a low number, such as “10”, which means that any category that has “critical” defects accounting for more than 10% of the total defects will “fail”, regardless of the actual category score.
On the other hand, users may stipulate that the criteria for a “trivial” defect is “100”, which means that the percentage of “trivial” defects in this category, compared to the grand total of defects, does not override the pass/fail computation of the actual score. (In other words, even if the trivial defects in this category were to account for 100% of all the defects, only the score deductions in combination with the global threshold are determining the pass/fail condition.)

If this second method of calculation is not used, the criteria can be set to “100” for all severities. This means that even if 100% of all defects appear in one category & severity, it will not automatically fail that category. (Fail score due to score deduction method compared to global threshold still applies independently.)

IF # of defects of this severity < Passing criteria x total defects → PASS
IF # of defects of this severity > Passing criteria x total defects → FAIL

Letter Grades

We provide this option for teams that wish to use letter grades instead of or in addition to the pass/fail status. Grades will be calculated based on each category’s score compared to that category’s maximum possible score. (There is no overriding fail mechanism related to total defects.)

Note: The grades and pass/fail calculations are independent of each other. It is possible for a category to have a grade of B and fail, while another category has a grade of C and passes. This is due to the overriding effect of the pass criteria setting mentioned above, by which a category could have exceeded the allowance of defects within any one severity level, while scoring above the fail threshold or within the grading percentage of a higher grade.

How to configure and use the gadget

Before utilizing the gadget, the user needs to have the following:

• Saved custom JIRA filter that specifies which data will be graded.
  (Please note: There is a limitation of 10,000 issues per filter being returned.)

The following items for a scoring model should be decided ahead of time:

• Defect Weights
• Severity Weights
• Pass/Fail Criteria
• Letter Grade ranges (optional – default values provided)

The Configuration screen:

Filter

Type part of the filter name and select the filter from the list. (Only the first 10 search results will be shown).

 

Total threshold, %

Preset at 80. This value can be changed to raise or lower the pass/fail threshold of the category scores.

 

Defect Type Field

Select the JIRA field to be used for the defect types – i.e. categories.

Once a field is selected, all the possible values for that field will be shown below the selection. The desired weights need to be entered for each defect type. The total must add up to 100.

 

Severity Field

Select the JIRA field to be used for the defect severity.

Once a field is selected, all the possible values for that field will be shown below the selection. Enter the desired score deductions as well as the pass criteria for each severity level.

As described in the Calculations section above, the score deduction determines how heavy a defect weighs against a category’s score. The pass criteria determines the maximum number of defects allowed in each category as a percentage of the grand total of all defects within the filter.

 

Grades

A letter grading system can also be set up. Default values are provided. Additional grades may be added or removed, or labels such as “great”, “okay”, “bad”, etc. can be used to replace the grades.

This can be useful for teams which require a more granular breakdown than the pass/fail status based on the threshold alone.

 

Load from file / Save to file

This option can be used to save the selected configuration to a json file and to load it again for later reuse.

Once all the information needed for the calculations has been filled in, click Save. The gadget will perform the calculations and display the results.

 

Results screen

The results screen displays the number of defects in each category, grouped by severity.

The Total column shows the total score of each category (max score minus deductions for each defect).

The Grade column shows the letter grade based on the set criteria for the score.

The Pass/Fail column shows the pass/fail status of each category.

The filters at the top can be used to refine the results:

Grades: Select to show only letter grades, pass/fail status, or both. Default is “both”.

Status: Select to show only open, closed, or all defects. Default is “all defects”.

Category / Values: Some additional criteria can be selected to narrow down results being shown. For example, “Language” and the values “Italian” and “French” could be selected, so that only the defects for those languages are displayed. Click Apply to apply this filter.