Pitfalls of Stack Ranking Software Engineers

“SAP boss wants to have bad employees identified” 

Sueddeutsche Zeitung (German newspaper)
December 7, 2023

“Stack Ranking Makes a Comeback”

        Virginia Backaitis

Stack Ranking for Software Enginners?

One of the announcements in the IT sector in Germany at the end of 2023 made headlines. SAP boss Christian Klein "obviously misses the idea of performance" and wants to categorise and evaluate employees as Performers, Achievers or Improvers [1].

This so-called stack ranking also seems to be making a comeback internationally [2]. There are enough reports that some digital companies have always used this method. Amazon is also explicitly mentioned in the article on the comeback.

The question now arises as to whether stack ranking is the right approach for software development and software developers. It is certainly possible, for example, to have project managers, product owners, even software architects or technical coordinators, as is practised at Amazon, evaluated by a committee, as their work can be assessed using metrics relating to product features and customer satisfaction.

But how should the committee for the evaluation of software developers be organised? We would like to draw attention in this and a subsequent blog to two pitfalls and show specifically which criteria should be taken into account in the stack ranking of software developers.

Back to the question of how the committee for the evaluation of software developers should be organised? Their work, the created software, is visible as source code, but difficult to understand for outsiders except for software architects or peers.

Other questions also arise: What is a top performer in this context? 

• The one who develops the most features? 
• The one who fixes the most bugs? 
• Or the person who writes the most code?

Even if one were to conceive reasonable measures of developer productivity, there is still the danger of a purely competitive evaluation. But we all know that software development is above all a kind of team sport.

Our blog on "Developer Contributions and Team Productivity" [3] shows a way out, 

1. by measuring the individual work of a developer as "Knowledge Islands" (KIs) and making it visible as such, so that a developer can claim sole "authorship" and be proud of it (which would increase individual developer productivity)
   
   
2. the individual developer's contributions to team resilience and knowledge sharing through bidirectional collaboration are recorded as "knowledge balances" (KBs) and
3. and a good mix of the two aspects based on a combination of these two criteria for each developer is taken into account

This means that a developer becomes a top performer if the person not only puts the greatest effort into further development, but also contributes to a comparable extent to knowledge sharing and thus to team motivation and team building.

Bidirectional collaboration can also take other forms, e.g. via review activities or pair programming, as mentioned in the blog above and the related blog Positive Impact of Developer Contributions (section "Taking the approach further")

Example of “co-valuation”

The following table refers to the developers of the Django open source web framework in 2023 who have contributed the most effort (Feat. Effort column) to the further development of features.

Table: "Top performer" candidates 

The following whitepaper [4] provides information on how we measure the development effort for features (or the bug-fix effort) with the DETANGLE analysis suite from Cape of Good Code.

Is Mariusz Felisiak a "top performer" with the greatest effort? He certainly worked 76% of his effort (Effort Knowledge Islands %) completely independently. But at 18%, the effort for bidirectional collaboration (Effort Knowledge Balances %) is just below the acceptable threshold of 20%. The value of 76% of Knowledge Island Effort is also definitely already in the critical range (over 60%) and any absence of Mariusz Felisiak already poses a project risk. Overall, he is a top performer, but the recommendation is to expand his collaboration and reduce his solo work.

A second category of developers consists of Nick Pope, Ben Lomax and Francesco Panico (fourth, fifth and eighth rows). They are in the top 10 contributing feature developers, but their Effort Knowledge Balance % scores are very low. As the score for collaboration has the same weight as the score for independent work, these developers are not considered "Top performers" but rather "Achievers" (or “Adequate Performers”)..

Jon Janzen (third row in the table) can certainly be rated as a “Top performer”, as he performs well to very well in both aspects with 48% Effort on Knowledge Islands (KIs) and 30% Effort on Knowledge Balances (KBs).

[0] Photo by Pixels (ROMAN ODINTSOV)

[1] SAP boss wants to identify bad employees

[2] Stack ranking Makes a Comeback

[3] Developer Contributions and Team Productivity

[4] Technical debt  - why the term causes more confusion than clarity and how to do it better