How A Metrics-Driven Culture Makes Great Engineering Leaders

A Must-Read Case Study On Engineering Leadership by Athenian, Faire & Google

Topics Discussed

True Value of Engineering

What the true value of engineering is to your customers and organization.

Best Serving Your Team

How engineering leaders should strive to lead their team by serving.

High Importance of Metrics

Why collecting data and analyzing metrics is crucial to a successful engineering operation.

Getting Better Work Done

How to get the most out of your engineering organization.

Supercharged Empowerment by Athenian

How using Athenian has empowered Paul to gain new insights into his company.
INDEX
1. INTRODUCTION

Executive Summary

Paul Poppert is the Head of Engineering at Faire, a hyper-growth startup bringing wholesale online and into a marketplace. In the past two years his team has grown from 20 to 90 engineers, and will be 120 by the end of 2020.

Paul has a mixed-discipline background, coming from both electrical and software engineering. He started his career in the 90s with Motorola, and grew as an engineering leader in parallel with the rise of software development and the internet. A vast background in manufacturing, an established industry with many parallels to software development, has given him an intricate understanding in the optimization of engineering processes.

He stresses two main factors as the recipe for a success engineering organization: metrics and culture.

Paul started a conversation with Athenian back in July of 2020. He immediately impressed us with the knowledge he had collected and first-hand experienced surrounding engineering leadership. We continue to learn a lot from him, and found it important to share his impressions on the topic with others leaders in the field.

This knowledge is around the areas of engineering leadership: the methodologies he uses, the mental models he has built, and how metrics have given him an edge in building an elite-performing engineering culture.

Together with Michael Winsberg, an engineering leader at Google Cloud, Athenian conducted a series of conversations with Paul about:

  • How an engineering leader should serve their team
  • How to get real engineering work done
  • What the true value of engineering is to your organization and customers
  • Why collecting metrics and analyzing data is critical to a successful operation
  • How Athenian has empowered him to gain new insights into his company

With a thank you to:

Paul Poppert

for being an awesome engineering leader that continues to teach us how awesome companies work

Michael Winser

for leading an insightful conversation at a company dedicated to improving the developer space

Everyone at Athenian

for working so hard to making amazing things happen

2. Paul’s Story

Journey To A High-Performing Engineering Leader

Paul started working with Motorola in 1996, writing software for charging algorithms. Within eight years, he would become the tech lead for Motorola’s battery software, heading 40+ engineers. Many of his early leadership skills came from Motorola, where organizational processes in quality and efficiency in manufacturing were pushed to the limits. His hunger for quicker and more tangible impacts grew alongside Motorola, as it would prove too big and deep rooted in old processes to adopt rising concepts (Extreme Programming, Agile, Scrum…)

And so he joined Sandvine, a smaller but sizable company doing 70 million in sales for networking equipment. They couldn’t just grow by adding more people, and so needed to become operationally efficient for profitability. Paul would run manufacturing. Daunted at first, with no background in inventory or finance, he was given the confidence by the CEO who understood they needed someone with an expertise in analyzing, improving, and driving quality. Paul had to step up once again to figure out how engineering operations could be improved. This was not just the technical process, but also the people. In the years after, he brought inventory down by 5x while revenues steadily increased. Later, he would take charge of the test engineering organization, which helped him understand how to test and measure quality.

The main question Paul continuously asked himself was: “How do I serve my team?” He consistently thought about operational frameworks that allowed him to measure, analyze, and improve. The goal wasn’t for him to personally have control, but rather for the environment to be controlled. And so he learned that, as an engineering leader, operations had to run smoothly enough for his daily activities to not be critical to the functioning of the company. 

The goal wasn’t for him to personally have control, but rather for the environment to be controlled.

Paul left Sandvine for Faire in 2019, an online wholesale marketplace. At the time, it had only 14 engineers with no engineering managers. Even though he had been in manufacturing for so long, he instinctively trusted that the same processes could apply to software. Roughly a year and a half later, his organization has grown 6x to 90 engineers pushing 15 software updates per day. By the end of 2020, Faire will have 120 engineers. Paul describes his team as “very high functioning, very high output, and scalable”.

Roughly a year and a half later, his organization has grown 6x to 90 engineers pushing 15 software updates per day.
3. Value Stream

The Importance Of Conscious Optimization

Before Paul joined Faire, he had moderate opinions on how engineering teams should be run. But he now strongly believes that goals are highly specific to the company. Engineering leaders need to make a conscious decision on not only where to optimize, but also how so and how to track.

Aligned with the philosophy of Athenian, he mentions that purely technical metrics such as lines of code written per engineer are not a representative view of the throughput of an engineering team. These types of data points can be easily skewed, misinterpreted, or prone to self-selection bias to give any outcome you want.

What he really cares about is the value stream, which takes into consideration the value proposition delivered to the end-user. At Athenian, we believe this should always answer the question: “How much time and effort are we spending on this feature, and what is the true value and outcome for our customers?” If these do not line up, there is room for improvement in the engineering process.

"How much time and effort are we spending on this feature, and what is the true value and outcome for our customers?"

At Faire, they do not only measure what they are optimizing for right now. They strongly believe that more data and metrics will always allow for better decision making in the long run, as you can not anticipate today what you might need in the future.

4. Six Pillars of Metrics

The Absolutely Essential Metric Groups

Six different groups of metrics are consistently tracked at Faire. At Athenian, we were very glad to hear they were almost identical to our own seven pillars. These are, with no special regards to order:

  1. Velocity: how long does it take to get comparable work done?
  2. Output: what work are we delivering? 
  3. Quality: are our end users getting quality work?
  4. Impact: how is our work impacting the core business objectives?
  5. Census: how are we structured to do our work?
  6. Testing: what is the impact of testing?
  7. Performance: are we meeting our expected service levels?
The Metric Groups tracked at Athenian, with examples of individual metrics.

Velocity

The first topic that Paul touched upon in our conversations was the relation between output and velocity. He mentions how Velocity is different from speed in engineering because it also considers direction. As opposed to a concept such as Story Points, which measures linear outputs, it sees the ability to deliver units of work within a set amount of time. 

Paul makes the distinction here that it’s not always about how much you have delivered, but also how much you could deliver. According to the flow of the business and engineering landscape, it is important for engineering leaders to be able to predict lead times based on specific criteria. Athenian’s Engineering Metrics Dashboard is built to help in these cases, allowing you to filter on specific teams, repositories, and tags.

According to the flow of the business and engineering landscape, it is important for engineering leaders to be able to predict lead times based on specific criteria.

Quality

At Faire, output is always measured in terms of quality. The main stress point is how the user experience changes with every update, for example by measuring the number of defects. While there are tools that can classify these metrics, Paul regards it as important to have a data analytics layer like Athenian on top of all of these to get a holistic view of the process. Interacting with each of the moving parts is important in understanding how and why things are engineered in the manner they are. He also mentions that Athenian was the first tool he saw that allowed him to properly measure Velocity, with no added overhead.

Census and Performance

At Faire, the human side of the engineering organization is considered incredibly important. Because they have a two-sided marketplace, they have to take into account engineering practices for both supply and demand. This means asking questions such as:

  • What is my ratio of engineering managers to UI specialists?
  • What is the contribution margin of investing in more engineers on the demand side?
  • How many engineers should I invest in this feature, taking into account it delays work on the supply side?

For product performance measuring, Paul uses tools such as DataDog to better understand the technicalities in areas such as latency and uptime. In analyzing these metrics, he again stresses the importance of ‘Conscious Optimization’. He does not believe in analyzing technicalities at the level of the individual. The question is never “How did Lisa’s code contribute to latency?”, but rather:

  • How does adding an extra engineer contribute to latency?
  • How many lines of code are written before a noticeable effect on latency?
  • How many engineers make for an efficient code review process in a team?

This lines up with the Athenian philosophy: the engineering team is a spectrum and must be analyzed at the process level. Stack ranking employees is ultimately detrimental to the company culture. It acts as an incentive to create quantity rather than quality. You don’t want your engineers working to achieve metrics, but rather always focused on developing great software. There is always a lot of unseen work that goes into the creative process of writing code. In topics like this, Paul’s vast background in manufacturing shines through.

You don’t want your engineers working to achieve metrics, but rather always focused on developing great software. There is always a lot of unseen work that goes into the creative process of writing code.
5. TRANSPARENCY

Why You Benefit From More Information

Paul strongly believes in transparency in the engineering hierarchy. If you measure something, you need to explain to people how and why. Furthermore, you should also show them how you’re using that input. This contributes to a positive work environment, where engineers understand that performance is being tracked rather than unique people.

If you measure something, you need to explain to people how and why.

Athenian allows engineering leaders to not only synthesize information for themselves, but to also share it with the team in a cohesive and objective manner. There are very few things that make sense for engineering managers to keep secret. It can be a great motivator to teams to see how umbrella metrics like lead time or cycle time improve.

Furthermore, by consistently tracking metrics in Athenian you avoid recency biases. Examples of these are:

  • A pandemic taking over the world and demotivating employees
  • Engineering leaders scrambling for metrics when they have a problem they need to solve

It is not enough to take a snapshot in time, for example with an employee engagement survey. Rather, Paul tells us that you need to understand what happens on a weekly basis. This can help you understand topics such as:

  • What teams are currently underutilized?
  • What teams have a very high output and how is this correlated to their impact?
  • What operational inefficiencies do we see when things get hectic?
It is not enough to take a snapshot in time… you need to understand what happens on a weekly basis.

This also avoids engineering leaders convincing themselves that they need to make radical changes such as reorganizing the organization. Rather, change can be seen in the lens of continuous improvement over time. Part by part, and engineer by engineer.

6. MISSION-BASED TEAMS

Work On Features, Not Roles

One of the main themes throughout Faire’s engineering organization is the presence of mission-based teams, called pods. These are highly flexible teams assembled to work on certain features or high-level concepts, rather than in specific domains. This allows them to quickly be created and shut down as needed, and staying lean helps them change according to customer needs. By doing this, you push decision making to those who have the most context: those who are actually working on the problem every single day.

Push decision making to those who have the most context: those who are actually working on the problem every single day.

By using a pod-like system, the leadership and the engineering teams can more closely collaborate by tackling parallel areas. While leadership focuses on cross-communicating with divisions such as product management and design, the engineers make decisions on technicalities. 

Paul mentions how engineering leaders need to consistently filter the information they receive based on the needs and scope of the project. Being able to quickly adjust these filters on the go in Athenian’s dashboard makes it easier to experiment with different directions. Better input (data) leads to better questions.

Better input (data) leads to better questions.

Pods can also include areas such as growth (for example expansion) and market segment (for example retail). It is part of the job of an engineering leader to help integrate the engineering team with the rest of the organization, so the organism can work as a cohesive structure.

7. USING ATHENIAN

How An Engineering Metrics Dashboard Improves Your Leadership

Paul believes that Athenian allows him to get a much better understanding of the whole product development process: tracking a software idea from concept to launch. He can properly map the value stream, which he argues is the key to empowering an engineering leader to make better decisions. 

Engineering metrics and insights not only give him vision, but also allow him to concretely act on this data. For example by:

  • Finding the bottlenecked stages of software development to increase Velocity
  • Discovering how long it takes for reviews to merge before launch
  • Calculating how many working hours per engineer it takes to go live on a new feature

He believes that once you dive into metrics, your attitude towards shipping software evolves. He mentions that new features at Faire take about 14 days from concept to launch, of which only 4 are coding. With Athenian, engineering leaders can beforehand make decisions on topics such as:

  • How they run their CI/CD process to make sure they don’t get stuck in merge
  • How early they should get their first code reviews pushed through
  • What areas of architecture they should stay on top of when working with pods of a certain size
Once you dive into metrics, your attitude towards shipping software evolves.

Athenian also helps him build an MVP quicker. It gives an aggregate understanding of how long it takes to deploy certain features based on its known characteristics. This is the essence of the continuous delivery mindset. It allows for great engineering organizations such as Paul’s to focus on continuously delivering software updates 15 times a day. And by doing so, you end up delivering a lot more value to your customers.

Athenian allows for the ‘Conscious Optimization’ of how projects and features contribute to the software delivery pipeline. Paul recently held a Live Trade Fair for the first time, which took a massive code deployment. Throughout this process, he had Athenian to help answer questions such as “Why is my merge time increasing?”. Seeing that his test time had not increased, he instantly knew that this wasn’t part of the problem. Tracking metrics allows for quicker problem-solving. 

Tracking metrics allows for quicker problem-solving.

Rolling out engineering metrics to Faire is done in phases by Paul. He considers the first to just be ‘Exposure’. Here, data is brought to the light, measured, and analyzed for 90 days. Conversations can be started with engineering managers surrounding the current situation, without the pressure to overthrow current processes. He believes it important to not forcibly alter the DNA of Faire, and to rather have it happen organically.

Over the last months of using Athenian, Paul has taken actions such as improving the review process by adding a second phase of reviewers. By having Athenian’s insight into these metrics, he concluded that the slight increase in review time ends up benefiting the value stream as well as other areas of shipping code. This is just one of the ways in which we can approach metrics: not everything has to be focused around number reduction. Rather, having a high-level of insight allows us to understand the engineering process from a holistic standpoint and make context-specific decisions.

Having a high-level of insight allows us to understand the engineering process from a holistic standpoint and make context-specific decisions.

Furthermore, Paul believes that Athenian adds to the culture rather than changing it. It allows everyone to speak the same language when discussing certain topics, with the mental comfort of objectivity in knowing that lead times are improving. If you have a bad day because you slipped in a muddy puddle, you are more likely to view things in a negative light. Having Athenian as the data layer on top of other tools helps separate personal emotions from the engineering process. 

The relationship between Engineering at Faire and other departments such as Executive and Finance has also improved. Athenian allows data to back up blanket statements such as “Engineering is doing great!”, increasing confidence and transparency throughout the organization.

Athenian allows data to back up blanket statements such as “Engineering is doing great!”.

Paul believes that engineering metrics and insights are still in its early stages, and will continue to grow in importance and complexity in the future as the tech progresses. He is incredibly happy with Athenian, because he sees it as an empowerment tool rather than a ‘Big Brother’. Instead of creating a culture where people feel like it’s their specific data being analyzed, decisions are made around the functioning of the engineering organism as a whole. 

He leaves us with one important thought:

“When the metrics become the goal, it loses all value.”

I am ready to get started!

Receive your getting started instructions via email.

Let me see that in action!

Get on a personal demo with us, and we'll answer all your questions.