Metrics have indisputable significance when it comes to Free and Open Source Software (FOSS) projects since it offers a situational awareness of the open source community, thus letting us identify what is working and what needs to be improved or fixed.
There are many questions that the community metrics will answer such as what are the problems and issues your community is having, how quickly is the community responding to different problems, is the community able to attract and retain contributors and so on. We know that community metrics is very important and good metrics can lead to action. But since most open source projects have a diverse community infrastructure with IRC, mailing lists, wikis, forums and the like, figuring out where to focus and deciding what to measure can be a daunting task. If you measure the wrong things, you end up with unintended results and this makes the entire process futile and sometimes detrimental.
However, if you are intending to get a multi-faceted view of your project community, then consider the following five metrics to track.
Activity
Activity metrics intends to measure the overall activity your community has undertaken and the changes occurred in it over time. The concept of activity metrics can be best understood with the help of an example in business context. A good activity metric in a tea shop will be the number of cups of tea sold in a year and in a salon, it will be the number of haircuts. In the context of the open source community, activity metrics focus on tracking all important kinds of activity the community is performing. Activity metrics reveal how much the community is doing. Note that defining and calculating activity metrics get complicated as the number of tasks performed by the community increases. In an open community, the number of commits gives insight into the volume of development effort. The number of tickets opened implies either the number of bugs reported or new features suggested. Similarly, the number of messages in mailing lists or forums reveals the extent of public discussion being held.
Size
Size metrics is concerned with measuring the size of the community. In other words, it is the total number of people participating in it. However, the size metrics vary according to the kind of participation. Generally, you focus on key community members who make a considerable amount of contribution. Highlighting the contributions of key project members is often considered a good practice. These active contributors may leave traces in the project repositories which make it easier to count them. In git repositories, the active contributors of the code can be counted by looking at the ‘author’ field. People participating in the resolution of tickets can be counted by viewing the contributors.
Performance
The above two metrics deal with measuring quantities of activities and contributors. The third is concerned with measuring how processes and people are performing. This type of metrics is called performance metrics. The time each process takes to be completed fall in this category of metrics. Performance metrics can enhance performance improvement, effectiveness, efficiency, and supports adequate levels of internal controls. In open source community, time to resolve or close tickets provides insight into how the project is responding to new data that demands action like fixing a reported bug or introducing a requested feature. Time taken for reviewing code tells how long it takes in implementing a proposed change. Performance metrics also throws light on how well the project is coping with pending work like the ratio of new to closed tickets and the backlog of unfinished code reviews.
Demographics
Demographic metrics measure the statistical aspects of the community population like the age. Contributors leave and enter communities over time and this is called the turnover. The success of a community depends on attracting new and efficient contributors while retaining old ones. If your project is matured enough, you can go on without any new members as long as you retain the old ones.
The turnover of a community reveals how interesting the community has appeared (i.e. to the newcomers) and how stable it has been (i.e. to the old members).
The age of the community can be seen as the time since the first members joined the community. Similarly, the age of each project is the length of time it has taken. The age also measures the time elapsed since each member has joined. This reveals the number of people the community has in different experience stages such as newbies and old-timers.
Diversity
Resilience of a community depends greatly on diversity of that community. It is believed that the resilience of the community is directly proportional to its diversity, i.e. the diversity of people and organizations participating in the community. Diversity can be used to bring strategic community advantage with proper quantitative and qualitative measurements linked with community objectives. For instance, losing an organization contributing 70% is much more destructive to the community than losing another which contributes only 5%. Diversity management becomes more systematic and efficient if it is metrics-driven. While measuring diversity, two factors are important. The Pony Factor is the minimum number of contributors performing 50% of the commits while Elephant Factor is the minimum number of companies whose employees perform 50% of the commits.
Bottom line
Keep in mind that the goals of each community is different and the metrics you decide upon should not be based on any general list of metrics, instead you need to understand the objectives of your community and identify and employ only those metrics which help you at least in some way to attain them.
