Process of metric approval
The CHAOSS Metrics Committee defines implementation-agnostic metrics for assessing open source communities' health and sustainability. The CHAOSS Metrics Committee's goals are to establish implementation-agnostic metrics for measuring community activity, contributions, and health; and optionally produce standardized metric exchange formats, detailed use cases, models, or recommendations to analyze specific issues in the industry/OSS world.
We already have many metrics listed under various working groups but you can also propose your own metric if you don't find listed in the existing metrics. Here is the list of all the existing metrics within CHAOSS
There are different working groups that foster around their focus areas. So once you have a valid plan for your metric, we find a working group that is best aligned and has an interest in the metric.
We add the metric to our Metrics Tracking Sheet
Someone writes a draft for the metric using our Template for Metric using google docs. The person proposing the metric could do this.
The working group discusses the metric and makes the required changes.
A markdown version of the metric is added to the working group repository.
The metric is ready for the Continuous Metric Contribution and Regular Release
Whenever there is a language or terminology change that would affect other metric references.
Changing the name of a metric (in any way)
Any changes that go beyond grammar or spelling fixes in the sections:
Question
Definition
Objectives
Implementation
Does not require review:
Grammar or spelling fixes (not including metric name)
Updates to sections References and Contributors
Step-by-step instructions for releasing CHAOSS Metrics
The version number is YYYY-MM of the release date. Continuous Metric Contributions do not get a separate version number.
The goal is to have short cycles of feedback and to get metrics out sooner.
Coordinate release of metric through release tracking spreadsheet
Decide in a working group that a metric is ready for release.
Create a release notes issue within the working group repository and update information for each new metric released. This issue will be used to create the regular release cadence release notes.
Create an issue for collecting feedback on that metric before release.
Include a message in the metric mark down file that the metric will be part of the next regular release. The message should be at the top metric markdown file using the following format:
###This metric is a release candidate To comment on this metric please see Issue #xx Following a comment period, this metric will be included in the next regular release.
Add metric to metric page with a "under review" tag and link to the feedback issue.
Add an item on the Release History page in the Continuous Metric Contributions Since Last Release section stating that the metric was added.
Announce the new metric on the mailing list and point to the issue and the markdown page for feedback. Update community on weekly zoom call.
Address feedback in issue and through edits to the markdown page.
The metric stays under review until after the next regular release
The regular release is when we update the version number, update the full release notes, and make a big announcement. These releases occur one to two times a year and may correspond with the dates for CHAOSScon North America and Europe.
Timeline:
2 months before release
Decide on a release date, coordinate with working groups
1 month +1 week before release
Finalize which metrics to include in the release
All metrics should be prepared as CMC process described above
Add information on the /metric page about the review period and planned release deadline
Announce Review Period on mailing list
Working Groups review and respond to comments as the come in
Request reviews in newsletter, Twitter, all community calls during review period
Working groups draft release notes
1 week before release
Announce on mailing list that review period is closed
Prepare the final release (see below)
Update /metrics page
Update release notes
Prepare the PDF release
Day of release
All the work was done before
Announce on mailinglist and Twitter and all other channels
Celebrate!
Determine release version number (<year>-<month>) because it will be used for several steps
Extend list of contributors to include those who contributed to the new and previous releases
Update list of governing board members at time of release
Working Groups respond to all comments in the issues and close them
Working Groups merge or close all pull requests related to the release
Working Groups remove the request for review from the top of all metrics
Release Engineer creates a release tag in all working groups to freeze metrics for release
Tag name: release-<year>-<month>
Description: Release notes for that working group, include full list of metrics released
Release Engineer updates metric pages to pull from release tag (always pull from master, snapshot exists in repo as a tagged commit and PDF)
Release Engineer cleans up /metrics by removing everything not needed in final release
Release Engineer cleans up the release notes page
Release Engineer creates a PDF of the release (see section below)
Release Engineer links the PDF on the /metrics and release notes page
The PDF is created from the website.
Make sure to not print menu, footer, "to-top" or any other elements that we don't want in the PDF
Here is a Tampermonkey user script that does most of it: CHAOSS Metric print clean (Tampermonkey).user.js
The Metrics are saved as PDFs from the browser (Chrome, print).
Name the files in order #) Metric Name.pdf (the user script will help with naming the pages like this, need to add the number manually after saving)
The CHAOSS Metrics by Working Groups and Focus Areas pages are the /metrics page saved as PDF
Remove the list of contributors and copyright notice from the bottom, they will be in the front-matter
Name the file CHAOSS Metrics by Working Groups and Focus Areas.pdf
Create the front-matter using the Word Document
update all of the information
save as PDF with any name
Create the License information using the Word Document, update the year for the copyright
Name the file The MIT License.pdf
Above steps should create the following files:
The front-matter PDF
CHAOSS Metrics by Working Groups and Focus Areas.pdf
#) Metric Name.pdf for each metric
Release History.pdf
The MIT License.pdf
Merge the files (except the front-matter) in the order listed above
Use the open source PDFsam Basic
Merge settings
YES: Add a footer
Bookmarks handling: Create one entry for each merged document
Table of contents: Generate from file names
After creating the merged document, add the front-matter
If you use PDFsam again, make sure to change the settings:
NO: Add a footer
Bookmarks handling: Retain bookmarks
Table of contents: Don't generate
As our process for developing metrics evolves, we will implement a process to go back to old metrics and review every two years. Specifically, after 4 release cycles, we should review metrics and see whether they are still aligned with
Our current metric structure,
Our current metric style, and
The original intention of the metric, and if so, whether the metric's language and/or intention requires updating.
What Types of Changes are suitable for continuous release?
Whenever there is a language or terminology change that would affect other metric references.
Changing the name of a metric (in any way)
Any changes that go beyond grammar or spelling fixes in the sections:
Question
Definition
Objectives
Implementation
Types of metric changes that do not require review:
Grammar or spelling fixes (not including metric name)
Updates to sections References and Contributors
Ultimately, we need a gardener who ensures consistency and flags metrics that need to be “updated”.
Metrics are continuously released (this includes edits to existing metrics) in English. During that time period, any metric may be edited (they are not static).
The English version of the metrics is officially released in April and October (approximately 6 months apart). The official release is preceded by a 30 day review period. When the metrics release candidates go into the 30 day review period, they are frozen and no edits are made. Following the review period, they may be edited prior to release to address comments from the review. The released metrics are a snapshot in time.
Following the official metrics release, all metrics can be edited for the next release as part of the continuous release process. Changes to the metrics are kept track of in working group release notes issues and the individual metrics release candidate issues.
The working groups need to make sure that the Metric Release Notes issue in the working groups’ repository is continuously updated with the latest changes in metrics. This will give an overview of all the changes that are occurring during a release cycle.
Working groups need to create an issue for each metric that has been created or edited - labeled as Metrics Candidate Release. This can signal that a metric needs to be translated.
Working groups or the release team need to create a corresponding issue in the translation repo that links to the individual Metrics Release Candidate issue. This issue is not targeted towards a specific language community but can be used by the individual language communities to coordinate work. The issue should be labeled with all applicable language tags.
The release manager will add the under review label to the English Metrics Release Candidates on the CHAOSS website. This also signals that a metric needs to be translated.
The first place the translation team should look is in the translation repository issues (all metrics that have been edited should have an issue created by the working group that created or edited a metric). Metrics that need attention should have a language label attached.
Additionally, the translations team could refer to the working groups’ Metric Release Notes issue. Example: - https://github.com/chaoss/wg-common/issues/106) and the Metrics Candidate Release labels in the working group repositories for guidance on which metrics may need to be translated during a release cycle.
The translations team could also look for the metrics with the label ‘under review’ on the website, which indicates that the metric was added/updated after the previous release.
Note: The metrics are prone to changes during the continuous release process and following the review period.
The release for translated metrics should happen within 2 weeks of the English metrics release. A simultaneous release is ideal.
All metrics undergo a 30-day community review period before the official release. This period should be utilized by the translations team to translate the metrics and share them for review.
The translated metrics should be released only when the translation team signals that the metrics are ready.
The review period for the translated metrics should follow the same timeline as the English review period.
Translation teams should remove the language tag from the translation repository issue when translation work has been completed for that language.
The YAML file used as the primary input for MARS can be used to signal that the translations are ready. The translations team can update the YAML file with new metrics signaling that metrics are ready to be released. As an example: https://github.com/chaoss/MARS/blob/main/automation/active_user_input/chinese_working-groups-config.yml
The translation team should inform the release team that they are ready for the release (by email, chat, or in Zoom Meetings)
The versioning of the PDFs should be done in the format - YYYY-MM_language-code. Example - `2021-09_EN, 2022-03_CN
The release version for the translations PDF should be the same as the English metrics PDF used as the basis for translations.
This sections provides the guidelines regarding the process of translating metrics into other various languages.
New metrics that need to be translated may be identified through translation repository issues, working group repository issues, and on the CHAOSS website release page.
Before adding translations in a new language, it is strongly recommended to have an active community of native speakers of that language.
To begin with translations into a new language, create a new directory in the translations repository. Name the directory as the language in which the metrics are to be translated.
Create subdirectories for each working group within the language directory. Note that the name of the sub-directories must match with the repository names of the working groups. (e.g., wg-common, wg-value, etc.)
The structure of the subdirectory for each working group must be the same as specified for the working group repositories on this page. Note:- The base files can be ignored except for the README.
Follow the templates of other files analogous to as specified in the governance repository
The metrics themselves also follow a standard template. Further, it is recommended to create a translated version of this template to ensure headings are consistent across all translated metrics.
All directories and files must be named in English only.
When the directory structure for translations in a new language is ready, the translated metrics can now be added to their respective working group and the focus area.
The above steps should be completed by the translating community before starting with the translation of the metrics.
Many of the above requirements are necessary for the M.A.R.S. release automation process.
Note: Optionally, an English - translated language comparison table or glossary can be created to ensure consistent naming of keywords in the translated language.
The working groups keep revising the focus areas and the metrics regularly. Therefore, it is necessary for the translations to be updated as per the current version of the release. The following steps help ensure this:
The translations should be updated whenever a change takes place in the original metrics including when:
A metric is added, updated, renamed, or removed
A focus area is added, updated, renamed, or removed
New metrics that need to be translated may be identified through translation repository issues, working group repository issues, and on the CHAOSS website release page.
An issue should be created for each metric that needs translation attention (this may be created by the working group) The issue should be labeled with the languages it affects.
The translation community should then investigate the issue and try to accommodate the changes in the translated metrics.
Optionally, the translation team may compare different versions of the metric. The relevant commit hashes can be used to easily identify the changes via the `git diff` command. Git history may be used to see changes to the metric markdown file and identify the relevant commit hashes.
The issue label for a specific language after it has been applied to metrics/focus areas in that particular language.
The issue may be closed when the change is implemented in all languages in which the metrics are maintained.
It is preferable for the translated metrics to get released biannually similar to their English counterparts in the form of a PDF report.
All metrics undergo a 1-month community review period before the official release. This period should be utilized by the translations team to translate the metrics and share them for review.
The translations team should update the YAML file in the MARS project when the metrics are ready for release. This serves as the signal to the Release Engineer to generate the release PDF.
The translations release should ideally happen within two weeks of English metrics release, however, the target for some translation teams may be a simultaneous release.
The version of the report should be the same as the version of the English report it is based on.
The version number for the report should be in the format: YYYY-MM_language-code
When the translation team signals to the metrics release team that translated metrics for a specific language are aligned with the English release, the release manager will use the MARS tool to create the release PDF for that language. The release PDF will be available on the CHAOSS website and an official announcement will be made on CHAOSS communication platforms
The translation team for each language should update and maintain a contributors list that can be added to the Release document
The translation teams and release team should ensure that the processes outlined below are executed in a timely manner.
1 month before the release.
Translate new metrics and update existing ones (identified during the continuous release process) with the most recent changes
The release team will announce on the mailing list that the review period is open
Request reviews on Twitter, newsletter, or any other social media platform
Translate or Draft any other required documents such as cover page and release notes.
1 week before the release
The release team will announce that the review period is closed.
Prepare the English PDF release
Buffer period (as long as is needed - translation team for each language will signal when ready)
Prepare the Translated PDF release
Day of release
Announce on all social platforms about the release
Celebrate! 🎉
The English version of the metrics is original and must be used as the only source during the translation process.
Presently, the templates are maintained only in English. These templates can be translated into different languages as and when required to ensure consistency among the translations.
Follow the general conventions related to naming and structure as specified here