A conflict of interest in peer review arises when a situation where you, the reviewer, are unable to make a fair judgment. The CHAOSS D&I Badging Project aims to promote a transparent and impartial environment to evaluate Diversity and Inclusion, as well as consciously avoiding situations where conflicts of interest may appear.
As a reviewer, Conflicts of Interest for event submission reviews are your present or previous associations with any event organizers and the event committee of a submission. The following situations are considered conflicts and should be avoided:
Organizing for the submitted event.
Having a personal relationship (e.g.family, close friend) with the event organizer(s).
Receiving personal benefit resulting from the review.
Recent association (past year) with the same organization of an applicant, for example, being employed at the same institution.
If you have a conflict of interest with a submission, you should disclose the specific reason to the moderator team. After the discussion within the moderator team, you may not be eligible to review the submission. Declaring actual, perceived, and potential conflicts of interest are required under professional ethics. If in doubt, ask the moderators.
The D&I Conflict of Interest Policy is derived from JOSS Conflict of Interest Policy and takes references from Elsevier Conflict of Interest Policy.
Thank you so much for showing interest in D&I Badging Project and the review system. We are delighted to have you here! This page tells you a bit about the participation as a reviewer and helps with some detailed principles you would need to know while reviewing.
A CHAOSS D&I Badging application starts when an event organizer opens an issue on the event badging repository. The organizer fills out the form and a GitHub issue is created that contains the content of the application. Upon being assigned an issue, a reviewer will be provided with a review checklist in the form of an issue comment.
You need a GitHub account to review for the D&I Badging program.
Once you become a reviewer of D&I Badging (see apply to review on how to be a reviewer), your GitHub handle will be added to the review list and you will be assigned to new issues.
You can read CHAOSS D&I metrics, especially metrics under the Event Diversity focus area to get a further grasp before reviewing.
Please make sure you don't miss an assignment notification. If a reviewer provides no response to the issue for more than one month, D&I Badging moderator team will reassign a new reviewer for that issue.
The Reviewer's task is to analyze the information given by an event organizer using the Review Checklist. The reviewer provides their observations according to the Review Checklist and feedback on how an application can be improved if certain checks are not met. You may find more guidance for this process in the review process section.
The reviewer's judgments are based on three main criteria:
A link to the event website is required from applicants. Reviewers should observe the Event website to verify everything is aligned with the submission answer. This will be the most valuable resource.
Each question in the application should be answered. The applicant should provide a sufficient description for the reviewer to understand how the metrics are fulfilled associated with the event.
Each link provided should be relevant and confirm the values of the description within a metric.
Special thanks to JOSS(Journal of Open Source Software) for the D&I Badge peer-review process. Their work has inspired our project's structure.
This page contains information on how to become a reviewer of the D&I badging program. To review submissions for D&I badging, you need to sign up as a reviewer first. The best way to start is with our sign-up form!
Please go to the Reviewer Application Form to sign up.
We no longer support Pull Requests as reviewer applications. We invite you now to sign up using the Reviewer Form.
The reviewer form requests the following personal information:
Your Name: Supply your full name. D&I Badging will not share the name with applicants or spread it unless you are willing to post your name on the website.
Your GitHub Handle: This will be later added to the reviewer list in reviewer.md file under the badge/event-diversity-and-inclusion repository once you become a member of the reviewer team.
Your Email Address: D&I Badging uses your email address to contact and communicate with you, and will not share or publish it on the website.
Your Organization(optional): You can choose not to share this information with us. D&I Badging asks for the organization to know the further background about you, and will not share or publish it on the website.
Below are the most important questions:
Who you are and why you're interested in the CHAOSS D&I Badging Program.
Your history with Diversity & Inclusion in Open Source.
We will also ask:
We want to know if you would like to include your name on our website as a CHAOSS D&I Badging Reviewer. If you agree with that, your name will show on the bottom of D&I Badging Page.
We also require you as a reviewer to identify and act upon any conflicts of interest. Please see the D&I Badging Conflict of Interest Policy for more information.
The D&I Badging team will be notified once you submit the form. They will contact you through email to set up a meeting to explain the reviewing process.
After the meeting, your GitHub handle will be added to the list of current reviewers. There is a badging bot that assigns reviewing tasks - don't forget to open email notifications for D&I badging repository to avoid missing an assignment!
This page contains information about going through the review process, especially about how a reviewer could work with the Review Checklist and collect the information required for the Review Checklist criteria.
The review checklist provides a guideline for the reviewers to verify the event's alignment with CHAOSS Diversity and Inclusion best practices.
Apart from providing viewpoints while reviewing, the checklist also plays the function of generating the badge state by calculating how many checkboxes are marked. This calculation includes the metric checkboxes from all reviews on the application.
The event review checklist for the CHAOSS Badging Project can be found here.
D&I Badging reviews are checklist-driven. A review checklist will be generated for each reviewer to work through when reviewing the submission. Reviewer tick the checkboxes which they think the event aligns with the criteria.
The review process completes when the applicant is satisfied with the current badge status or the applicant will no longer provide information. Reviewers are encouraged to ask applicants for more information through issue comments.
Below is an explanation of each initial check:
The Event is about Open Source technologies and systems. This is not always easy to understand without talking to the event team. In most cases, this can be explained by the event website in combination with the applicant's description. Ask the applicant if you need more information.
The Event information is publicly available on a website. Observe the event website to understand whether the event information is public.
The Event Code of Conduct is publicly available. Try to find the Event Code of Conduct on the event's website or other public places.
The applicant is the organizer of the event. The applicant must be involved in planning and developing the event.
Ensure all the initial checks are marked before proceeding with Metric based checks. If the event does not qualify from the initial checks, thank the applicant for their time and connect with the moderator team to end the review.
Use the website provided in the application to everyone's advantage! Most of the data you collect to make your decisions should come from the event website.
Measuring demographics: Ensure that the website clearly states the process by which they measure and use their demographic data of speakers.
Displaying demographics: This information could be in a PR statement, press release, or otherwise, but it must be easy to access.
Measuring Demographics: Ensure that the website clearly states the process by which they measure and use their demographic data of attendees.
Displaying demographics: This information could be in a PR statement, press release, or otherwise, but it must be easy to access.
Attendee Inclusivity: Ensure the event possesses explicit measures to promote attendee inclusivity. An event should at least requests feedback from attendees. Ask the applicant if you need more information.
Findability: Observe the event website to see whether the CoC is easy to find.
Clarity: Read the CoC, make sure the CoC provides a clear definition of proper conduct at the event and behaviors that should not be tolerated.
Reporting venue: Ensure that the event has a venue for all event participants to report violations of CoC. The information about the reporting venue should be public on the event website or other places.
Support at Event: Read the CoC and ensure it contains information on possible methods to provide support to victims of inappropriate behavior.
Enforcement: Ensure the event has a definite process to display the CoC and a clear request for them to accept it.
Availability: Ensure that the event provides one or more Diversity Access Tickets.
Ticket allocation: Allocating different kinds of tickets may always be a process that operates internally, try to find out the process based on the description provided by the applicant and ask for more information if required.
Findability: The information on diversity access tickets is not necessarily posted on the event website, it can also be on a ticketing system or other places, but this should be at least public and findable.
The Review Checklist for Virtual Event won't have this metric.
Availability: Ensure that the event provides one or more services/facilities for families.
Findability: Observe the event website to see whether information regarding family friendly services provided at the event is easy to find.