The FAQs are developed by the MAG chair and the Secretariat and members of the MAG Workshop Process Working Group in June 2021.
Q1: How many proposals does each MAG member have to evaluate and how do MAG members know which proposals they have to evaluate?
When each individual MAG member logs into the evaluation platform they will see the list of proposals that have been assigned to them. This list is based on the “issue area” group they are a part of. In 2021 it is expected that each MAG member will have to read and evaluate about 65 to 70 proposals.
Q2: What does a MAG member do if they are connected in some way to a proposal?
MAG members should check if they have any conflict of interests related to any of the proposals allocated to them. For example, they might be a proposed speaker at the workshop, or if the proposal is from the organisation they work for, or a network they are affiliated with. If they do have a conflict they have to indicate this on the form. Scanning all allocated workshop proposals for potential conflicts of interests should be done as early as possible in the process; if a conflict is identified, the proposal will be removed from the list.
Q3: What is the timeline for workshop evaluation by individual MAG members?
The deadline for individual evaluation is 17 June. The initial evaluation results will be presented to the MAG during the MAG meeting on 22 June. Then during the MAG meeting on 22, 23 and 30 June (tbc) the MAG will polish the evaluation results and agree on the final list which will be made public on 1 July.
The overall workshop evaluation timeline is as follows:
1 June – 17 June: Individual evaluation by each MAG member
18 June – 21 June: Processing of evaluation results by Secretariat
22 June: Open Consultations (community input on Intersessional work and also online phase of IGF 2021)
23 and 30 June: 1st Day of MAG meeting initial discussion of evaluation results and agreement on way forward
Q4: Why does the MAG work in thematic or issue groups when evaluation workshops?
Evaluation groups are meant to provide a space for MAG members to (a) support each other during the evaluation process and discuss any issues or uncertainties they come across, and (b) once the individual evaluation is finalised, look at the overall scoring results (to be provided by the Secretariat), discuss proposals that receive very different scoring within the group, and, generally, propose a way forward for finalising the selection of workshops. In the course of this final selection they will also assess to which extent the selected proposal covers the issue area and the pre-identified policy questions in this area.
Q5: Does each group have to agree on some standards in evaluating workshops?
No, this is not necessary. MAG members all use the criteria developed by the MAG Working Group on Workshop Process. What MAG members can do in the working group is talk about how to interpret and apply the criteria when they evaluate a proposal.
Q6: Do MAG members need to consult one another when they evaluate proposals?
MAG members are expected to evaluate each proposal on their own and should not ideally not share scores with other MAG members in their group until everyone has completed their individual scoring, or until the initial compilation of the evaluation results has been done by the Secretariat. At the same time, if a MAG member has questions or would like to discuss a particular proposal, they should definitely consult others in their group using the mailing list, keeping in mind that it is important not to influence how other members of the group score proposals, nor be influenced by them.
Q7: When a MAG member does feel a need to consult others in their group, what is the best way to do this?
Using the mailing list that has been set up for the evaluation groups. These lists are private; only subscribers have access to them. Remember, if you have a question, ask your WG colleagues, we can help each other.
Q8: Can MAG members keep their evaluations in draft format so that they can go back and look at them before finalising the “score”?
Yes, MAG members always can go back and revise evaluations and update them as necessary, until closure of the form on 17 June which is the deadline for completing evaluation of all the proposals assigned to each MAG member.
Q9: How do I change an evaluation that I am still busy finalising?
The evaluations appear on the same platform in 2 views: a. table view (similar to Excel, and sortable by columns) and b. textual view which can also be sorted. Evaluators can use whatever view they prefer. In each evaluation there is a “Change” link which can be selected and which will then allow evaluators to update a given evaluation at any time until 17 June.
Q10: What do I do if I accidentally submitted an evaluation before I had completed it?
It is possible to easily change the submitted evaluation until 17 June, as many times as needed. Please see the response to Q9 above for how to do this.
Q11: What happens if there is a discrepancy in results between workshops evaluated by our group? In other words, if different members of the group assign very different scores to the proposal?
This is a common scenario for some proposals and it is to be expected that different MAG members may score proposals differently because they have different perspectives. During the initial evaluation MAG members are expected to evaluate the proposals individually, not as an evaluation group, or in stakeholder groups. Once the initial evaluation period closes (17 June) the Secretariat provides the MAG with an analysis of the scores and they will highlight workshops where scores diverged dramatically (e.g. some MAG members gave a proposal a high score and others gave comparatively lower scores). This is where the role of the Workshop Evaluation Group is vital. You can then discuss these variances in your group and decide on how to proceed.
Q12: What do we do as a group after the 17 June deadline?
After the initial evaluation process ends on 17 June the Secretariat will compile the scoring and share the result with each group. They will also flag workshops where scores diverged dramatically.
The group will then meet to discuss the evaluation results. During this call it can be useful to:
Discuss the lowest scoring proposals and agree on which should be excluded (i.e. not considered further)
Discuss on the highest scoring proposals and agree on which should be kept
Identify proposals which might need further input from the proposers, for example, a proposal that scored very well for content, but not on format or participation could be returned to the proposers with a request to update the relevant fields.
Assess to which extent the proposals that scored well cover the issue area and all the pre-identified policy questions in that issue area.
Identify proposals which MAG members feel would be good to include in the preparatory phase of the IGF (provided the proposers had agreed to this being an option).
The goal of the group discussions in the period after 17 June is to agree on a list of workshops recommended for inclusion in the IGF 2021 programme.
Q13: How do we agree on the final list of accepted workshops?
Initially individual MAG members and Evaluation Working Groups should only focus on the quality of the proposals and the extent to which they respond to the issues and policy questions. There is no need to be concerned by the numbers at this point.
Q14: How many proposals can we accept?
The number of proposals to be included in the IGF2021 programme - as part of either the preparatory phase or the main event, will be finalised during the MAG meeting of 30 June.
Q15: What is the precise procedure for requesting a proposer to revise their proposal? Who does it? What is the timeline for this?
Organisers will only be approached with feedback on their proposals, and requests for updating them, after the final selection has been made - so after 30 June. The Secretariat communicates the MAG's feedback to the proposers.
Q16: What would be the procedure for proposals which a group feels would fit better into one of the other groups’ issue areas?
MAG members should continue to score the proposal irrespective of whether they feel it is in the right issue area or not. Proposals cannot be re-categorized once the evaluation process has been launched. MAG members can make notes in the evaluation form should they think a proposal might fit better into a different issue area and discuss this in their group once the initial scores have been compiled. Should they agree that a proposal should move to a different area they can then consult with the relevant evaluation group. The final step will be for the Secretariat to be notified so that they can then consult the proposer on moving the proposal to a different issue area.