[IGFmaglist] Evaluating workshop proposals - Standardized reasons for scores of 3 or less

Flavio Rech Wagner flavio at inf.ufrgs.br
Fri Apr 17 10:59:34 EDT 2015


Dear Marilyn

You are absolutely right when you state that we must be, and want to be, 
as inclusive as possible. We do not want to dismiss interests of any 
proposers. Our objective, of course, is to welcome many more proposers 
and to have them all come to João Pessoa. But that would be an ideal 
situation.

We have to face reality. We cannot accommodate all 247 proposals in the 
programme of IGF. We cannot even think of a merger process that 
considers a too large number of proposals. That would be simply 
unfeasible as a process, simply for lack of time and resources (and also 
for many other practical reasons already commented in our list).

We are not MAG members to "reject" proposals. This is not a language we 
shall use in our communication with the community. We shall not announce 
that we have "rejected" or "dismissed" 150 proposals. But we will have 
anyway to announce the 100 proposals that were kept in the programme, so 
that others may feel "rejected".

And then we need criteria to select proposals, because we cannot accept 
or merge all of them. Even after detecting that some proposals do not 
fit well to IGF, or fail on sufficiency, and even after an effort for 
merging as many proposals as feasible, I am fairly convinced that we 
will still have much more that 100 proposals left to consider. And then 
we will have to use criteria to select those that can be accommodated in 
the programme. We cannot escape the fact that these criteria have to be 
"comparative" and based on (1) some sort of scoring (because a 
subjective evaluation of all 247 proposals, without some scoring, is, 
unfortunately, simply unfeasible) and (2) agreed priorities (first-time 
proposers, developing countries, new formats, diversity, etc.).

This is not like having a goal of "rejecting" the others, but it is 
unavoidable that many proposers, even if we take utmost care with the 
language we use, will feel like "rejected". Those proposers whose 
workshops could not be accepted because of lack of space and time will 
like to know why their proposals could not be accepted. So we need to 
give them some feedback, of course using positive language and helping 
them, as much as possible, to improve their proposals for the next 
years. The proposed "buttons" are meant to facilitate both the "scoring" 
and the feedback.

Would you like to propose some concrete, feasible improvement to the 
evaluation and selection process, so that we become more inclusive and 
positive?

Best

Flavio


> I am hoping that through mergers, we will be able to be more open and 
> receptive to bringing in more proposers.
>
> All of the submitters want to attend, and contribute.
>
> IF the MAG earns a reputation for negativity toward applicants, rather 
> than an accommodating perspective. We must
> make efforts to be inclusive, while maintaining a focus on a high 
> quality IGF.
>
> BUT, we need to also ensure that we are being the open, and 'place to 
> come'.
> Suggesting we would reject 150 workshops shocks me.
>
> IF that is the MAG objective, we should stop with welcoming proposals, 
> and write a script that says
> only those who meet MAG criteria need 'enter here'.
>
> That is not why I am a MAG member.
>
> NO one who submitted did so thinking that we would casually dismiss 
> their interests.
>
> SOME will not fit, and some can be merged, and some may fail on 
> sufficiency.
>
> BUT, to just announce that we will reject 150 of the workshops seems 
> quite negative.
>
> Of course, our MAG list is posted, so all can see what we discuss, and 
> for me, I hope for as much
> flexibility and openness as we can provide, while achieving a high 
> quality event, focused on the
> main and sub themes, but allowing for some flexibility in Emerging 
> Issues.
>
> This is a critical year for inclusiveness.
>
> Marilyn Cade
>
>
> ------------------------------------------------------------------------
> Date: Fri, 17 Apr 2015 16:38:38 +1200
> From: susan at chalmers.associates
> To: mark.carvell at culture.gov.uk
> CC: igfmaglist at intgovforum.org
> Subject: Re: [IGFmaglist] Evaluating workshop proposals - Standardized 
> reasons for scores of 3 or less
>
> Dear Flávio, Mark,
>
> Flávio, I appreciate your approach, thank you kindly for suggesting 
> standardised reasons for workshop proposals that have a score of 3 or 
> less.
>
> For those workshops which must be respectfully declined - which will 
> be about 150 - proponents will be provided with this feedback.
>
> These tick box options will exist alongside a free text box, where we 
> can submit additional thoughts on the proposal. Please word your 
> thoughts constructively and compassionately for the proponent.
>
> Unless there are any strenuous objections, I suggest that we submit 
> these to the Secretariat, so that the IT person can get started on 
> adding these to the website. Is that okay with everyone?
>
> Mark, in regards to your observation, there are a few "positive 
> discrimination" elements - in favour of developing countries, first 
> time proposers, new session formats - upon which the MAG has found 
> rough consensus. MAG members are asked to reflect these elements in 
> their overall score. These elements are not individually 
> weighted. That is a conversation that the next MAG may wish to entertain.
>
> Sincere regards,
> Susan
>
>
>
>
> On Wednesday, April 15, 2015, Mark Carvell 
> <mark.carvell at culture.gov.uk <mailto:mark.carvell at culture.gov.uk>> wrote:
>
>     Dear Flavio
>
>     Many thanks. This list captures very effectively and
>     comprehensively the key evaluation criteria so is very helpful and
>     I support.
>
>     I have just one query regarding 4 in both lists: I accept that
>     "first-time proposers are preferred over repeat-proposers" serves
>     to re-fresh active stakeholder participation in the IGF but I hope
>     the intention is not that having previously submitted a proposal
>     should be a decisive negative criterion when comparing with other
>     similar proposals. If it were, we would have to explain that to
>     the applicant.
>
>     Kind regards
>
>     Mark
>
>     On 15 April 2015 at 11:33, Flavio Rech Wagner
>     <flavio at inf.ufrgs.br> wrote:
>
>         Dear MAG members
>
>         As already mentioned by Susan, I volunteered during the
>         yesterday's virtual meeting on workshop evaluation to propose
>         "standardized reasons" - kind of "buttons" the evaluators
>         could easily click on - for why a proposal received a score of
>         3 or less.
>
>         These reasons would match the "Considerations when Evaluating
>         Proposals", which have been already defined by the MAG and are
>         copied at the end of this message for your reference.
>
>         Based on the "considerations", I suggest that the online
>         evaluation form includes two sets of "buttons". The reasons
>         directly match each of the considerations.
>
>         For each proposal, each evaluator may click on one or more
>         buttons, as s/he thinks appropriate.
>         *
>         FIRST SET OF REASONS*
>
>         The first set gives reasons that, after aggregation by the
>         secretariat, will be sent to the proposers to explain the
>         final average scores their proposals received (numbers before
>         the reasons correspond to the numbers of the "considerations").
>
>         (1) The proposal is either (a) not well thought-through or (b)
>         incomplete.
>         (2) The proposal is not relevant to Internet Governance.
>         (3) The proposal does not include either (a) a list of
>         proposed speakers, participating individuals and
>         organizations, or (b) a description of how different
>         stakeholder perspectives will be represented.
>         (5) The workshop description is not consistent with the format
>         listed.
>         (7) There is no diversity amongst the participants (gender,
>         geography, stakeholder group, perspective).
>         (9) The description does not clearly specify the Internet
>         Governance problem/question/challenge to be addressed during
>         the workshop.
>         (10) The proposal does not include a well-considered plan for
>         remote participation.
>
>         *SECOND SET OF REASONS*
>
>         The second set corresponds to reasons that justify the
>         subjective score given by the evaluator, in comparison with
>         other proposals, but do not necessarily represent a weakness
>         of the proposal, so that these reasons do not need to be sent
>         to the proposers:
>
>         (4) This is not the first time this individual or organization
>         has submitted a workshop proposal to the IGF.
>         (6) This is not a proposal for a new format (Break-out Group
>         Discussions, Debates, Flash Sessions, Birds of a Feather,
>         Roundtables and Other formats are encouraged over the Panel
>         format).
>         (8) There is no participation from developing countries.
>
>         I am looking for your feedback.
>
>         Best
>
>         Flavio
>
>         ----------
>
>         *Considerations when Evaluating Proposals
>
>         *from*
>         *http://www.intgovforum.org/cms/workshop-proposals/mag-workshop-review-and-evaluation-process-for-igf-2015*
>
>         *
>
>         1.    Is the proposal well thought-through and complete?
>         2.    Is the proposal relevant to Internet Governance?
>         3.    Does the proposal contain a list of proposed speakers,
>         participating individuals and organizations, or a description
>         of how different stakeholder perspectives will be represented?
>         4.    Is this the first time this individual or organization
>         has submitted a workshop proposal to the IGF? (first-time
>         proposers are preferred over repeat-proposers),
>         5.    Is the Workshop description consistent with the format
>         listed (for example, if the format is Debate, then does the
>         proposal describe how the debate will be set up, with timings,
>         etc.,  indicated, are all sides of the issues represented)?
>         6.    Is the proposal for a new format? (Break-out Group
>         Discussions, Debates, Flash Sessions, Birds of a Feather,
>         Roundtables and Other formats are encouraged over the Panel
>         format),
>         7.    Is there diversity amongst the participants (gender,
>         geography, stakeholder group, perspective)? (as a general
>         matter, greater diversity is encouraged),
>         8.    Is there developing country participation? (as a general
>         matter, developing country participation is encouraged),
>         9.    Does the description clearly specify the Internet
>         Governance problem/question/challenged to be addressed during
>         the workshop?
>         10.    Does the proposal include a well-considered plan for
>         remote participation?
>
>
>
>         _______________________________________________
>         Igfmaglist mailing list
>         Igfmaglist at intgovforum.org
>         http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
>
>
>
>
>     -- 
>     Mark Carvell
>     Global Internet Governance Policy
>     Department for Culture, Media and Sport
>     mark.carvell at culture.gov.uk
>     tel +44 (0) 20 7211 6062
>
>
>
> -- 
>
>
> Susan Chalmers
> susan at chalmers.associates
>
> *CHALMERS* & ASSOCIATES
> http://chalmers.associates
>
>
> _______________________________________________ Igfmaglist mailing 
> list Igfmaglist at intgovforum.org 
> http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
>
>
> _______________________________________________
> Igfmaglist mailing list
> Igfmaglist at intgovforum.org
> http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://intgovforum.org/pipermail/igfmaglist_intgovforum.org/attachments/20150417/4a71a6d5/attachment-0001.html>


More information about the Igfmaglist mailing list