[IGFmaglist] suggested workshop grading scale descriptions

Flavio Rech Wagner flavio at inf.ufrgs.br
Tue Apr 14 08:53:29 EDT 2015


Dear Subi, Makane, Susan and all

We can accept only around 100 of 250 proposals. Therefore, we need to 
give scores that clearly make a difference between the proposals. If we 
give a score of 4 or 5 to all proposals, we will end up with no 
objective criteria to decide which 100 proposals are to be accepted.

Each of us must remember that only around 40% of the proposals can be 
accepted. I would thus strongly suggest that each of us does not assign 
a score 4 or 5 to more than 40% of the proposals s/he is evaluating, 
although MAG members evaluating only a small subset of the proposals may 
find more than 40% of very good proposals in her/his sample.

For the same reason, the scores must have clear different meanings. For 
the final decision in our meeting in Geneva, we need to differentiate 
the proposals in order to have a clear, objective procedure to decide on 
the final selection.

I thus suggest the following correspondence between scores and selection 
criteria:

     5) *proposals that MUST be accepted*, because they are excellent 
and cannot be rejected,
          independently of any other conditions;

     4) *proposals that COULD be accepted*, because they are good, _/but 
maybe some of them /__/
/_//_/are not kept in the final selection/_ if there are no more time 
slots available or because
         we want to achieve a better balance among stakeholder groups, 
or give priority to
         first-time proposers or to  developing countries, so that 
unfortunately we have to select
         other proposals;

     3) *proposals that are not so good as the ones in groups 4 and 5 
above*, _/but some of /__/
/_/_them can still be included in the final selection_/ because we want 
to achieve a better
         balance among stakeholder groups, or give priority to 
first-time proposers or to
         developing countries;

     2) *proposals that are interesting but not so good as the ones in 
groups 3, 4 and 5 above*,
         so that unfortunately we cannot keep them in the final selection;

     1) *proposals that cannot be accepted* because they either have 
problems in their
         presentation or do not fiit well into IGF.

So, I suggest that MAG members give their scores based on the above 
rationale, trying to remember that only 40% of the proposals can be 
accepted and that the scores given by each of us must reflect this 
"acceptance rate".

I, personally, would try to classifiy into bins #4 and #5 at most 40% of 
the proposals I will evaluate. I would try to assign other 40% to bins 
#1 and #2. I would classify the remaining 20% into bin #3, to be 
possibly traded off against proposals in bin #2.

If each of us adopts the same rationale, then the procedure to be 
followed in the Geneva meeting could be as follows, after calculating 
the average score of all proposals (and again supposing we can accept 
100 proposals, or around 40% of all proposals):
- identify 20% to 30% of the proposals that fit in bin #5 (those with 
average scores possibly above 4.0) and accept them;
- identify 30 to 40% of the proposals that fit in bins #1 and #2 (those 
with average scores well below 3.0) and do not accept them;
- complete the acceptance of 100 proposals by assessing proposals in 
bins #4 and #3, giving priority to proposals with higher average scores, 
but seeking a balance among stakeholder groups and giving priority, 
whenever possible, to first-time proposers and to developing countries.

Best

Flavio


> Dear Makane and Susan,
>
> It will ultimately depend upon the number of slots available and the 
> venue.
>
> This is more a process of elimination than of selection. After we've 
> applied ourselves to what should an ideal number prof workshops be, 
> which addresses the theme and the sub themes adequately.
>
> If you had 200 + high quality proposals all scoring 5 we will still 
> need to select or eliminate. Because we cannot have so many workshops. 
> The limitation is of logistical resources available and not crowding 
> the schedule so much that it leaves participants breathless and confused.
>
> Hence each year there is a cut off score shared by the secretariat 
> keeping a variety of factors in mind. Post which all of as us MAG 
> collectively review each proposal to strive for balance across dimensions.
>
> In view of the above I would prefer to stay away from subjective 
> interpretations of scores allocated.
>
>
> warmest
>
> Subi
> ----
>
> Subi Chaturvedi
>
>
>
>
> On 14 April 2015 at 15:56, Makane Faye <fmakane at gmail.com 
> <mailto:fmakane at gmail.com>> wrote:
>
>     Dear Susan,
>
>     Thank you for your proposal, however seeing that the maximum
>     number of points is 5, we should be more lenient and ensure that 4
>     and 5 are accepted and 3, which is above the mid range could be
>     accepted Hence I propose the following:
>
>     1 = proposal has serious problems
>     2 = should not be accepted
>     (2.5 = should be merged with another)
>     3 = may be accepted
>     4 = must be accepted
>     5 = must be accepted
>
>     Best regards,
>
>     ----------------
>
>     Makane Faye (Mr.)
>     Chief, Knowledge Services Section  ---
>     United Nations Economic Commission for Africa (ECA) ---
>     Limpopo Building ---
>     P.O. Box 3001Addis Ababa, Ethiopia---
>     Tel: +251 11 5443563; Ext. 33563 // Fax: +251 11 5514416 / +251 11
>     54411458 ---
>     Email: mfaye at uneca.org <mailto:mfaye at uneca.org>;
>     faymakane at gmail.com <mailto:faymakane at gmail.com>---
>     http://facebook.com/makane.faye.eca // twitter: @makaneeca //
>     Skype: makaneaddis
>
>     Makane Faye (M.)
>     Président, Fédération des Syndicats et Association du Personnel
>     des Nations Unies (FUNSA
>     Chef de la Section des Connaissances ---
>     Commission Economique des Nations Unies pour l’Afrique (CEA) ---
>     Bâtiment Limpopo ---
>     BP. 3001 Addis Abeba, Ethiopie ---
>     Tel: +251 11 5443563; Ext. 33563 // Fax: +251 11 5514416 / +251 11
>     54411458 ---
>     Courriel: mfaye at uneca.org <mailto:mfaye at uneca.org>;
>     faymakane at gmail.com <mailto:faymakane at gmail.com> ---
>     http://facebook.com/makane.faye.eca // twitter: @makaneeca //
>     Skype: makaneaddis
>
>
>     On Tue, Apr 14, 2015 at 11:34 AM, Susan Chalmers
>     <susan at chalmers.associates <mailto:susan at chalmers.associates>> wrote:
>
>         Dear MAG colleagues,
>
>         Based upon input from the last virtual meeting, how do the
>         following grading scale descriptions sit with everyone?
>
>         1 = proposal has serious problems
>         2 = should not be accepted
>         (2.5 = should be merged with another)
>         3 = borderline
>         4 = could be accepted
>         5 = must be accepted
>
>         Please don't hesitate to share your thoughts.
>
>         Sincerely,
>         Susan
>
>
>         Susan Chalmers
>         susan at chalmers.associates
>
>         *CHALMERS* & ASSOCIATES
>         http://chalmers.associates
>
>         _______________________________________________
>         Igfmaglist mailing list
>         Igfmaglist at intgovforum.org <mailto:Igfmaglist at intgovforum.org>
>         http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
>
>
>
>     _______________________________________________
>     Igfmaglist mailing list
>     Igfmaglist at intgovforum.org <mailto:Igfmaglist at intgovforum.org>
>     http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
>
>
>
>
> _______________________________________________
> Igfmaglist mailing list
> Igfmaglist at intgovforum.org
> http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://intgovforum.org/pipermail/igfmaglist_intgovforum.org/attachments/20150414/6b2054d1/attachment-0001.html>


More information about the Igfmaglist mailing list