Difference between revisions of "Creating Assignments (assign IPC members to submissions)"

From SRMv2 Wiki
Jump to navigation Jump to search
(Created page with "SRMV2 provides an automatic reviewer assignment process. There are two entry points: (for the correct links, replace "event" by your event. https://srmv2.eg.org/COMFy/Conference/event/showSuggestionsForPaperAssignment and https://srmv2.eg.org/COMFy/Conference/event/showSuggestionsForPaperAssignmentBySubmission Requirements and steps: 1) There are IPC members with status "accepted" (https://srmv2.eg.org/COMFy/Ipc/event/manageIPC). For clarity, pending or rejected...")
 
 
(38 intermediate revisions by the same user not shown)
Line 1: Line 1:
SRMV2 provides an automatic reviewer assignment process. There are two entry points:
SRMV2 provides an automatic reviewer assignment process. There are two entry points:


(for the correct links, replace "event" by your event.
(for the correct links, replace "event" by your event.)


https://srmv2.eg.org/COMFy/Conference/event/showSuggestionsForPaperAssignment
https://srmv2.eg.org/COMFy/Conference/event/showSuggestionsForPaperAssignment
and
and
https://srmv2.eg.org/COMFy/Conference/event/showSuggestionsForPaperAssignmentBySubmission
https://srmv2.eg.org/COMFy/Conference/event/showSuggestionsForPaperAssignmentBySubmission


Requirements and steps:
At first the system shows a list of all IPC members with along with every review type which is currently assigned to the conference.
1) There are IPC members with status "accepted" (https://srmv2.eg.org/COMFy/Ipc/event/manageIPC).
[[Image:ShowSuggestionBlank.png|thumb|350px|none|Show Suggestion for IPC without data]]
  For clarity, pending or rejected IPC members should be removed.
 
2) Different reviewer types exist (usually Primary, Secondary, Tertiary)
 
3) IPCuploader and bidding are completed.
 
4) SARA score calculation should be started
 
As chair you now have the possibility to either use the built-in solution or to use an external script for creating assignments. This script is owned and maintained by Prof. M. Wimmer. (https://gitlab.com/salzrat/eg-sig-sort)
 
5 A) Build-in
5 A1)) Now a new suggestion can be created: https://srmv2.eg.org/COMFy/Conference/event/createNewSuggestionForIPC
This suggestion can be edited, exported (then edited offline, if comfortable), and imported again.
 
5 B) External Script


5 B1) Initialize Ratings
== Requirements and steps:== 
# There are IPC members with status "accepted" (https://srmv2.eg.org/COMFy/Ipc/event/manageIPC). For clarity, pending or rejected IPC members should be removed.
# Different reviewer types exist (usually Primary, Secondary, Tertiary). 
# IPCuploader and bidding are completed.
# SARA score calculation should be started.


Initializes Rating and RatingOrg for Sort.  For this purpose, the score (=affinity score, from the IPCuploader) is combined with the bidding.
As chair, you now have the option of either using the built-in solution or using an external script to create assignments. This script is owned and maintained by Prof. M. Wimmer. (https://gitlab.com/salzrat/eg-sig-sort)
For “want review” the score is multiplied with 10, and for “could review” with 5.


5 B2) IPC members can help with sorting. Use "Manage Sorter" to invite them and assign them to submissions.  
=== Built-in ===
* Now a new suggestion can be created: https://srmv2.eg.org/COMFy/Conference/event/createNewSuggestionForIPC
The goal is to set the ratings so that the order determined by the ratings corresponds to the preferences of a submission for IPC members.
::This suggestion can be edited, exported (and then edited offline if necessary) and imported again..
[[Image:CreatenewSuggestion.png]]


This type of sorting is not about assigning reviewers, but rather about whether the rating, which is formed from bidding and affinity scores, is in line with the wishes and preferences of the IPC members. The sorters can change the rating for this purpose (initially, rating = RatingORG). This new order can then be used to generate the assignment.
When you press create New Suggestion you set the preferences for the new matching. If you are interested how the matching is created: [[calculate Suggestion]]
==== Modify the matching ====
If you are not satisfied with the current mapping, you can start over to get a better mapping (the mapping is randomly shuffled) or adjust the system's mapping. To do this, click on the document you want to correct. You will be taken to the following page:
[[Image:EditSuggestion.png|thumb|350px|none|Edit Suggestion]]


5 B3) After the sorting, export the ratings https://srmv2.eg.org/COMFy/Review/event/ExportSuggestions and use the external script, the result can then be imported: https://srmv2.eg.org/COMFy/Review/event/ImportSuggestions
There you can specify who is assigned as which review type. (One person cannot be in multiple review types in one submission!)
There you can access the submission, see every review type, who is currently assigned in this review type, a checkbox for every review type, the internally generated rating, the person's bid, his global expertise list. And how much expertise the person has that matches the paper.
In this table:
Score: value from IPCUploader (SARA TFIDF),
RatingOrg: value, generated through "Initialize Rating", calculated from SARA TFIDF combined with bidding. (*5 could review, *10 want review)


6) Last step (for both alternatives) when everything is set: "take Suggestion Live": the assignments are now active, IPC member should be informed.
=== External Script===
Use this button only once.
* Initialize Ratings [[Image:InitializeRatings.png]]
*:Initializes Rating and RatingOrg for Sort.  For this purpose, the score (=affinity score, from the IPCuploader) is combined with the bidding.
*:For “want review” the score is multiplied with 10, and for “could review” with 5.
* IPC members can help with sorting. Use "Manage Sorter" to invite them and assign them to submissions.
[[Image:ManageSorter.png]]
:The goal is to set the ratings so that the order determined by the ratings corresponds to the preferences of a submission for IPC members.
:This type of sorting is not about assigning reviewers, but rather about whether the rating, which is formed from bidding and affinity scores, is in line with the wishes and preferences of the IPC members. The sorters can change the rating for this purpose (initially, rating = RatingORG). This new order can then be used to generate the assignment.
* After the sorting, export the ratings https://srmv2.eg.org/COMFy/Review/event/ExportSuggestions and use the external script, the result can then be imported: https://srmv2.eg.org/COMFy/Review/event/ImportSuggestions


There is no way to change active assignments through this menu.
=== Last step (for both alternatives)===
when everything is set:  "take Suggestion Live": the assignments are now active, IPC member should be informed.
:Use this button only once.
:There is no way to change active assignments through this menu.

Latest revision as of 09:36, 7 June 2024

SRMV2 provides an automatic reviewer assignment process. There are two entry points:

(for the correct links, replace "event" by your event.)

https://srmv2.eg.org/COMFy/Conference/event/showSuggestionsForPaperAssignment and https://srmv2.eg.org/COMFy/Conference/event/showSuggestionsForPaperAssignmentBySubmission

At first the system shows a list of all IPC members with along with every review type which is currently assigned to the conference.

Show Suggestion for IPC without data

Requirements and steps:

  1. There are IPC members with status "accepted" (https://srmv2.eg.org/COMFy/Ipc/event/manageIPC). For clarity, pending or rejected IPC members should be removed.
  2. Different reviewer types exist (usually Primary, Secondary, Tertiary).
  3. IPCuploader and bidding are completed.
  4. SARA score calculation should be started.

As chair, you now have the option of either using the built-in solution or using an external script to create assignments. This script is owned and maintained by Prof. M. Wimmer. (https://gitlab.com/salzrat/eg-sig-sort)

Built-in

This suggestion can be edited, exported (and then edited offline if necessary) and imported again..

CreatenewSuggestion.png

When you press create New Suggestion you set the preferences for the new matching. If you are interested how the matching is created: calculate Suggestion

Modify the matching

If you are not satisfied with the current mapping, you can start over to get a better mapping (the mapping is randomly shuffled) or adjust the system's mapping. To do this, click on the document you want to correct. You will be taken to the following page:

Edit Suggestion

There you can specify who is assigned as which review type. (One person cannot be in multiple review types in one submission!) There you can access the submission, see every review type, who is currently assigned in this review type, a checkbox for every review type, the internally generated rating, the person's bid, his global expertise list. And how much expertise the person has that matches the paper. In this table: Score: value from IPCUploader (SARA TFIDF), RatingOrg: value, generated through "Initialize Rating", calculated from SARA TFIDF combined with bidding. (*5 could review, *10 want review)

External Script

  • Initialize Ratings InitializeRatings.png
    Initializes Rating and RatingOrg for Sort. For this purpose, the score (=affinity score, from the IPCuploader) is combined with the bidding.
    For “want review” the score is multiplied with 10, and for “could review” with 5.
  • IPC members can help with sorting. Use "Manage Sorter" to invite them and assign them to submissions.

ManageSorter.png

The goal is to set the ratings so that the order determined by the ratings corresponds to the preferences of a submission for IPC members.
This type of sorting is not about assigning reviewers, but rather about whether the rating, which is formed from bidding and affinity scores, is in line with the wishes and preferences of the IPC members. The sorters can change the rating for this purpose (initially, rating = RatingORG). This new order can then be used to generate the assignment.

Last step (for both alternatives)

when everything is set: "take Suggestion Live": the assignments are now active, IPC member should be informed.

Use this button only once.
There is no way to change active assignments through this menu.