Creating Assignments (assign IPC members to submissions)
SRMV2 provides an automatic reviewer assignment process. There are two entry points:
(for the correct links, replace "event" by your event.)
https://srmv2.eg.org/COMFy/Conference/event/showSuggestionsForPaperAssignment and https://srmv2.eg.org/COMFy/Conference/event/showSuggestionsForPaperAssignmentBySubmission
At first the system shows a list of all IPC members with along with every review type which is currently assigned to the conference.
Requirements and steps:
- There are IPC members with status "accepted" (https://srmv2.eg.org/COMFy/Ipc/event/manageIPC). For clarity, pending or rejected IPC members should be removed.
- Different reviewer types exist (usually Primary, Secondary, Tertiary).
- IPCuploader and bidding are completed.
- SARA score calculation should be started.
As chair, you now have the option of either using the built-in solution or using an external script to create assignments. This script is owned and maintained by Prof. M. Wimmer. (https://gitlab.com/salzrat/eg-sig-sort)
Built-in
- Now a new suggestion can be created: https://srmv2.eg.org/COMFy/Conference/event/createNewSuggestionForIPC
- This suggestion can be edited, exported (and then edited offline if necessary) and imported again..
1. Bidding (only if flag Uses Bidding is on)
| Reviewing | Meaning | Points |
|---|---|---|
| 0 | want to review | 100 |
| 1 | could review | 80 |
| 2 | not competent | 0 |
If there's no bid, nothing is added.
2. Expertise match (only if flag Uses Area of Expertise is on)
The paper is tagged with categories. For each of those topics, the code looks up the expertise level the user gave themselves and counts how many fall into each level:
| ExpertiseLevel | Counter | Points |
|---|---|---|
| 3 | expert | 4 |
| 2 | knowledge | 2 |
| 1 | passable | 1 |
aoe = (4·expert + 2·knowledge + 1·passable) / numberOfPaperTopics → 0 … 4 aoe = aoe × 25 → 0 … 100
- Each level is worth double the one below it.
- Dividing by the number of topics gives the average per topic, so a paper with many topics doesn't score higher just for having more of them.
- Multiplying by 25 scales it so that "expert in every topic" comes out at exactly 100.
- A topic the user hasn't rated still counts in the divisor, so it pulls the average down like a 0.
3. PaperMatchingScore (only if flag Uses IPCUploader Scores is on)
The PaperMatchingScore measures how similar the submission's text is to the representational papers that the IPC member uploaded to the IPCUploader. It is calculated using TF/IDF:
- TF (term frequency): how often a term occurs in a document.
- IDF (inverse document frequency): terms that occur in only a few documents get a higher weight than common terms that appear almost everywhere.
The submission and the IPC member's representational papers are compared based on these TF/IDF-weighted terms. If they share many distinctive terms, the PaperMatchingScore is high. The score is computed beforehand and stored per user and submission, and the rating code uses it as is.
Combining the parts with configurable weights
Each part is multiplied by a weight that can be configured, and the results are added up:
IpcRating = AoEModifier · ExpertiseScore (0 … 100)
+ BiddingModifier · BidScore (0, 80 or 100)
+ IPCUploaderModifier · PaperMatchingScore
The weights decide how much each source of information counts toward the final IpcRating:
- Equal weights (all 1): a "want to review" bid counts exactly as much as being expert in every topic of the paper.
- Raising one weight, for example
BiddingModifier = 2, makes the reviewers' own bids count twice as much as their stated expertise. - Setting a weight to 0 has the same effect as switching that part off. The
uses…flags do the same thing explicitly.
Example
A paper has 3 topics. The user is expert in 1 and knowledgeable in 1, and didn't rate the third. They bid "could review", and their PaperMatchingScore is 40.
- ExpertiseScore = (4·1 + 2·1 + 0) / 3 · 25 = 50
- BidScore = 80
- PaperMatchingScore = 40
| Weights (AoE / Bidding / IPCUploader) | IpcRating |
|---|---|
| 1 / 1 / 1 | 50 + 80 + 40 = 170 |
| 2 / 1 / 0.5 | 100 + 80 + 20 = 200 |
| 1 / 0 / 2 | 50 + 0 + 80 = 130 |
When you press create New Suggestion you set the preferences for the new matching. If you are interested how the matching is created: calculate Suggestion
Modify the matching
If you are not satisfied with the current mapping, you can start over to get a better mapping (the mapping is randomly shuffled) or adjust the system's mapping. To do this, click on the document you want to correct. You will be taken to the following page:
There you can specify who is assigned as which review type. (One person cannot be in multiple review types in one submission!) There you can access the submission, see every review type, who is currently assigned in this review type, a checkbox for every review type, the internally generated rating, the person's bid, his global expertise list. And how much expertise the person has that matches the paper. In this table: Score: value from IPCUploader (SARA TFIDF), RatingOrg: value, generated through "Initialize Rating", calculated from SARA TFIDF combined with bidding. (*5 could review, *10 want review)
External Script
- Initialize Ratings
- Initializes Rating and RatingOrg for Sort. For this purpose, the score (=affinity score, from the IPCuploader) is combined with the bidding.
- For “want review” the score is multiplied with 10, and for “could review” with 5.
- IPC members can help with sorting. Use "Manage Sorter" to invite them and assign them to submissions. Sorters are IPC members, they cannot see the authors.
- The goal is to set the ratings so that the order determined by the ratings corresponds to the preferences of a submission for IPC members.
- This type of sorting is not about assigning reviewers, but rather about whether the rating, which is formed from bidding and affinity scores, is in line with the wishes and preferences of the IPC members. The sorters can change the rating for this purpose (initially, rating = RatingORG). This new order can then be used to generate the assignment.
- After the sorting, export the ratings https://srmv2.eg.org/COMFy/Review/event/ExportSuggestions and use the external script, the result can then be imported: https://srmv2.eg.org/COMFy/Review/event/ImportSuggestions
⚠️ Warning: There is no way to change active assignments through this menu. Changing assignment is done using Edit Reviewer on the paper submissions page.
https://srmv2.eg.org/COMFy/Ipc/<ConferenceShortName>/ShowByAoeTable?format=json
⚠️ Warning: For external processing the AoE data of the IPCs can be downloaded using https.
Last step (for both alternatives)
when everything is set: "take Suggestion Live": the assignments are now active, IPC member should be informed.
- Use this button only once.
- There is no way to change active assignments through this menu.
⚠️ Warning: There is no way to change active assignments through this menu. Changing assignment is done using Edit Reviewer on the paper submissions page.