Request Tracker

From SRMv2 Wiki
Jump to navigation Jump to search

Update April 2015:

  • “Send Email To Reviewer”, now distinction of reviewers possible concerning review cycles.
  • Contact chair has been introduced; a new person can get chair access for a specific paper, for example if all chairs are conflicted.
  • Additional export for meeting data is possible, all submission fields can be exported.
  • Email preview is available.
  • The conflict management has been revised, it is now event dependent, but existing conflicts can be carried over.
  • Access to user database has been restricted.
  • Sorting of review fields can be done, no fixed fields.
  • Upload of documents has become phase-sensitive: only one link is used for uploading submission, revised version, and CRC with drop down box for choosing pdf, multimedia, or ELF etc.
  • Multiple email addresses are possible for one user account.

request of EG2015 chairs, Dec. 19., 2014: possibility to have different dropdown-boxes (Accept Decline Dropdown)for further cycles

request of EG2015 chairs, Dec. 04., 2014: concerning meeting data export: additional options:

  • #pages (if available)
  • #rebuttal discussion posts
  • #IPC discussion posts
  • #IPCReviewer
  • #IPCReviewer, assigned via "self assign"
  • # of discussion postings on IPC discussion boards, coming not from primary or secondary

request of EG2015 chairs, Dec. 04., 2014:

  • change "author summary" --> "summary review", only textfield, no decision dropdown (currently with)
  • change "Senior recommendation comment" --> "Confidential Comments" with description:"Confidential comments for the IPC discussion and the chairs only - authors won't see this."

With that modifications the reviewer with the appropriate rights can give -- one recommendation -- confidentional comments -- comments to the authors

request of IPCmember of EG2015, Nov. 04., 2014:

  • Add title of paper to XML/HTML form.

request of EG2015 chairs, Oct. 15., 2014:

request of EG2015 chairs, Oct. 8., 2014:

  • More filters in "send email to"
  • "Review status" should be available in "My Reviews" for IPC members with the appropriate rights done

request of EG2015 chairs, Oct. 5., 2014:

  • IPC-Member should have the possibility to remove a reviewer he/she has assigned
  • IPC-Member should have the possibility to remind a reviewer he/she has assigned (send email again)

request of EG2015 chairs, Oct. 3., 2014:

  • Batch-Download also for Additional Material (MultiMedia files) currently not feasible due to the accepted size of the MM files
  • Navigation: Home -->My Reviews-->confname-->paperid-->Show

Click on "My Reviews" should show all reviews; click on confname should show reviews in confname

request of EG2015 chairs, Oct. 2., 2014:

  • reviewer invitation should contain paperID in the subject done
  • no double-assigning of reviewers possible done
  • adapt IPC sight on paper page, x/arrows should not be visible done
  • strict undocking of settings of reviewertypes (can see reviews, can see reviewers etc.) done
  • review overview for authors
  • introducing of contact chair, would be helpful in cases of conflicts

request of EG2015 chairs, Sep. 30., 2014:

  • prevent IPC member from assigning another IPC member done

SRMv2 wishlist EG2015 chairs, Aug. 2014

  • Affinity scores are not shown and available for sort when doing manual paper assignment

Done: In paper edit suggestion columns “scores” (IPC Uploader) and “Current Workload” are inserted.

  • Review List: all reviews from all events are shown! should have a list from IPC page! (same goes for any reviewer) -> fixed, but conference should be shown in header somewhere done
  • IPC view problematic (this refers to “My reviews” page in IPC mode, i.e., the same “Review List” ):
    • The column “Accepted” in the table list of all review assignments is strange -- should be called “review assignment accepted?” or so, and display “yes/no” instead of “accepted” done
    • no overview of grades, other reviews etc. -> the “my reviews” page is now better and shows other grades etc., but is still worse than SRM1, where you had: paperid, title, role, avg., stddev, Status (of own review), Tertiary (did your tertiaries submit accept the review request and do the reviews), Overalls (shows review scores with primary/tertiary in differently dashed boxes). done,view has been improved'
    • On the other hand, we don’t need “conference” as a column there if not all reviews are shown. The sight should be the same as the one when all reviews are shown.
    • The column “other reviews” is misleading because it also includes the own review. Change to “All Reviews”
    • The review deadline does not need to be in the list if only one event is shown. There may be different deadlines for different types of reviewers.
    • review overview is too much info, need a quick overview like previous list with scores the review overview has been cleaned and is now configurable, so that a user can click what info he want to see: The first column of Review Overview gives the possibility to display the content in columns or in the whole textwidth. So after some making familiar with this display system, everybody can use this type he/she likes more. The second “+” displays the content in columns.
  • Save bidding throws you back to SRM homepage! done
  • Bidding minor suggestion: could use radio button instead of dropdown. This would also allow making the rows less high and fit more rows on a page
  • Bidding by AoE: show #papers also for main keywords (Animation, …)
  • Bidding by AoE: SRMv1 had an interesting feature that also showed you the number of papers that you “haven’t looked at” yet. Since many papers use many keywords, when you go down the list of AoEs, you keep look at the same papers again and again. This is inefficient. (Michi 6.8.2014)
  • Bidding by preferences: AoEs are unsorted! should have same (hierarchical) sorting as Bidding by AoE!
  • Bidding by preferences/AoE: requires “Save” after each modification (before clicking on another AoE). This is very tedious! Should remember changes temporarily and “Save” saves everything!

the different bidding views have been revised

  • ordering of review fields? -> possible, however, the “Overall Recommendation” field seems to always go on top (below expertise). The order of that field should also be editable! (so that “Overall recommendation” can go after the individual comments, and the “Justification for recommendation” can come directly after) It is possible to set the order of the fields and to mark which fields should come after the mandatory fields, overall revcommendation and evaluation confidence
  • batch download of papers -> is great, however: should have a directory structure (e.g.: “papers\paper_1008”; should include additional material (maybe depending on a checkbox); should directly include offline review form (maybe also depending on a checkbox); note that for Siggraph, you can select which files you want included in the batch download (shows a list of all submissions with all their files, each line is a file with a checkbox); should also be available for chairs Batch download for MM files will be considered later.
  • remember me button doesn’t work is browser and browser setting dependent; may be due to regular updates, these will kill the session.
  • review in the paper page; here, each reviewer is listed with a list of fields, but this could be done much better in a table like it was in SRM1. If it looks just as in SRM1, then it would already be perfect. There the table was 2-column: first column: reviewer-ID (first row) and role (second row), and second column: reviewer name (first row) and review availability and score in the second row. Please change it to something like this! done,view has been adapted similiar to SRMv1-view
  • IPCUploader: when you get to that page, there is no button to go back to SRM2 (and it’s very annoying to push “back” all the time because of uploading 10 papers, the “back” of the browser takes you through all those upload forms again in reverse order…) done
  • Inspect and fix the submission form (upload of additional files and multimedia files is confusing; separate upload field for data+code?) -> fixed? -> still confusing (“Media” could also be pdf files; additional files are only required for CRC phase and should only be avilable there?) done: Only if there are required files to be uploaded, like ELF, Cover Letter etc. the link “Additional files” is visible.
  • Submission form: Additional non-anonymous files possible? They can be also uploaded as “additional media”, or as “additional files” with the label “non-anonymous material”
  • Review form: overall recommendation doesn’t show the numerical value - still not fixed; also: numerical value is not shown in review overview!
  • numbers in review overview disagree with numbers in review -> not fixed! just tested with EG2014! see here: https://eganon.cgv.tugraz.at/COMFy/Conference/EG_2014/Submission/paper1018/Show in review form overall recommendation is “Good”, that should be “7”, but review overview and the paper page shows “8”

Is a setting problem, now possible to set start value of score-/combo-boxes

  • show my review - the review is shown with boxes that disturb the reading and necessitate scrolling. done
  • When the reviews of cycle 1 are e-mailed to the authors, the “summary review” (or “senior recommendation”) is not included! -> intended behavior, many event chairs don’t want to show the senior recommendation to the authors, so we decided only to send out the reviews themselves, and make the senior recommendation visible for the authors if chairs want it.
  • check what the “senior recommendation” really means in what phase! The senior gives his recommendation to the chairs, accept, reject, discuss, or marked the paper as tabled.
  • After the first review cycle, only the primary reviewer sees the decision, secondary and tertiary don’t. Secondaries should be able to see the decision. Is configurable in the “Reviewers Types” menu (admins and elevated chairs)
  • Also, the decision is only displayed at the overview table of all reviews, not at the individual paper’s page. done
  • possibility to export bidding matrix, affinity matrix, aoe matrix -> bidding matrix can be exported to excel using copy/paste, but not IPCuploader scores.

Exporting using xml/json- Interface:

http://eganon.cgv.tugraz.at/COMFy/Conference/EG_2014/showBiddingResult?data=xml

http://eganon.cgv.tugraz.at/COMFy/Conference/EG_2014/showBiddingResult?data=json

It also works with Affinity Scores: http://eganon.cgv.tugraz.at/COMFy/Conference/EG_2014/setPreferencesForConferenceForIPC?data=json

and to add the userid: http://eganon.cgv.tugraz.at/COMFy/Conference/EG_2014/setPreferencesForConferenceForIPC?UserId=772bf90f-3173-45a6-ac8e-06da6a9f0ef2&data=json

http://eganon.cgv.tugraz.at/COMFy/Conference/EG_2014/setPreferencesForConferenceForIPC?UserId=772bf90f-3173-45a6-ac8e-06da6a9f0ef2&data=xml

  • Might be available by “IPC Scores”, but that currently leads to a “404” error; now it works again, but this list can hardly be exported to anything useful! The IPC scores are now also visible in the „Edit Suggestion“
  • im paper view sind die file sizes in bytes angegeben (z.b. 38955869). Besser wäre ein human readable format (MB oder KB je nachdem) -> better now, but what does “04,404” mean exactly? should use english decimal notation, and no leading zeroes (4.4MB for example). Better (but this is a minor suggestion): use adaptive display, so if > 1MB use MB, if <1MB use KB. done
  • menu for reviewer area always closes by itself if an item is chosen from the menu, it closes, not before
  • Siggraph-style notification emails
    • - who has posted (reviewer/author#)
    • - paper number already in the subject
    • - conf already in the subject!!
    • - paper# and(!) paper title in body
    • - message body(!!!) in body

the emails have been revised.

  • show own review in review overview, and show also reviewer ids (and they should be consistent): own name still not shown (have to memorize own reviewer id!); why are there separate review-ids? If the reviewer right is set to “see reviewer”, there will be names otherwise reviewer ids, but it will be shown the own name
  • name of “rebuttal discussion” Discussion between reviewers after the authors finished their rebuttal.
  • decline review: allow suggestion/text!
  • xml form generated by offline html: has no line breaks at all! should be formatted in same way as downloaded offline xml! this form should only be for upload, not for editing
  • Sort:
    • show # of assigned papers per IPC in each view
    • allow sorting by affinity and rating in paper view
    • allow manual score assignment

the sort has been revised.

  • AoEs: are they carried over to next event? no, not yet
  • Representative Image: fix name in review form (should be “Representative image”). done
  • Can submission fields be made mandatory (Representative image, at least one keyword)? no
  • Assign IPC reviewer: sorting by “IPCUploader” score alone not possible (only by combined rating) Is now available.
  • “Elevated IPC”: does not correspond to what we said:
    • abstract
    • representative image (!!)
    • title
    • anonymisierte scores (including average)
    • summary of the primary
    • discussion board (rebuttal discussion)
    • but NOT authors and other reviewer names or individual reviews!

done

  • Average scores are still missing from reviewstatus and “my reviews” page. done
  • Option to manually change rating for IPC members for a paper before “Create new suggestion” to influence the result of the algorithm; also, allow sorting by pure IPCUploader scores, not only combined rating check only "Use IPC Uploader scored
  • Show current IPC member load in different views, not only “by IPC member” view Now also visible in the Edit suggestion view
  • Contact chair per paper?

SRMv2 fixed issues:

  • + system doesn't know about primary/secondary
  • + no balancing of reviewing workload
  • + reviewer numbers of IPC were <100
  • reintroduce “justification for overall recommendation” field (can be manually introduced)
  • check whether “justification for overall recommendation” field is default for new conferences
  • get affinity scores (are integrated by “IPCUploader”)
  • use affinity scores for bidding! (available if bidding starts after paper submission deadline!)
  • declined reviews are shown in current reviews -> fixed (declined reviews now disappear.)
  • bidding: by preferences list hard to use (because of having to expand multiple cycles be define? (to allow anyonymous reviewer discussion)
  • general review form or batch download of all review forms? - done
  • Bidding -- is very inconvenient (abstract appears in a thin column) -- fixed
  • offline reviewing - entering the numerical value for the intermediate questions should be enough, but currently it fails! -> fixed using offline html form
  • review invitations don’t have review deadline (check templates or remind IPCs to include it); review deadline should be in system as well! -> fixed
  • bidding: list jumps around when opening/closing abstracts! fixed
  • IPCUploader: typo: header reads “IPC uplader”! fixed

request of SR2014 chairs, Apr. 25, 2014: It would be good if a 'contact' chair can be assigned to each paper in the system. When emails are send through the system to reviewers/IPC regarding this paper, this contact email should be used. This is to ensure when reviewers respond to such a reminder, that the conflicted chair does not receive the email (and thus learns about the identity of the reviewer/IPC member. Maybe even include a feature that if a shared email address is used, that this would be replaced by the contact address for conflicted papers.

request of SR2014 chairs, Apr. 09, 2014:

  • It would be great if the chair can change the role of invited reviewers. done
  • It would be really helpful if the affinity scores for the papers vs IPC members was available to the chairs.
  • It would be great if there was a way to send a reminder to for a particular review invitation. People sometimes 'loose' the invitation email, or forgot to reply to it.
  • When searching for a reviewer (add reviewer function), it would be great if the list highlights IPC members, so that we don't accidentally send a request to a IPC member.

feature request of HPG2014 chairs:

  • a "download all papers as zip file"
  • "make a list of all (titles+abstract)'s (ideally one per page)

request of SCA2014 chairs, Mar. 31, 2014:

submission instructions and template should be available without login

request of SR2014 chairs, Mar. 25, 2014:

  • Is there a way for us to check which IPC members have already used the Author Uploader, and which still need to complete this step?

Please ask the SRM admins, they can provide you the list of completed IPCUploader files.

  • change position of Evaluation Confidence and OVERALL RECOMMENDATION in the review form

that is now possible, please provide the SRM admins with the order of the review fields you want to have

request of SR2014 chairs, Mar. 21, 2014:

Reviewers should not be able to see the names of other reviewers. The discussion board should however identify whether the reviewer is a primary or secondary (on the IPC), or tertiary.

done

request of Mar. 21, 2014: done

Old events should be invisible for users.

We have now the possibility to hide events from users, if events are not completely setup or if only a limited group of users should have access, and also we can hide links from users of finished events.

request of SR2014 chairs, Mar. 04, 2014: done

„the review summary by the primary should have the options of: accept, borderline leaning accept, borderline leaning reject, reject, refer to CGF (it is ok if the available options for chair decision are the same -- we would only select accept, reject, or refer to CGF).”


New Layout: Feb. 20, 2014

  • improved usability by better structured navigation
  • review cycles and CRC visible in tabs at paper page, first cycle starts with "1"
  • possibility to set visibility of Review Cycles, Chair Cycle, Senior Recommendation Cycle independently
  • adaptable dropdown lists for "decision", "evaluation confidence" and "overall recommendation"
  • user friendly error messages
  • phase swapping only possible for admins or chairs with admin rights
  • CRC deadline, no new submissions possible, uploaded papers of conditionally accepted submissions are marked with "_n", where n is the number of current review cycle, in the CRC phase papers are marked with "_CRC"
  • in "currentReviewStatus" names of reviewers are visible
  • Adaptability of "AddSubmissionText": so the instruction text how to add a new submission can be adapted individually
  • MeetingData: creating of a dynamic semicolon separated list of the conference data


feedback by EG2014 chairs, Feb. 02, 2014:

Submission page: done

  • be able to show _all_ submission on one page (not just up to 100)
  • previous/next button should be at top and bottom
  • sort submission according to the different fields: ID, reviewing status, average score, etc.
  • show more fields: number of tertiary reviewers, number of completed reviews, average score -> currentReviewStatus
  • for chairs: show the senior recommendation / final recommendation on this page

paper sort:

  • allow chairs to change IPCs selection (with a new "chair selected" category)

recommendation:

  • senior recommendation -> put in more prominent place done: see currentReviewStatus

paper page:

  • add/remove reviewer, why on top and not (also) below? done: see navigation left
  • 'mayor revisions' -> 'major revisions' done
  • EMails to IPC: also send a (single) copy to chairs.


requests from a reviewer, sent to SRM on: Feb. 02, 2014:

  • There should be a “save draft” button when filling in the review. Now, if you want to save it’s not clear looking at the system that it’s just a temporary draft. (You have to have indicated a recommendation, I believe, but you don’t have to have filled all the fields for this to happen.)

done: Enter a value in the review, click on "submit" and you can use "Show My Review" to show your review and "Edit Review" to edit your review at any time before the deadline.

  • I received the same review twice. I declined one of the instances but the “home” page where the blue squares are keeps saying that I have 2 reviews, 0 review requests, 1 review pending, and 1 review completed. I really only had one so that’s not a big deal for me but for reviewers with more reviews that can be very confusing.

fixed

  • It seems that the authors see different reviewer numbers than the reviewers see? In the review overview it seems that I am R1146 but from the comments of the authors they seem to refer to me as R555? Really difficult to track critiques and responses…
  • As far as ease of navigation goes, I think that the required improvements are quite a few, but I’ll limit myself to the 2-3 most problematic ones:
    • From this page https://srmv2.eg.org/COMFy/Conference/EG_2014 I can go for a “new submission” but I can’t access the papers that were assigned to me, I have to go back to home, and then select the conference.
    • There should really be a “order by reviewer” view of the reviews. I find that that’s most helpful to understand the general tone of a review.
    • When looking at the rebuttal there’s no link to the reviews


requests of EG2014 chairs before Dec. 10, 2013: done

  • ordering of authors can easily be changed by clicking on up or down
  • IPC members can choose in their profile whether or not they want to get automatically created emails
  • invitation emails to reviewers – which are always sent from the SRM event email address – are now augmented with the email address of the person making the invitation
  • special email address for the chair of each event: this email address works as sender address in “send Email to IPC”, “send Email to Reviewer”, “send Email to Authors”
  • IPC specific links (to bidding and reviewing) not only on the home/ entry page, but also on the event site, so that they are easier to reach
  • interface to the bidding has been improved: expanding of abstracts and sorting by “want to review”, “could review”, “not competent”
  • “remove reviewer” link added at the top of the page as well next to the reviewer entry
  • “show my review” link for double checking and archiving instead of using the “edit review” function
  • text areas are displayed with the formatting in which they are inserted
  • hiding of the reviewer ID from authors
  • authentication cookies (two days valid) and session keys for forms (one day valid) prevent users from being kicked off and from losing their uncompleted review
Retrieved from "https://wiki.eg.org/SRMv2Wiki/index.php?title=Request_Tracker&oldid=115"