Getting early and wide review of a document is very important, yet in practice it can be challenging. This document provides some best practices for getting document review; it does not define explicit mandatory steps.
See also the Wide Review section in the W3C Process document.
When should wide review be requested?A document is available for review from the moment it is first published. Working Groups should formally address any substantive review comment about a technical report in a timely manner.
Wide review should or must be requested:
After a First Public Working Draft is published (for most documents).
Working Groups are often reluctant to make substantive changes to a mature document, so reviewers should get a chance to send substantive technical reviews as early as possible. This is especially important for horizontal reviews.
Before a document gets advanced to Candidate Recommendation, gets updated as a Candidate Recommendation Snapshot or gets updated as a Recommendation.
For those, the W3C Process requires a Group to show that the specification has received wide review.
The objective is to ensure that the entire set of stakeholders of the Web community, including the general public, have adequate notice of the progress of the Working Group and are genuinely able to perform reviews of and provide comments on the specification. When considering requests to advance the maturity level of the document, the Team will consider who has been explicitly offered a reasonable opportunity to review the document.
You must publish an updated technical report, with the Status of the Document indicating clearly that you’re looking for wide review, before inviting review from other stakeholders. Our tooling monitors publications and propagates notifications to public-review-announce@w3.org automatically (for example, Candidate Recommendation Snapshot: Payment Request API (Call for Wide Review)).
You should then inform, and request reviews from:
Tip: consider tracking your wide review progress using a GitHub issue, such as issue #299 of the Sensors API. You can then simply point the Team to the issue.
Generate a meta-issue to track wide review steps in a GitHub repositoryYou may find it useful to create an issue in the GitHub repository of your spec to track your progress. Add the name of your GitHub repository to the field below and hit the "Create GitHub issue" button. This opens the "new issue" form in your repository, and pre-fills the body with review steps as a list of checkboxes.
Note: You will be able to edit the issue's title and body before it gets created.
The reviews provided by the horizontal groups, a subset of a full wide review, do receive special attention and much of the rest of this document focuses on how and when to conduct horizontal reviews.
How to get horizontal reviewWhen you have published a First Public Working Draft, you should work through available “self-review” materials and request review of your results in at least the following areas.
Long enough before you request a transition to CR, you should do the same again, identifying substantive specification changes since the first review. During the Last Call for Review of Proposed Amendment, you should also do the same.
The meaning of “Long enough” depends on how many changes there are, how clearly you have explained them, and how much discussion is needed to resolve issues. Pointing to 14 concise points for a small spec means a short time if they are simple fixes, pointing to 900 diffs from commits and hoping people understand them in a 300 page spec means it will take a long time to get review, and potentially a long time to also discuss and agree on how to solve the issues. If you have effectively identified issues for review during development and received feedback on them, the review time will probably be shorter. Horizontal review groups sometimes get bogged down; planning in advance is useful.
You should familiarize yourself with the rest of this document. This section is just a quick reminder for when you are in the middle of doing the work.
Recognize that horizontal review groups may be resource limited and may only be able to do one review or may have difficulty scheduling your review quickly. Give them as much time as you can, consistent with asking for review while it is still reasonable to change the technology to accommodate the issues they find.
Working with Horizontal Review labelsApplying these labels doesn’t replace the need to schedule a review of your spec at an appropriate time. (See How to Get Horizontal Review above.) Horizontal groups participants can find detailed process information here.
Day-to-day use of labelsApply the *-tracker label in your own repository to draw a horizontal review group’s attention to an issue in one of your own repositories. Horizontal review groups may also apply the label if they are interested in tracking a particular issue. Tooling will automatically notify the horizontal group that you attached the label.
If you want some specific advice from the horizontal group, describe that request in the issue thread.
Horizontal review groups may apply the *-needs-resolution label to issues they expect to be resolved before the specification moves to a new maturity level. Working Groups must not remove or add this label (not even when you close your issue).
If the horizontal group believes that an issue with a *-tracker label needs to be resolved before a transition, they may apply a *-needs-resolution label to the issue. Automatic tooling will later remove the *-tracker label.
If you close an issue with a *-tracker or *-needs-resolution label attached, do not remove the label. Keeping the label maintains the tracking if the issue is reopened, but also provides potentially useful information about what was tracked. (Closed issues in your repository have no effect on tools that check for unresolved issues.)
At the end of a review, and before attempting to transition the spec, you should clarify that a resolution is described for all of the issues with a *-needs-resolution label, and check that the horizontal group is aware of those resolutions. You don’t have to do this for issues with the *-tracker label. (The horizontal group was only tracking those issues, not expecting a particular resolution.)
Note that the label may be applied by setting it directly on the issue if you have proper rights, or by appending it preceded with a PLUS sign (+) in the issue description, +*-tracker or +*-needs-resolution.
*-needs-resolution issues at transition?
When a Working Group requests a new Maturity level, the transition will not be approved if a horizontal group has an open *-needs-resolution issue showing on the tracker boards. Note that the tracker monitors the horizontal group’s copy of the issue. Before requesting a new Maturity level, the Working Group is advised to review the tracker and contact the horizontal group to close any lingering issues.
*-needs-resolution?
As lead technical architect, the W3C Council is tasked (among many things) to assess consensus within W3C for architectural issues and to decide on the outcome of Formal Objections. When a horizontal issue gets flagged as *-needs-resolution and a Group chooses to request a new Maturity level despite the lack of consensus with the horizontal group, it is the task of the W3C Team to assess the issue and the outcome of the request. A horizontal group MAY choose to elevate an horizontal issue as a Formal Objection to elevate further the importance of an issue per the W3C Process.
In the case where an horizontal issue hasn’t been addressed and the document was allowed to move forward, it is recommended that the issue remains open in the horizontal group repository (it MAY get closed in the specification repository unless the W3C Team requests otherwise). Some issues may take years to get resolved, but that doesn’t mean those should be forgotten.
Issue trackersThe horizontal groups maintain repositories containing issues that track those raised in the WG repos. You can see a list of tracked issues on the tracker boards, one per horizontal area.
Horizontal groups participants can find detailed process information here.
FAQKeep a Disposition of Comments (DoC) Document showing comments received and a summary of their resolution.
This is also the reason that the Process clearly suggests there should be (TR) Working Drafts published when there are “significant changes that would benefit from review beyond the Working Group”, rather than every day or only twice in the life of a spec…
TR Working Drafts are also useful for reviews since they provide a dated snapshot which can be recovered when the review comments are being discussed. Trying to discuss review comments against a document which has changed out of all recognition can be a frustrating and inefficient experience.
Never ever exclude some horizontal groups from your review requests because you concluded it was irrelevant for them or they haven’t responded to your last request.
Let them make the decision that something is irrelevant to their field of expertise instead. You’re welcome to time out if you don’t hear back, and request to move forward anyway.
Don’t flag your issues with one of those *-needs-resolution labels, and don’t remove one which has been applied (you can close the issue though, if it is resolved).
Those are intended solely to be used by horizontal groups to bring special attention.
Don’t assume that the horizontal group will be able to schedule and complete a review within 2 weeks so that you can proceed to CR.
They may not even be able to find someone with availability to do the review in that time, and then they need a week or two to discuss their response after the review, and then they’ll send you comments that may require you to make substantive changes.
Note that this is a bad way to get review. In general, features should be reviewed as they are developed. Waiting for a “Last Call” for most review means that when reviews suggest changes it is far harder to make them, due to a commonly observed and logical reluctance to break deployed systems or content. – Charles McCathie Nevile 11:18, 13 October 2014 (UTC)
Abbreviations:
See the Document Review Dashboard document for information about creating a dashboard type service to facilitate document reviews.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.3