Showing content from https://github.com/w3c/web-platform-tests/issues/10053 below:
Extra CSS testsuite requirements · Issue #10053 · web-platform-tests/wpt · GitHub
#9718 broke the CSS WG's test harness. See discussion in #9718 (comment), #9940, and #9953, as well as from IRC.
The status quo is that we have a number of lints to keep the CSS test harness (and the build system it relies on working), however these do not suffice to ensure everything keeps working.
Ideally, we'd get rid of all the extra requirements the test harness (directly and indirectly through the build system), as this appears to be the only achievable path to have a common set of requirements across the entire repo (I think it's clear from prior discussions that extending the currently CSS-only ones to the whole repo is not a feasible option).
Things the build system relies on we don't lint for
Things I'm aware of missing that are required to keep the built output correct:
- We don't check that all links to files in tests point to a file in the adjacent
support/ or whitelisted top-level directories (this is, of course, impossible in the general case once JS is involved because it trivially reduces to the halting problem). (This is basically Lint tool should report broken links and documentation should reflect this #10017, I think?)
- We don't check that we have
.htaccess files that match .headers files, especially after the moving files around. ( Add support for .headers file to CSS build system #5467)
- We don't check that no wptserve server-side features are relied upon.
- The build system as run on https://test.csswg.org runs at the top-level of WPT which means any files with a
<link rel=help> could potentially be included (and the extra lints are only run in css/).
- Links in
<link rel=help> need to point to an existent anchor ( lint on css files should check spec links against bikeshed #5646)
Possible solutions
There are, as far as I'm aware, a number of ways to solve this:
- Add lints for all of the above (as far as is computationally possible).
- Address the above in other ways (e.g., 2&3 can be addressed by making the build system copy the needed files around and using wptserve on test.csswg.org).
- Make the test harness not rely on the built testsuite (this is primarily difficult when it comes to reftests, from memory of previous exploratory work).
- Replace the test harness with something that doesn't rely on the built testsuite (probably wpt.fyi).
- Don't address these and continue to best-effort fix things when someone notices they're broken.
- Don't address these and accept changes that are known to break the test harness (to e.g. reduce duplication of files).
As far as I'm aware:
- we don't have any real plan for 1, 2, 3 (all of which require someone who cares about the build system or its dependents to put up some money to resolve);
- 4 is the direction I think most people around WPT would like to see happens, but this depends on how much is needed to make wpt.fyi work for that use-case and funding that work;
- I think we can discount 6 as potentially having bad outcomes (both politically and potentially fragmenting contributions to the CSS testsuite if they fork);
- thereby leaving us with option 5 in the immediate near future.
Ways in which wpt.fyi doesn't suffice
When it comes to option 4, my understanding of the reasons why wpt.fyi doesn't currently work is (ignoring current infra issues like reliability of test runs and the fact we're currently running old versions of browsers):
- There's no way to store (or submit) results of manual test runs (which, e.g., for css-ui is a big deal because a large proportion of the testsuite is manual due to Unable to test which cursor is used #7750 which there's no real plan to fix).
- There's no way to have results of non-browser impls (which has been needed for CSS specs to reach Proposed Recommendation before, especially for features that rely on paged media).
- There's no easy view of "tests that don't have two passing implementations".
- It doesn't support grouping tests by spec (e.g., there's no view that will show both
css/css-writing-modes/ and css/vendor-imports/mozilla/mozilla-central-reftests/writing-modes-3/), yet alone tests that apply to multiple specs (e.g., css/CSS2/visuren/inherit-static-offset-001.xht which has links to both CSS2 and css-cascade).
- It doesn't support grouping tests by spec section (to show that there is coverage of the whole spec).
- It doesn't support excluding specs by spec level (e.g., for a level 3 implemenation report we need to exclude all level 4 tests).
It would be nice to know how important each of these is considered and whether it blocks moving away from the existing test harness.
RetroSearch is an open source project built by @garambo
| Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4