As the number of pipelines and datasets in the benchmark increase, the runtime also increases making the process of benchmarking longer and more prone to failed runs. As a consequence, the benchmark might halt due to timeouts or even resource allocation issues.
To ease the process of benchmarking, we should allow continued runs to be made in order to continue from where the last benchmark stopped. This will help reducing repetitive experiments.
Design
Introduce a new parameter to benchmark resume to indicate whether the run should start from the beginning or to resume from where it last stopped (by default it should be False).
benchmark(.., cache_dir, resume=True)
The benchmark will then use cache_dir to find previously completed experiments and skip the ones that already exist.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4