Showing content from https://github.com/mlr-org/rush below:
mlr-org/rush: Package designed to solve large-scale problems asynchronously across a distributed network
Package website: release | dev
rush is a package designed to solve large-scale problems asynchronously across a distributed network. Employing a database centric model, rush enables workers to communicate tasks and their results over a shared Redis database. Key features include low task overhead, efficient caching, and robust error handling. The package powers asynchronous optimization algorithms in the bbotk and mlr3tuning packages.
- Database centric model for robust scalability.
- Efficient communication between workers using
Redis.
- Maintains low overhead, limiting delays to just a millisecond per task.
- Reduces read/write operations with a lightweight and efficient caching system.
- Offers centralized system features, such as task queues.
- Provides fast data transformation from Redis to
data.table.
- Simplifies local worker setup with
processx.
- Enables scaling to large remote worker networks via the
mirai package.
- Automatically detects and recovers from worker failures for high reliability.
- Logs worker messages directly into the Redis database using
lgr.
- Designed with minimal dependencies for lightweight integration.
Install the latest release from CRAN.
Install the development version from GitHub.
And install Redis.
- The rrq package is a task queue system for R using Redis.
- The future package provides a simple and uniform way of evaluating R expressions asynchronously across a range of backends.
- batchtools is a package for the execution of long-running tasks on high-performance computing clusters.
- The mirai package evaluates an R expression asynchronously in a parallel process, locally or distributed over the network.
RetroSearch is an open source project built by @garambo
| Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4