A RetroSearch Logo

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Search Query:

Showing content from https://github.com/smarr/are-we-fast-yet below:

smarr/are-we-fast-yet: Are We Fast Yet? Comparing Language Implementations with Objects, Closures, and Arrays

Are We Fast Yet? Comparing Language Implementations with Objects, Closures, Arrays, and Strings

The goal of this project is to assess whether a language implementation is highly optimizing and thus able to remove the overhead of programming abstractions and frameworks. We are interested in comparing language implementations (not languages!) with each other and optimize their compilers as well as the run-time representation of objects, closures, arrays, and strings.

This is in contrast to other projects such as the Computer Language Benchmark game, which encourage finding the smartest possible way to express a problem in a language to achieve best performance, an equally interesting but different problem.

To allow us to compare the degree of optimization done by the implementations as well as the absolute performance achieved, we set the following basic rules:

  1. The benchmark is 'identical' for all languages.
    This is achieved by relying only on a widely available and commonly used subset of language features and data types.

  2. The benchmarks should use language 'idiomatically'.
    This means, they should be realized as much as possible with idiomatic code in each language, while relying only on the core set of abstractions.

For the detailed set of rules see the guidelines document. For a description of the set of common language abstractions see the core language document.

The initial publication describing the project is Cross-Language Compiler Benchmarking: Are We Fast Yet? and can be cited as (bib file):

Stefan Marr, Benoit Daloze, Hanspeter Mössenböck. 2016. Cross-Language Compiler Benchmarking: Are We Fast Yet? In Proceedings of the 12th Symposium on Dynamic Languages (DLS '16). ACM.

Disclaimer: This is an Academic Project to Facilitate Research on Languages

To facilitate our research, we want to be able assess the effectiveness of compiler and runtime optimizations for a common set of abstractions between languages. As such, many other relevant aspects such as GC, standard libraries, and language-specific abstractions are not included here. However, by focusing on this one aspect, we know exactly what is compared.

Currently, we have 14 benchmarks ported to ten different languages, including C++, Crystal, Java, JavaScript, Lua, Python, Ruby, SOM Smalltalk, SOMns (a Newspeak implementation), and Smalltalk (Squeak/Pharo).

The graph below shows some older results for different implementations after warmup, to ensure peak performance is reported:

A detailed overview of the results is in docs/performance.md.

The benchmarks are listed below. A detailed analysis including metrics for the benchmarks is in docs/metrics.md.

Micro benchmarks are based on SOM Smalltalk benchmarks unless noted otherwise.

Considering the large number of languages out there, we are open to contributions of benchmark ports to new languages. We would also be interested in new benchmarks that are in the range of 300 to 1000 lines of code.

When porting to a new language, please carefully consider the guidelines and description of the core language to ensure that we can compare results.

A list of languages we would definitely be interested in is on the issues tracker.

This includes languages like Dart, Scala, and Go. Other interesting ports could be for Racket, Clojure, or CLOS, but might require more carefully thought-out rules for porting. Similarly, a port to Rust need additional care to account for the absence of a garbage collector and should be guided by our C++ port.

Getting the Code and Running Benchmarks

To obtain the code, benchmarks, and documentation, checkout the git repository:

git clone --depth 1 https://github.com/smarr/are-we-fast-yet.git
Run Benchmarks for a Specific Language

The benchmarks are sorted by language in the benchmarks folder. Each language has its own harness. For JavaScript and Ruby, the benchmarks are executed like this:

cd benchmarks/JavaScript
node harness.js Richards 5 10
cd ../Ruby
ruby harness.rb Queens 5 20

The harness takes three parameters: benchmark name, number of iterations, and problem size. The benchmark name corresponds to a class or file of a benchmark. The number of iterations defines how often a benchmark should be executed. The problem size can be used to influence how long a benchmark takes. Note that some benchmarks rely on magic numbers to verify their results. Those might not be included for all possible problem sizes.

The rebench.conf file specifies the supported problem sizes for each benchmark.

Using the Full Benchmark Setup

Each port of the benchmarks comes with a build.sh file, which either runs any build steps needed, or with ./build.sh style runs code style checks. Though, the repository does not contain setup steps for the various languages anymore. We abandoned the idea of maintaining a full setup, since it took too much work.

Benchmark are configured and executed with the ReBench tool.

ReBench can be installed via the Python package manager pip:

The benchmarks can be executed with the following command in the root folder, assuming they have be previously built:

rebench -d --without-nice rebench.conf all

The -d gives more output during execution, and --without-nice means that the nice tool enforcing high process priority is not used. We don't use it here to avoid requiring root rights.

Note: The rebench.conf file specifies how and which benchmarks to execute. It also defines the arguments to be passed to the benchmarks.

Academic Work using this benchmark suite

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4