Starting a new REST API project? A performance benchmark of frameworks and execution environments

Abstract

REST APIs have become widely adopted in the software industry, finding extensive usage for business-critical purposes such as data exchange, mobile app development, and microservice architectures. Such popularity has led to a proliferation of dedicated frameworks, making it challenging for developers and organizations to choose which one to use for developing resource-efficient solutions. Adding to the complexity is the possibility of adopting also different execution environments, such as GraalVM, that offer advantages such as faster startup times, lower memory footprint, and polyglot capabilities.

Some prior works have investigated the performance of various frameworks for REST APIs. Still, these studies often consider simplistic scenarios with a single endpoint, which fail to capture the complexity and diversity of real-world REST API applications. Furthermore, the impact of different execution environments on performance was often overlooked. Consequently, there remains a significant knowledge gap in comprehensively assessing the combined influence of frameworks and execution environments on the performance of REST APIs. This study aims to move a first step towards bridging that gap, by conducting a thorough performance benchmark that encompasses real-world REST APIs and considers also the effects of different execution environments.

More in detail, the study focuses on two of the most popular programming language and framework combinations for REST APIs, namely JavaScript with the Express framework and Java with the Spring framework. As for the execution environment, we consider both mainstream execution environments for JavaScript and Java (Node and OpenJDK, respectively), and GraalVM, which can execute both Java and JavaScript software. The benchmarking process involves conducting realistic load and stress tests using state-of-the-art tools. Results reveal significant differences in performance across the considered combinations, providing insights that could support developers and system architects in making more informed decisions on the technologies to use for their REST API projects.

Replication Package

The subject REST API implementations, the Docker environments to run them in the different execution environments, as well as the JMeter scripts to run the load and stress tests, are available in the replication package at . The replication package also includes the outputs of each execution of the load and stress tests on each configuration, as well as all the code necessary to compute the considered metrics, and data analytics scripts we used to analyze the raw data and produce plots and results presented in this paper.