How fast is Saltcorn? If you are self hosting, does it matter how big your server is?

Very little work has gone into optimising the performance of Saltcorn, as I am a strong believer in the saying that “premature optimisation is the root of all evil.” (I think there are worse evils, but premature optimisation is very bad!) As we are consolidating the main features in Saltcorn and moving into a phase of focusing on usability, robustness, security and indeed performance, I decided to create some benchmarks and run them now before the optimisation work begins, to establish a baseline for performance. Since Saltcorn is implemented in node.js, express and PostgreSQL, you can also take this as the performance evaluation of a unoptimised application of that type.

I established five benchmarks and ran these on DigitalOcean droplets of different core counts, as well as my desktop development machine, a Mac Mini M1 running Saltcorn under Rosetta (x86 emulation). The five benchmarks are:

• static_file: serve a static file built into Saltcorn, in this case the CSS file saltcorn.css. Although this file will grow slightly with different versions, comparing to serving a larger file shows very little impact on the request throughput. There are no database lookups in the response cycle, so this should represent the maximum throughput for the current server configuration
• stored_file: serve a small file from the uploaded files. Similar to the above framework but currently this requires a database look up.
• form_view: display a view of the edit view template, with a GET request
• simple_page: display a page with a single text element as contents.
• complex page: display a page with several embedded views. I am using the front page from the forum application.

Benchmarks were run with the wrk tool through the Saltcorn CLI commands setup-benchmark and run-benchmark, introduced in 0.4.4-beta.0

In addition, to measure in browser user experience, I am generating Lighthouse reports from Google Chrome. This shows the performance as perceived by the user and accessibility and SEO issues. For this evaluation I am using the example landing page application.

If you want a quick peek at the results I have created a dashboard here: Saltcorn Performance. If you look at the benchmark dashboard, you will see the results displayed for different server core counts, systems, benchmarks and currently only one version tested. From a first peak it looks as if the throughput scales with core count all the way up to 8 cores. This is however an artefact because the macOS system reports eight cores but is also very fast in other respects. If you click the Ubuntu bar to select only be DigitalOcean droplets, and select either one benchmark or one server configuration, then you will get the unbiased view across different benchmarks or different server sizes. Let's start by looking at the different benchmarks for 1 core, the $5/m droplet:

As you can see this droplet will serve static files at 470 req/s and complex pages at 37 req/s. I think that is very decent if you are creating an internal tool to be used in a small to medium size company, or if you are creating a site for a club. I reiterate that this is a baseline and that these numbers will go up.

Let's look at how the complex page benchmark scales with the core count:

Here you see very good scalability up until 4 cores, and then saturation. The scalability is similar for the other benchmarks. The Saltcorn server is single threaded and makes no use of multiple cores (we have not yet used the cluster facility that allows an express.js application to span multiple cores), but my hypothesis for the increased performance is that node.js offloads work to the kernel and to to the database which runs on the same server. It is also possible that performance is constrained not only by cores but also by memory, which also grows in the larger servers.

The results for Lighthouse are very encouraging for an unoptimised application:

This is certainly nothing to be ashamed of and is already fairly decent. Both accessibility and best practices have easy to resolve issues, and the performance score will benefit from future work to increase the throughput as well as some reorganisation of how assets are loaded on Pages.

Conclusion about where the performance work will have the most impact: caching metadata to reduce the number of database queries for serving a single page is likely to have the largest impact. Using cluster to use multiple cores inside the node application is likely to have less impact right now and can be delayed.