Run it yourself

Your Mac, your clients, the same instrument

An afternoon of setup, most of it adding an account to each client and waiting for it to sync. After that, each client is one command. Everything stays on your machine until you open a pull request.

The harness clicks, types and scrolls in real windows. It never touches your own mail — the benchmark account is a separate account on a local server — but run it on a Mac you can leave alone for a while.

1. 1

   Get the harness

   A Mac with Xcode’s command line tools (for swiftc), Python 3.9 or later, and Homebrew. The instrument is built on your machine the first time it runs: a binary built on another machine’s macOS would be the instrument from a different experiment.

   git clone https://github.com/wes/mailbench.git
   cd mailbench
   xcode-select --install     # if swiftc is missing
   brew install dovecot
   ./mailbench doctor         # what this machine has and is missing

2. 2

   Generate the mailbox and start the server

   About ten seconds for 50,000 messages. It lands in ~/mailbench/maildir-50k, and the server checks it against the published digest every time it starts. For the larger mailbox, prefix both commands with CORPUS=250k; it runs beside the first, on port 10144.

   ./mailbench corpus 50k
   ./mailbench server start   # Dovecot on 127.0.0.1:10143, warmed, behind the meter

3. 3

   Grant the permissions, then calibrate

   The instrument needs Screen Recording (to watch windows) and Accessibility (to post clicks and keys), granted to the terminal you run it from. Calibration measures the instrument’s own overhead on this machine and display; results without it say so.

   ./mailbench permissions    # opens both panes in System Settings
   ./mailbench calibrate      # about a minute; a window flips colour on its own

4. 4

   Add the account to each client

   Server 127.0.0.1, port 10143, no encryption, user bench@example.com, password bench. Each client’s page lists what else it needs — above all, the settings that make it keep the whole mailbox locally. Let every client finish syncing before measuring it.

   ./mailbench clients            # which are installed, and which are ready
   ./mailbench account applemail  # adds it for you where the client allows; prints the steps where not

5. 5

   Check where the clicks land

   Coordinates are per client and per version, and a click two pixels off a row measures nothing. The snapshot outlines the watched rectangle in red and marks every click in blue. If they are wrong, fix the client’s JSON — and send the fix.

   ./mailbench snapshot mailmate

6. 6

   Measure

   One client at a time, with every other mail client quit and nothing else moving on screen. Each takes a few minutes and ends by quitting and relaunching it. Plug the Mac in, and don’t touch the mouse.

   ./mailbench run mailmate
   ./mailbench run thunderbird
   ./mailbench report             # every client on this machine, in one table

7. 7

   First sync (optional)

   Timed at the server: empty the client’s store, follow the prompt, and the meter times the download from its first busy second to its last. It also reports what share of the mailbox the client downloaded.

   ./mailbench sync thunderbird

8. 8

   Submit

   submit copies this machine’s current results into results/, removes your home directory from every path, writes the machine’s details, and generates the summary the site shows. It asks for your GitHub handle and whether you work on any of the clients measured — answered either way, and shown with the run.

   ./mailbench submit
   git checkout -b results/my-mac
   git add results && git commit -m "Results: <machine>, 50k"
   gh pr create --fill

TAG=before APP=~/builds/Before.app ./mailbench run myclient
TAG=after  APP=~/builds/After.app  ./mailbench run myclient
./mailbench report