The point is that we all get faster

Add a client

Any macOS mail client that can be pointed at an IMAP server can be measured. You do not need to be its author to add it — but if you are, you are especially welcome, and you can maintain its definition from then on.

1. Set up the harness and the server as in Run it yourself, and add the benchmark account to the client.
2. Copy clients/_template.json to clients/<id>.json and fill it in. Coordinates are window-relative points, for a 1400 × 900 window; ./mailbench snapshot <id> resizes the window to that and draws what you have, so you can adjust until the red box is the reader pane (or the list) and the blue crosses sit on message rows.
3. If the client needs anything beyond clicks — an account written to a config file, an isolated profile, an installer — add clients/<id>.sh with hooks. See clients/spool.sh for one that uses all of them.
4. Run it: ./mailbench run <id>. Then open a pull request with the definition — and, if you like, the results, from ./mailbench submit.

A complete definition looks like this:

{
  "$schema": "../schema/client.schema.json",
  "id": "myclient",
  "name": "My Client",
  "vendor": "My Company",
  "homepage": "https://example.com",
  "platform": "macos",
  "source": "open",
  "maintainers": ["your-github-handle"],
  "vendor_maintained": true,
  "install": "`brew install --cask myclient`.",
  "app": {
    "path": "/Applications/My Client.app",
    "bundle_id": "com.example.myclient",
    "process": "My Client",
    "window_title": "Inbox"
  },
  "account": ["Settings ▸ Accounts ▸ Add ▸ IMAP. No encryption."],
  "parity": ["Turn on Settings ▸ Offline ▸ Download all messages."],
  "reset": ["Remove the account and quit."],
  "store": "~/Library/Application Support/My Client",
  "layout": {
    "status": "verified",
    "verified": { "version": "2.1", "os": "macOS 26.1", "date": "2026-11" },
    "row": [420, 300],
    "row_dy": 64,
    "reader": [780, 90, 590, 760],
    "list": [220, 70, 540, 800],
    "search": { "field": [1250, 20], "clear": "cmd+a,backspace", "leave": "escape" }
  },
  "caveats": []
}

Every field is described in clients/README.md and checked against the schema, which editors pick up from the $schema line.

The rules for a definition

• Measure it as it ships. The only settings a definition may change are the ones that make the client hold the whole mailbox locally (parity), and any that give its reader the same room as everyone else’s. Each one is listed, and shown on the client’s page.
• Say what a reader needs to know. If the client searches less than the full text, keeps only recent mail, or does something no other client does, add a caveat. It appears beside the number it affects.
• Watch the pane that changes. The reader for opening a message, the list for search and scroll — not the whole window.
• Declare yourself. Set vendor_maintained if you work on or for the client.
• Your client only. Changes to another client’s definition go through an issue, or a pull request its maintainer reviews.

Submit results

New hardware is always welcome: an Intel Mac, a base M-series laptop, a 60 Hz external display and a 120 Hz built-in one. Follow Run it yourself, then ./mailbench submit. It writes a run into results/ with the raw output, the machine’s details and a generated summary; you add notes on anything unusual and open a pull request. CI checks the schema, the corpus, that no home directory leaked into the files, and that the summary is exactly what the raw output gives.

A run is accepted when:

• the mailbox matched the published digest;
• the machine was calibrated, and every client in the run was measured on the same display;
• each client held the whole mailbox, or parity in run.json says what it held;
• it was measured with a released build of each client, not a development one;
• the submitter has said which clients, if any, they work on.

Results are never removed for being unflattering. They can be superseded by a later run on the same machine, and a run with a mistake in it is corrected in the open, with the history kept.

Challenge a result

If a number looks wrong, a coordinate misses, a setting gives one client an advantage, or a word on this site leans toward anyone: open an issue. A client’s author disputing their own result is the most useful kind of report there is — usually it means the definition is wrong, and the fix makes the benchmark better for everyone.

Changes to the method itself — what is measured, and how — start as an issue too, so they are discussed before they are made. One that changes results becomes a new version of the method, and old runs keep the version they were measured with.

The code

The harness is Bash, Python and Swift with no dependencies to install beyond Dovecot; the site is Waku and reads everything from the repository at build time. Tests: python3 -m unittest discover -s harness/tests, and ./mailbench validate for the data. The code is MIT-licensed; the results are CC BY 4.0, so you can quote and chart them anywhere with a link back.