Skip to content
Writing

Blacksmith Testbox for iOS development

4 min read

Last week I wrote about the macOS sandbox I actually want. A few days before that, I had already tried something close to it: Blacksmith Testbox.

My problem is simple. Coding agents are great at writing code, but for iOS they need a Mac to build, run tests, and open the simulator. My M1 Pro is out of room, and cloud agents run on Linux. So the agent writes code but can’t check it.

What Testbox is

Testbox is in early beta. It takes a GitHub Actions job, boots it on a Blacksmith runner (Linux or Apple Silicon macOS), and keeps it alive. Your local changes sync to it, and you run commands on it. Output streams back to your terminal.

Because it is a real Actions job, it already has your secrets, your setup steps, and your dependencies. The CLI is built for agents, not humans, so the commands and flags are explicit.

It is two commands:

# install once
curl -fsSL https://get.blacksmith.sh | sh

# generate a testbox workflow and an agent skill for the repo
blacksmith testbox init

# boot a box, returns an id right away
blacksmith testbox warmup ci-testbox.yml

# sync changed files and run a command
blacksmith testbox run --id <id> "scripts/build.sh"

# done
blacksmith testbox stop --id <id>

Here is what happens between my laptop and the Mac at each step. Click through or press play.

$0.00runner bill at $0.08/min

The sync uses rsync over SSH, so only changed files go across. After the first run, each run takes a second or two to start.

init also writes a skill file for the agent. It tells the agent to warm up a box as soon as it starts a task, never run tests locally, reuse the same box, and stop it at the end.

What I tried

I tested it with Claude Code on an iOS app. My ask was small: build the app with Xcode 27.1, launch it in the simulator, and take a screenshot.

The first warmup failed. blacksmith-6vcpu-macos-latest was still macOS 26, with no Xcode 27.1. Claude read the failed run log, found that the macos-27 runner has it, switched the label, and pinned the simulator to iOS 27.0 since that was the only runtime on that image. The second box built the app and sent back a screenshot.

Then I pushed it further. I wanted screenshots of every screen I could reach, to see how the app looks with Liquid Glass on iOS 27.

Testbox only had node, so Claude installed agent-device on the box and used it to drive the simulator: open each screen from the menu, switch the language to Arabic for RTL, and take a screenshot of each screen. Then it downloaded the images and built a gallery page.

19 screens in about 10 minutes. I didn’t touch the simulator once.

It wasn’t perfect. One screen hid its rows from the accessibility tree, so the agent tapped by coordinates. Once it tapped blind and switched the app to Dutch. It noticed from the screenshot and fixed it.

What it cost

The 6 vCPU macOS runner is $0.08 per minute, billed the same as normal Blacksmith CI.

Run Time Cost
The box I used ~18 min ~$1.44
Failed first warmup ~1 min ~$0.08
Boxes started by workflow PR triggers ~2 min ~$0.16
Total 21 min ~$1.68

The catch is idle time. A box keeps billing until you stop it or it times out, and the default idle timeout is 30 minutes. A forgotten macOS box costs about $2.40 that way. So I changed my testbox workflow to:

  • drop the default idle timeout to 5 minutes,
  • install agent-device during warmup, so the box is ready to drive the app,
  • remove the pull_request trigger that init added, which started boxes nobody used.

Why I like it

  • The agent gets a real Mac only when it needs one, and I pay by the minute.
  • It reuses my CI setup, so there’s no separate machine to maintain.
  • Each agent can get its own box, so parallel worktrees don’t fight over one simulator.
  • The blacksmith CLI is just a client. It runs on Linux too, so a Claude Code cloud session can write code on Linux and build and test on the Mac.

What you give up: no Xcode previews or debugger, and a couple of minutes of warmup on a fresh box. Out of the box you only get screenshots and recordings of the simulator, but you can install simserv on the box and watch the simulator live.

If you already use Blacksmith for CI, give Testbox a try.

aiiostools