MAREF's Performance Numbers Are Public — Here's How We Measure Them
By MAREF Engineering
"Fast" is a marketing word. P95 < 200ms is a number you can verify. MAREF publishes its enterprise SLA benchmarks as executable test code — not a slide deck. Any engineer can clone the repo, run one command, and see the gates we assert in CI.
This post lists every performance target MAREF asserts, points at the exact source file, and explains what each one means for a production agent deployment. We'd rather you verify us than trust us.
Reproduce it yourself
All targets below are enforced by tests/benchmark/performance_benchmarks.py in the MAREF repository. Clone, install, run:
git clone https://github.com/maref-org/maref
cd maref
pip install -e ".[dev]"
pytest tests/benchmark/performance_benchmarks.py -v -m benchmark
Every test is marked benchmark and slow so it's excluded from the fast unit-test suite and run deliberately. If any gate fails, the build fails — these are hard thresholds, not aspirational targets.
The targets we assert
| Metric | Target | Why it matters |
|---|---|---|
| Screen capture | P95 < 200 ms | A desktop agent that stalls on screenshots feels broken. |
| Safety-gate decision | P99 < 10 ms | Governance must be invisible at input-event frequency. |
| 4-tier policy decision tree | P95 < 100 ms | Rule → Mode → SafetyGate → User must not add perceptible latency. |
| Audit-log throughput | > 100 ops/s | A tamper-evident trail you can actually keep up with. |
| State-machine throughput | > 500 transitions/s | Governance state transitions must scale past burst traffic. |
| GC pause | < 100 ms | Latency spikes from the garbage collector are unacceptable. |
| Idle agent footprint | < 10 MB object | A governance layer shouldn't cost you a VM. |
These are the assertion targets shipped in the repo. The benchmark module also sanity-checks the circuit breaker's failure threshold (between 1 and 10 consecutive failures) so auto-isolation triggers before an outage compounds.
Why publish benchmarks as code?
Three reasons, in decreasing order of self-interest:
- Reproducibility. A benchmark you can't run is a wish. Ours ships in the repo and runs on any CI.
- Regression protection. Every PR that would push P99 over 10ms on the safety gate fails CI. Performance can't silently rot.
- Trust. We're asking teams to put a governance layer between their agents and the world. We owe them evidence, not adjectives.
What these numbers do and don't claim
What they do claim: on the CI hardware, the governance primitives meet these thresholds, and any future commit that breaks them fails the build.
What they don't claim: they are not end-to-end latency measurements of a specific production workload on your hardware. Every deployment has its own network, model latency, and contention. That's precisely why we open the benchmarks — you can adapt them to your environment instead of trusting ours.
🛡️ Sources: MAREF repository — tests/benchmark/performance_benchmarks.py (all targets listed above; run via pytest -m benchmark). Project status: 11,000+ tests — About MAREF. Run them yourself.