Parse + evaluate a boolean metric language: precedence, parens, good errors. A model passes this trial only when every test goes green — retries are included in the price, and the clock runs until done.
| Model | Verdict | Tries | Time | Cost to done | Tokens |
|---|---|---|---|---|---|
| claude-haiku-4-5 | GREEN | x2 | 24s | $0.0181 | 5639 |
| gpt-5-mini | GREEN | x3 | 62s | $0.0261 | 11881 |
| gpt-5.2 | GREEN | x1 | 36s | $0.0316 | 2304 |
| gpt-5.4 | GREEN | x2 | 19s | $0.0375 | 4103 |
| gpt-4.1 | GREEN | x2 | 33s | $0.0528 | 5092 |
| claude-opus-4-8 | GREEN | x2 | 41s | $0.0660 | 6660 |
| gpt-5.5 | GREEN | x2 | 29s | $0.0730 | 4033 |
| claude-sonnet-4-6 | GREEN | x3 | 74s | $0.1126 | 13161 |
| gpt-5.1 | GREEN | x4 | 60s | $0.1301 | 19944 |
| gpt-5 | GREEN | x4 | 67s | $0.1734 | 20735 |
| claude-fable-5 | REFUSED | x6 | 31s | $0.0088 | 2874 |
| claude-sonnet-5 | DNF | x6 | 2m05s | $0.1937 | 15402 |
Green row = cheapest to done · blue time = fastest to done. REFUSED = the model declined the task (a failure mode token prices never show). claude-fable-5's line is high-variance: follow-up probes saw it stochastically refuse benign coding prompts it had previously attempted. One trial per model per round; replies capped at 2,048 output tokens uniformly. Costs metered per session by cerver.
Write `solution.py` with `evaluate(expr: str, lookup) -> bool` for a tiny DSL:
avg(cpu, 5m) > 0.8 AND NOT deploy_in_progress
rate(errors, 1m) > 10 OR (p99(latency, 30s) > 250)
Grammar:
- comparisons: <fn>(<name>, <duration>) <op> <number> with op ∈ {>, <, >=, <=}
- bare identifiers are boolean flags (e.g. deploy_in_progress)
- boolean ops: NOT > AND > OR (that precedence), parentheses allowed
- durations look like 5m / 30s / 1h (pass through to lookup as strings)
- `lookup(fn, name, duration)` returns a float; `lookup("flag", name, None)`
returns a bool for bare identifiers
- raise ValueError with the offending token in the message for bad input
(e.g. ">>", unbalanced parens)
Return the COMPLETE `solution.py` in one ```python block, nothing else.
import pytest
from solution import evaluate
DATA = {("avg","cpu","5m"): 0.9, ("rate","errors","1m"): 3.0, ("p99","latency","30s"): 300.0}
FLAGS = {"deploy_in_progress": False, "maintenance": True}
def lookup(fn, name, duration):
if fn == "flag": return FLAGS[name]
return DATA[(fn, name, duration)]
def test_simple_true():
assert evaluate("avg(cpu, 5m) > 0.8", lookup) is True
def test_ops():
assert evaluate("rate(errors, 1m) <= 3", lookup) is True
assert evaluate("rate(errors, 1m) < 3", lookup) is False
def test_not_and_precedence():
assert evaluate("avg(cpu, 5m) > 0.8 AND NOT deploy_in_progress", lookup) is True
assert evaluate("NOT maintenance OR maintenance", lookup) is True
def test_and_binds_tighter_than_or():
# false AND x OR true → (false AND x) OR true
assert evaluate("deploy_in_progress AND maintenance OR maintenance", lookup) is True
def test_parens():
assert evaluate("deploy_in_progress AND (maintenance OR maintenance)", lookup) is False
assert evaluate("(avg(cpu, 5m) > 0.8) OR (p99(latency, 30s) > 250)", lookup) is True
def test_bad_op_names_token():
with pytest.raises(ValueError) as e:
evaluate("avg(cpu, 5m) >> 0.8", lookup)
assert ">" in str(e.value)
def test_unbalanced_parens():
with pytest.raises(ValueError):
evaluate("(avg(cpu, 5m) > 0.8", lookup)