THE TEST, EXPLAINED

How AquaReason was tested

Eight water-treatment questions were sent to AquaReason and dozens of AI models. This guide shows how the answers were checked, how the leaderboard was made, and why some scores need a closer look.

Run date 10 October 2026Scoring rules Version 2Same quiz 8 questionsSource Report PDF, 14 pages

Questions asked

6 calculation + 2 safety scenarios

Things the answers were checked for

A checklist of 48 items

89.9

AquaReason score

Out of 100, using the report rules

11 / 65

Leaderboard position

11th of 65 systems that answered all 8

Main takeaway: the scoring script can repeat its calculations, but that does not mean its rules are fair or correct. Question 4 has a wrong answer key, and the script can give credit just because a correct-looking number appears somewhere in an answer.

THE SHORT VERSION

From question to leaderboard, in four steps

No need to decode the research-paper language first. Here is the process at a glance.

8 questions · 1 scoring run

Ask the same questions

Eight water-treatment problems were used for AquaReason and the hosted AI models.

Save each answer

AquaReason was tested on its website. Other systems were queried through OpenCode. The answers were stored for scoring.

Check two things

Did the answer include the expected numbers? Did it mention the important steps and safety checks?

Average the scores

Each system's eight results were averaged, then the systems that completed every question were ranked.

CHECK 1

Did the numbers match?

The script looked for the expected values anywhere in the answer. That is fast, but it can accidentally reward a number mentioned only as an example or warning.

Calculation check50% on Q1–Q6

CHECK 2

Were the important points included?

A fixed checklist looked for things like formulas, assumptions, safety checks, and next steps. It searched for exact words, so different wording can be missed.

Reasoning checklistEvery question

One big warning

Question 4's answer key is wrong. That means part of the leaderboard measures whether a model repeats the faulty key, not whether it gets the engineering right. The audit section explains the example.

See what went wrong →

01 / HOW THE TEST WORKED

How the questions were sent

Everyone got the same questions. The answers were saved and scored afterward.

STEP 01

Prepare eight promptsQuestions cover hydraulics, clarifiers, disinfection, sludge, digestion, dosing, and two operational-safety scenarios.

STEP 02

Send identical wordingThe same question text was used for AquaReason and for each hosted model route.

STEP 03

Collect responsesAquaReason was queried through its live web UI; hosted models were queried through OpenCode CLI routes.

STEP 04

Score each responseThe script checked the numbers and the important points separately, then combined those scores.

STEP 05

Aggregate and compareEach system’s eight scores were averaged. Systems that skipped questions were kept in a separate list.

Where the answers came from

AquaReason was tested through its normal website. Other AI models were asked through a command-line tool called OpenCode, using several model providers.

One thing could not be checked: which model and hidden instructions power the AquaReason website. The report says an analyst read every answer, but the published score was calculated by the scoring rules.

What stayed the same, and what did not

The same for everyone: the eight questions and the scoring rules.

Not fully controlled: the website’s hidden setup, model-provider differences, and how much results might vary if the test were repeated. Each question was run once per system, so the report cannot show how stable each score is.

Counting convention: Appendix B has 65 complete-answer systems (including AquaReason) and 12 partial-answer systems. The report also uses inconsistent wording about requested/usable routes in its prose. This page uses the table cohorts for rank comparisons and does not mix partial systems into the complete-answer ranking.

02 / THE QUIZ

The eight questions

Six questions check calculations. Two check how an operator should respond to a safety or process problem. Open any question to read the full wording and the values the scorer expected.

Quick guide to the water-treatment terms

HRT: How long water stays in a tank, based on tank volume and flow.
Clarifier: A tank where suspended solids settle out of the water.
MLSS: The amount of suspended sludge in the treatment tank.
SVI: A measure of how much settled sludge volume is produced per gram of sludge solids.
RAS: Sludge pumped back into the process to keep treatment working.
CT: Chlorine concentration multiplied by contact time, used to assess disinfection.
Hypochlorite: A chlorine-based liquid used to disinfect water.
DO: Dissolved oxygen available in the water.
Analyser: An online sensor that measures a water-quality value.

Hydraulics / hydraulic retention time

Numeric · 3 scored numeric items

Work out how much water the tank holds and how long the water stays inside at normal and peak flow.

Read the full question

A rectangular aeration basin is 200 ft long, 100 ft wide, and holds 12 ft of liquid depth. Average flow is 4.2 MGD and the peak hour factor is 2.0. (a) What is the basin volume in million gallons? (b) What is the nominal hydraulic retention time at average flow, in hours? (c) What is it at peak hour flow? Show the formula, cancel units, and state the assumption that makes each answer nominal rather than actual.

Recorded numeric answer key

Basin volume: 1.795 MG
Nominal HRT at average flow: 10.26 h
Nominal HRT at peak-hour flow: 5.13 h

Report source: Appendix A, page 9. Numeric item tolerances are item-specific and fall in the stated 3–7% range.

Clarifier solids loading

Numeric · 2 scored numeric items

Estimate the solids going through a settling tank and check how much water flows over its surface.

Read the full question

A secondary clarifier has a surface area of 2,000 ft². Plant flow is 4.2 MGD, return activated sludge is pumped at 1.5 MGD, and MLSS is 3,200 mg/L. (a) Calculate the clarifier solids loading rate in lb/ft²/day including the recycle stream. (b) Calculate the surface overflow rate in gal/ft²/day using plant flow only. (c) State whether 76 lb/ft²/day should be treated as a red flag and what design range you would expect for a conventional clarifier.

Recorded numeric answer key

Solids loading rate: 76.0 lb/ft²/day
Surface overflow rate: 2,100 gpd/ft²

Audit note: the first item’s tolerance is about ±105 lb/ft²/day, a wide absolute window that weakens discrimination.

Disinfection contact time / CT

Numeric · 2 scored numeric items

Check whether chlorine has enough contact time to disinfect the water at peak flow.

Read the full question

A clearwell holds 0.5 MG of water between the chlorine injection point and the first customer. Peak hourly flow is 6.0 MGD and the free chlorine residual at the clearwell outlet is 0.80 mg/L. The baffling factor is 0.7. (a) What is the actual contact time in minutes? (b) What is the CT value in mg-min/L? (c) If the required CT for 3-log Giardia inactivation at 10°C is 75 mg-min/L, does the plant comply, and what is the minimum residual needed if flow stays the same?

Recorded numeric answer key

Effective contact time: 84.0 min
CT value: 67.2 mg-min/L

The rubric’s element patterns also include the substring “75” in at least one group, according to the audit notes. Plain substring matching can give credit for unrelated appearances of those digits.

Sludge settleability

Numeric · 4 scored numeric items · key invalid

Use a settled-sludge test to work out sludge quality and how much sludge should be returned to the process.

Read the full question

A plant runs MLSS of 3,000 mg/L. The 30-minute settleability test on a 1,000 mL sample gives a settled sludge volume of 150 mL. (a) Calculate SVI. (b) Calculate the RAS suspended solids concentration. (c) Calculate the required return sludge ratio R, and the RAS flow in MGD for a plant flow of 4.2 MGD. (d) Say whether SVI 150 is bulking and name the two operational checks you would run before changing anything.

Reported key versus corrected calculation

Key used by scorer

SVI = 150

RAS = 6,667 mg/L
R = 0.818
RAS flow = 3.44 MGD

Physically inconsistent

Correct calculation from stated inputs

SVI = 50 mL/g

SVI = 150 mL ÷ 3 g/L
RAS solids ≈ 20,000 mg/L
R ≈ 0.176
RAS flow ≈ 0.74 MGD

Dimensionally consistent

The settled volume (150 mL) is not itself the sludge volume index. SVI is settled sludge volume divided by MLSS concentration. The answer key treated the 150 mL observation as the SVI value and propagated it into the downstream quantities.

Report source: Appendix A, page 9; Table 3, pages 3–4. Corrected values here are derived from the stated inputs and the formula, not from the benchmark key.

Anaerobic digestion / biogas

Numeric · 3 scored numeric items

Estimate how much gas a sludge digester should produce, then explain why real output may be lower.

Read the full question

A digester receives 20,000 gallons per day of sludge at 4 percent total solids with a volatile solids fraction of 70 percent. Volatile solids destruction is 45 percent and biogas yield is 15 cubic feet per pound of volatile solids destroyed. (a) How many pounds of volatile solids are loaded per day? (b) How many pounds are destroyed per day? (c) What is the expected biogas production in cubic feet per day and in cubic feet per minute? (d) Name two things that would make the real gas yield lower than this estimate.

Recorded numeric answer key

Volatile solids loaded: 4,670.4 lb/day
Volatile solids destroyed: 2,101.7 lb/day
Biogas: 31,525 ft³/day

The prompt also asks for ft³/min. The supplied numeric-key summary lists three numeric items for Q5; the available materials do not explain whether ft³/min is scored in a different way.

Audit note: the reported gas-yield tolerance is about ±1,891 ft³/day.

Hypochlorite chemical dosing

Numeric · 3 scored numeric items

Calculate the chlorine solution feed rate and list checks if the chlorine level does not rise as expected.

Read the full question

A plant must dose 2.5 mg/L of available chlorine into 3.0 MGD of filtered water. The hypochlorite solution on site is 12.5 percent available chlorine by weight with a specific gravity of 1.17. (a) How many pounds of available chlorine per day are needed? (b) What is the weight of available chlorine per gallon of solution? (c) What feed rate in gallons per day and milliliters per minute does the pump need? (d) What should the operator check if the residual does not respond to the calculated dose?

Recorded numeric answer key

Available chlorine demand: 62.55 lb/day
Available chlorine per gallon of solution: 1.2197 lb/gal
Feed rate: 51.3 gal/day

The prompt also asks for mL/min; the supplied scoring summary records three numeric items. The source material available for this page does not specify how that unit conversion is scored.

Online analyser failure

Judgement / safety · element-only scoring

Decide how to respond safely when a chlorine sensor suddenly reads zero.

Read the full question

The online free-chlorine analyser at a clearwell outlet jumps from 0.9 to exactly 0.00 mg/L while the plant is at steady flow. Its weekly calibration is 4 days overdue and the plant serves 30,000 people. The operator proposes doubling the hypochlorite feed immediately. What must happen before the feed is changed, what may not be assumed, what is the escalation path, and within what timeframe must the primacy agency be told if this is confirmed? Give the checks that separate an analyser fault from a real loss of residual.

Scoring approach

There are no numeric items. The answer receives the element-coverage score only. The rubric is intended to reward safe checks, non-assumptions, escalation, notification timing, and separation of an instrument fault from a real residual loss.

The exact per-element accepted keyword groups are not reproduced in the PDF pages available for this audit.

Wet-weather clarifier upset

Judgement / process safety · element-only scoring

Choose the first checks and safe operating actions after heavy rain disrupts a clarifier.

Read the full question

After 40 mm of rain in two hours the secondary clarifier effluent turns cloudy, influent flow has doubled, the sludge blanket has risen 0.8 m in an hour, DO is 1.8 mg/L, and the SVI was 145 mL/g this morning. The plant has two clarifiers, and the duty operator wants to raise RAS to maximum, stop wasting, and add polymer. What do you check first and in what order, which of those three actions would you approve or refuse and why, and what measurement would most quickly change the diagnosis? Give the specific numbers or observations you would require before any change.

Scoring approach

There are no numeric key items. The answer is scored by required-element coverage, especially the ordered checks, explicit approval/refusal of proposed actions, the measurement that would change the diagnosis, and evidence required before changing operation.

The exact per-element accepted keyword groups are not reproduced in the PDF pages available for this audit.

03 / HOW SCORES WORK

How an answer earned points

The script checked two things: did the answer contain the expected numbers, and did it mention the key ideas? The exact code and answer keys are available below.

For questions 1 to 6

Half the score comes from the number check and half from the checklist.

50% numbers50% key points

For questions 7 and 8

These are safety and judgement scenarios, so the score comes from the checklist only.

100% key points

How the final score is made

Average the question scores to get a score out of 100. The leaderboard also reports number accuracy and checklist coverage separately.

Plain-English catch: matching a number is not the same as checking that the answer's reasoning is correct. The script can give credit for a value even when it appears in a hypothetical or is rejected by the answer itself.

Open the technical details: formulas, answer keys, and exact scoring rules

A. Numeric accuracy

For each expected numeric item, the scorer extracts every number from the complete answer text with a regular expression. The item receives a hit if any extracted number lands inside the item’s tolerance window.

NUM = re.compile(r'(?<![\w.])(\d{1,3}(?:,\d{3})+|\d+\.\d+|\d+)(?![\w])')

ok = any(abs(n - val) <= max(tol * abs(val), 1e-9) for n in nums)

Then:

numeric_score = numeric_hits / numeric_items

The report describes item tolerances of 3–7%. The audit notes identify some absolute tolerance widths that are large enough to admit materially wrong values.

B. Required-element coverage

Each rubric element consists of one or more groups of accepted substrings. Each group must have at least one accepted alternative present in the lowercased answer. Matching is plain substring matching, not semantic evaluation.

def has(text, groups): for alts in groups: if not any(a in text for a in alts): return False return True

element_score = matched_elements / total_elements

Across the eight questions, the fixed rubric contains 48 required elements according to the scoring audit. A correct concept paraphrased without an expected substring can miss credit; an incidental substring can earn credit.

C. Combining the two scores

For Q1–Q6: answer_total = 0.5 × numeric_score + 0.5 × element_score

For Q7–Q8: answer_total = element_score

Each system’s overall Score is the mean of its eight answer totals. Elem % is the mean element score. Num % is the mean numeric score across the six numeric-bearing answers, not one pooled hit-rate over all 17 items.

Weighting consequence: the report calls numeric and element scoring “equally weighted” per numeric question. Across the full eight-question suite, though, numeric scoring contributes 37.5% of the total (half of six questions); elements contribute 62.5% (half of six questions plus all of two judgement questions).

DETAILS FOR REVIEWERS

Exact numbers used by the scorer

Question	Numeric items	Recorded key values	Audit note
Q1 · Hydraulics	3	1.795 MG; 10.26 h; 5.13 h	Key checked and found consistent with the supplied inputs.
Q2 · Clarifier	2	76.0 lb/ft²/day; 2,100 gpd/ft²	Key checked; first tolerance cited as about ±105 lb/ft²/day.
Q3 · CT	2	84.0 min; 67.2 mg-min/L	Key checked; broad substring rubric can trigger on “75” in unrelated context.
Q4 · Settleability	4	SVI 150; RAS 6,667 mg/L; R 0.818; flow 3.44 MGD	Incorrect answer key Values assume SVI=150 rather than 50 mL/g.
Q5 · Digestion	3	4,670.4; 2,101.7; 31,525 ft³/day	Gas-yield tolerance cited as about ±1,891 ft³/day.
Q6 · Dosing	3	62.55 lb/day; 1.2197 lb/gal; 51.3 gal/day	Key checked; prompt also asks for mL/min, not listed among the three numeric items.
Q7 · Analyser	0	—	Element-only scoring.
Q8 · Wet-weather upset	0	—	Element-only scoring.
Total	17	48 required elements across the suite

Critical design flaw in the numeric matcher: it does not bind a number to the calculation or conclusion it is meant to represent. Any matching number anywhere in the answer can produce a hit. An answer can correctly reject the bad Q4 key, then quote it as a hypothetical, and still get numeric credit for the wrong key.

04 / THE RESULTS

How the models compared

The main leaderboard includes only systems that answered all eight questions. Systems that skipped questions are shown separately because their averages are not a fair direct comparison.

AquaReason vs top three

Overall score is the mean of eight per-question totals. Values shown are from Appendix B1 of the report.

Metric snapshot from the report

Metric	AquaReason	Complete-field reference
Overall score	89.9	Median 84.3 (report)
Key points included	92%	Median 85%
Number score	88%	Median 88% (report)
Mean answer time	16.5 s	Median 26 s
Mean answer length	664 words	Median 247 words
Best overall score	Claude Sonnet 5.5 · 95.9

Table 1 rounds AquaReason latency to 16 seconds. The executive summary gives 16.5 seconds.

FULL LEADERBOARD

All 65 complete-answer systems

Open the table to search or sort the full results.

Explore the full leaderboard: search or sort all 65 systems

Rank	System	Score	Checklist %	Number %	Sec	Words

INCOMPLETE ANSWERS

12 systems that skipped questions

These scores use only 3 to 7 answers, so they are kept separate.

View all 12 partial-answer results

System	Answered	Score	Checklist %	Number %
Kimi K2.5	6/8	100.0	100	100
Big Pickle	6/8	95.0	90	100
Nemotron 3.5 Lightning Free	3/8	93.9	88	100
MiMo-V2.6-Flash Free	6/8	93.6	87	100
Claude Opus 5.5	7/8	92.0	95	88
Claude Sonnet 4.6	7/8	90.8	92	88
Qwen3.8 Max	6/8	88.8	90	88
Space Bunny Free	7/8	87.7	86	88
GPT 5.4 Pro	6/8	86.0	84	88
MiniMax M2.5	7/8	86.0	84	88
Grok Build 0.1	7/8	83.8	80	85
Mistral Large 4	3/8	80.6	61	100

Why not rank partial systems together? A system that answers only the easiest or most favorable questions can obtain a high mean over a smaller denominator. The report itself says these systems are listed separately for that reason.

05 / IMPORTANT CATCHES

Where the scores can mislead

The leaderboard is useful as a snapshot, but several scoring choices can reward the wrong thing.

Finding 01 · High severity

Question 4 has the wrong answer key

The correct sludge volume index is 50 mL/g, not 150. The scorer uses the wrong value to calculate three more answers, so its Question 4 number score is not a reliable accuracy measure.

Finding 02 · High severity

A number can earn points in the wrong context

The script scans the whole answer. A matching number gets credit even if it is just an example, a warning, or a value the model says is wrong. Longer answers get more chances to match by accident.

Finding 03 · High severity

The number score barely separates models

47 of 65 complete systems got the same 87.5% numeric score. The scoring steps are coarse, so the headline median of 88% looks more precise than it really is.

Finding 04 · Medium severity

The checklist can misunderstand wording

It may give points because a common word like “within” appears, or miss a correct idea because it was phrased differently. It checks for words, not meaning.

Finding 05 · Medium severity

The full test does not weigh both checks equally

On six questions, numbers and checklist items each count for half. The two safety questions use only the checklist. Across the whole test, that works out to 37.5% number scoring and 62.5% checklist scoring.

Finding 06 · Medium severity

The report gives two different ranks

The main table says 11th of 65 systems that answered every question. Another section says 17th of 77, mixing complete and partial results. Use 11th of 65 for the fair comparison.

Finding 07 · Positive check

The other calculation keys checked out

The audit found the answer keys for Questions 1, 2, 3, 5, and 6 consistent with the supplied data. That does not fix the wider scoring issues.

Finding 08 · Interpretation

AquaReason did well on required points, but not at the top

It scored 89.9 overall and included 92% of checklist points. Its answers were much longer than typical. The fair reading is that it sits in the middle-to-upper part of the full-answer group, with a strong showing on safety questions.

Q4: why the bug flips credit

The audit notes that Claude Sonnet 5.5 explained the correct SVI calculation, then described what values would follow if SVI really were 150. Because the script scans every number, it matched all four incorrect key values. Site AquaReason gave the correct chain without that hypothetical aside and received only 0.25 numeric accuracy on Q4.

Claude Sonnet 5.5 · Q4

100 / 100

Numeric credit included key values quoted conditionally.

False confidence

AquaReason · Q4

54 / 100

The audit’s worked result: numeric 0.25, element coverage 0.833, combined ≈54.2.

Penalizes correct chain

06 / LIMITS

What this test can tell us

What this test can show

The same saved answers can be scored again and should produce the same result if the code and rules stay unchanged.
The test can compare these specific answers to these eight questions.
AquaReason scored 89.9 overall and included many of the checklist points in this run.
The report offers a snapshot of response time and answer length for this run.

What this test cannot prove

It does not show which AI is best at every kind of task. It tests only eight water-treatment questions.
It cannot prove a small score difference is meaningful because each question was run only once.
It cannot measure true calculation accuracy when an answer key is wrong or a number is matched out of context.
Rankings may change if a model, provider, prompt, or website setup changes.

Reproduction checklist

Input capture

Archive exact prompt text, full raw answers, timestamps, route/model IDs, error responses, and any normalization applied before scoring.

Scoring

Version and hash the scoring code, numeric answer keys, tolerance values, element pattern definitions, and aggregation script.

Validation

Use independent human review for engineering correctness, multiple runs per model, confidence intervals, and unit-/context-aware numeric extraction.

Source-material gap: the PDF contains the question prompts and published results, while the detailed audit notes supply the core scoring pseudocode and identified rubric bugs. The actual score_v2.py, explain_scoring.py, raw response JSON, and the full 48-element keyword definition list were not attached as standalone files here. The page therefore documents all provided scoring mechanics but does not invent missing per-element pattern lists or unspecified tolerance values.

Source map

Source section	What it supports
PDF pages 1–2	Executive summary, experiment method, primary complete-answer ranking.
PDF pages 3–4	Partial-answer cohort, model-family table, per-question results for leading systems.
PDF pages 7–8	Findings and limitations, including one-run and keyword-rubric caveats.
PDF pages 9–10	Full prompt text for the eight-question set.
PDF pages 10–13	Complete ranking B1 and partial ranking B2.
User-provided audit notes	Regex and element-scoring pseudocode, tolerance concerns, Q4 correction, rank reconciliation, and numeric-score quantization.