We Tested 10 AI Vision Models on Real Construction Plans: The Models Were Never the Problem

Last Updated: August 6, 2026

By AJ Awan | Former EY management consultant, TOGAF certified enterprise architect, Founder of Flowtivity

We gave ten frontier AI vision models a set of real architect drawings and asked each to produce the same ten quantities a human estimator reads off by hand. The models were good, often within 6% on wall lengths and exact on door counts. Almost everything that went wrong was our harness, not their eyesight. Here is a detailed write-up of what we ran, what broke, and what we would build differently.

The headline finding: a median of four models achieved 0.03% error on a hospitality fit-out wall measurement, and a model costing $0.10 per million input tokens matched one costing $5.00. Total inference spend across the entire experiment was $15.

Can AI Vision Models Read Architectural Plans?

AI vision models can accurately extract construction quantities from architectural drawings. In our testing of ten frontier models against two completed fit-out projects, four of six models achieved within 10% accuracy on wall length measurements. Five of six models returned exact door counts matching the professional quote. Crucially, no model invented scope that was not present in the drawings, returning zero for all absent line items. The bottleneck was not model vision but harness engineering: payload limits, token ceilings, and ambiguous quantity definitions caused most failures.

The Setup: Real Drawings, Real Quotes, Ten Models

A commercial fit-out contractor quotes jobs the way most of the industry does. An estimator opens a set of architectural drawings, reads off the dimensions, and types quantities into a spreadsheet. Everything downstream of that is fixed arithmetic: rates, labour, margin, overheads. The spreadsheet does not need help with the maths. What it needs help with is the reading.

The estimator's manual work reduces to ten numbers: walls in lineal metres, ceilings in square metres, bulkheads in square metres, timber doors as a count, windows in lineal metres, feature wall paint in square metres, grid ceiling in square metres, aluminium doors as a count, frameless doors as a count, and mirrors in square metres.

That framing is what makes the problem tractable. There is no ambiguity about what "good" means. The AI's entire job is those ten numbers, and scoring is a straight comparison against what a professional estimator actually quoted. No rubric, no LLM judge, no vibes.

We had two completed jobs to test against: a hospitality fit-out with curved walls, and a smaller office fit-out with straight runs. Both came with the drawings and the finished quote.

What We Tested: 10 Models Across the Price Curve

We ran everything through OpenRouter so a single harness could fan out to every vendor. Ten models, chosen to span the price curve on purpose, because the interesting question is not only "which is most accurate" but "how cheap can you go before accuracy breaks."

Every model got an identical prompt, identical images, temperature 0, and had to return a fixed ten-key JSON object. PDF pages were rendered to images and sent as vision input, because the dimensioned set-out sheet is the whole point and it only exists as a drawing.

A full ten-model pass over a 45-page plan set cost about $1.60. Total spend across the entire experiment, including every failed run and every rerun, was roughly $15. According to our API usage logs, cost was never a constraint. That surprised us, and it changes the shape of what is worth trying.

Why Did Most Models Fail the First Run?

Seven of ten models returned nothing on the larger plan set in the first runs. The terminal filled with empty responses and parse errors. If we had stopped there, the write-up would have said frontier vision models cannot handle real construction drawings. That conclusion would have been completely wrong. The failures were all engineering problems in our harness, not model limitations.

Failure one: we blew the payload limit. A 25-page A1 plan set rendered to PNG at 150 dpi is 28 MB. The API rejects image payloads over 30 MB with an HTTP 413. Two models burned 1,120 seconds each before failing. Switching to JPEG at quality 88 took the full 45-page set to 14.1 MB and the 413s disappeared.

Failure two: we starved the models of output tokens. The original harness set max_tokens to 1500. Every model on our list is a reasoning model, and reasoning tokens are charged against that budget before a single character of JSON is emitted. Models were spending the entire allowance thinking and returning empty strings. We caught one mid-sentence, truncated while working through the take-off:

"walls: ~38m. Let's re-verify. CD 2.3 Set Out Plan. PWD: 2.84 + 2.15 = 4.99. Bar: 2.016 + 0.6 = 2.616..."

That is a model doing exactly the right work and running out of room to answer. We raised the ceiling to 8,000, then 32,000.

Failure three: we diagnosed failure two wrongly and believed our own error message. After raising the ceiling to 32,000, the same models failed again in 63 to 80 seconds. A model cannot burn 32,000 reasoning tokens in 63 seconds. The token theory was dead, but our error string said "empty content - model likely spent max_tokens on reasoning" as hardcoded text on any empty response, so every log kept confirming a cause nobody had verified.

We fixed the diagnostics to record finish_reason, the usage block, and the raw response envelope. The answer came back immediately: HTTP 200 with a completely empty body after 1,859 seconds. The request was simply too large to complete.

The failure rate tracked payload size, not model quality:

According to our debugging logs, the lesson for anyone building this is clear: instrument the failure path before you instrument the success path. We lost hours to an error message that asserted a cause it had not checked. An empty response has several very different causes, and they are indistinguishable without the envelope.

How Accurate Were the Models on Real Construction Drawings?

Once the harness stopped losing data, the actual results arrived, and they were strong. On the office job, where the quote recorded 28.3 lineal metres of new partition wall, four of six models achieved within 10% accuracy. Five of six models returned exactly 4 for timber doors, matching the professional quote. Every model correctly returned zero for the five line items not part of that job.

The result we did not expect concerns the five line items that genuinely were not part of that job. No windows, no mirrors, no aluminium doors, no new plasterboard ceilings, no bulkheads. Every single model correctly returned zero for all five. Not one invented work that was not there.

For an estimating tool, that matters more than precision. A tool that quietly adds scope is worse than useless, because the error is invisible until someone loses money on it. Every model passed that test.

Do You Need Expensive Models for Construction Takeoffs?

No. Our testing revealed that the cheapest model matched the most expensive one on accuracy. OpenAI GPT-5.6 Luna at $0.10 per million input tokens achieved 3.3% error on measured items, while OpenAI GPT-5.6 Sol at $5.00 per million input tokens achieved 3.2%. That is a 50x price gap with statistically identical accuracy. The cheaper model also ran in 63 seconds versus 1,038 seconds for the premium model.

According to our API cost analysis, if the cheap tier is competitive at reading plans, the economics of running this in production change completely. So does the case for running several models at once rather than picking one.

The Breakthrough: Ensembling AI Models for Construction Takeoffs

The most interesting finding from our research was that an ensemble of models dramatically outperformed any single model. On the hospitality job, where the quote recorded 32.56 metres of wall, the individual models scattered badly with errors between 9% and 49%. But their median was accurate to two decimal places: 32.55 metres, a 0.03% error.

This also gave us a way to check our own ground truth. The models had no knowledge of the quoted figure, and their consensus landed on it independently. According to cross-model agreement analysis, when you are working with sparse ground truth, this consensus is a genuinely useful signal that costs nothing but an extra API call or two.

But ensembling does not always work. On the office job the median was 6.6% off while the best single model was 0.7% off. The difference is bias. On the hospitality job the errors bracketed the truth, two high and two low, so the median cancelled them. On the office job five of seven models under-measured, so the median inherited that bias.

That is actionable. Where errors are unbiased noise, ensemble. Where they are systematically low, you need a calibration factor instead. Systematic under-measuring is the commercially dangerous direction because it eats margin.

The Hidden Lever: Extracting Text from Vector PDFs

Architectural PDFs are usually vector files. The dimension strings are not pictures of numbers. They are real, extractable text. One set-out sheet in our test contained 43,272 characters of extractable text and 135 numeric tokens: 2,067, 1,100, 6,223, 1,500, 2,125, and so on, in millimetres.

We were rendering those to JPEG and asking models to read them off pixels. Every wall-length error we measured might have been an OCR error we manufactured ourselves.

So we built a hybrid approach: send the page image and that page's extracted text, paired together, with instructions to use the text for exact values and the image for spatial layout (which dimension belongs to which wall).

On the office job the effect was dramatic:

Cross-model spread collapsed from 31.0 m to 10.5 m. That told us a large share of the error was reading, not interpretation. Give models exact values and they converge.

The clearest case was qwen3-vl-235b. It had returned all zeros on every image-only run and we had written it off as broken. With the text layer it scored best in the run. It was never incapable. It simply could not read the pixels. Latency improved too: three models got 12x to 120x faster, because they had been burning enormous reasoning effort squinting at dimension strings. One model went from 3,671 seconds to 30.

What Went Wrong: Findings That Reversed on the Second Job

We ran the same text-extraction technique on the hospitality job and it made everything worse. The ensemble median went from 0.03% error to 11.03%. The spread widened from 23.9 m to 36.0 m.

The likely culprit was our own optimisation. The raw text layer for 45 pages was 189,000 tokens, enough to overflow the smaller-context models outright, so we collapsed the whitespace to get it to 27,000 tokens. That whitespace was not padding. In layout-preserved extraction, column positions encode which dimension sits next to which wall. Collapsing it turned an aligned drawing schedule into an undifferentiated stream of numbers: exact values with nothing to attach them to.

We changed two variables at once, plan set and text compression, on a run where we had no clean baseline. That is a self-inflicted wound.

The same thing happened with page curation. Reasoning that most sheets were noise, we cut the 45-page set to the nine pages densest in dimensions and raised resolution from 130 to 200 dpi. It fixed reliability: every model that had never returned on that plan set came back. But it also moved the cross-model consensus from 32.55 m (essentially exact) to 21.0 m, and tripled the disagreement between models. The pages we discarded as noise were load-bearing.

With two jobs, treat any result that has not held on both as provisional. We learned that twice, the hard way.

How to Build AI Construction Takeoff Systems: What We Would Change

The models can read the drawings. That is settled. What we would change is everything around them. Based on our first-hand testing across ten models and two completed construction projects, here are the five changes that would deliver the most value.

1. Ask for the take-off, not the total

The real bar is not percentage error. It is whether an estimator would correct the output rather than redo it. A single total that is 10% off is not correctable, because nobody can tell which wall is wrong. A list of eight wall segments where one is wrong is correctable in thirty seconds. Same accuracy, completely different value. The models already do this internally, as the truncated output above shows. We were discarding the working and keeping only the answer.

2. Decompose the call

Asking one model for ten different quantities from 45 sheets in a single pass is not how estimators work. Per-item calls allow a precise definition per quantity, page selection tuned to each one, and granular failure instead of all-or-nothing.

3. Fix the definitions before blaming the model

One line item produced a 241% spread across models. That is not a vision problem. It is four models answering four different questions. "Lineal metres of new stud wall" is ambiguous on a drawing showing existing, demolished and new walls. Sharpening definitions costs nothing.

4. Measure something the business cares about

Mean absolute percentage error hides everything that matters. It cannot distinguish a model that is uniformly 10% off from one that nailed nine walls and missed a tenth entirely, and it treats over-measuring and under-measuring as equally bad when only one of them destroys margin.

5. Get more jobs, not more line items

Two jobs with ten quantities each is twenty data points and a sample size that already produced two reversals. Ten jobs measuring one well-defined quantity would be worth far more, and it is a much smaller ask of the person who has to supply the ground truth.

The Bottom Line: Are AI Vision Models Ready for Construction Takeoffs?

Frontier vision models can extract usable quantities from real architectural drawings. On every quantity where we had trustworthy ground truth, they cleared the bar we set in advance, and none of them hallucinated scope that was not in the job. The bottleneck was never their eyesight. It was payload limits, token ceilings, an error message that lied to us, ambiguous definitions, and a ground truth thin enough that most of the task could not be scored at all.

According to our total spend analysis, we spent roughly $15 on inference and considerably more than that on debugging our own scaffolding. The measurement problem is the hard part. The models are ready before your evaluation harness is.

If you are a contractor or estimator evaluating AI for your own workflow, budget your effort accordingly. The models will surprise you. Your harness will not.

AJ Awan is a former EY management consultant, TOGAF certified enterprise architect, and founder of Flowtivity, an AI consultancy specialising in workflow automation for growing businesses. This research was conducted using real architectural drawings from completed commercial fit-out projects.