What this page is, and what it is not
The pricing page promises two things that are easy to say and hard to show: every build ships with its own evaluation suite, and operations includes model migration tested against that suite before anything changes in production. This page is how those promises are kept current. It is a dated record of engineering decisions, newest first, each with its sources, what was checked, what the result does and does not support, and whether it changed how I deliver the work.
It moves at two speeds. When an engineer publishes something that changes what a buyer should expect from a system like the ones I build, the target is a dated note within 24 hours of the source, marked evaluating until a bounded test has run, with the source date, the verification date and the draft date all shown. A late discovery says it was late. Slower notes record what a test found once it has been run. In both cases a note appears only when there is something supported to say, and a quiet week gets nothing. Three words are kept apart throughout: reviewed means I read it, tested means I ran it under stated conditions, and in use means it runs in a system of mine today. A note never promotes itself from one to the next without saying so.
The operating record is the record of how my systems behave. The receipts library is the record of how other people's AI failed. This page is the record of how my methods change. The three are kept separate on purpose.
Recent engineering decisions
Two notes to start. Both are about something that was wrong before it was right, because that is the part a buyer cannot get from a sales call.
September 7, 2026. Two public demo readers could exceed their own spending caps.
Business question. A system that spends money per call needs a cap that holds under load, and a system that reads documents must never present a failed read as an empty one. The two live demos on this site, the certificate reader and the safety data sheet reader, each make one paid model call per run and advertise a cap of 5 reads per visitor and 50 per day.
Source and review date. An independent audit of the site by a second reviewer, delivered September 7, 2026, with local tests against synthetic responses that made no paid calls. Reviewed and fixed the same day.
Status and scope. In use, on this site's own demos. This is not a client system.
What was checked. The original code counted the day's ledger, called the model for several seconds, and wrote the ledger row afterward. Every request arriving inside that window saw the same count and passed, so the caps were soft under concurrent load. The same audit found that a model reply which was not the JSON asked for came back as a successful read with no rows, and that a diagnostic log line quoted the model's restatement of the uploaded page, which the page had promised never to store. The fix reverses the order, so the ledger row is written first as a reservation, the window is recounted with it in place, and an over-cap request withdraws its reservation and answers 429. A reply that does not parse is now a failed read, reported as such. The log records the shape of an empty reply, never its text.
Result and limits. Against a fake ledger with realistic latency, a burst of 20 requests from one visitor passed 5 and a burst of 120 across visitors left 50 rows. The first version of the fix shipped with a bug of its own: one reader's filter string was quoted so that it matched no rows, and the same reviewer caught it within the hour. The fix now refuses a malformed filter before any row is written. The test runs against a stand-in for the database, so it proves the ordering, not the database's own behavior under two truly simultaneous writes; that window is now one round trip instead of one model call, which is the claim, and no more than the claim.
Decision. Adopt. Reserve, then count, then spend is now the rule for anything of mine that pays per call.
Offering impact. Build implementation and operations procedure. Every build that spends per call gets a reservation-first cap and a test that hammers it, and a reply that fails to parse is a failure state, never an abstain.
September 6, 2026. A test that could not prove its own reads were complete.
Business question. An upgrade decision built on a benchmark is only as good as the benchmark's bookkeeping. If a reply cut short by an output limit is scored as the model declining to answer, the bench measures the test setup and calls it the model.
Source and review date. Found in my own code on September 6, 2026 while reading run records from a sister bench, then checked against the archived runs of the safety data sheet bench that feeds the disclosure demo. A second reviewer recomputed both benches' score files on September 7 and corrected one caveat of mine.
Status and scope. Tested on synthetic data only. The bench reads 22 synthetic safety data sheets under two conditions, a clean PDF and a degraded scan, with a known answer key of 102 ingredient rows. It is not a customer accuracy figure.
What was checked. The bench called the model with an output cap of 1,500 tokens and scored any reply that failed to parse as every row on that page missed. A reply cut at the cap fails to parse. The archived runs from September 4 show the model reader missed 0 of 102 rows under both conditions, which is consistent with no reply having been cut, but the bench recorded no stop reason, so it could not prove that. The cap was raised to 4,000 tokens and the stop reason and output length are now recorded for every call.
Result and limits. No number on this site changed, because no safety data sheet read had been cut. The point of the note is the gap it exposed, a bench whose only evidence of completeness was the absence of misses. The run records live in the bench's local archive and are not published, so this is my account of them, corroborated by the second reviewer's recomputation. The sister bench, which had been cut, was re-run into a new dated directory beside the old one; its numbers are held for a trade publication and are not on this site yet.
Decision. Adopt. Every evaluation records the stop reason of every call, and a capped reply is scored as a failure of the test setup, never as an abstention by the model.
Offering impact. Build acceptance and operations regression checks. The evaluation suite that ships with a build records completion evidence per call, and a model migration that produces capped replies fails the migration test rather than lowering the accuracy figure quietly.
Methods in use, and their scope
What each method rests on, and the honest label for it. A row only moves up this table with a note above that says why.
| Capability | What it rests on | Scope label |
|---|---|---|
| Evaluation with archived runs | Two document benches with committed manifests naming the model, the cap, the spend and the caveats. A number that appears anywhere comes from an archived run or it does not appear. | In use, on synthetic data |
| Two-reader agreement with a person on exceptions | An OCR or text-layer reader and a vision model read the same page; a row passes only when both are confident and agree, and anything else goes to a person with the page beside them. | In use, in the public demos; tested on synthetic data |
| Outcome verification | A job is called verified only when the state it was supposed to produce is checked against a source of truth by a read that could contradict it. Seven scheduled jobs are held to this on the operating record. | In use, own infrastructure |
| Bounded agent work | An internal coordination store where an autonomous worker may take only tasks marked read-only research, under a dollar ceiling per start and hourly and daily start caps, with every action logged. Anything outward-facing waits for a person. | In use, internal only |
| Spend caps that hold under load | Reserve, then count, then spend, with a test that sends bursts. See the first note above. | In use, on the public demos |
| Model migration against the evaluation suite | Promised in operations: a model change is a re-run of the build's evaluation or the old numbers stay with the old model. The rule is applied to my own benches; no client migration has been run yet. | Method defined; no client run |
| Retrieval over documents | Nothing here implements a retrieval pipeline or an embedding index. If a workflow needs one, the comparison will include the simplest baseline and will count retrieval misses, unsupported answers and stale documents separately. | A question, not a method |
How a change earns adoption
An idea advances only if it has a named buyer problem, touches work I actually do or am preparing to sell, and admits a measurable comparison. Avoided wrong actions and reduced review burden come before novelty. Then five steps, in order.
- Source review. Where the idea came from and when it was published, kept separate from when I reviewed it. A vendor announcement establishes that a vendor announced something, nothing more.
- Bounded test. A named baseline, representative cases, acceptance criteria and a budget, all written down before the run. Repeated trials where model variability matters.
- Evidence review. What the result supports and what it does not. A second reviewer where the claim will be public.
- Decision. Adopt, keep the current method, run a bounded follow-on, or reject. Rejections are recorded too.
- Controlled release. A change to a running system goes through the same gates as any other change, with rollback and maintenance cost counted in the decision.
Discovery is watched every four hours from engineer-authored posts and primary sources, with a fixed effort ceiling per day. Watching is not a response guarantee: a source can be missed, and a note waits for verification. A confirmed deprecation or security issue affecting something installed gets an impact assessment as soon as it is found; a useful architecture can wait for the weekly review.
Questions people ask
Is this a blog?
It is not a blog in the sense of a publishing schedule. A material development, usually an engineer's own post before any formal write-up, gets a dated evaluating note quickly, with a target of 24 hours from the source, and a late discovery is labeled as late. What it does not get is an adoption claim: evaluating stays evaluating until a bounded test has run, and the note is updated with the result rather than replaced. Every note keeps its original date; a later correction or reversal is linked from it, never written over it.
Why publish the failures?
Because the failures are the part a buyer cannot get from a sales call. A method that has never been wrong in public has usually never been checked in public. Both notes on this page were found by a check, one by a second reviewer's audit and one by reading a run record, and both say what was wrong before they say what was fixed.
Do you use retrieval, or RAG, in your systems?
Not today. Nothing this practice runs has a retrieval pipeline or an embedding index, and nothing on this page will call retrieval tested until a real document workflow needs it and a bounded comparison against the simplest baseline has been run. The methods table records it as a question, not a method.
Are any of these results from a client system?
No. Every result here is from the practice's own infrastructure, its public demos, or a synthetic test corpus, and each note says which. No client engagement has been signed, and nothing here is presented as one.
How does a note change what you sell?
Usually it does not. An engineering improvement changes how the existing assessment, build or operations work is delivered. The offering impact field on each note says whether it touched the assessment method, a build implementation, an operations procedure, or nothing. A new paid offer would need demand, repeatable delivery and pricing first, and an article is none of those.