For CSV leads, QA and process engineers

How we test it.

For software that reads values, one error matters most: a value read wrong that reaches your export without anyone being asked. This page says how we measure it, on which records, and what we will publish.

No figures yet. Everything we have measured so far is on synthetic records, so we publish none.

The measure: confident misreads, per form

A value Batchread is unsure of goes to a person, with the reason, and has no number in the export until it is checked. That costs a look, but nothing wrong gets through. The error that gets through is the other kind: a value read wrong, read with confidence, within its limits and possible for its unit, so no check sends it to a person. It reaches the export as a number.

So the figure we work to is the share of values let through without a check that differ from what is written, counted per form. A rate averaged over many forms can hide the one form your site uses most. A character error rate or a page-level accuracy says little about it.

We report it beside the share of values sent to a person. Sending every value to check is one way to have no misreads, and it saves nobody any work, so neither number means much alone.

What we test on today

Synthetic records made by our own generator:

  • Blank master batch records in three layouts (boxed entries, tables ruled into cells, entries on ruled lines), and forms varied in wording, order and layout, filled in with generated handwriting.
  • Scanned the ways real records are: shifted, turned, zoomed, cropped, photographed, with a curled page, and read against a blank master batch record from an older version of the form.
  • Forms held out. Some forms are never used to tune the software; they are only measured, so the results on them say how it does on a form it has not seen before.

Generated handwriting is not your operators’ handwriting, and a generated form is not your form. That is why we publish no figures from these records: they guide the work, they do not describe what you will see.

Next: forms filled in by hand, then real records

In development. Blank forms from the generator are printed, filled in by hand by people, scanned on ordinary office scanners and measured against what they wrote. Then real executed records, with the written permission of the company that owns them; we will publish results from them only with that permission.

What we will publish with every figure

  • Which records: synthetic, filled in by hand, or real, and how many forms, records and values.
  • Whether those forms were ever used to tune the software.
  • The confident misreads per form, with the uncertainty that comes from the number of values measured.
  • The share of values sent to a person to check.
  • The software and model versions measured.

No figure without its test set. If you see an accuracy figure from us without these, ask for them.

Measure it on your own forms

Our figures will be on our records. What counts for you is your forms and your operators’ handwriting, so the software gives you the means to measure it there:

  • Past batches you have already checked. Read a few and compare value by value with the values you trust.
  • Spot checks. A few values read with confidence come to you in each batch, to estimate how often confident readings on that form are wrong.
  • Typing without seeing the reading. You can choose to type a value before the software’s reading is shown, so a check is not led by it.

What ships with each release

  • The same reading every time. The same scan read again by the same versions on the same computer gives the same values, and every software and model version is recorded with what it read. Whether different processors give identical readings is still being measured.
  • A qualification pack: intended use, known limitations, test scripts and a traceability matrix, with a fixed sample of synthetic records and the results the release produces on it, misses included, so your own tests can repeat them.

The known limitations are also on the security page, and what is built today is on the status page.