Refusing to Build on Guessed Values

The previous three posts built two checks that keep the training code and the device firmware in agreement. The first generates shared constants from one file. The second, golden vectors, makes both languages reproduce the same saved results, and the previous post proved that the golden test catches ten realistic mistakes.

This post adds the third and simplest check, which handles values that nobody has decided yet. It then covers the order the tools must run in, lists the kinds of disagreement worth checking for in any project, and summarises Part 1.

The code is in edge-ai-toolkit/01-contract.

Values that are not known yet

Some settings cannot be chosen at the start of a project. Two examples from the companion code:

• The detection threshold. The model outputs a score between 0 and 1, and the device reports a detection when the score is above some value. The right value depends on how the model performs on real data, so it should be chosen by testing on a validation set, a set of recordings kept aside for exactly this kind of tuning.
• The alert hold time. Once the device raises an alert, it should stay raised for a while so that it does not flicker on and off. The right length depends on how the device behaves in practice, so it has to be measured on the bench.

The tempting approach is to type in a reasonable-looking value, such as 0.5, and move on. The problem shows up later. Six weeks from now, nobody will be able to tell whether 0.5 was a carefully tested choice or a placeholder someone meant to replace. A guessed value looks exactly like a decided one.

Mark them as undecided

The contract marks these values explicitly, using Python’s None:

"DETECTION_THRESHOLD": None,      # <-- pick on a validation set
"ALERT_HOLD_S": None,             # <-- measure on the bench

For each value that is still None, the generator writes a C #error directive into contract_runtime.h. An #error stops the compiler with a message, so any firmware that includes this header cannot be built:

#error "DETECTION_THRESHOLD is unset in data_contract.py. Refusing to build on a guessed constant."
#error "ALERT_HOLD_S is unset in data_contract.py. Refusing to build on a guessed constant."

Here is what happens when you try:

$ echo '#include "contract_runtime.h"' | cc -x c -c - -o /dev/null
In file included from <stdin>:1:
contract_runtime.h:18:2: error: #error "DETECTION_THRESHOLD is unset in data_contract.py. Refusing to build on a guessed constant."
contract_runtime.h:20:2: error: #error "ALERT_HOLD_S is unset in data_contract.py. Refusing to build on a guessed constant."

Once someone makes a decision and enters it in the contract, for example "DETECTION_THRESHOLD": 0.62, the generator writes an ordinary definition in place of that error:

#define DETECTION_THRESHOLD 0.62f

The other #error stays until the hold time is decided too.

This can feel heavy-handed, since it stops the firmware from building at all. That is the point. A missing decision becomes a loud failure that someone has to deal with, instead of a quiet default that nobody remembers choosing.

The Python tests check this behaviour as well. One test confirms that each open value is either None or a real number, and another tries to compile contract_runtime.h and checks that it fails while any value is undecided.

Why the contract needs two headers

The first time I built this, I put everything in one header. Then I tried to build the golden test, and it refused to compile, because the detection threshold was still undecided.

The #error was doing exactly its job, and it revealed a real design problem. The values in the contract fall into two groups that become known at different times:

• The shape of the data is decided at the start: the channel order, the units, the feature layout, the missing-reading marker and the test tolerances. None of it waits on a measurement.
• Product settings, such as thresholds and hold times, can only be decided later, once there is real data to measure.

With one header, nothing could be tested until every product setting was final. That is the wrong way round, because the shape of the data is the thing you want to check first and most often.

So the contract generates two C headers:

contract_data.h describes the shape of the data and never contains an #error. The feature code in features.c includes only this header, so the golden test builds and runs from the first day. contract_runtime.h holds the product settings. Only the parts of the firmware that act on them, such as the code that raises an alert, should include it. Those parts cannot be built until the decisions are made, while the feature code is tested all along. The companion repo contains no device firmware, so nothing in it includes contract_runtime.h yet.

The tools must run in order

The tools in 01-contract depend on each other, so they have to run in a fixed order:

python data_contract.py       # 1. generate the contract files
python gen_golden.py          # 2. generate golden vectors from them
make run                      # 3. check that the C code reproduces them
python test_contract.py       # 4. check that the Python code does too

Step 2 imports the Python file that step 1 generates, and it records the contract’s hash in both golden vector files. So whenever the contract changes, both steps must run again. make generate runs steps 1 and 2 together.

If someone changes the contract and runs step 1 but forgets step 2, the golden vectors still describe the old contract. The C test checks for this before comparing anything, and it refuses to run:

CONTRACT HASH MISMATCH
  contract_data.h    80e5c36f1fd8
  golden_vectors.h   d5a07fe0a05f

  The golden vectors were made from a different version of the contract.
  Run: python data_contract.py && python gen_golden.py

This check matters more than it might seem. Golden vectors made from an old contract would compare the C code against outdated answers. If the C code happened to match them, the test would report success while checking the wrong thing, which is worse than having no test.

In everyday use, make check runs all of these checks in the right order, and python gen_golden.py --check usually reports the problem first. The hash check inside the C test is there for when the test is built on its own, for example when you compile it for the microcontroller.

What to check in your own project

The examples in this series come from one project, but the same kinds of disagreement appear in almost every system that trains in one language and runs in another. Here are the ones I have seen most often:

• Channel order, especially when the firmware fills its buffer in whatever order a loop happens to read the sensors.
• Units and scaling, meaning raw sensor readings against real units, and the factor that converts between them.
• The missing-reading marker, particularly when a cleared buffer quietly supplies zeros.
• Quantisation settings. Models that run on microcontrollers often use 8-bit integers instead of decimal numbers. Converting between the two uses a scale and a zero point, and both sides must use the same values.
• Normalisation values. Many models expect inputs adjusted using a mean and spread calculated during training. The device must use exactly the same numbers.
• Window length, step and overlap, meaning how far each window moves forward, and how much consecutive windows share.
• The standard deviation formula, where numpy and pandas use different defaults.
• 64-bit numbers in training and 32-bit numbers on the device, and any calculation where that small difference grows.
• Feature layout, meaning the order in which features are placed in the list the model receives.
• Byte layout of data sent between devices, including how a C structure is packed in memory and the order of bytes within each number.

If you have used tools such as TensorFlow Lite for Microcontrollers or Edge Impulse, several of these will be familiar. Each one belongs in one of the checks from this series: a value in the contract, or a scene in the golden vectors.

What these checks do not do

It is worth being clear about the limits, so that nobody assumes more protection than exists.

They do not check that your data is good. The checks guarantee that both sides process data the same way. They will carry a poor dataset from one end to the other just as faithfully. Labelling, how the data was collected, and honest evaluation are separate problems.

They do not check that your model is good. Nothing here measures accuracy or judges the model’s design.

They do not make your C code fast. The companion code is written to be easy to read. If you optimise it, the golden vectors are what make that safe, because they show immediately if an optimisation changed a result.

Part 1 in summary

A model is trained in Python and runs on a device in C. If the two sides disagree about the data, nothing crashes. The model simply gives wrong answers that look right, for as long as the device runs.

Part 1 built three checks that stop this from happening silently:

1. Shared constants come from one file. The contract generates the Python and C versions, and --check fails the build if either one drifts.
2. Golden vectors check the shared calculation. Both languages must reproduce saved results within a measured tolerance, and the sabotage script proves the test catches realistic mistakes.
3. Undecided values stop the build. The firmware cannot be compiled until someone makes the decision, and the two-header split keeps everything else testable in the meantime.

None of this makes a model better. What it does is let you trust that the number the device produces is the number the model was trained to produce. In my experience, that trust is usually what is missing.

Try it yourself

Everything in Part 1 runs on a laptop with Python, numpy and a C compiler. From the 01-contract folder:

make check           # run all three levels of checks
python sabotage.py   # break the C code ten ways and watch the test catch each one

Part 2 moves from the code to the data itself. It starts with why the file format you store your training data in is also part of the contract, and what ten storage options actually cost when measured on a Raspberry Pi.