Golden Vectors: Testing That Python and C Calculate the Same Features

The previous post made sure that the fixed numbers shared by the training code and the firmware, such as the window length and the channel order, come from a single file. It ended with a problem that approach cannot solve: most projects also share a calculation, written once in Python and once in C, and a hash cannot tell you whether two functions calculate the same thing.

This post introduces golden vectors, the second of the three checks in Part 1. The idea is short to state, and most of this post is about the details that make it work in a real project.

The code is in edge-ai-toolkit/01-contract. To run the C test you need a C compiler, such as gcc or clang, which most Linux and macOS machines already have.

The calculation both sides share

The example project calculates features from each window of sensor data. For each of the six sensor channels, it does the following:

1. It skips readings marked as missing.
2. It converts the raw readings (ticks) into real units.
3. It calculates four statistics: the mean, the standard deviation, the minimum and the maximum.
4. It writes each result into a fixed position in a list of 24 features.

This logic exists twice. features.py uses it to prepare the training data, and features.c uses it on the device to prepare live data for the model. The two versions are written in different languages, often by different people, and every step above is a place where they can differ: a missing reading handled differently, a unit conversion applied in the wrong place, or a result written into the wrong position.

The idea

Run the Python version on a set of chosen input windows and save both the inputs and the exact outputs. Commit those numbers to the repository. Then require both the Python and the C versions to reproduce them, and fail the build if either one does not.

These saved input and output pairs are the golden vectors. The Python version is the reference because the model was trained on its output, so its output defines what the model expects to receive.

This changes what the C code is. It is no longer the same logic written a second time and trusted to be correct. It is code that must produce these exact numbers, or the build fails.

A comparison with recipes

Think about translating a recipe into another language.

You can check the list of ingredients word by word: 200 g of flour on both sides, three eggs on both sides. That is what the check in the previous post does, and it works well for ingredients.

The method cannot be checked that way. An instruction like “fold gently until just combined” can be translated into something that reads correctly but produces a different cake.

A golden vector is like baking the cake once with the original recipe, weighing and measuring the result, and writing it down: 500 g and 8 cm tall. The translated recipe now has to produce a cake of 500 g and 8 cm. How the instructions were worded no longer matters, because the result is what you check.

What a golden vector looks like

Here is one of the ten saved examples, which the repository calls scenes. In this scene, called channel_distinct, every channel holds a different constant value for all 100 readings in the window: 1000 ticks on the first channel, 2000 on the second, and so on up to 6000.

{
  "name": "channel_distinct",
  "catches": "channel order and feature layout",
  "input": [[1000, 2000, 3000, 4000, 5000, 6000],
            [1000, 2000, 3000, 4000, 5000, 6000],
            ... 98 more rows ...],
  "expected": [0.244141, 0.0, 0.244141, 0.244141,
               0.488281, 0.0, 0.488281, 0.488281,
               ...
               366.210938, 0.0, 366.210938, 366.210938]
}

The expected output is easy to check by hand. The first channel is an accelerometer axis, where 4,096 ticks equal one g, so 1000 ticks is 0.244141 g. The value never changes, so the mean, the minimum and the maximum are all 0.244141, and the standard deviation is zero. The last channel is a gyroscope axis, where 16.384 ticks equal one degree per second, so 6000 ticks gives 366.210938.

Because every channel has a different value, this scene shows immediately if two channels are swapped or a result lands in the wrong position. Each scene is designed to catch particular mistakes, which is what the catches field records. The next post explains how the ten scenes were chosen.

One generator, two files

The script gen_golden.py runs the Python reference on every scene and writes the results into two files:

File	Used by	Why this format
golden_vectors.json	The Python tests	Easy to read, and changes show up clearly when a change is reviewed
golden_vectors.h	The C test	A C header needs no JSON parser and no file system, so the test can also run on a microcontroller

Both files come from the same generator in the same run, so they cannot disagree with each other. Like the files in the previous post, both are committed to the repository and both carry the contract’s hash. Post 1.5 uses that hash to make sure the golden vectors were made from the current contract.

Why the comparison needs a tolerance

This detail decides whether golden vectors survive in a real project.

As post 1.1 mentioned, Python usually calculates with 64-bit floating point numbers (float64), while microcontrollers usually use 32-bit ones (float32). A 32-bit number keeps about seven significant digits, so the two sides almost never agree to the last digit. They are not supposed to.

A test that demands exact equality would fail all the time, and within a week somebody would switch it off. A test that has been switched off is worse than no test, because people still believe it is protecting them.

So the contract defines two tolerances, and both languages use the same values:

"GOLDEN_REL_TOL": 1e-4,
"GOLDEN_ABS_TOL": 1e-3,

A value passes if it is within either tolerance of the expected value. The relative tolerance of 1e-4 allows a difference of 0.01% of the expected value, which suits large numbers. The absolute tolerance of 1e-3 allows a fixed difference of 0.001, which is needed for values that should be exactly zero, where a percentage has no meaning.

How big is the gap?

A tolerance only works if ordinary rounding differences fall well below it and real mistakes fall well above it. I measured both on the companion code.

Across all 240 comparisons, the largest relative difference between the C results and the Python results was about 0.00001% (1.3e-7). That is roughly 800 times smaller than the tolerance.

On the other side, I measured the ten deliberate mistakes from the next post. The smallest one changes the standard deviation by 0.5%, which is 50 times larger than the tolerance. Every other mistake produces a larger difference.

The space between ordinary rounding and the smallest real mistake is about 40,000 times wide, and the tolerance sits inside it with room on both sides.

There is one honest limit to mention. The absolute tolerance of 0.001 is generous for small values. Accelerometer features are measured in g and are often well below 1, so a 0.5% error in a small accelerometer value can be smaller than 0.001 and pass. In the companion code this does not let any mistake through, because the same mistake also affects the gyroscope features, whose values are much larger and fail clearly. It does mean that the test catches some mistakes in only one group of features, so it is worth knowing how the tolerances interact with the size of your values.

Running the C test without hardware

The C test, test_golden.c, includes golden_vectors.h, runs features.c on every scene and compares the results. It needs nothing beyond a C compiler, so it runs on a laptop or a CI server in about a second, with no board connected:

$ make run
cc -std=c99 -Wall -Wextra -Wpedantic -O2 -o test_golden test_golden.c features.c -lm
./test_golden
ok   all_zero           zero is a valid reading, not the sentinel
ok   channel_distinct   channel order and feature layout
ok   scale_split        accel vs gyro scale factors not swapped
...
ok   realistic          ordinary motion, nothing round or symmetric

10 scenes, 240 feature comparisons, 0 failures

When the C code is wrong, the test names the scene, the feature, both values, and the difference. Here is part of the output after the standard deviation formula in features.c was changed on purpose:

FAIL alternating        (ddof: population vs sample std)
       ax_std     [ 1]  C 0.245371        Python 0.244141        diff 0.00122994
       ...
       gx_std     [13]  C 61.342636       Python 61.035156       diff 0.30748

This is what makes golden vectors practical. The feature code on the device is checked on every change, without anyone needing to find a board, flash it and watch it. Because the vectors are a plain C header, you can also compile the same test for the microcontroller itself, which is worth doing occasionally, since a device compiler can handle floating point slightly differently from a laptop compiler.

The Python side is checked too

Golden vectors also protect against changes on the Python side. The test test_python_reproduces_the_golden_vectors in test_contract.py runs features.py on every scene and compares the results with the saved outputs, using the same tolerances.

This matters because the saved outputs only describe the Python code as it was when they were generated. If someone changes features.py later without regenerating the golden vectors, this test fails. They then have to decide deliberately: either the change was a mistake, or the feature calculation really should change, in which case they regenerate the vectors and update the C code to match.

You can confirm that the saved vectors still match what the generator would produce now:

$ python gen_golden.py --check
  golden vectors in sync  (10 scenes, hash d5a07fe0a05f)

Regenerating golden vectors should always be a deliberate decision, made when the feature calculation is meant to change. Running the generator just to make a failing test pass removes the protection the test was giving you.

What comes next

A golden test is only as good as the inputs it checks. A test with ten random windows could pass even if the C code has a serious mistake that those windows happen not to reveal. The next post explains how the ten scenes were chosen, each one designed to catch a specific mistake, and then breaks the C code on purpose in ten different ways to prove that the test catches every one.