One Source for Python and C: Generating Shared Constants

The previous post described a kind of bug that never crashes. A model is trained in Python and runs on a device programmed in C, and if the two programs disagree about the data, the model quietly gives wrong answers. That post also outlined three checks that stop the two sides from drifting apart.

This post builds the first check. It covers the simplest kind of disagreement: two copies of the same fixed number that no longer match.

The code is in edge-ai-toolkit/01-contract. You only need Python to follow this post.

The numbers both sides need

Some values have to be identical in the training code and in the firmware. In a typical sensor project they include:

• The sample rate, meaning how many readings the sensor takes per second.
• The window length, meaning how much time one window covers.
• The order of the sensor channels, for example which position holds the X axis of the accelerometer.
• The unit conversions, which turn the raw numbers a sensor reports into real units such as g (acceleration due to gravity).
• The value that marks a missing reading.

The most obvious approach is to type these values twice, once in a Python file and once in a C header, and try to keep them matching. A slightly better approach is to type them once and copy them across. Both approaches fail the same way: sooner or later, someone changes one copy and forgets the other. Nothing warns them, and the program on each side keeps working.

One file that generates the others

The approach that works is to make one file the only place where these values are written by hand, and to have that file generate the files each language actually uses.

I call this source file the data contract, because it records what both sides have agreed the data looks like. It produces three files:

• contract.py is imported by the Python training code.
• contract_data.h is included by the C firmware. It holds everything about the shape of the data.
• contract_runtime.h holds product settings that have not been decided yet, such as a detection threshold. It works differently from the other two and is the subject of post 1.5.

Nobody edits the three generated files by hand. To change a value, you edit the data contract and run it again.

What the contract looks like

The data contract is an ordinary Python dictionary. Here is part of it:

CONTRACT = {
    "SAMPLE_RATE_HZ": 50,
    "WINDOW_MS": 2000,

    # The order matters. The firmware fills its buffer in this order and the
    # model was trained on this order. If two entries are swapped, nothing
    # crashes; the model simply reads one sensor axis where it expects another.
    "CHANNELS": ["ax", "ay", "az", "gx", "gy", "gz"],

    "STATS": ["mean", "std", "min", "max"],
    "FEATURE_LAYOUT": "channel_major",

    "ACCEL_TICKS_PER_G": 4096.0,
    "GYRO_TICKS_PER_DPS": 16.384,
    ...
}

The six channels are the three axes of an accelerometer (ax, ay, az) and the three axes of a gyroscope (gx, gy, gz). The sensor reports raw whole numbers called ticks, and the last two values convert ticks into real units: 4,096 ticks equal one g of acceleration, and 16.384 ticks equal one degree per second of rotation.

The full file contains a few more entries. Each one is explained in the post where it matters.

Values that can be calculated are never typed

Some values follow from others. A window of 2,000 milliseconds at 50 readings per second always holds 100 readings. Six channels with four statistics each always give 24 features. If you typed these numbers in as well, they could contradict the values they come from. So the contract calculates them instead:

def derive(c):
    d = dict(c)
    d["N_CHANNELS"] = len(c["CHANNELS"])
    d["N_STATS"] = len(c["STATS"])
    d["WINDOW_SAMPLES"] = int(round(c["WINDOW_MS"] * c["SAMPLE_RATE_HZ"] / 1000))
    d["N_FEATURES"] = d["N_CHANNELS"] * d["N_STATS"]
    ...

A value that is calculated from others cannot disagree with them, because it has no independent existence.

The contract checks itself first

Before generating anything, the contract checks that its own values make sense together. One of these checks catches a window length that does not contain a whole number of readings:

if d["WINDOW_MS"] * d["SAMPLE_RATE_HZ"] % 1000 != 0:
    problems.append(
        f"WINDOW_MS={d['WINDOW_MS']} at {d['SAMPLE_RATE_HZ']} Hz is not a "
        f"whole number of samples. Rounding here means Python and C may "
        f"round differently.")

If you set the window to 2,010 milliseconds, the window would hold 100.5 readings, and the contract refuses to continue:

CONTRACT IS INTERNALLY INCONSISTENT:
  - WINDOW_MS=2010 at 50 Hz is not a whole number of samples. Rounding here means Python and C may round differently.

The reason is that the two languages round fractions differently. Python’s round() sends 100.5 to the nearest even number, which is 100, while a common C conversion such as (int)(100.5 + 0.5) gives 101. Each side would build windows of a different size, and nothing would report it. A contract that contradicts itself is worse than having no contract, because both sides would faithfully build the contradiction.

What the generated files look like

Here is the start of the generated C header:

/*
 * GENERATED FILE -- DO NOT EDIT
 * Source: data_contract.py   Contract: DC-001 r1.0   Hash: d5a07fe0a05f
 ...
 */

#define CONTRACT_HASH "d5a07fe0a05f"

/* --- sampling --- */
#define SAMPLE_RATE_HZ      50
#define WINDOW_MS           2000
#define WINDOW_SAMPLES      100

#define N_CHANNELS          6
#define CH_AX               0
#define CH_AY               1
...

The Python file holds the same values as Python variables. Both start with a warning not to edit them, and both carry the same hash: a short code calculated from every value in the contract. Change any value and the hash changes too, so the hash works like a version label that cannot be forgotten or typed wrongly.

Catching drift with one command

The contract has a --check option:

$ python data_contract.py --check
  contract DC-001 r1.0  hash d5a07fe0a05f  -- 3 generated files in sync

It generates all three files again in memory, without writing anything, and compares them with the files on disk. If they match, it reports that they are in sync. If they do not, it explains what went wrong and exits with an error code, which makes an automated build stop.

There are two common ways for the files to drift, and the check reports them differently.

Someone edited a generated file by hand. For example, someone changes WINDOW_SAMPLES to 120 directly in the C header:

CONTRACT CHECK FAILED (hash d5a07fe0a05f)
  x contract_data.h HAS DRIFTED -- someone hand-edited a generated file

  Regenerate BOTH sides; do not patch one:
      python data_contract.py

Someone changed the contract but forgot to regenerate. For example, someone changes the window to 2,400 milliseconds in the data contract and commits it without running the generator:

CONTRACT CHECK FAILED (hash 80e5c36f1fd8)
  x contract.py is OUT OF DATE -- generated from contract d5a07fe0a05f, not the current one; the contract changed and nobody regenerated
  x contract_data.h is OUT OF DATE -- generated from contract d5a07fe0a05f, not the current one; the contract changed and nobody regenerated
  x contract_runtime.h is OUT OF DATE -- generated from contract d5a07fe0a05f, not the current one; the contract changed and nobody regenerated

  Regenerate BOTH sides; do not patch one:
      python data_contract.py

This is where the hash earns its place. Each generated file records the hash of the contract it was made from. If that hash differs from the current one, the file is simply old. If the hash matches but the content does not, the contract has not changed, so the file itself must have been edited.

The fix is the same in both cases: run the generator, so that both languages are rebuilt from the same source. You never fix one side by hand.

Run it automatically

The check only helps if it runs every time, so add it to your continuous integration (CI), the automated system that builds and tests your project whenever someone pushes a change. In the companion repo, make check runs this check together with the tests from later posts.

Once the check is part of CI, a shared constant can no longer differ between the Python and the C without the build failing.

Commit the generated files

This part goes against a common habit. Most projects keep generated files out of version control, on the reasoning that anyone can regenerate them. Here, you should commit them.

The check works by comparing the generated files in the repository with what the contract produces now. If the generated files are not in the repository, there is nothing to compare against, and the check silently stops protecting you. Committing them has a second benefit: when someone changes the contract, the reviewer of their change sees exactly which values changed in both the Python and the C.

The companion repo’s .gitignore includes a comment explaining this, so that nobody “tidies up” by excluding the files later.

Why this check is not enough on its own

This check is enough for a project that feeds raw sensor readings straight into a model. In that case, everything the two sides must agree on is a fixed number: the rate, the window length, the channel order, and the units.

Most real projects do something in between. They calculate statistics over each window, convert units, combine channels, or rescale values before the model sees them. That processing has to exist in both languages: in Python to prepare the training data, and in C to prepare live data on the device. It is the same logic written twice.

A hash can check that WINDOW_SAMPLES is 100 on both sides. It cannot check that two functions, written separately in two languages, calculate the same thing.

Two people can write the feature calculation in slightly different ways. Both versions pass every check in this post, and the model on the device receives data it was never trained on.

Try it yourself

From the 01-contract folder:

python data_contract.py --check   # should report the files are in sync

Then break it on purpose. Change a number in contract_data.h and run the check again, and you will see the first failure above. Restore the file, change WINDOW_MS in data_contract.py to 2010 instead, and run python data_contract.py to see the contract refuse its own values. Afterwards, undo your changes with git checkout -- contract_data.h data_contract.py.

The next post deals with the function problem, using saved example inputs and outputs that both languages must reproduce.