Python Hardware

Blackchirp lets contributors and users write hardware drivers in Python without modifying or recompiling the application. From the contributor’s side, a Python driver is a trampoline: a C++ class that inherits from one of the hardware base classes (AWG, Clock, FlowController, FtmwDigitizer, …) and from PythonHardwareBase via multiple inheritance. The hardware base supplies the Qt slot/signal API the rest of Blackchirp consumes; the mixin owns a child Python interpreter, a PythonProcess that talks to it over JSON-lines IPC, and the lifecycle plumbing that ties the two together. Each pure virtual on the hardware base is reimplemented as a JSON method dispatch through pu_process->sendRequest().

This page documents what a contributor needs to add new C++ behavior to the Python hardware stack: the IPC architecture, the proxy injection model, the three state-management patterns the trampolines fall into, and the recipes for adding a new trampoline class or a new push-style proxy. The user-facing perspective — writing a driver script, picking a profile, hot-reloading from the UI — lives on Python Hardware and its sub-pages, and the class-level API is on PythonHardwareBase and PythonProcess.

Subprocess and JSON IPC

A Python trampoline does not embed an interpreter. Each instance launches a fresh python3 (or per-profile environment, see Per-profile Python environment) under a Qt QProcess running python_hw_host.py, which loads the user driver, injects a small set of proxy objects onto it, and dispatches calls received on stdin. The Python heap therefore lives in a separate OS process, so a script crash cannot corrupt Qt state, no GIL ever touches the Blackchirp main thread, and the C++ side has no compile-time dependency on a particular Python version. The cost is one IPC round trip per call, on the order of a millisecond, which is negligible against typical instrument I/O latencies of tens to hundreds of milliseconds.

The wire format is one compact JSON object per line in each direction. Four message kinds travel over the channel:

Method calls (C++ → Python). Carry an integer id and a method name; any other keys are forwarded as keyword arguments to the snake_case method on the user driver. Responses carry the same id and either result on success or error plus traceback on failure. The host script’s generic dispatch means adding a new hardware-specific method requires no host-script change — declaring a Python method with the same name as the trampoline’s IPC payload is enough.
Relay requests (Python → C++, interleaved). The host script uses these to reach back through the C++ side for services it cannot perform itself: self.comm.query/write/read_bytes are relayed as "relay": "comm_query" (etc.) and serviced against the trampoline’s CommunicationProtocol; self.settings.get/set are relayed against the SettingsStorage callbacks the mixin installs.
Log messages (Python → C++, unsolicited). Lines containing "log" and "level" are forwarded to bcLog() with the appropriate severity, so script output flows into the hardware log panel beside C++ driver output without per-trampoline wiring.
Waveform pushes (Python → C++, unsolicited). Push-style hardware (the digitizer trampolines) sends raw shot data as "waveform": "<base64>" with a shots count; the bytes decode and surface on the PythonProcess::waveformReceived() signal for the trampoline to drain into the WaveformBuffer.

PythonProcess push model

PythonProcess is the C++-side endpoint of the IPC channel. It owns the QProcess, holds the CommunicationProtocol pointer used to service relay requests, and exposes the synchronous PythonProcess::sendRequest() API.

Reads are event-driven. QProcess::readyReadStandardOutput is connected to onReadyRead, which appends to a buffer, splits on \n, and dispatches each complete JSON line by message kind:

line contains "log"      → emit bcLog at the parsed severity
line contains "waveform" → base64-decode, emit waveformReceived
line contains "relay"    → handle relay, write response back
line contains "id"       → store, emit responseReady

sendRequest does not poll. It writes the request, sets up the expected id, and runs a nested QEventLoop until the matching response arrives, the subprocess dies, or the configured timeout (30 s by default) fires:

QEventLoop loop;
connect(this, &PythonProcess::responseReady,
        &loop, &QEventLoop::quit);
connect(p_process, &QProcess::finished,
        &loop, [this, &loop]() {
            onReadyRead();   // drain remaining stdout
            loop.quit();
        });
QTimer::singleShot(d_timeoutMs, &loop, &QEventLoop::quit);
loop.exec();

Because the loop continues to process events, relay requests, log messages, and waveform pushes are all dispatched correctly while a method call is in flight. readyReadStandardOutput may deliver partial lines, so the buffer accumulates bytes until a newline appears.

Reentrancy contract. The nested event loop fires waveformReceived from inside sendRequest. The trampoline’s slot must therefore not re-enter sendRequest: the only safe operation is to forward the bytes into the WaveformBuffer. PythonFtmwDigitizer::onWaveformReceived calls FtmwDigitizer::emitShot() and returns; that is the shape every push-style handler must follow. Reentering sendRequest from a slot the nested loop has dispatched serializes incorrectly with the in-flight call and is undefined.

Proxy injection

The host script attaches proxy objects to the user driver before its initialize() runs. The three standard proxies (self.comm, self.settings, self.log) are injected on every _init and are always available. Optional, hardware-type-specific proxies are gated.

The trampoline opts in by calling PythonProcess::setEnabledProxies() between PythonHardwareBase::initPythonProcess() and the first PythonProcess::sendRequest(). The chosen names ride on the _init payload:

{"method": "_init", "key": "...", "model": "...",
 "proxies": ["digi"]}

On the Python side, the host script keeps a factory map and only instantiates the proxies the C++ side requested:

_OPTIONAL_PROXY_FACTORIES = {
    "digi": DigitizerProxy,
}
for name in request.get("proxies", []):
    factory = _OPTIONAL_PROXY_FACTORIES.get(name)
    if factory:
        setattr(user_obj, name, factory())

The single optional proxy that ships today is DigitizerProxy, which push-streams base64-encoded waveforms to C++:

class DigitizerProxy:
    def emit_shot(self, raw_bytes, shots=1):
        b64 = base64.b64encode(bytes(raw_bytes)).decode('ascii')
        _send_json({"waveform": b64, "shots": shots})

_send_json holds an internal stdout lock, so emit_shot is safe to call from the driver’s acquisition thread.

Adding a new push-style proxy

To add a push channel for a new hardware kind:

Implement the proxy class on the Python side in python_hw_host.py. The proxy’s job is to package a payload and call _send_json with a unique top-level key.
Add the class to _OPTIONAL_PROXY_FACTORIES under that key (lowercase string, matching what the trampoline will request).
In PythonProcess::onReadyRead(), add a branch that matches the new key, parses the payload, and emits a trampoline-facing Q_SIGNAL. Follow the waveform branch as the model: decode, then emit.
In the trampoline, call pu_process->setEnabledProxies({"yourproxy"}) from initialize (or the type-specific helper virtual) immediately after PythonHardwareBase::initPythonProcess() returns, and before any PythonProcess::sendRequest() call would force the subprocess to start.
Connect the new signal to a handler slot in the trampoline. The slot follows the reentrancy contract: it dispatches the payload synchronously and never re-enters sendRequest.

PythonHardwareBase mixin

PythonHardwareBase carries the boilerplate every trampoline needs: the owned subprocess, the lazy-start hook, the helpers that turn HardwareObject::sleep() and the trampoline’s read-settings hook (HardwareObject::hwReadSettings() or the per-base variant on the intermediate bases that final-override hwReadSettings — fcReadSettings, pcReadSettings, tcReadSettings, pgReadSettings, awgReadSettings, clockReadSettings, ftmwReadSettings, gpibReadSettings, ioReadSettings, lifLaserReadSettings, lifDigitizerReadSettings) into IPC dispatches, and the static helpers that find the host script and resolve a Python interpreter. The mixin’s constructor takes the hardware key and model strings and stores them; it does not need a back-pointer to the HardwareObject, because the IPC and the settings relay are funneled through callbacks the trampoline installs.

The members the mixin owns and exposes to subclasses:

pu_process — the PythonProcess instance, owned by std::unique_ptr. Non-null after initPythonProcess returns; the subprocess inside it is started lazily on the first testPythonConnection.
PythonHardwareBase::initPythonProcess() — constructs pu_process, binds the comm pointer, and installs the settings get/set callbacks. The callbacks are typically lambdas capturing SettingsStorage::get() and SettingsStorage::set() on the trampoline. The setter lambda is the bridge that lets the script update persistent settings across the relay, since SettingsStorage::set() is protected and otherwise inaccessible from outside the owning class.
PythonHardwareBase::testPythonConnection() — lazily starts the subprocess via startPythonProcess(), refreshes the comm pointer (in case the protocol has been swapped since the previous call), then sends test_connection and returns the boolean result.
PythonHardwareBase::startPythonProcess() — looks up the per-profile pythonScriptPath, pythonClassName, and pythonEnvPath on HardwareProfileManager, resolves the interpreter, and delegates to PythonProcess::start(). Refuses to start (and sets PythonHardwareBase::pythonErrorString()) if the script path or the class name is empty rather than substituting a default.
PythonHardwareBase::findHostScript() — locates python_hw_host.py in the application directory or in the share/blackchirp/ install location. Returns an empty string if neither exists.
PythonHardwareBase::resolvePythonExecutable() — probes envPath for the standard venv and conda layouts (bin/python3, bin/python, Scripts/python.exe) and falls back to the literal "python3" (resolved through PATH) when envPath is empty or contains no interpreter.
PythonHardwareBase::pythonSleep() and PythonHardwareBase::pythonReadSettings() — the trampoline’s HardwareObject::sleep() override and its read-settings hook (HardwareObject::hwReadSettings() for trampolines that inherit the default, or the per-base variant — fcReadSettings, pcReadSettings, tcReadSettings, pgReadSettings, awgReadSettings, clockReadSettings, ftmwReadSettings, gpibReadSettings, ioReadSettings, lifLaserReadSettings, or lifDigitizerReadSettings — when the parent base final-overrides hwReadSettings) delegate to these helpers. pythonReadSettings deliberately sends read_settings over IPC rather than restarting the subprocess, because a restart would re-run initialize() and disrupt connected state.
PythonHardwareBase::pythonErrorString() — exposes the human-readable error from the most recent failed startPythonProcess or testPythonConnection. Trampolines copy it into d_errorString on failure so the connection result reaches the GUI with a useful message.
The destructor stops pu_process if the subprocess is running.

A concrete trampoline therefore looks like a thin layer on top of the mixin: an initialize hook that calls initPythonProcess, a testConnection hook that calls testPythonConnection, delegations to the sleep helper and the appropriate read-settings hook helper, and one IPC dispatch per hardware-specific virtual.

Three state-management patterns

How a trampoline implements its hardware-specific virtuals depends on how its hardware base class manages config state. The three patterns below cover every trampoline that ships today.

Pattern A — Bulk Configure

The base class inherits from a complex config class (a DigitizerConfig with channel maps, trigger settings, sample rates, multi-record state, and so on). Per-call setters do not exist; the experiment hands the trampoline a desired config, and the trampoline applies it in one shot via a virtual configure (or, for FtmwDigitizer, an override of HardwareObject::prepareForExperiment()). The trampoline serializes the full config to JSON, sends a configure IPC call, parses a validated config back from the response, and copies it onto the C++ object so any clamped or substituted values persist. The Python side decides which keys to clamp.

Examples: PythonIOBoard overriding IOBoard::configure(), PythonLifDigitizer overriding LifDigitizer::configure(). FtmwDigitizer does not expose a configure virtual; each subclass overrides HardwareObject::prepareForExperiment() directly. PythonFtmwDigitizer follows that convention: it overrides prepareForExperiment to serialize the FtmwDigitizerConfig over JSON IPC, then leaves the final hwPrepareForExperiment in FtmwDigitizer to construct the WaveformBuffer from the validated config in the usual way. No bespoke buffer wiring is needed in the trampoline.

Pattern B — Granular Methods

The base class contains a config object as a member and exposes per-channel or per-parameter getter/setter slots. Each slot delegates to a hw* pure virtual that the trampoline implements. The base class owns the polling sequence, the validity checks, and the signal emission; the trampoline only ever sees one value at a time.

Examples: PythonFlowController (per-channel flow and pressure reads/writes, the per-channel enable toggle, and the pressure-control mode), PythonPulseGenerator (~22 setters and readers across channel and global state), PythonTemperatureController, PythonPressureController. The trampoline does not override the polling cadence — FlowController::poll() is non-virtual, and the IPC round trip per hw* call is the right granularity for the slow serial links these instruments typically sit behind.

Pattern C — Stateless / Pass-Through

The base class has no internal config object to manage. The trampoline receives experiment-supplied data (chirp segments and markers for an AWG; frequency assignments for a clock) at HardwareObject::prepareForExperiment() time, serializes it into JSON, and hands it to the Python script. There is no configure virtual and no bulk read-back: the experiment data is the truth, and the script programs the hardware to match.

Examples: PythonAwg (serializes ChirpConfig segments, markers, RF chain parameters, and clock assignments), PythonClock (per-output hw_set_frequency/hw_read_frequency).

The mapping from each Python* trampoline that ships with Blackchirp to its pattern, default driver class name, and hardware-specific entry points lives in the user-guide table at Trampoline Overview; the companion sections on Per-Type Capabilities walk through the per-method signatures.

Trampoline implementation contract

A new Python<Type> trampoline follows a fixed recipe.

Inherit from both bases. Multiple-inherit from the hardware base class and from PythonHardwareBase. Initialize the mixin in the constructor’s initializer list with d_key and d_model:

PythonFlowController::PythonFlowController(const QString &label, QObject *parent) :
    FlowController(QString(PythonFlowController::staticMetaObject.className()),
                   label, parent),
    PythonHardwareBase(d_key, d_model)
{ d_threaded = true; save(); }

Set d_threaded = true so the device runs on its own QThread; the IPC round trips would otherwise block the hardware-manager thread.

Wire the mixin in the type-specific initialize hook. For a plain HardwareObject subclass that is HardwareObject::initialize(); for hardware bases with a typed helper virtual it is the helper (initializeClock, fcInitialize, initializePGen, …):
```
void PythonFlowController::fcInitialize()
{
    initPythonProcess(p_comm,
        [this](const QString &k, const QVariant &dv) -> QVariant {
            return get(k, dv);
        },
        [this](const QString &k, const QVariant &v) {
            set(k, v, true);
        });
}
```
initPythonProcess does not start the subprocess. The profile-creation flow runs the registry-driven default-settings pass through every HardwareObject constructor, and spawning a Python interpreter on every dialog open would be gratuitous.

Drive the test-connection hook. Call testPythonConnection(p_comm) from HardwareObject::testConnection() (or its typed helper). The first call starts the subprocess; subsequent calls reuse it:

bool PythonFlowController::fcTestConnection()
{
    if (!testPythonConnection(p_comm)) {
        d_errorString = pythonErrorString();
        return false;
    }
    return true;
}

Delegate sleep and the read-settings hook. Override HardwareObject::sleep() to call PythonHardwareBase::pythonSleep(), and override the appropriate read-settings hook to call PythonHardwareBase::pythonReadSettings(). The hook is HardwareObject::hwReadSettings() for trampolines whose parent base does not final-override it, and the per-base variant (fcReadSettings, pcReadSettings, tcReadSettings, pgReadSettings, awgReadSettings, clockReadSettings, ftmwReadSettings, gpibReadSettings, ioReadSettings, lifLaserReadSettings, or lifDigitizerReadSettings) for the intermediate bases that do.
Implement the hardware-specific virtuals as IPC dispatches. The pattern from Three state-management patterns chooses the shape:
- Pattern A — implement the configure(...) virtual. Serialize the full config to JSON, send a configure IPC call, deserialize the validated dict from result.config back onto the C++ side. PythonFtmwDigitizer overrides HardwareObject::prepareForExperiment() directly because FtmwDigitizer does not expose a configure virtual.
- Pattern B — implement each hw* pure virtual as a standalone PythonProcess::sendRequest() call. The C++ side keeps the config object; PythonProcess::sendRequest() returns the result on success and an error-bearing object on failure, so each method ends with a default-value fall-through for the failure case.
- Pattern C — typically only an override of HardwareObject::prepareForExperiment() plus any per-output IPC calls (clock frequencies, AWG markers).
Push-style hardware: enable the optional proxy. For digitizer trampolines, after initPythonProcess returns:
```
pu_process->setEnabledProxies({"digi"_L1});
connect(pu_process.get(), &PythonProcess::waveformReceived,
        this, &PythonFtmwDigitizer::onWaveformReceived);
```
The handler slot honors the reentrancy contract from Reentrancy contract for push handlers — it forwards the bytes to FtmwDigitizer::emitShot() (FTMW) or to LifDigitizer::emitWaveform() (LIF) and returns.
Register the class. REGISTER_HARDWARE_META, REGISTER_HARDWARE_PROTOCOLS, and any REGISTER_HARDWARE_SETTINGS / REGISTER_HARDWARE_ARRAY declarations go in the .cpp next to the trampoline. The protocol set is Rs232 + Tcp + Gpib + Custom + Virtual, omitting any value that is meaningless for the hardware (PythonGpibController omits Gpib because it is the GPIB controller). Custom is the explicit “comm is handled by the .py script” indicator: the Python trampolines do not register REGISTER_CUSTOM_COMM field descriptors, and the user-facing CustomProtocolWidget detects the driver prefix and shows a note rather than a generic empty-fields form.
Ship a template script. Add python_<type>_template.py next to the trampoline source. The template defines a single class — AwgDriver, FtmwDigitizerDriver, etc. — that works out of the box on the Virtual protocol and documents every method. The host script’s generic-keyword dispatch means the template is the contract the user code must satisfy; no host-side change is needed when adding a new hardware type.
CMake wires up automatically. The hardware globs in cmake/BlackchirpHardware.cmake pick up python*.cpp/.h and python_*_template.py without editing the build file. Trampoline headers are appended to HARDWARE_IMPLEMENTATION_HEADERS so AUTOMOC generates the meta-object code for each Q_OBJECT and the static registration initializers are pulled out of the static library at link time (without that step the registrations are silently dropped).

Reentrancy contract for push handlers

A push-style trampoline’s PythonProcess::waveformReceived() slot must not call PythonProcess::sendRequest(). The signal fires from inside the nested QEventLoop of an in-flight sendRequest; a recursive call would serialize incorrectly with the pending response. The slot’s only legitimate work is to forward the bytes into the WaveformBuffer.

When a method call is in flight, the C++ side is sitting inside loop.exec() waiting for the matching id. The event loop dispatches every other message kind — relay, log, waveform — to its handler slot. If a waveform handler issued another sendRequest it would push a second nested loop on top of the first, and the second loop’s response would be the one the outer loop unblocks on. That is undefined behavior. The trampolines today obey the contract by writing only to the buffer.

QSettings key paths

Every persistent setting for a hardware object — the commType chosen at profile creation, every Required/Important/Optional value declared by REGISTER_HARDWARE_SETTINGS, and any custom-comm parameters — lives directly under the <hwType>:<label> QSettings group. That group is the SettingsStorage root for the HardwareObject. There is no driver-name subgroup, no configParams subtree, and no two-layer fallback. A Python trampoline reads and writes settings through the same SettingsStorage::get() / SettingsStorage::set() calls that any other HardwareObject subclass uses; the relay across self.settings reaches that root from the script side.

Required parameters live in the same group as ordinary settings, declared with HwSettingPriority::Required — there is no separate parameter store for them. The settings registry is the single source of truth; see Hardware Configuration for the full declaration model and the create-vs-edit semantics that determine when Required parameters are writable. A common slip is to write commType or a Required parameter under <hwType>:<label>/<impl>/...: that path is wrong, it shadows nothing on read, and the value the trampoline actually sees is whatever default the registry pass installed.

Hot reload

Reloading a script is a subprocess operation, not a hardware-object operation. The user clicks Reload Script on PythonHardwareControlWidget, which routes the request through HardwareManager::reloadPythonScript() to the device’s thread; the manager runs a single lambda there that calls PythonProcess::stop() and immediately HardwareObject::bcTestConnection(). bcTestConnection calls back through startPythonProcess, which launches a fresh subprocess and re-runs the _init → initialize() → test_connection handshake. Everything on the C++ side — the SettingsStorage cache, the CommunicationProtocol, the QThread and signal connections — is untouched; only the subprocess is replaced. The user-facing flow and what does not survive a reload are documented on Reloading and Editing a Script. The runtime-side of the reload signal (the HardwareManager::pythonScriptReloadResult() reporting hop) is on Hardware Runtime.

Per-profile Python environment

Each Python profile may carry an optional pythonEnvPath field on HardwareProfileManager pointing at a venv or conda environment directory. PythonHardwareBase::resolvePythonExecutable() probes that directory for bin/python3, bin/python, and Scripts/python.exe in order; an empty path falls back to "python3" (resolved through PATH). This lets a script that depends on a vendor SDK ship with the SDK installed in a dedicated environment without polluting the system Python — the trampoline launches the user’s interpreter and the user’s Python package set without further configuration.