Posted on — Leave a comment

Rooting (or failing to root) a Heights HT-8766BE — with nothing but a UART

Status: in progress, but a lot further along than the title suggests. Ethernet, Wi-Fi, VLAN-aware switching and LuCI are all genuinely working on this hardware now — just not yet all stitched together under a single unattended OpenWrt boot. I’m publishing this as a running account, written as it happens, warts and all, because the honest version of this story is more useful than a tidy one.

The target

The Heights HT-8766BE is a Wi-Fi 7 gateway supplied by UK ISPs — the kind of box that turns up, pre-provisioned, bolted to the wall, and never thought about again. This one is brand new out of the box. There is essentially zero public information about it: nothing in the OpenWrt Table of Hardware, nothing on the OpenWrt forum, no FCC filing I could find, no teardown photos. The closest public relatives are the Heights HT-6766BE (same silicon family, different cosmetic SKU) and its ISP-branded variants sold by BeFibre and Zzoomm in the UK.

Heights states its product line is built on Broadcom chipsets and SDK. That turns out to be exactly right, and it shapes everything that follows.

The one constraint that defines this whole project: the PCB sits under a heatsink/shield I’m not opening. The only access point is a UART header. Everything here — the recon, the backup, the porting, the debugging — happens over a 115200-baud serial link, nothing else, until very late in the piece when a wired Ethernet port came alive enough to test with a direct cable.

Tools

  • An ESP32 “Bus Pirate” board used purely as a UART-to-USB bridge (115200 8N1, 3.3 V).
  • A small Python daemon (uartd.py) that logs everything to a timestamped file and exposes a FIFO for sending commands — built because the browser-based serial terminal I started with missed the first ~21 seconds of boot, and because the router’s 1-second autoboot window needs something hammering the keyboard faster than a human can.
  • Standard Linux tools on the laptop side: objdump (ARM cross), readelf, Python for ad-hoc binary parsing (FDT walking, LZO/LZOP decompression via ctypes against liblzo2, FIT image parsing).

First boot: what’s actually in this box

With the UART logger running from power-on, the full boot chain is visible:

BTRM
V4.0
...
U-Boot SPL 2019.07 (Apr 29 2026 - 13:55:51 +0800)
RevID: 67660A0
Board is non secure

“Board is non secure” is the single most important line in the whole log. No secure boot, no signature verification on the boot chain. That’s the thing that makes the rest of this project possible at all.

The hardware, pieced together from the boot log and later confirmed live:

  • SoC: Broadcom BCM96766 — 4× ARM Cortex (32-bit, armv7l) at 2 GHz, 1 GiB DDR4 at 1600 MHz.
  • Wi-Fi: two SoC-integrated radios, wl0 (BCM6039, 4×4, 2.4/5/6 GHz — Wi-Fi 7 capable) and wl1 (BCM6038, 2×2), both using Broadcom’s proprietary wl driver (version 17.10.369.39076).
  • Switch/Ethernet: a Broadcom-internal “SF2” switch fabric, plus a separate external AN8855 switch chip handling three of the Gigabit copper ports, plus two SERDES-capable ports (labelled “2.5G WAN/LAN” on the rear panel) driven directly by the SF2.
  • Voice: an Intel/MaxLinear SLIC for FXS/analogue phone lines.
  • Storage: eMMC, 3.64 GiB, with the usual boot0/boot1 hardware boot partitions (4 MiB each) plus an RPMB region.
  • Boot chain: boot ROM → U-Boot SPL → U-Boot TPL → full U-Boot 2019.07 → a FIT image containing the Linux kernel and device tree.
  • Software: Broadcom’s BDK/SDK 5.04L.04p3, Linux 4.19.294 (armv7, BusyBox 1.36.1 userland, built with Buildroot 2021.02.4 / GCC 10.3), TR-069 (usp_md) and OpenPlat for ISP remote management.

The firmware identifies itself as 2.0.0.14-HT-6766BE-EM, customer config HT_6766BE_EM_BeFiber_default.conf — the BeFibre rebrand of the same hardware family, consistent with the sparse public information available.

U-Boot gives a one-second window to interrupt autoboot (Hit any key to stop autoboot: 1 0), with no authentication at all. The boot environment itself prints several things in cleartext during boot that I am deliberately not reproducing here, including an admin password, a default Wi-Fi key, the device serial number, and a set of TR-069 ACS credentials for acs-heights.fullfibre.systems. None of that belongs in a public writeup, regardless of how it was obtained.

The eMMC layout is a plain GPT table with a dual-bank design — two complete copies of bootfs/rootfs (100 MiB / 200 MiB squashfs each), so the device can fail an update and fall back. This dual-bank structure turned out to be the single most useful thing about the whole device, for reasons that become obvious below.

Getting a root shell — without any credentials at all

U-Boot’s bootcmd is printenv;run once;sdk boot_img — a Broadcom-specific helper (sdk boot_img) that loads the active bank’s FIT image and boots it. Critically, U-Boot lets you append arbitrary extra kernel command-line arguments before that boot, and nothing validates them:

setenv bootargs_append 'cma=72M init=/bin/sh'
sdk boot_img

That’s it. The kernel boots its normal squashfs root filesystem, but instead of running the vendor’s real init system, it drops straight to a BusyBox ash shell as PID 1 — fully root, no login, no password, nothing persisted (since we never saveenv, this reverts to the normal boot on the next power cycle). This became the base working environment for essentially the entire project.

Two gotchas showed up immediately:

  1. The hardware watchdog resets the board after about a minute, because init=/bin/sh never feeds it (the real init system normally has a dedicated watchdog-feeding service, wdtd.sh, running very early). Fix, run as the very first thing after getting a shell: �CODE2� V is the magic-close character that disarms the Broadcom watchdog driver cleanly. Follow it with a tiny background loop that writes to it every 10 seconds, just in case something else re-arms it.
  1. USB isn’t live in this minimal boot. The USB PHY power-up module (/lib/modules/4.19.294/extra/bcm_bca_usb.ko) and usb-storage/xhci-plat-hcd all need loading by hand — none of this is in modules.dep for the vendor’s out-of-tree modules at all; load order is only encoded in the vendor’s own (never-run, in this boot path) init scripts.

Backing up the whole eMMC, over USB

Before touching anything, I wanted a full backup. Over raw UART that’s impractically slow for 3.6 GiB, so the plan was: get USB mass storage working in the minimal shell, then dd everything onto a USB stick plugged straight into the router.

A few things bit along the way:

  • The kernel has no exFAT support — the first stick I tried, pre-formatted exFAT, just sat there unmountable. Switched to a blank ext4 stick instead.
  • BusyBox’s mount -o remount,rw silently fails on this kernel’s ext4 driver (it passes a relatime mount option the kernel rejects) — umount and a fresh mount -o rw works around it.
  • No od, wc, cat -v, base64, or blkid in this BusyBox build; dd, gzip, sha256sum, xxd and hexdump (from a fuller toolset found elsewhere on the real rootfs), ifconfig/ip, wget, curl, tftp, nc, fdisk are all present.

With USB sorted, the backup was three dd passes — mmcblk0boot0, mmcblk0boot1 (4 MiB each), and the full 3.64 GiB user area — each verified by hashing the source device and the resulting image file separately and comparing. (Lesson: reading 3.6 GB back off a USB stick to verify it takes noticeably longer than writing it did; budget for that.) RPMB wasn’t dumped; it needs an authentication key I don’t have, and nothing in this project needed it.

Two copies of the backup now exist, on a USB stick and on the laptop, both checksum-verified against the live device. Everything that follows is reversible because of this.

Getting OpenWrt to boot

The realistic goal, decided early: mainline OpenWrt has no target for the BCM96766 at all – I checked the current release tree directly, nothing. Writing one from scratch would mean reimplementing the DDR controller bring-up, the PMC, the eMMC controller, the Archer packet-offload engine, and the AN8855 switch glue, entirely from binary reverse-engineering, with zero vendor source available (no GPL source-code offer exists anywhere in this firmware as of the time of writing). For scale: Linux’s one genuinely open driver for a comparable chip in this family, the AN8855 switch, took over a year and more than twenty patch revisions from an experienced upstream kernel developer, working with a different vendor who at least cooperated. Writing a whole SoC BSP blind is not a realistic solo undertaking.

So: keep the vendor’s already-working, already-signed-off kernel and device tree. Replace only the userland. The vendor kernel already has everything procd (OpenWrt’s init) needs – CONFIG_NAMESPACES, cgroups, overlayfs, devtmpfs, ext4, loop devices, all present, confirmed by pulling the live .config via gunzip -c /proc/config.gz. A quick ABI check (readelf -A on a vendor binary) confirmed soft-float EABI, matching OpenWrt’s armsr/armv7 target build exactly.

OpenWrt ships a prebuilt image for exactly this scenario – openwrt-25.12.6-armsr-armv7-generic-ext4-rootfs.img, a 104 MiB plain ext4 filesystem meant to be written straight to a block device and booted via root=, with no squashfs-plus-overlay switch_root dance needed. Its /etc/inittab already has an entry for ttyAMA0 – the exact UART this board’s console uses – so it needed zero editing.

Using the spare bank (bank 1) as the target, leaving the running stock bank (2) completely untouched:

dd if=/dev/mmcblk0p5 of=/dev/mmcblk0p3 bs=1M      # copy the known-good kernel/FDT onto the spare slot
dd if=<usb-stick>/openwrt-....img of=/dev/mmcblk0p4 bs=1M   # write OpenWrt's rootfs onto the spare slot

then, from U-Boot:

setenv bootargs_append 'root=/dev/mmcblk0p4 rootfstype=ext4 rootwait rw'
sdk boot_img 1

sdk boot_img 1 boots bank 1 specifically, without touching the metadata/commit state — so a plain power cycle, with no further action, always lands back on the stock firmware. This is a one-off test boot, not a bank switch.

It worked first try:

OpenWrt 25.12.6, r33311-981c978eff Dave's Guitar
root@OpenWrt:~#

A real procd/ubus OpenWrt userland, running on this SoC’s own vendor kernel, over the serial console. No Ethernet yet – deliberately not attempted in this first test, to isolate variables.

Ethernet: the kernel panic, and the long way round

Loading the vendor’s Ethernet module chain under OpenWrt (same .ko files, copied in from the read-only-mounted stock rootfs – same kernel build, so the module ABI matches exactly) goes fine, right up until the actual driver:

insmod bcm_enet.ko

→ a hard kernel panic. Full ARM backtrace, tainted kernel, forced reboot in three seconds. The crash backtrace pointed at two symbols: _phy_drv_init and phy_drv_serdes_146class_init_lock. Checking bcm_enet.ko‘s own symbol table showed neither of them actually exist in that module — they’re kernel-builtin symbols, confirmed by cross-checking the live kernel’s /proc/kallsyms (c0922504 t _phy_drv_init, address in the 0xc0... vmlinux range, not the 0xbf1... module range). So this wasn’t a missing-module problem at all: bcm_enet was calling into a SERDES PHY-class framework compiled directly into the kernel, and some piece of that framework’s internal state wasn’t set up the way the driver’s probe path expected.

The device tree detour

With the router’s entire eMMC backed up, I could work entirely offline against the backup image. The FIT image inside the boot partition is itself a device-tree-structured container (U-Boot’s “Flattened Image Tree” format), parseable with the same FDT-walking logic used for the main device tree – so I wrote a small parser and found the real port topology:

port_gphy    reg=0  phy-mode=gmii    status=okay
port_sgmii0  reg=5  phy-mode=serdes  status=okay
port_sgmii1  reg=6  phy-mode=serdes  status=okay
port_imp     reg=8  phy-mode=gmii    status=okay   (management/CPU port)

Both SERDES ports – the two “2.5G WAN/LAN” ports on the rear panel — share the PHY-class code path that was crashing. The devicetree status property is a standard way to disable a node; "okay" and "disabled" are both five bytes including the terminator, and "fail" is a value the kernel treats as equally unavailable and is also exactly five bytes – meaning a two-port fix could, in principle, be a trivial in-place byte patch with no FDT restructuring at all.

It wasn’t quite that simple: FIT images store a SHA-256 hash of each sub-image, and U-Boot checks it. Patch the device tree, and the stored hash no longer matches – Bad Data Hash, followed by a hang that needs a hard power-cycle. The actual fix needed two patches: the five bytes of the status string, and the stored SHA-256 of the whole device tree blob, recomputed and written to its own location in the FIT header. Both applied only to the spare bank’s boot partition – the running stock bank was never touched.

With one SERDES port disabled this way, bcm_enet loaded clean. A genuinely new milestone: the first physical Ethernet link lights of the whole project, auto-negotiated at 1000 Mb/s full duplex, confirmed on the actual hardware.

The correction

Later the same day, while working on Wi-Fi in parallel (see below), I loaded the Wi-Fi driver prerequisite chain – hnd.ko and friends – before bcm_enet, out of simple necessity (it turns out bcm_enet and the Wi-Fi driver share dependencies; more on that below). With that load order, bcm_enet loaded cleanly with the device tree fully reverted to its original, unpatched state – both SERDES ports enabled, original hash restored, no hack in place at all.

That was a real “oh” moment: the device-tree patch was never the actual fix. Something in the Wi-Fi prerequisite chain – almost certainly hnd.ko, Broadcom’s shared hardware-abstraction module – populates the same kernel-builtin PHY-class state that bcm_enet‘s probe path needs, as a side effect of its own initialisation. Loading it first was the real fix, and it’s much cleaner: no FIT-image patching needed at all, and all three wired ports are available simultaneously rather than having to sacrifice one.

I’m leaving the device-tree-patch story in this writeup rather than editing it out, because it’s honestly representative of how this kind of work actually goes: a plausible, working-seeming fix that turns out, on broader testing, to have been coincidental.

The part that was still broken — and then wasn’t

With real PHY-level link established – auto-negotiation correct, LEDs on, speed and duplex both correctly negotiated – actual traffic still didn’t pass. Sending a ping got no reply, in either direction. The question was whether this was a signal-integrity/calibration problem or something else.

A previously-undocumented debug interface answered that cleanly:

echo 'stats 2' > /proc/driver/enet/cmd      # argument is a MAC index, not an interface name
dmesg | tail -n 40                           # output goes to the kernel log, not back to the proc file

This gives the full hardware MIB counters per port. The result was unambiguous: rx_dropped equalled rx_packet exactly, across every traffic class and every frame-size bucket, with zero FCS, alignment, fragment, jabber, or overflow errors. The switch hardware was receiving perfectly clean, valid frames and discarding every single one before they reached Linux’s networking stack.

I spent a long time tracing this in disassembly, following a hooks-table early-return inside bcm_enet.ko‘s enetxapi_rx_pkt, convinced the bug lived somewhere in Broadcom’s proprietary Archer/Crossbow packet-acceleration engine. It didn’t. The actual fix was absurdly small, and I only found it by going back to the vendor’s own debug CLI (ethswctl) and working through its command list methodically rather than guessing inside a disassembler:

ethswctl -c hw-switching -o enable

That’s a global switch-fabric enable, independent of any per-port setting, VLAN membership, or STP state – and it was simply off. Our manual boot path never calls it, because it’s normally asserted by a proprietary userspace management daemon during a full stock boot, which we skip entirely by booting straight to a shell. One command. Immediate effect: the next hardware MIB read showed rx_dropped: 0, and the first real ping (10.99.99.2 ↔ 10.99.99.1) came back clean – 20/20, 0% loss, ~0.7 ms RTT.

(There’s a sharp gotcha hiding in the same debug tool worth flagging: ethswctl -c portctrl -i <if> has no pure query mode. Omitting its -v flag doesn’t read the port’s state – it silently defaults to -v 0 and disables the port. I “checked” a port’s status this way more than once and immediately un-did my own fix by doing so.)

With that one flag in place, all five physical Ethernet ports work: the two SERDES-driven “2.5G WAN/LAN” ports on the rear panel (driven by the SF2 fabric directly), plus the three plain Gigabit LAN ports (driven by a separate AN8855 switch chip, confirmed via the MediaTek/Airoha mt753x lineage hinted at by its own CLI tool, /bin/switch) — tested with sustained TCP throughput, not just ping.

Full per-port VLAN isolation, for free

The AN8855 turned out to have a complete, working control path that I’d initially assumed didn’t exist. /bin/switch talks to a Generic Netlink family (an8855_nl_family) compiled straight into bcm_enet.ko – confirmed by grepping /proc/kallsyms for an8855_* symbols (an8855_reg_read, an8855_hw_reset, an8855_gsw_nl_init, and so on). It’s a genuinely capable tool: register-level read/write, full clause-22/45 PHY access, MAC/ARL table management, port mirroring, a complete ACL engine, and the important part – full 802.1Q VLAN control (vlan dump/set, per-port PVID, egress-tag modes).

Better still, I found /etc/switch-airoha.sh – the vendor’s own AN8855 setup script, sitting unused on the stock rootfs because our manual boot path skips vendor userspace init entirely. Running it directly configures exactly what you’d want from a router like this: one AN8855 port is a tagged uplink trunk to the SoC, and the other three are each an untagged, isolated member of their own VLAN. Confirmed by moving one cable between all three physical LAN jacks in turn, creating a standard Linux 802.1Q sub-interface (eth0.1/eth0.2/eth0.3) for each VLAN, and getting a clean, independent ping on each one. No DSA kernel driver involved at all – this is standard 802.1Q tagging over a single physical uplink, handled entirely by the switch chip itself.

One command found by deep-diving this same tool turned out to be the single most dangerous thing in this whole project: switch ... reset visibly reset the real chip (confirmed by a port’s link light dropping instantly) – but the very next call into any AN8855-related code afterward hung the kernel completely, uninterruptibly, immune even to Ctrl-C. Recoverable only with a full power cycle. Everything else about this tool, including register writes, has been completely safe; reset specifically never gets called again.

Wi-Fi: solved, and a real client connected

(The next several sections were the previous day’s “equally unresolved” story – kept below because the dead end and the eventual fix are both worth reading. Skip ahead to “The actual fix” if you just want the resolution.)

The dead end

The Wi-Fi driver has the same “nothing’s in modules.dep” problem as Ethernet – the real module load order only exists encoded in a vendor init script (bcm-wlan-drivers.sh) that never runs in the minimal boot path. Recovered from reading that script directly:

wlshared → firmware_class → cfg80211 → hnd → bcmmcast → emf → igs → bcm_pcie_hcd → wl

with real parameters (intf_name=wl%d instance_base=<n> for wl) also recovered from the same script.

One genuinely surprising discovery: Wi-Fi and Ethernet are not independent subsystems on this SoC. wl.ko needs bcm_pcie_config_bar_addr/bcm_pcie_map_bar_addr from bcm_pcie_hcd.ko – and bcm_pcie_hcd.ko itself needs enet_init_done, an export from bcm_enet.ko. Trying to test Wi-Fi in true isolation, without any Ethernet modules loaded, hits a hard dependency wall immediately. They share the same underlying packet-acceleration fabric.

With the correct module chain and the Ethernet prerequisite loaded first, wl.ko loads with zero missing symbols – real, measurable progress. The actual hardware attach still fails, though:

si_doattach: Embedded WLAN core (devid:0x6039) pseudo-PCI ...   <- succeeds
M2MDMA / BME / M2M ring alloc ...                                 <- all succeed
M2MDMA: core 0/1 deinit                                           <- rolled back
wl driver 17.10.369.39076 (...) failed with code 15

Disassembling wl.ko traced this to wl_pci_probe calling wlc_attach(..., &err); on failure it prints exactly this message, with err (15) read back through an output parameter. wlc_attach is the actual radio/firmware/calibration attach routine – and it is, and apparently always has been, completely closed. I confirmed this isn’t specific to our build by checking the equivalent open-source glue code from the long-running broadcom-sta Linux driver project: the exact same call pattern exists there too (wlc_attach(..., &err) → printf("... driver failed with code %d", err)), with wlc_attach itself still a sealed blob even in that project, across every chip generation it’s supported. There has never been a public table mapping these error codes to causes.

I tried restoring the device’s real NVRAM/calibration data properly, via the actual supported mechanism rather than guessing – /bin/nvram kernelset /etc/wlan/nvram/HT6766BDM.nvm (the nvram binary has an alarming ~30 shared-library dependencies, but they’re all just .so files already present on the vendor’s own rootfs; nothing needs to be running, so it executes fine standalone, prints restore done!, and is a clean, safe way to apply calibration data, unlike the ad-hoc route described below). It made no difference – same error 15 before and after, so whatever’s missing isn’t (just) calibration.

At the time, this looked like the Ethernet story before its resolution: genuine progress, blocked on an opaque, closed function with no public reference anywhere. I even tried the obvious alternative driver, dhd.ko (Broadcom ships two WLAN drivers historically) – it loaded cleanly but explicitly rejected this chip’s PCI device ID (dhdpcie_pci_probe: chipmatch failed!! Unsupported vendor 14e4 device 6039), conclusively ruling it out: dhd.ko is built for discrete PCIe WiFi dongles, not this SoC-integrated radio, so wl.ko genuinely is the correct driver, and the error 15 was a real problem inside it.

The actual fix

The breakthrough came from asking the obvious question I’d somehow skipped: why can’t we just look at the vendor’s own boot process for how it brings Wi-Fi up, given it demonstrably works in stock firmware? I’d read the Wi-Fi driver load order script already, but never the NVRAM population script, /etc/init.d/hndnvram.sh.

It calls a function, nvram_mfg_conf_mlo(), that runs a real binary (wifi_init_mlomode, deep in the vendor’s TR-069/CMS management framework, too heavy to replicate standalone) to get a numeric “MLO mode” value, then uses it to write a wl_mlo_config=... line into the NVRAM settings file – before the calibration restore we’d already been doing correctly. Checked the raw calibration blob directly: it contains no wl_mlo_config entry at all. For context, MLO (Multi-Link Operation) is a genuinely core Wi-Fi 7 feature (using multiple bands simultaneously for one client connection), and this chip is Wi-Fi 7 hardware. A missing, uninitialised MLO link configuration is an extremely plausible reason for a hard attach failure on exactly this kind of radio.

Read the small helper script (mlo_nvram_setting.sh) that consumes that mode value rather than replicating the whole binary chain: mode 1 sets wl_mlo_config="0 1 -1 -1"; anything else leaves it at "-1 -1 -1 -1". Tried the 1 case directly — which meant first discovering that the raw .nvm calibration file isn’t the newline-separated text format I’d assumed; it’s NUL-separated (confirmed with hexdump: ... $Rev: 83751 $\0boardnum=1780\0 ...), while the plain-text file the vendor’s own scripts edit with sed is a different, converted representation. tr '\0' '\n' converts it cleanly. Appended the one missing line, ran the real restore mechanism (nvram kernelset), and loaded wl.ko.

Both radios attached immediately, with zero errors. wl0 (BCM6039, 4×4) and wl1 (BCM6038, 2×2) both registered real network interfaces, both explicitly logging MLO_BA64/MLO_BA256 buffer pool configuration on attach, confirming the MLO subsystem itself was now correctly active, not just skipped. ip link set wl0 up brought the radio fully up: SMAC/CM7 firmware downloaded, PHY DSP firmware downloaded, MAC clock configured clean, with no crash. One missing NVRAM variable was the entire root cause of the WiFi blocker, from the very start of this project.

A real phone, a real connection

Getting an access point running from here hit two more real bugs, each worth its own paragraph:

Wrong band. My first hostapd config asked for 2.4 GHz (hw_mode=g) and got “Could not determine operating frequency” – because wl0 was already sitting in AP mode on channel 36, 5180 MHz, by default. Matched the config to the hardware instead of the other way round, and the AP started cleanly, genuinely visible over the air from a laptop’s own Wi-Fi scan (correct BSSID, correct channel, WPA2).

Stuck handshake. A real phone associated and got through message 2 of the WPA 4-way handshake cleanly (so the passphrase was right), but message 4 never arrived back, hostapd timed out and disconnected, every single retry. Root-caused to forcing pure legacy 802.11a mode on a chip that’s natively Wi-Fi 7/EHT: the driver’s key-install path evidently isn’t reliable in that mode. Adding ieee80211n=1 (ordinary HT mode, nothing exotic) fixed it on the very next attempt – EAPOL-4WAY-HS-COMPLETED, genuine two-way traffic counters climbing.

The only thing left after that was DHCP – the phone reported “timeout, could not connect” for a few minutes even though it was fully authenticated and passing real traffic at Layer 2, simply because nothing was serving leases on the Wi-Fi interface yet. A scoped dnsmasq instance later: DHCPACK in the log, a real lease handed out, and confirmation, in person, on the phone: it works.

Porting it all onto OpenWrt itself

Everything above was proven on the vendor rootfs, driven by hand over UART. The real goal is OpenWrt with all of it working unattended, so the next step was converting the whole manual sequence into real OpenWrt /etc/init.d services.

OpenWrt runs on the exact same vendor-built kernel as the stock firmware (only the bootloader’s root= argument differs), so the proprietary .ko files are 100% kernel-ABI compatible and insmod under OpenWrt with zero changes. The vendor’s own CLI tools (ethswctl, switch, nvram, hostapd) are a different story – they’re glibc-linked, and OpenWrt itself is musl. Copying the glibc runtime pieces (ld-linux.so.3, libc.so.6, and around forty other shared libraries) straight into OpenWrt’s real /lib, alongside musl’s own differently-named equivalents, turned out to just work for most of them. One genuinely nasty exception: nvram‘s dependency chain pulls in libgcc_s.so.1 — a filename musl also ships, at the same path, with an incompatible ABI, producing a cryptic internal error from the dynamic linker. Fixed by shadowing just that one library in an isolated override directory and replacing the real binary with a tiny wrapper that explicitly points the dynamic linker at it.

A live, unattended cold-boot test caught one more real bug the hard way: hostapd failing with libatomic.so.1: cannot open shared object file – a transitive dependency of libcrypto.so.3 I’d missed entirely by hand-tracing the dependency list. Copied the one missing file (mounting the vendor’s read-only rootfs directly from inside the running OpenWrt session, no reboot needed) and re-ran the exact same cold boot: both Wi-Fi radios came up and both successfully joined the LAN bridge automatically, confirmed in the system log, zero manual intervention. Wi-Fi is now a genuinely automated, reboot-proof OpenWrt feature.

The one thing still not reliable: wired LAN under OpenWrt specifically

The wired side of this same automation is written, and is proven correct on the vendor rootfs, the exact same module chain and VLAN sub-interfaces work every time. Under OpenWrt specifically, there’s a genuine, reproducible bug: netifd (OpenWrt’s own network management daemon) can never successfully create the LAN bridge itself. Isolated this properly rather than guessing: netifd’s own debug log shows precisely which internal call fails:

system_bridge_addbr(1573): Error adding bridge 'br-lan': -10

– and manually performing the exact same operation (ip link add br-lan type bridge, or the older brctl addbr, under any name, including br-lan itself) works instantly, every time, at the kernel level. So it’s not a kernel limitation and not a driver problem – it’s something specific to netifd’s own bridge-creation call path on this particular kernel/config combination, returning an error code that doesn’t obviously correspond to anything that should go wrong here. Diagnosing further would mean strace-ing netifd’s actual syscalls, which is a reasonable next step but a step too far for a serial console in one sitting. The wired side works completely reliably by hand; it just isn’t automated yet.

A mistake worth documenting honestly

Partway through the Wi-Fi investigation, trying to supply calibration data the wrong way, I wrote roughly 2,500 lines of real board NVRAM data into /proc/nvram/set – a kernel interface that turned out to be capped at 16 KB, far smaller than the ~76 KB calibration file. This didn’t just fail cleanly: it left malformed entries in the persistent flash-backed NVRAM store (not RAM-only, as I’d assumed), which then caused a flood of kernel warnings on every subsequent boot, severe enough in one case to blow the boot-time watchdog before a shell was even reachable.

The fix was a straightforward image restore – the corrupted region lives entirely inside the small mmcblk0boot0/mmcblk0boot1 hardware partitions, both of which were in the very first backup taken, before any of this. dd-ing the pristine 4 MiB images back (after clearing the kernel’s force_ro write-protect flag on those block devices, then re-setting it afterward) restored normal boot behaviour completely, checksum-verified against the original backup. Nothing was lost; the stock bank’s reliability as a fallback was never actually compromised, just the thing I’d been testing with recklessly. The lesson – verify the actual capacity of a kernel interface before bulk-writing to it, and never assume a knob called /proc/something is RAM-only without checking – earned its place in this writeup.

Where this stands

  • Rooted: completely, trivially, via an unauthenticated U-Boot bootarg injection. No exploit needed.
  • OpenWrt: genuinely boots, on the real hardware, on a spare bank that leaves stock firmware completely intact and recoverable with a plain power cycle.
  • Ethernet: all five physical ports working with real, sustained traffic – the two SERDES “2.5G” ports and the three AN8855-driven LAN ports, the latter with full per-port 802.1Q VLAN isolation. Proven solid on the vendor rootfs; the same automation under OpenWrt itself hits a reproducible netifd bridge-creation bug, not yet root-caused.
  • Wi-Fi: fully working. Both radios attach cleanly (the real fix was a single missing NVRAM variable, wl_mlo_config, that the vendor’s own boot process sets and ours never had). A real phone associated, completed a WPA2 handshake, and got a DHCP lease – confirmed in person. Automated as a proper OpenWrt /etc/init.d service, reboot-proof, confirmed working unattended.
  • LuCI: reachable and rendering correctly over the router’s own Wi-Fi/Ethernet, the moment Ethernet started passing traffic – nothing extra needed installing.
  • Full switch/VLAN control: a complete, working control path for the external AN8855 switch chip (register access, ACLs, full VLAN management) – not a kernel DSA driver, but genuinely capable and confirmed live.
  • Full eMMC backup, taken and verified before any of this, in two separate locations – meaning everything above is undoable, and the default boot bank is still deliberately left as stock firmware until the wired-LAN automation under OpenWrt is sorted.

This has gone from “rooted, with a working OpenWrt shell and nothing else” to “every subsystem the original goal named is individually proven working on real hardware,” in one extended session. What’s left is less “can this work at all” and more “stitching it into one unattended boot.” I’ll keep this updated as it moves.

Posted on — Leave a comment

Repurposing a Hive Hub: Backing Up the eMMC and Getting a Root Shell Over USB

The Hive Hub (model SHA-533, the Centrica/British Gas “Hive Nano3 Hub”) turns up cheap and unwanted now that the service it was built for has moved on. Inside it is a surprisingly capable little computer: a Texas Instruments AM6232A (dual Cortex-A53 Sitara, the AM62x family), 8 GB of Samsung eMMC, an ESP32-MINI-1 for Wi-Fi and Bluetooth, and a Silicon Labs EFR32MG21 Zigbee chip, in a case with three LEDs. I wanted one as a general-purpose Linux box on my network. This post is how I got a root shell, working SSH and my own Wi-Fi onto it, without soldering anything except one jumper wire, and without leaving the device unrecoverable. It includes a script that does the rewriting automatically, with the caveats about what I have and have not tested.

Read this first. This works on a device you own, by editing its storage over USB. A mistake can leave it unbootable, and the hub holds a certificate-based identity for a cloud service that no longer matters to me, but might to you. Make the full backup in step 2 and keep it. Everything below was done by hand on my own hub; the script that chains the steps together is new and has been tested only against a fake disk image (details at the end).

What you need

  • A Linux PC with dfu-util, e2fsprogs (for debugfs), openssl, python3 and curl.
  • A USB-C cable. The hub is powered from the USB-C port, so it needs nothing else.
  • One short jumper wire to tie the test pad labelled B7 to 3.3 V. That pulls the boot-mode strap so the AM6232’s mask ROM waits for USB DFU instead of booting from eMMC.
  • Strongly recommended: a USB-serial adapter on the AM6232’s UART header. I did my debugging through one, and the UART is the only way in if networking is wrong.
  • A patched first-stage bootloader (tiboot3.bin), explained next.

How the entry point works

With B7 high, the ROM boot code in the AM6232 exposes a DFU device and will accept a first-stage bootloader over USB. It must be a correctly signed image for this silicon, and the production hub uses a “HS-FS” part, so the build target is tiboot3-am62x-hs-fs-evm.bin. TI’s own U-Boot already knows how to be a DFU target from that stage, but it can only offer RAM, not the eMMC. So I patched TI’s U-Boot (the 2023.04 branch, pinned to the November 2023 commit that matches the hub’s SYSFW 9.1.8 firmware) to do two extra things from inside the SPL, the early loader that runs on the Cortex-R5:

  1. Initialise the eMMC and expose it over DFU as raw block targets: the whole main area, the two hardware boot partitions, and the rootfs partition by itself.
  2. Allow writes to it.

That avoids the rest of the boot chain (TF-A, OP-TEE, full U-Boot) entirely. I did try to build the whole chain and hit a TF-A error I never solved (Timeout waiting for thread SP_RESPONSE to fill), so the stock chain on the eMMC is what I use for actual boots. The pivot was deliberate: it is much safer to touch only the storage than to replace the boot chain.

The patch, in words

  • common/spl/spl_ram.c: initialise MMC device 0 before starting DFU (the DFU boot path never probes the eMMC itself), then register the dfu_alt_info_emmc targets alongside the RAM one.
  • drivers/dfu/dfu_mmc.c: stub out the file-based DFU back end in SPL builds (raw only), and fix a 32-bit overflow in the size calculation. Without this, a 7.28 GiB pull silently stops at 3,523,215,360 bytes and dfu-util reports a clean “Upload done”. Cast to u64 before multiplying.
  • common/spl/Kconfig: new SPL_DFU_MMC option. common/cli.c: two #ifdef CONFIG_CMDLINE changed to CONFIG_IS_ENABLED(CMDLINE) so SPL builds compile.
  • include/environment/ti/k3_dfu.env: the target list. The size fields are in 512-byte sectors, not bytes (my first attempt passed bytes, got 16 million sectors, and overflowed):
    dfu_alt_info_emmc=
    	backup.mainarea raw 0x0 0xe90000 mmcpart 0;
    	backup.boot0 raw 0x0 0x2000 mmcpart 1;
    	backup.boot1 raw 0x0 0x2000 mmcpart 2;
    	install.rootfsA raw 0x7be8 0x600000 mmcpart 0;
    	restore.bootpart raw 0x8 0x7be0 mmcpart 0
  • In the config, start from am62x_evm_r5_usbdfu_defconfig and enable CONFIG_SPL_DFU_MMC, CONFIG_DFU_MMC, CONFIG_SPL_MMC_WRITE (without it, blk_dwrite() silently writes zero blocks and you only learn from a final “MMC operation failed”) and CONFIG_SPL_DFU_NO_RESET.

One more SPL gotcha: its tiny printf mangles %llu and corrupts every argument after it on the same line, so print sector counts, not byte counts, in debug output.

To build, you need the pinned TI U-Boot, the matching ti-linux-firmware commit (532db4d1703f, “Update firmwares to v09.01.08”) and an arm-none-eabi or arm-linux-gnueabihf cross compiler:

git clone https://git.ti.com/git/ti-u-boot/ti-u-boot.git && cd ti-u-boot
git checkout c1c2bdf2725            # ti-u-boot-2023.04-next, 2023-11-29
git apply hivehub-spl-dfu-mmc.patch
export ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-
make am62x_evm_r5_usbdfu_defconfig
scripts/config -e SPL_DFU_MMC -e DFU_MMC -e SPL_MMC_WRITE -e SPL_DFU_NO_RESET
make olddefconfig
make BINMAN_INDIRS=/path/to/ti-linux-firmware -j8
# result: tiboot3-am62x-hs-fs-evm.bin  (use this as tiboot3.bin)

I should be straight about that recipe: it is reconstructed from the config of the build I used (the options are the ones in my working .config) and from the patch, and I have not re-run a clean build from these exact commands while writing this post. If your tiboot3.bin does not offer backup.mainarea as a DFU target, it is not the right build.

Step 1: Put the hub in DFU mode

  1. Unplug the hub. Tie pad B7 to 3.3 V.
  2. Plug the USB-C cable into the PC. dfu-util -l should show a TI device with bootloader and SocId alt-settings.

The board is powered from the cable, so there is no hot-swap: every time you want to go back to this state you unplug and replug. After the SPL starts, it stays in its own DFU session until you cut the power.

Step 2: Take a full backup (don’t skip it)

dfu-util -a bootloader -D tiboot3.bin
dfu-util -a backup.mainarea -U backup.mainarea.img   # 7,818,182,656 bytes
dfu-util -a backup.boot0   -U backup.boot0.img       # 4 MiB
dfu-util -a backup.boot1   -U backup.boot1.img       # 4 MiB

Check the sizes exactly. The eMMC has 15,269,888 sectors of 512 bytes in the main area, plus two 4 MiB boot partitions where the ROM actually finds the production tiboot3.bin (boot0 is FAT32, boot1 FAT12). A pull that is a few percent short means the overflow bug is back. The RPMB area (512 KB) needs the authenticated RPMB protocol and is not included; the script does not touch it either.

What is on the disk

The stock system is Linux 6.1 (TI’s kernel) with OSTree: mmcblk0p1 is a FAT boot partition, mmcblk0p2 is the ext4 rootfs holding an OSTree deployment, p3 is a legacy recovery image, and p4 holds /var and /persistent. Two things are worth knowing before you change anything:

  • The real boot configuration is /boot/loader/uEnv.txt inside the rootfs (a symlink to loader.0), not in the FAT partition. I wasted a recovery cycle restoring the FAT partition to fix a boot problem it had nothing to do with.
  • U-Boot’s environment is not persisted (it prints Loading Environment from nowhere on every boot), so fw_setenv and the A/B boot counter do nothing useful. Keep your DFU backup as the safety net.
  • The stock firmware has a factory-restore.service that, if /boot/uboot/factory.rst exists at boot, runs blkdiscard -s on mmcblk0p4. Don’t create that file.

Step 3: Open it up

The approach that worked is to edit the OSTree deployment’s /etc in place, offline, on a copy of the rootfs partition, and write it back. That keeps Hive’s own boot chain, kernel and device tree (which already drive the Wi-Fi, the LEDs and the Zigbee chip), and avoids the unsolved bootloader problem. The changes are:

  1. Root password. /etc/shadow says root:*, so nothing can match. Replace it with an openssl passwd -6 hash.
  2. Serial login. Only the framebuffer getty was enabled. Link serial-getty@ttyS2.service into getty.target.wants; the AM6232’s debug UART then gives you a login prompt at 115200 8N1.
  3. SSH. There is no sshd. I used dropbear from Debian 11 (bullseye) arm64 packages with their three libraries (libtomcrypt1, libtommath1, libcrypt1). Bullseye specifically, because the hub’s glibc is 2.35 and a binary built against an older glibc runs on a newer one but not the other way round. One more trap: host keys generated on a PC in OpenSSH/PEM format crash dropbear with Early exit: String too long and systemd restarts it in a loop. Dropbear wants its own key format. The script runs it with -R, which generates the host keys itself on the first connection.
  4. The firewall. /etc/nftables.conf allows only the cloud ports (svc_tcp_ports = {8883, 9124}) and rejects everything else, by design. Add 22.
  5. Wi-Fi. The stock Wi-Fi is a standard wpa_supplicant, but wrapped in Hive’s own scripts, and something in it (most likely supplicant-error-action.sh, which has a .restore backup file) quietly restored the factory access-point config (ap_scan=2, open network “Hivehub-sha-…”) at some point during my reboot loops. I lost a whole evening believing the board had stopped booting; it was booting fine and had left my network. Rejoining is not enough on its own: ap_scan is read only at start-up. The fix is to bypass Hive’s wrapper completely: a normal wpa_supplicant-wlan0.conf in /etc/wpa_supplicant, the Debian wpa_supplicant@wlan0 template unit enabled, and the Hive wrapper (wpa-supplicant-ctl.service) masked.
  6. The Hive cloud services (optional). The AlertMe/Centrica stack, including a Java daemon, a cloud registration service and telemetry timers, can be masked. Masking is reversible: it is just a symlink to /dev/null. I disabled these live over SSH once I had a shell, and left nanohub-hostname.service alone because it only sets the hostname.

The script

The source, with my notes on the boot chain and stock firmware, is on GitLab: git.experimental-engineering.co.uk/root/hive-hub-tools. You can also download both files here (the script, and the U-Boot patch described above):

This does steps 2 and 3. It takes the backup with size checks; refuses to carry on if the rootfs it pulls does not match the backup; applies the edits to a copy; runs e2fsck on the copy; and only then asks you to type YES before it writes anything. After the write it reads the partition back and compares it.

export TIBOOT3=./tiboot3.bin
export WIFI_SSID='your-network' WIFI_PSK='your-passphrase'
export SSH_PUBKEY=~/.ssh/id_ed25519.pub STRIP_CLOUD=1
./hivehub-jailbreak.sh check
./hivehub-jailbreak.sh selftest            # no hardware needed
./hivehub-jailbreak.sh backup ~/hivehub    # replug into DFU mode after this if asked
./hivehub-jailbreak.sh patch  ~/hivehub
#!/bin/bash
# Hive Hub SHA-533 (TI AM6232A): back up the eMMC and unlock SSH/Wi-Fi, over USB only.
#
#   hivehub-jailbreak.sh check                 tools + files present?
#   hivehub-jailbreak.sh backup  DIR           pull main area, boot0, boot1 into DIR
#   hivehub-jailbreak.sh patch   DIR           pull rootfs, edit a COPY, verify, (ask) write back
#   hivehub-jailbreak.sh selftest              run the edit logic against a fake rootfs image
#
# Needs: dfu-util, e2fsprogs (debugfs), openssl, python3, ar/tar/xz, curl.
# The board must be in ROM DFU mode (test pad B7 tied to 3V3, USB-C to this PC) and
# SPL_TIBOOT3 must be the patched R5 SPL (see the build notes in the blog post).
# Settings come from the environment:
#   TIBOOT3=path   WIFI_SSID=...  WIFI_PSK=...  SSH_PUBKEY=~/.ssh/id_ed25519.pub
#   ROOT_PASSWORD=...  (optional, default: ask)  STRIP_CLOUD=1 (mask the Centrica services)
set -euo pipefail

TIBOOT3=${TIBOOT3:-./tiboot3.bin}
ROOTFS_SECTOR_COUNT=$((0x600000))      # install.rootfsA = 3 GiB at sector 0x7be8
BACKUP_SIZES="backup.mainarea:7818182656 backup.boot0:4194304 backup.boot1:4194304"
CLOUD_UNITS="device-abstraction reflector hubreg hub-control-client.timer hub-metrics.timer \
alertme-hub-delete-angstrom alertmehub-osrelease-logger hub-state-monitor mqtt_mdns greengrass \
hub-bluetooth2 hsr wpa-supplicant-ctl"

die() { echo "ERROR: $*" >&2; exit 1; }
need() { command -v "$1" >/dev/null || die "missing tool: $1"; }

dfu_ready() { dfu-util -l 2>/dev/null | grep -q "$1"; }

push_spl() {
  [ -f "$TIBOOT3" ] || die "TIBOOT3 not found: $TIBOOT3"
  if dfu_ready 'name="backup.mainarea"'; then echo "SPL DFU session already running."; return; fi
  dfu_ready 'name="bootloader"' || die "no ROM DFU device (B7 high, USB-C connected?)"
  dfu-util -a bootloader -D "$TIBOOT3"
  for _ in $(seq 20); do dfu_ready 'name="backup.mainarea"' && return; sleep 1; done
  die "SPL did not expose the backup.* targets (wrong tiboot3.bin?)"
}

cmd_check() {
  for t in dfu-util debugfs openssl python3 ar tar xz curl; do need $t; echo "ok  $t"; done
  [ -f "$TIBOOT3" ] && echo "ok  $TIBOOT3" || echo "-- $TIBOOT3 missing"
}

cmd_backup() {
  local d=${1:?directory}; mkdir -p "$d"; need dfu-util; push_spl
  for e in $BACKUP_SIZES; do
    n=${e%%:*}; want=${e##*:}
    dfu-util -a "$n" -U "$d/$n.img"
    got=$(stat -c %s "$d/$n.img")
    [ "$got" -eq "$want" ] || die "$n: got $got bytes, expected $want (truncated pull - do not continue)"
    echo "ok  $n  $got bytes"
  done
  (cd "$d" && sha256sum backup.*.img > SHA256SUMS)
  echo "Backup complete. Keep it somewhere safe; it contains the device's identity and certs."
}

# ---- offline edit of an ext4 image (the OSTree rootfs) ------------------------------
dbg() { debugfs -w -R "$1" "$IMG" 2>&1 | grep -v '^debugfs ' || true; }
dput() { # local-file target-path mode(octal, e.g. 0100644)
  dbg "rm $2" >/dev/null; dbg "write $1 $2" >/dev/null
  dbg "sif $2 mode $3" >/dev/null; dbg "sif $2 uid 0" >/dev/null; dbg "sif $2 gid 0" >/dev/null
}
dexists() { ! debugfs -R "stat $1" "$IMG" 2>&1 | grep -q "not found"; }   # debugfs exits 0 either way
dmkdir() { dexists "$1" || { dmkdir "$(dirname "$1")"; dbg "mkdir $1" >/dev/null; }; }
dread() { debugfs -R "cat $1" "$IMG" 2>/dev/null; }
dlink() { dmkdir "$(dirname "$2")"; dbg "rm $2" >/dev/null; dbg "symlink $2 $1" >/dev/null; }

find_deploy() {
  local os d
  os=$(debugfs -R "ls -p /ostree/deploy" "$IMG" 2>/dev/null | awk -F/ '$6!="." && $6!=".." && $6!="" {print $6}' | head -1)
  [ -n "$os" ] || die "no /ostree/deploy/* found - is this the rootfs partition?"
  d=$(debugfs -R "ls -p /ostree/deploy/$os/deploy" "$IMG" 2>/dev/null | awk -F/ '$6 ~ /\.0$/ {print $6}' | head -1)
  [ -n "$d" ] || die "no deployment under /ostree/deploy/$os/deploy"
  echo "/ostree/deploy/$os/deploy/$d"
}

fetch_deb() { # package -> path to .deb (Debian 11 arm64: older glibc than the device, so it runs)
  local pkg=$1 w=$WORK/debs idx rel
  mkdir -p "$w"
  for base in http://deb.debian.org/debian http://archive.debian.org/debian; do
    [ -f "$w/Packages" ] && break
    curl -fsSL "$base/dists/bullseye/main/binary-arm64/Packages.xz" 2>/dev/null | xz -dc >"$w/Packages.tmp" \
      && [ -s "$w/Packages.tmp" ] && mv "$w/Packages.tmp" "$w/Packages" && echo "$base" >"$w/base"
  done
  [ -f "$w/Packages" ] || die "cannot fetch the Debian 11 package index"
  rel=$(awk -v p="$pkg" '$1=="Package:"{c=($2==p)} c&&$1=="Filename:"{print $2; exit}' "$w/Packages")
  [ -n "$rel" ] || die "package $pkg not found"
  [ -f "$w/$(basename "$rel")" ] || curl -fsSL "$(cat "$w/base")/$rel" -o "$w/$(basename "$rel")"
  echo "$w/$(basename "$rel")"
}

apply_edits() {
  local DEP; DEP=$(find_deploy); echo "deployment: $DEP"
  local E=$DEP/etc U=$DEP/usr
  mkdir -p "$WORK/x"

  # 1. root password (stock shadow has root:* so nothing can ever match)
  local pw=${ROOT_PASSWORD:-}
  [ -n "$pw" ] || { read -rsp "New root password: " pw; echo; }
  local hash; hash=$(openssl passwd -6 "$pw")
  dread "$E/shadow" | awk -v h="$hash" -F: 'BEGIN{OFS=":"} $1=="root"{$2=h} {print}' >"$WORK/x/shadow"
  grep -q "^root:\\\$6\\\$" "$WORK/x/shadow" || die "shadow edit failed"
  dput "$WORK/x/shadow" "$E/shadow" 0100640

  # 2. serial console login on the AM62 UART (ttyS2)
  dlink /lib/systemd/system/serial-getty@.service "$E/systemd/system/getty.target.wants/serial-getty@ttyS2.service"

  # 3. dropbear + its three runtime libraries from Debian 11 arm64
  local pkg deb
  for pkg in dropbear-bin libtomcrypt1 libtommath1 libcrypt1; do
    deb=$(fetch_deb "$pkg"); rm -rf "$WORK/x/$pkg"; mkdir -p "$WORK/x/$pkg"
    (cd "$WORK/x/$pkg" && ar x "$deb" && tar xf data.tar.* )
  done
  put_tree() { # srcdir/relpath -> $U/relpath-ish (files only)
    local src=$1 rel=$2 tgt=$3 mode=$4
    [ -e "$src/$rel" ] || die "missing in deb: $rel"
    dmkdir "$(dirname "$tgt")"; dput "$src/$rel" "$tgt" "$mode"
  }
  put_tree "$WORK/x/dropbear-bin" usr/sbin/dropbear "$U/sbin/dropbear" 0100755
  put_tree "$WORK/x/dropbear-bin" usr/bin/dropbearkey "$U/bin/dropbearkey" 0100755
  for lib in "$WORK"/x/libtomcrypt1/usr/lib/aarch64-linux-gnu/libtomcrypt.so.1.* \
             "$WORK"/x/libtommath1/usr/lib/aarch64-linux-gnu/libtommath.so.1.* \
             "$WORK"/x/libcrypt1/lib/aarch64-linux-gnu/libcrypt.so.1.*; do
    [ -f "$lib" ] || die "library not found in deb: $lib"
    b=$(basename "$lib"); dput "$lib" "$U/lib/aarch64-linux-gnu/$b" 0100644
    dlink "$b" "$U/lib/aarch64-linux-gnu/${b%.*.*}" 2>/dev/null || true
  done

  # 4. systemd unit. -R makes dropbear create its own host keys on first connection (the
  #    PEM/OpenSSH keys made on a PC do NOT load: "String too long").
  cat >"$WORK/x/dropbear.service" <<'UNIT'
[Unit]
Description=Dropbear SSH server
After=network.target
[Service]
ExecStartPre=/bin/mkdir -p /etc/dropbear
ExecStart=/usr/sbin/dropbear -F -R -E
Restart=always
[Install]
WantedBy=multi-user.target
UNIT
  dput "$WORK/x/dropbear.service" "$E/systemd/system/dropbear.service" 0100644
  dlink /etc/systemd/system/dropbear.service "$E/systemd/system/multi-user.target.wants/dropbear.service"

  # 5. authorised key for root (home on this image is /home/root)
  if [ -n "${SSH_PUBKEY:-}" ] && [ -f "$SSH_PUBKEY" ]; then
    local home; home=$(dread "$E/passwd" | awk -F: '$1=="root"{print $6}')
    [ -n "$home" ] || home=/home/root
    dmkdir "$(dirname "$DEP$home")/root"   # no-op if present
    dmkdir "$DEP$home/.ssh"; dbg "sif $DEP$home/.ssh mode 040700" >/dev/null
    dput "$SSH_PUBKEY" "$DEP$home/.ssh/authorized_keys" 0100600
  fi

  # 6. firewall: stock nftables allows only 8883/9124 then rejects everything
  dread "$E/nftables.conf" >"$WORK/x/nft.conf"
  python3 - "$WORK/x/nft.conf" <<'PY'
import re, sys
p = sys.argv[1]; s = open(p).read()
m = re.search(r'(svc_tcp_ports\s*=\s*\{)([^}]*)(\})', s)
if not m: sys.exit("svc_tcp_ports not found in nftables.conf")
if not re.search(r'\b22\b', m.group(2)):
    s = s[:m.end(2)] + ", 22" + s[m.end(2):]
open(p, "w").write(s)
PY
  dput "$WORK/x/nft.conf" "$E/nftables.conf" 0100644

  # 7. Wi-Fi with the standard Debian template unit, and stop the AlertMe script reverting it
  if [ -n "${WIFI_SSID:-}" ]; then
    mkdir -p "$WORK/x" && {
      echo "ctrl_interface=DIR=/run/wpa_supplicant GROUP=netdev"
      echo "update_config=1"; echo "country=GB"
      printf 'network={\n\tssid="%s"\n\tpsk="%s"\n}\n' "$WIFI_SSID" "${WIFI_PSK:?WIFI_PSK}"
    } >"$WORK/x/wpa.conf"
    dput "$WORK/x/wpa.conf" "$E/wpa_supplicant/wpa_supplicant-wlan0.conf" 0100600
    dlink /lib/systemd/system/wpa_supplicant@.service "$E/systemd/system/multi-user.target.wants/wpa_supplicant@wlan0.service"
    dbg "rm $E/systemd/system/multi-user.target.wants/wpa-supplicant-ctl.service" >/dev/null
    dlink /dev/null "$E/systemd/system/wpa-supplicant-ctl.service"      # mask
  fi

  # 8. optional: mask the Hive/Centrica cloud services (masking is reversible: delete the link)
  if [ "${STRIP_CLOUD:-0}" = 1 ]; then
    for u in $CLOUD_UNITS; do
      case $u in *.timer|*.service) n=$u;; *) n=$u.service;; esac
      dlink /dev/null "$E/systemd/system/$n"
    done
  fi
  echo "edits applied"
}

cmd_patch() {
  local d=${1:?directory}; mkdir -p "$d"; need debugfs; need openssl; need python3
  WORK=$(mktemp -d); trap 'rm -rf "$WORK"' EXIT
  [ -f "$d/backup.mainarea.img" ] || die "run 'backup' first (and keep that backup)"
  push_spl
  dfu-util -a install.rootfsA -U "$d/rootfsA.orig"
  [ "$(stat -c %s "$d/rootfsA.orig")" -eq $((ROOTFS_SECTOR_COUNT*512)) ] || die "short rootfs pull"
  cmp -n $((ROOTFS_SECTOR_COUNT*512)) "$d/rootfsA.orig" \
      <(dd if="$d/backup.mainarea.img" bs=512 skip=$((0x7be8)) count=$ROOTFS_SECTOR_COUNT 2>/dev/null) \
      || die "rootfs differs from the main-area backup (is the board running something else?)"
  cp "$d/rootfsA.orig" "$d/rootfsA.new"; IMG=$d/rootfsA.new
  apply_edits
  e2fsck -fn "$IMG" || die "filesystem check failed on the edited copy - NOT writing"
  echo; echo "Edited image: $IMG  (original untouched: $d/rootfsA.orig)"
  read -rp "Write it to the board's eMMC now? Type YES: " a; [ "$a" = YES ] || { echo "not written"; return; }
  dfu-util -a install.rootfsA -D "$IMG"
  dfu-util -a install.rootfsA -U "$d/rootfsA.readback"
  cmp "$IMG" "$d/rootfsA.readback" && echo "written and verified. Unplug, release B7, power up."
}

cmd_selftest() {
  need debugfs; need mkfs.ext4; need e2fsck; need python3; need openssl
  WORK=$(mktemp -d); trap 'rm -rf "$WORK"' EXIT
  local t=$WORK/tree dep=$WORK/tree/ostree/deploy/os/deploy/abc123.0
  mkdir -p "$dep/etc/systemd/system/multi-user.target.wants" "$dep/etc/wpa_supplicant" "$dep/usr/sbin" "$dep/usr/bin" \
           "$dep/usr/lib/aarch64-linux-gnu" "$dep/home/root"
  printf 'root:*:19000:0:99999:7:::\n' >"$dep/etc/shadow"
  printf 'root:x:0:0:root:/home/root:/bin/sh\n' >"$dep/etc/passwd"
  printf 'define svc_tcp_ports = {8883, 9124}\n' >"$dep/etc/nftables.conf"
  printf 'x' >"$dep/etc/systemd/system/multi-user.target.wants/wpa-supplicant-ctl.service"
  IMG=$WORK/fake.img; mkfs.ext4 -q -F -d "$t" "$IMG" 64M
  ssh-keygen -q -t ed25519 -N '' -f "$WORK/k" ; export SSH_PUBKEY=$WORK/k.pub
  export ROOT_PASSWORD=testpw WIFI_SSID=testnet WIFI_PSK=testpass STRIP_CLOUD=1
  apply_edits
  e2fsck -fn "$IMG" >/dev/null && echo "PASS fsck"
  local D=/ostree/deploy/os/deploy/abc123.0
  chk() { debugfs -R "$1" "$IMG" 2>/dev/null | grep -q "$2" && echo "PASS $3" || { echo "FAIL $3"; FAIL=1; }; }
  FAIL=0
  chk "cat $D/etc/shadow" 'root:\$6\$' "root hash set"
  chk "cat $D/etc/nftables.conf" '9124, 22}' "port 22 allowed"
  chk "stat $D/usr/sbin/dropbear" 'Mode:  0755' "dropbear mode"
  chk "stat $D/etc/shadow" 'Mode:  0640' "shadow mode"
  chk "ls $D/etc/systemd/system/getty.target.wants" 'serial-getty@ttyS2' "serial getty enabled"
  chk "ls $D/etc/systemd/system/multi-user.target.wants" 'dropbear.service' "dropbear enabled"
  chk "ls $D/etc/systemd/system" 'hubreg.service' "cloud masked"
  chk "cat $D/home/root/.ssh/authorized_keys" 'ssh-ed25519' "ssh key installed"
  chk "cat $D/etc/wpa_supplicant/wpa_supplicant-wlan0.conf" 'ssid="testnet"' "wifi config"
  [ -n "${KEEP:-}" ] && cp "$IMG" "$KEEP"
  [ "$FAIL" = 0 ] && echo "SELFTEST OK" || { echo "SELFTEST FAILED"; exit 1; }
}

case "${1:-}" in
  check) cmd_check;; backup) shift; cmd_backup "$@";; patch) shift; cmd_patch "$@";;
  selftest) cmd_selftest;; *) sed -n '2,13p' "$0"; exit 1;;
esac

What I have and haven’t verified

  • Verified on my hub, by hand: the SPL DFU read and write (byte-for-byte read-back comparison, including a full 7.28 GiB pull), editing the rootfs with debugfs, a working root login, dropbear SSH, joining my Wi-Fi, and stripping the cloud services.
  • Verified for the script: its selftest builds a fake ext4 image with an OSTree-style layout and checks that every edit lands with the right mode and ownership, that the nftables edit works, and that e2fsck is clean afterwards. That test downloads the real Debian packages. It proves the file-editing logic, not the hardware.
  • Not verified: running the whole script end to end against a hub. In particular, the by-hand version I used enabled the Wi-Fi live, over the UART; the offline Wi-Fi route in the script (a new config plus the template unit) is based on what I found afterwards on the running system and has not been tried from cold. It also assumes the same OSTree layout, /home/root home directory and unit names as my unit. If it fails, you still have the UART shell and the backup.
  • Bullseye has reached end of life, so the script falls back to archive.debian.org for the dropbear packages if the main mirror no longer carries them.

After it boots

You get a Debian-ish OSTree system with a root shell on the UART and SSH over Wi-Fi. A few things I found useful:

  • LEDs. There are three: hub:green:led1, hub:blue:led2, hub:red:led3. The blinking blue is just the kernel’s timer trigger set in the device tree, not an application. Write none to trigger before you set brightness or the trigger overrides you. The kernel also has panic (lights on a kernel panic, no userspace needed) and netdev (flashes with interface activity), which suit this nicely.
  • A real distribution. I built a Debian 12 arm64 root with debootstrap --foreign on the PC, copied it across, and ran the second stage on the hub itself in a busybox chroot, which avoids needing qemu. mkfs.ext4 -d populates an image from a directory with no loop mount. Note that dropbear has no sftp-server, so use cat file | ssh host "cat > dest" rather than scp.

What doesn’t work

  • Zigbee. The EFR32MG21 is wired to the SoC over SPI (EZSP-SPI) and runs Hive’s own NCP firmware. The kernel driver gets into a retry loop (SPI response failed: -11) at around 180 s of uptime on every boot I observed. My goal was to use the hub as a router extending my existing Zigbee network. I could not get it to join as one: the host-side library I used has no router-join path, and replicating Silicon Labs’ own join sequence still failed against the stock NCP firmware. Every community EFR32 firmware image I could find is UART-only, so nobody has published SPI firmware to replace it. Reflashing it needs the 10-pin programming header and EFR32-side pin names I don’t have. I stopped there on purpose.
  • A full custom boot chain. Still blocked by the TF-A SP_RESPONSE timeout.

If you try this and find anything the script gets wrong, I’d like to hear about it, particularly if you have the schematic for the Zigbee side.

Posted on — Leave a comment

HP 8xSFF SAS Backplane Reverse Engineering, and the Pico Bay Controller Behind It

The whole backplane: eight drive connectors in two banks of four, with the HP logo in the middle.
The whole backplane: eight drive connectors in two banks of four, with the HP logo in the middle.

I’m building a 2U NAS around two HP DL380 G7 eight-bay small form factor drive cages, and each cage comes with an HP 496074-001 SAS backplane. The drives sit happily on it, but the three little LEDs in each bay (blue, green and amber) are controlled by two tiny microcontrollers speaking a proprietary protocol, and I wanted to drive them myself. So this turned into a reverse engineering project: work out how the backplane really works, replace its firmware with something I understand, and draw the schematic, because I couldn’t find a published one.

This is a long post because it covers a few days of poking. Everything here was measured on my own boards. Where I’m guessing, I say so.

What’s on the board

Each backplane has two PIC18F24K20 microcontrollers (28 pins, no crystal, running from their internal oscillator). PIC A looks after bays 1 to 4 and PIC B looks after bays 5 to 8. The 3.3 V rail comes from a small linear regulator, a TC1264-3.3 (800 mA, SOT-223, marked 1264-33 / VDB1006) fed from the 5 V pins of the power connector.

There is a 10-pin Mini-Fit Jr power connector (J13, the only Molex part on the board) that carries more than power: two of its pins are an I2C bus and another is an “address strap” analogue input. There are also a 3-pin I2C header (J44), two plain 0.1 inch 2-pin headers (J14 beside PIC A, and an unpopulated J43 beside PIC B), and two ICSP programming headers (J11 for PIC A and J12 for PIC B). The two SAS connectors are SFF-8087 types, J1 (silkscreen IPASS1) feeding PIC A and bays 1 to 4, and J6 (IPASS2) feeding PIC B and bays 5 to 8.

The top of the board, with the power connector, regulator and the two PICs.
The top of the board, with the power connector, regulator and the two PICs.
PIC A (U1 on the silkscreen) with its ICSP header J11 and the small J14 header, next to the power connector.
PIC A (U1 on the silkscreen) with its ICSP header J11 and the small J14 header, next to the power connector.
The second PIC (U2) and the small resistors around it.
The second PIC (U2) and the small resistors around it.
A PIC beside one of the SFF-8087 connectors (the silkscreen reads IPASS), with the sideband traces running to the chip.
A PIC beside one of the SFF-8087 connectors (the silkscreen reads IPASS), with the sideband traces running to the chip.
The silkscreen: 'SAS Backplane', copyright 2007-2008 Hewlett-Packard, with the PCB part number and the HDD1 and HDD2 connectors.
The silkscreen: ‘SAS Backplane’, copyright 2007-2008 Hewlett-Packard, with the PCB part number and the HDD1 and HDD2 connectors.
Block diagram of the backplane
The block diagram. Green lines are verified on the board, amber dashed lines are inferred, grey dotted lines are not traced yet.

Reading the chips

I used a Microchip ICD 4 with MPLAB X’s command-line programmer. A few practical notes, since they cost me time:

  • The ICD 4 does not power the target, so the backplane needs its own bench supply, and I ended up needing the 12 V rail connected as well.
  • Pin 1 orientation on the ICSP header matters, and I got it wrong the first time.
  • While the ICD 4 is attached it holds the PIC in reset. Unplug the lead to let it run.
  • The programming lines are the same pins the PICs use to detect whether a drive is fitted in two of the bays, so pull any drives out of those bays before you programme.

The first good news was that neither chip is read protected. Before doing anything else I took a full backup of both chips (program memory, configuration bits, user ID and the little data EEPROM) and kept it somewhere safe. I’m not publishing the dumps, because it’s HP’s firmware, but I did compare them: the program code on the two chips is identical, and only the EEPROM differs, in a handful of bytes that tell each chip which half of the backplane it is.

Finding out what the firmware does

There’s no source, so I wrote a small PIC18 disassembler and read the start-up code. That gave me the list of which pins are outputs and which are inputs, and the rest followed from watching the board:

  • Thirteen outputs: the LED pins, plus one that toggles constantly (I’ll come back to that).
  • An I2C slave on the usual hardware pins.
  • Two analogue inputs, and a handful of digital inputs.

The oddest discovery was how the chip decides its I2C address. It doesn’t use a fixed address or a jumper. It measures a voltage on one pin, averages four ADC readings, and uses the result to pick an address from a table. That voltage comes from pin 9 of the power connector. It goes straight to PIC A, and through a 750 Ω resistor to PIC B, with 5.6 kΩ to ground on each chip’s input. PIC B also reads a voltage with pin 9 grounded, which that network can’t produce on its own, so I think there is a pull-up on it that I haven’t found yet. With pin 9 floating, both chips answered at the same address and collided. Grounding pin 9 live, with the board running, moved them to two separate addresses and I could talk to each chip individually.

The protocol itself is a simple framed command set: a type byte, a length, a command and a checksum that is the two’s complement of the sum. Replies start with a status byte. The documented behaviour I found was a firmware version query, an LED self-test that chases through every LED in each bay, a command that lights the blue “locate” LEDs, and some housekeeping queries.

The mistake

One command in the stock firmware writes to the chip’s EEPROM. I sent it a few times with junk while trying to work out what it did, and the chip’s behaviour changed afterwards. I’d corrupted its configuration area. It was recoverable only because I had backed everything up first.

The restore itself taught me something worth passing on: with Microchip’s ipecmd, the option that programs only the EEPROM (-ME) also erases the flash and resets the configuration bits. The safe way to restore is to programme the whole original hex file with the plain programme option and then read everything back and compare. I did that, and the round trip came back byte-for-byte identical.

The lesson is the usual one: back up first, and don’t send commands you don’t understand to firmware you can’t easily reflash.

Mapping the LEDs and the presence pins

Each bay has three LEDs and one “is a drive fitted” input. I found them by lighting one pin at a time and watching, and by hot-swapping drives while reading the port registers. All of the LED pins are active low (pulling the pin low lights the LED), and the drive-detect pins are active low too.

Bay (PIC A / PIC B) Blue (locate) Green (activity) Amber (error) Drive detect
1 / 5 RA2 (pin 4) RA4 (pin 6) RC2 (pin 13) RA7 (pin 9)
2 / 6 RA3 (pin 5) RA5 (pin 7) RB3 (pin 24) RA6 (pin 10)
3 / 7 RB5 (pin 26) RC0 (pin 11) RC7 (pin 18) RB7 (pin 28)
4 / 8 RB4 (pin 25) RC1 (pin 12) RC6 (pin 17) RB6 (pin 27)

Drive detect turned out to be neat. Pin 5 of each drive’s power connector (a ground pin on the drive side) goes through a 330 Ω resistor to the PIC’s detect pin, and the pin sits at about 7.5 to 10 kΩ to ground with no drive fitted, so a pull-up is implied. Fit a drive and its ground pulls the pin low. All eight bays are the same.

One thing I want to be clear about: drive activity is not wired to any input on the PICs. A drive with no data link never changed any pin apart from insert and remove. Early on I wrongly concluded that the PICs couldn’t see drives at all, from a run where I hadn’t actually moved the drive, and I invented a theory to explain it. I retracted both when I realised. Bad science on my part, and I’m glad I caught it.

Replacing the firmware

I wrote a replacement in C for the XC8 compiler. It is an I2C slave with a pointer-and-auto-increment register map, so a host can set a colour mask for each LED group, read the raw port pins (which tells you which bays have drives fitted), read the two ADC channels, and, for debugging, take direct control of the port latches. The two chips of one backplane get consecutive addresses (0x30 and 0x31 on the first, 0x32 and 0x33 on the second), set at build time. I flashed all four chips across the two backplanes, verified each byte for byte, and ran a slow colour twinkle across both as a test. The original images remain restorable.

The host side started life as an ESP32 acting as an I2C master. It has since been replaced by a Raspberry Pi Pico 2, which also runs the fans and the front panel; that is the second half of this post. On the first cage the register interface works end to end (colour masks, drive-detect and the ADC channels). The chips on the second cage are flashed and verified, but I haven’t exercised their registers yet.

The side channel: SGPIO

The weird output that toggles constantly bothered me. In the stock firmware, one interrupt handler on a single input pin runs a long state machine that alternates between sampling one input and driving that output. Four pins, one clock, one load strobe, one data in and one data out. That is exactly the shape of SGPIO (SFF-8485), the serial bus a SAS controller uses to tell a backplane which LEDs to light.

The decoded behaviour per bay is three bits each way. From the controller, the three bits drive the green activity, blue locate and amber error LEDs. Back to the controller, the chip reports drive presence. That also explains why the stock I2C interface only exposes the blue LED: green and amber were meant to come from the HBA.

The pin mapping, which I confirmed with a meter:

PIC pin Role SFF-8087 pin on J1 / J6 Verified how
23 (RB2) SClock A8 (SB0) matched-length trace pair seen, not buzzed (the row is hidden under the connector)
21 (RB0) SLoad A9 (SB1) same matched pair
22 (RB1) SDataOut B10 (SB4) buzzed
16 (RC5) SDataIn B11 (SB5) buzzed

The standard sideband assignment for these connectors comes from a 2007 T10 document, and the pins I could reach (B row) matched it exactly: B7, B9 and B12 are ground, B10 goes to pin 22 and B11 goes to pin 16. The A row sits underneath the surface-mount connector body where I couldn’t get a probe in, but the two lines that I couldn’t reach arrive as a length-matched pair at PIC pins 23 and 21, which fits a clock and a strobe.

Does it work with my HBAs? I don’t know. The cards are LSI 9211-8i (SAS2008), and reports of whether those drive the sideband in IT mode are mixed. I’ve built a passive capture mode for the replacement firmware that counts clock edges and records raw samples over I2C, so the first test will be a definite yes or no. It isn’t flashed yet.

The LED circuit

Then I started tracing the LED network by hand, with the board powered down, measuring resistance and diode drops in circuit. This is the awkward part: much of the board is hidden under surface-mount parts and I only managed one bay properly.

  • Green and amber: the anode goes to the 3.3 V net, and the cathode goes through a 68 Ω resistor straight to the PIC pin. That is about 19 mA per LED, comfortably inside the PIC’s limit.
  • Blue: the anode goes directly to the 5 V rail. The cathode goes through a 120 Ω resistor to the drain of a small SOT-23 N-channel MOSFET. The source of that MOSFET goes straight to the PIC pin, and its gate is tied permanently to 3.3 V.

That FET is a nice little piece of design. A blue LED needs about 3 V, which a 3.3 V supply can barely give, so HP runs it from 5 V. But a 3.3 V microcontroller pin must not see 5 V. With the gate held at 3.3 V, the transistor conducts when the PIC pulls its pin low (lighting the LED) and switches off when the pin is high, so the 5 V side never reaches the chip. There are eight of them, one per blue LED: Q7, Q17, Q6, Q8, Q3, Q9, Q4 and Q5 for bays 1 to 8. Each bay also has a dual LED (CR1, CR3, CR5 and so on) and a separate green one (CR2, CR4, CR6 and so on). The part is marked 701 on mine, while the ON Semiconductor 2N7002L datasheet lists the marking as 702, so I’m treating it as a 2N7002L-class device until I find out otherwise.

There are also two 100 µF electrolytic capacitors by every drive connector, one on the 5 V rail and one on the 12 V rail.

The 100 µF electrolytic capacitors beside a drive connector.
The 100 µF electrolytic capacitors beside a drive connector.

I’ve only measured bay 1 on PIC A. The rest of the board is drawn on the schematic as assumed identical (amber and dashed) rather than measured, and I’d rather the drawing said so than pretended otherwise.

The schematic

I generate the drawing from a script, so it can be corrected as I trace more of the board. There are nine A3 sheets in the PDF. The first is the block diagram above, the next three are the wired component-level schematics, and the remaining sheets are the pin maps, the connector pinouts, the per-bay table and the firmware interface.

Schematic sheet 2: power, I2C and drive power
Sheet 2: power connector, regulator, I2C bus and the drive power circuit.
Schematic sheet 3: PIC A circuit
Sheet 3: the PIC A circuit, with every LED branch, the drive-detect lines and the J1 sideband.
Schematic sheet 4: PIC B circuit
Sheet 4: the PIC B circuit, the same layout for bays 5 to 8 and J6.
Connector pinouts, with a render of the power connector
The connector pinouts, including a render of the female 2×5 Mini-Fit Jr power connector with its pin numbers.
Per-bay pin table and the SGPIO frame
The per-bay pin table and the SGPIO frame as the stock firmware handles it.

Reading it is straightforward: black solid lines are measured, amber dashed lines are inferred, and grey dotted lines have not been traced. The reference designators J1, J6, J11, J12, J14 and J44 come from the board’s silkscreen. Anything with an asterisk, like Q1*, RB1* or DG2*, is a name I invented to keep the drawing readable.

The PDF is at the end of the post.

Controlling it all: from an ESP32 to a Pico 2

The backplanes now speak a simple I2C register protocol, but something has to talk to them, and the same something has to run four case fans, read temperature probes and press the server’s power button. I started with an ESP32, then realised I had Raspberry Pi Pico 2 boards on the shelf and no WiFi requirement, so the controller is now a plain Pico 2 that plugs into a spare internal USB header on the motherboard. USB powers it and carries all the traffic: status, configuration, alerts and firmware updates.

What hangs off it:

  • I2C to the two backplanes (the J44 headers), which gives the NAS control of every bay LED.
  • Four PWM fans with tach, each through one 2N3904 that pulls the PWM line low (open collector), so it works with any fan pull-up. A jumper decides what the fans do if the Pico is off or hung: they go to full speed.
  • A 1-Wire bus for DS18B20 temperature probes.
  • Power and reset: two transistors across the front-panel header pins, in parallel with the real buttons, plus drivers for the 12 V LED rings in the case buttons and sensing of the motherboard’s own power and drive-activity LED signals.
Pinout of the Pico 2 bay controller, with every used pin labelled and wired to its connector.
The pinout, in the style of the official Pico diagram: fan PWM and tach interleaved on GP6 to GP13, 1-Wire on GP14, I2C on GP4 and GP5.

A few design decisions that are worth passing on:

  • I don’t rely on the RP2350’s internal pull resistors. The A2 stepping has a known erratum that makes the pull-downs unreliable, and pull-ups are weak for a fan tach anyway, so the tach lines have a real 10 kΩ and the 1-Wire bus a real 4.7 kΩ.
  • I swapped some pins from my first plan so that fan PWM and tach pins alternate. That makes the wiring to each fan header run straight across instead of crossing, which matters when you are building it on stripboard.
  • The fan headers are stacked in one column, so the four 12 V pins (and the four grounds) can be joined by a heavy wire under the board.
The Pico bay controller schematic, drawn with real wires in functional blocks.
The controller schematic, drawn with real wires rather than net labels: one block per fan, the I2C and 1-Wire headers, the two panel LED stages, the two button circuits and the 12 V input. The full-size PDF is at the end of the post.

I generated the drawing from a script and checked it electrically: it knows the intended connections, and it fails if any wire shorts two nets, leaves one open or crosses another. A stripboard layout then came out of the stripboard-editor.com solver at 31 by 49 holes, after a couple of attempts. None of this has been built yet, so treat it as a design, not a tested board.

Fans that learn

The Pico runs the fans itself, so they keep working if the NAS is off or crashed. Each fan has a temperature curve of up to eight points, hysteresis so it doesn’t hunt, and slew limiting so it doesn’t lurch. Three temperatures matter: a target the fan should settle at, a limit above which it adds 10 per cent per degree on top of the curve, and a critical temperature at which it goes to full speed.

On top of the curve there is a learned correction. Every thirty seconds, if the zone is warmer than the target, the correction for the current temperature band (2 °C wide) goes up a step; if it is comfortably cooler, it goes down. Over hours the fan finds the quietest speed that holds the target, band by band. It is saved to flash at most every fifteen minutes, so the flash isn’t worn out.

Calibration measures each fan’s start speed and its RPM at every speed, and it also tests the polarity. Fans differ: some run faster at a low PWM duty (inverted). It runs the fan at 20 per cent and then 80 per cent, and if the fan is faster at 20 it flips the software invert flag for that fan.

When a fan dies

A fan that is being driven but sends no tach pulses for eight seconds is treated as failed. Most servers respond by sending every other fan to full power, which is loud. I wanted something calmer: fans are grouped by what they cool, and when one fails the healthy fans in the group are raised by the ratio of total to healthy (so with four fans and one failed, the others go up by a third), capped at 85 per cent by default. Learning pauses while they are covering, so the learned curve isn’t polluted. A critical temperature still means full speed.

The Pico also notices a fan that is turning, but far too slowly (under half its calibrated speed for thirty seconds) and reports it. Every change of state goes into a numbered ring of events and is sent to the NAS straight away, so a host that was busy can still ask for what it missed.

NAS//OS: the control panel

The NAS operating system I’m writing is the control panel for all this. You create zones: a zone is a set of fans plus whatever they cool, drives by cage or bay and temperature probes by their unique ID. The NAS sends the Pico the hottest drive temperature in each zone from SMART, and keeps the Pico’s configuration in step with the zones by comparing what it reads back, so a Pico that has lost its settings is put right on the next poll. Failed fans, slow fans, a zone above its limit, a missing probe or a lost link raise an e-mail alert, and the alert clears when the problem does.

Telling the probes apart is the awkward bit with 1-Wire, because they are identical. The NAS has a “find a probe” button: it remembers every probe’s temperature, you warm one in your hand, and the one that heats up is highlighted so you can name it and say where it sits.

I was caught out on one thing: I’d written my own 1-Wire routines, JSON writer and checksum in the Pico firmware. Then I looked at what the ecosystem already provides and replaced them with the OneWire, DallasTemperature, ArduinoJson and CRC32 libraries. It’s less code to maintain and far less to get wrong.

It has been tested against simulations, not hardware: unit tests on the control logic (failure cover, limits, polarity detection), a whole-firmware simulation on the PC with a model fan and a thermal model, and 38 checks of the NAS side against a simulated Pico. The real test starts when I build the board.

What I still don’t know

  • Whether my 9211-8i cards drive the SGPIO lines at all (test pending).
  • The pad-level confirmation of the sideband clock and load pins (hidden row).
  • Where the SAS lanes go (they bypass the PICs).
  • The LED network on every bay except the first.
  • What J14 and J43 are for. Pin 1 is ground and pin 2 goes through a 10 kΩ resistor to pin 26 (RB5) of the nearest PIC, with no divider to ground. That pin is also the blue LED output for bay 3 (bay 7 on the second chip), which makes me suspect a factory test tap, but I haven’t proved it.
  • What the PICs’ analogue input RA1 is for. On the second chip it is a 1 kΩ and roughly 8.2 kΩ divider with a capacitor, reading a constant 0.891 of the supply, and on the first chip the trace vanishes down a blind via. My guess is a fixed hardware-ID strap, a bit like the address strap, but I haven’t proved it.
  • Pull-up values, the regulator’s capacitors and the PIC B strap divider.
  • How the Pico controller behaves on real hardware: the board is not built yet, and I haven’t run the 1-Wire library against a real probe on the RP2350.

What I’d do differently

  • Back up the chips before touching anything, and verify the backup by reading it back.
  • Don’t experiment with write commands on the only copy of the firmware.
  • Check that I have actually changed the physical state before drawing conclusions from a log.
  • Draw the schematic as you go. Doing it properly exposed a couple of places where I’d misattributed a connection in my notes, such as putting the interrupt on the wrong pin and the FETs on the wrong LED colour.

I’ll follow this up when the SGPIO capture test is done, when I’ve been through the rest of the bays with a meter, and when the controller board has been built and has met a real fan failure.

Downloads

Posted on — Leave a comment

ESPHome Environmental Sensor Prototype – ESP32-C3, SCD41, SEN55 & OLED

Environmental Sensor Prototype
Environmental Sensor Prototype

I’ve had a CO₂ monitor in the master bedroom for a while now, built around a Sensirion SCD41 on a plain ESP32 dev board running ESPHome, and feeding Home Assistant. Since then it has picked up a Sensirion SEN55 for particulates, VOC and NOx (I’ve had one of those apart too – [link to the SEN55 teardown once it’s published]), and I finally got round to what I really wanted – a display on the unit itself, so I can see the numbers without opening a dashboard. That meant a different board: an ESP32-C3 with a 0.42″ OLED built in, and a new name to match, since it’s no longer just a CO₂ sensor. To be clear up front, this is still a prototype – everything is on a breadboard for now, with no enclosure – so treat this as a build log rather than a finished design.

The OLED on these boards is tiny – a 72×40 pixel window in the middle of a standard SSD1306’s 128×64 RAM – and it’s hard-wired to I²C on GPIO5 (SDA) and GPIO6 (SCL). Those two pins are broken out to the header as well, so the sensors just share the same bus: the OLED at 0x3C, the SCD41 at 0x62 and the SEN55 at 0x69. One thing to watch is that the SEN55 needs 5V on its supply pin (from the board’s 5V pin), with the I²C lines pulled up to 3.3V. The SCD41 is happy on 3.3V.

As it stands, everything is just plugged into a breadboard, with the big SEN55 module out at one end and the SCD41 breakout sat next to the ESP32 board. It works well enough for developing the software and finding out what needs fixing, which is exactly what this stage is for.

ESPHome already has a “SSD1306 72×40” display model, which applies the column offset for the odd panel position by itself, so the display config is short. Two differences from the old board: the C3 has native USB rather than a USB-serial chip, so the logger has to be pointed at USB_SERIAL_JTAG, and the 26MHz crystal override the old board needed doesn’t apply any more (this one reports 40MHz like it should).

i2c:
  id: i2c_bus
  sda: GPIO5
  scl: GPIO6
  scan: true

logger:
  hardware_uart: USB_SERIAL_JTAG

display:
  - platform: ssd1306_i2c
    id: oled
    model: "SSD1306 72x40"
    address: 0x3C
    update_interval: 1s

There isn’t room for more than one reading at a time, so the display cycles through nine pages, three seconds each: CO₂, temperature, humidity, PM1.0, PM2.5, PM4.0, PM10, VOC index and NOx index. Rather than nine separate pages, the drawing code is a single lambda with a small table of label, unit, sensor and decimal places, and a global page counter that a 3 second interval bumps along. Anything without a value yet shows dashes instead of a bogus number.

Readings Close Up
Readings Close Up

The first thing I noticed was that the zeros looked like the letter D. On a 1-bit display with no anti-aliasing, at 10 or 11 pixels high, Roboto renders a zero as a plain rounded box – and that’s exactly what a D looks like too. I rendered the glyphs at the real pixel size with no smoothing before touching the hardware (below). Switching to Roboto Mono, which has a slashed zero, doesn’t help at that size: the slash collapses into a horizontal bar and the zero looks like an 8 instead. Inconsolata Bold keeps a clearly recognisable slashed zero at this size, so that’s used for the small text (at 11px). Roboto Mono Medium works fine for the big value at 22px, where there’s room for the slash to survive. At that size the widest thing the display needs to show is a five character value like 123.4, which is 66 pixels wide, so it just fits in the 72.

Zeroes At 10-11px – Roboto, Roboto Mono, Inconsolata Bold
Zeroes At 10-11px – Roboto, Roboto Mono, Inconsolata Bold

The next problem is more about physics than fonts. Humidity depends on temperature, and the SCD41 works its relative humidity out using its own reported temperature. If that temperature is reading warm – from the sensor’s own self-heating, say – the humidity comes out low. I’d been correcting this with a flat offset on each: a temperature offset, and a humidity offset of +3%. But that’s only right at one particular temperature and humidity.

What actually stays constant is the amount of water vapour in the air. Correct the temperature, and the relative humidity follows from the ratio of saturation vapour pressures at the two temperatures, using the Magnus formula for the saturation pressure:

RH_corrected = RH_raw * es(T_raw) / es(T_corrected)
es(T) = 6.112 * exp(17.62 * T / (243.12 + T))   [hPa]

In ESPHome, that’s a template sensor that reads the raw temperature and humidity and the live-tunable temperature offset:

lambda: |-
  if (id(humidity_raw).has_state() && id(temp_raw).has_state()) {
    auto es = [](float t) { return 6.112f * expf(17.62f * t / (243.12f + t)); };
    float t_raw = id(temp_raw).state;
    float t_corr = t_raw + id(temp_cal_offset).state;
    float rh = id(humidity_raw).state * es(t_raw) / es(t_corr)
             + id(humidity_cal_offset).state;
    return clamp(rh, 0.0f, 100.0f);
  } else {
    return {};
  }

Checking this against the numbers I had from the old board: with a -0.8°C temperature offset, the physics alone gives +2.68% RH at those conditions, against the flat +3.0% I’d dialled in by hand. So the old offset was almost entirely doing this job anyway, and the humidity offset is now just a small residual trim (I’ve set it to 0.5%). The difference is that this now follows the temperature: at 26°C and 45% RH the correction is about +2.2%, at 20°C and 60% it’s about +3.1%, and a flat offset can’t do that.

To check the calibration I sat the sensor about a metre from a Sonoff temperature and humidity sensor in the living room. I’d like to say the readings matched straight away, but they didn’t – I’d just been handling it, and the sensors take a good while to settle after being picked up. Body heat warmed the temperature reading, and my breath sent the CO₂ up to 2005ppm on the first readings, which had dropped to around 660ppm in under half an hour. It’s worth waiting for things to settle before touching any offsets. Once it had, the numbers agreed well. This is all with the bare breadboard layout, in free air. The SCD41’s temperature offset is a compile-time setting, and ESPHome’s default of 4°C is intended for a bare board in free air – once this goes in an enclosure, self-heating will be worse (more like 6-10°C as a rule of thumb), so all of these numbers will need going through again.

One thing the SEN55 can do that I hadn’t exposed is a forced fan clean, which spins the fan at maximum speed for about 10 seconds to blow dust off the blades. The sensor does this by itself every week by default, but it’s useful to trigger manually after a fan speed warning or a dusty spell. ESPHome has an action for it (sen5x.start_fan_autoclean), so there’s now a button in Home Assistant, and also a long press (2 seconds or more) of the BOOT button on the board – GPIO9, so it’s just a normal input while the chip is running. Both call the same script, which sets a flag so the OLED shows “Cleaning” for the duration instead of PM readings that the cleaning burst would skew, and ignores a second trigger if one is already running.

script:
  - id: sen55_fan_clean
    mode: single
    then:
      - globals.set:
          id: fan_cleaning
          value: 'true'
      - sen5x.start_fan_autoclean: sen55
      - delay: 12s
      - globals.set:
          id: fan_cleaning
          value: 'false'

binary_sensor:
  - platform: gpio
    internal: true
    pin:
      number: GPIO9
      mode: INPUT_PULLUP
      inverted: true
    filters:
      - delayed_on: 30ms
    on_click:
      min_length: 2s
      max_length: 60s
      then:
        - script.execute: sen55_fan_clean

ESPHome’s SEN5x component doesn’t expose the sensor’s device status register, so I read it directly over I²C once a minute and turn the bits that matter into diagnostic binary sensors in Home Assistant: fan speed warning, fan failure, laser failure and gas sensor error. The fan and laser failure bits are sticky – they stay set after the fault clears until the register is cleared or the sensor is power cycled – so there’s a button that sends the clear command as well. There’s also a “Sensor Runtime” counter, stored in flash and added to every five minutes, since the uptime sensor resets on every boot and these sensors have a finite service life. It was seeded with the hours from the old board when the sensors moved over, and there’s a button to zero it when a sensor gets swapped.

A couple of gotchas from the changeover. Home Assistant treats a new board as a completely new device, with all-new entities and no history. I removed the old device, added the new one, and then renamed the new entities back to the old entity IDs, so the automations, dashboard cards and recorded history carried straight over. And on the very first boot the SCD41 didn’t respond at all – ESPHome marked it as failed in the log, and a failed component isn’t retried until the next reboot. A reset was all it needed, and it’s been fine since.

The full config is below, in case it’s useful. As this is still a prototype, expect it to change as the build develops.

Full Config

esphome:
  # Node name is capped at 31 chars by ESPHome, so "environmental" is abbreviated
  # here - the user-facing name below is the full one.
  name: master-bedroom-env-sensor
  friendly_name: "Master Bedroom Environmental Sensor"

esp32:
  # ESP32-C3 board with the integrated 0.42" (72x40) SSD1306 OLED. No board
  # definition exists for the specific module, so the generic C3 devkit is used.
  # (The old esp32dev board's 26MHz crystal override doesn't apply to the C3.)
  board: esp32-c3-devkitm-1
  variant: esp32c3
  framework:
    type: esp-idf

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

logger:
  # The C3 has native USB and no UART bridge, so the default UART0 logger is not
  # what the USB port is wired to.
  hardware_uart: USB_SERIAL_JTAG

api:

ota:
  - platform: esphome
    password: !secret ota_password

web_server:
  version: 3

# The OLED is hard-wired to GPIO5 (SDA) / GPIO6 (SCL) on this board, so the
# sensors share that bus: SCD41 0x62, SEN55 0x69, OLED 0x3C.
i2c:
  id: i2c_bus
  sda: GPIO5
  scl: GPIO6
  scan: true

# Cumulative powered-on time, surviving reboots - the uptime sensor resets on
# every boot, which is no use for tracking sensor service life.
#
# The runtime tracks the SCD41/SEN55 hardware, not the board, and both were moved
# over from the old ESP32, so this is seeded with the old board's last reported
# value (790.917h at 2026-09-20 11:46 UTC, from HA history, just before it was
# unplugged). initial_value only takes effect on the first flash - restore_value
# takes over from flash after that.
globals:
  - id: sensor_runtime_seconds
    type: uint32_t
    restore_value: yes
    initial_value: '2847300'

  # Raw SEN55 Device Status Register (I2C command 0xD206) - not exposed by
  # ESPHome's sen5x component, read directly on the bus. Bit layout per the
  # Sensirion SEN5x datasheet section 5.4: bit 21 speed warning, bit 7 gas
  # sensor error, bit 6 RHT comms error, bit 5 laser failure, bit 4 fan
  # failure. Bits 4 and 5 are sticky - they do NOT self-clear even once the
  # fault condition resolves, only via Clear Device Status (0xD210) or a
  # power cycle.
  - id: sen55_device_status
    type: uint32_t
    restore_value: no
    initial_value: '0'

  # Index of the reading currently shown on the OLED.
  - id: display_page
    type: int
    restore_value: no
    initial_value: '0'

  # True while a SEN55 fan cleaning cycle is running, so the OLED can say so
  # instead of showing PM readings that the cleaning burst will skew.
  - id: fan_cleaning
    type: bool
    restore_value: no
    initial_value: 'false'

interval:
  # Accumulated in coarse 5-minute ticks rather than continuously: this global is
  # flash-backed, and every change to it eventually costs an NVS write. A tick
  # this size keeps writes down to ~288/day while losing at most 5 minutes of
  # the partial period on reboot - irrelevant against a service life in years.
  - interval: 5min
    then:
      - lambda: 'id(sensor_runtime_seconds) += 300;'

  - interval: 60s
    then:
      - lambda: |-
          if (id(sen55)->is_failed()) return;
          uint16_t raw[2];
          // MSW first, then LSW, per the datasheet's Read Device Status response layout.
          if (id(sen55)->get_register(0xD206, raw, 2, 20)) {
            id(sen55_device_status) = (uint32_t(raw[0]) << 16) | raw[1];
          }

  # Advance the OLED to the next reading. Keep in step with the table length in
  # the display lambda below.
  - interval: 3s
    then:
      - lambda: 'id(display_page) = (id(display_page) + 1) % 9;'

# SEN55 fan cleaning, shared by the "Clean SEN55 Fan" HA button and the long-press
# of the board's BOOT button. mode: single ignores a second trigger while a clean
# is already running instead of restarting the sequence.
script:
  - id: sen55_fan_clean
    mode: single
    then:
      - globals.set:
          id: fan_cleaning
          value: 'true'
      - sen5x.start_fan_autoclean: sen55
      # Datasheet: cleaning takes 10 seconds; hold the OLED message a little longer.
      - delay: 12s
      - globals.set:
          id: fan_cleaning
          value: 'false'

font:
  # Inconsolata Bold for the small text: at 10-11px on a 1-bit OLED both Roboto's
  # and Roboto Mono's zero collapse into a box that looks like a "D". Inconsolata's
  # slashed zero stays distinct at this size. 'PM2.5' + 'µg/m³' is 55px wide at
  # size 11, inside the 72px window.
  - file: "gfonts://Inconsolata@700"
    id: font_small
    size: 11
    glyphs: ' !%()+-_.,:/°µ³²0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
  # Roboto Mono for the big value: on a 1-bit OLED, Roboto's zero renders as a
  # rounded rectangle that reads as a "D". Roboto Mono's zero is slashed. Size 22
  # keeps a 5-character value ("123.4") at 66px, inside the 72px window.
  - file: "gfonts://Roboto Mono@500"
    id: font_big
    size: 22
    glyphs: '0123456789.-'

# 0.42" OLED: a 72x40 window in the middle of the SSD1306's 128x64 RAM. ESPHome's
# "SSD1306 72x40" model applies the column offset itself.
display:
  - platform: ssd1306_i2c
    id: oled
    model: "SSD1306 72x40"
    address: 0x3C
    update_interval: 1s
    lambda: |-
      if (id(fan_cleaning)) {
        it.printf(36, 14, id(font_small), TextAlign::CENTER, "Cleaning");
        it.printf(36, 27, id(font_small), TextAlign::CENTER, "SEN55 fan");
        return;
      }

      struct Reading {
        const char *label;
        const char *unit;
        sensor::Sensor *sensor;
        int decimals;
      };
      // One entry per OLED page - if this length changes, update the modulus in
      // the 3s interval above.
      const Reading pages[] = {
        {"CO2",      "ppm",   id(co2_sensor),      0},
        {"Temp",     "°C",    id(temp_corrected),  1},
        {"Humidity", "%",     id(humidity_corrected), 1},
        {"PM1.0",    "µg/m³", id(pm_1_0),          1},
        {"PM2.5",    "µg/m³", id(pm_2_5),          1},
        {"PM4.0",    "µg/m³", id(pm_4_0),          1},
        {"PM10",     "µg/m³", id(pm_10_0),         1},
        {"VOC",      "index", id(voc_index),       0},
        {"NOx",      "index", id(nox_index),       0},
      };
      const Reading &r = pages[id(display_page) % 9];

      // Label top-left, unit top-right, value filling the rest.
      it.printf(0, 0, id(font_small), TextAlign::TOP_LEFT, "%s", r.label);
      it.printf(71, 0, id(font_small), TextAlign::TOP_RIGHT, "%s", r.unit);
      if (r.sensor->has_state() && !std::isnan(r.sensor->state)) {
        it.printf(36, 27, id(font_big), TextAlign::CENTER, "%.*f", r.decimals, r.sensor->state);
      } else {
        it.printf(36, 27, id(font_big), TextAlign::CENTER, "--");
      }

# Live-tunable fine calibration, exposed to HA so you can dial these in against
# a reference thermometer/hygrometer without reflashing. Values persist across reboots.
#
# The temperature offset carries over what was dialled in on the old board
# (-0.8C). The humidity offset is now only a RESIDUAL trim: Humidity (Corrected)
# is recomputed from the corrected temperature (see below), which accounts for
# almost all of what the old flat +3.0% was doing (+2.68% at the old conditions,
# leaving ~+0.3%; 0.5 is the nearest step). initial_value only applies on first
# flash, so an already-flashed board keeps whatever value is stored on it.
# Re-check both against a reference once the sensor has settled.
number:
  - platform: template
    name: "Temperature Calibration Offset"
    id: temp_cal_offset
    icon: mdi:thermometer-plus
    unit_of_measurement: "°C"
    min_value: -10
    max_value: 10
    step: 0.1
    initial_value: -0.8
    restore_value: true
    optimistic: true

  - platform: template
    name: "Humidity Calibration Offset"
    id: humidity_cal_offset
    icon: mdi:water-percent
    unit_of_measurement: "%"
    min_value: -20
    max_value: 20
    step: 0.5
    initial_value: 0.5
    restore_value: true
    optimistic: true

sensor:
  - platform: scd4x
    id: scd41
    co2:
      name: "CO2"
      id: co2_sensor
    temperature:
      name: "Temperature (Raw)"
      id: temp_raw
    humidity:
      name: "Humidity (Raw)"
      id: humidity_raw
    address: 0x62
    update_interval: 60s
    automatic_self_calibration: true
    # Onboard self-heating compensation, fed back into the sensor's own RH calc.
    # 4C is ESPHome's own default for a bare board in free air - if this ends up
    # in an enclosure, expect more like 6-10C. Re-tune against a reference and
    # reflash once you have real comparison data; this one's compile-time only.
    temperature_offset: 4°C
    measurement_mode: periodic

  - platform: template
    name: "Temperature (Corrected)"
    id: temp_corrected
    unit_of_measurement: "°C"
    device_class: temperature
    state_class: measurement
    accuracy_decimals: 1
    lambda: |-
      if (id(temp_raw).has_state()) {
        return id(temp_raw).state + id(temp_cal_offset).state;
      } else {
        return {};
      }
    update_interval: 60s

  # RH is a function of temperature: the SCD41 computed its RH using the (biased)
  # temperature it reports as Temperature (Raw), so when the temperature is
  # corrected, the RH has to be re-derived, not just nudged by a flat offset. The
  # vapour pressure in the air doesn't change; only the saturation pressure at
  # the true temperature does:
  #     RH_corrected = RH_raw * es(T_raw) / es(T_corrected)
  # with es from the Magnus formula (hPa, valid -45..60C). The Humidity Calibration
  # Offset is then only a small residual trim on top of that.
  - platform: template
    name: "Humidity (Corrected)"
    id: humidity_corrected
    unit_of_measurement: "%"
    device_class: humidity
    state_class: measurement
    accuracy_decimals: 1
    lambda: |-
      if (id(humidity_raw).has_state() && id(temp_raw).has_state()) {
        auto es = [](float t) { return 6.112f * expf(17.62f * t / (243.12f + t)); };
        float t_raw = id(temp_raw).state;
        float t_corr = t_raw + id(temp_cal_offset).state;
        float rh = id(humidity_raw).state * es(t_raw) / es(t_corr) + id(humidity_cal_offset).state;
        return clamp(rh, 0.0f, 100.0f);
      } else {
        return {};
      }
    update_interval: 60s

  - platform: sen5x
    id: sen55
    pm_1_0:
      name: "PM 1.0"
      id: pm_1_0
    pm_2_5:
      name: "PM 2.5"
      id: pm_2_5
    pm_4_0:
      name: "PM 4.0"
      id: pm_4_0
    pm_10_0:
      name: "PM 10.0"
      id: pm_10_0
    voc:
      name: "VOC Index"
      id: voc_index
    nox:
      name: "NOx Index"
      id: nox_index
    temperature:
      name: "Temperature (SEN55)"
    humidity:
      name: "Humidity (SEN55)"
    address: 0x69
    update_interval: 60s
    acceleration_mode: low
    store_baseline: true

  - platform: template
    name: "Sensor Runtime"
    id: sensor_runtime
    icon: mdi:timer-sand
    unit_of_measurement: "h"
    device_class: duration
    state_class: total_increasing
    accuracy_decimals: 1
    entity_category: diagnostic
    lambda: 'return id(sensor_runtime_seconds) / 3600.0f;'
    update_interval: 60s

  - platform: uptime
    name: "Uptime"

  - platform: wifi_signal
    name: "WiFi Signal"
    update_interval: 60s

  - platform: internal_temperature
    name: "ESP32 Internal Temperature"
    entity_category: diagnostic
    # Chip die temperature, not ambient - runs hotter than the room due to
    # CPU/WiFi load. Diagnostic only, not a substitute for the SCD41 reading.

binary_sensor:
  # The board's BOOT button (GPIO9, active low with an external pull-up). Holding
  # it for 2s or more, then releasing, runs a SEN55 fan clean. Internal: it needs
  # no HA entity. GPIO9 is a strapping pin, but that only matters if it is held
  # low while the chip resets - pressing it while running is harmless.
  - platform: gpio
    id: boot_button
    internal: true
    pin:
      number: GPIO9
      mode: INPUT_PULLUP
      inverted: true
    filters:
      - delayed_on: 30ms
    on_click:
      min_length: 2s
      max_length: 60s
      then:
        - script.execute: sen55_fan_clean

  - platform: template
    name: "SEN55 Fan Speed Warning"
    icon: mdi:fan-alert
    device_class: problem
    entity_category: diagnostic
    lambda: 'return id(sen55_device_status) & (1UL << 21);'

  - platform: template
    name: "SEN55 Fan Failure"
    icon: mdi:fan-off
    device_class: problem
    entity_category: diagnostic
    # Sticky - stays on after a one-off blockage clears until Clear Sensor Faults is pressed.
    lambda: 'return id(sen55_device_status) & (1UL << 4);'

  - platform: template
    name: "SEN55 Laser Failure"
    icon: mdi:alert-decagram
    device_class: problem
    entity_category: diagnostic
    # Sticky - datasheet: not cleared automatically even once current is back in range.
    lambda: 'return id(sen55_device_status) & (1UL << 5);'

  - platform: template
    name: "SEN55 Gas Sensor Error"
    icon: mdi:molecule
    device_class: problem
    entity_category: diagnostic
    lambda: 'return id(sen55_device_status) & (1UL << 7);'

button:
  # Manual trigger for the SEN55's fan-cleaning routine (spins the fan at maximum
  # speed for ~10s to blow dust off the blades). The sensor already does this by
  # itself weekly; this is for after a Fan Speed Warning or a dusty stretch.
  # Only valid while the sensor is measuring, which it always is here.
  - platform: template
    name: "Clean SEN55 Fan"
    icon: mdi:fan-auto
    entity_category: config
    on_press:
      - script.execute: sen55_fan_clean

  - platform: template
    name: "Clear SEN55 Sensor Faults"
    icon: mdi:alert-remove
    entity_category: config
    on_press:
      - lambda: 'id(sen55)->write_command(uint16_t(0xD210));'

  - platform: template
    name: "CO2 Forced Calibration (420ppm outdoor air)"
    icon: mdi:target
    on_press:
      - scd4x.perform_forced_calibration:
          value: 420
          id: scd41

  # Zero the hour counter when a sensor is physically swapped out, so the
  # figure keeps tracking the hardware actually installed rather than the board.
  - platform: template
    name: "Reset Sensor Runtime"
    icon: mdi:timer-refresh
    entity_category: config
    on_press:
      - globals.set:
          id: sensor_runtime_seconds
          value: '0'
      - component.update: sensor_runtime

  - platform: restart
    name: "Restart"
Posted on — Leave a comment

Brsk Updates – A New Network Link

I posted a couple of weeks back that I was soon to be the recipient of a new gigabit symmetric link to the intertubes – and there was a bit of a caveat of their use of CG-NAT due to the exhaustion of the IPv4 address space.

To compensate I’ve routed everything over a VPN link in prep for this new connection, which does work fantastically well.

After a chat with their support guys – and I did actually manage to speak to someone in their Networks team, not just a customer support droid(!), they will soon be introducing a service where I can add a public IPv4 to my account, so I’ll be able to run at full gigabit speeds without the 100Mbit restriction of this VPN connection!

I’m told that this will be within the next couple of months, which is around the same lead time as the connection itself, so hopefully this will be live just in time for my new fibre.

More updates to come as this progresses!

Posted on — Leave a comment

Infrastructure Updates & The Cost Of Living

Everyone who reads this website will know about the cost of living crisis – this is a global problem brought about by the combination of the Covid-19 pandemic, and the unfortunate situation in Ukraine at the present time.

For us in the UK, the costs of energy have skyrocketed in recent months, with our government permitting the energy companies to charge us all an extra £600 per year in electricity alone. Given I run servers, this isn’t particularly good news.

The cost of running the network infrastructure behind this blog is not insignificant – just the server alone runs me around £30-£40 per month just in power, and add onto that the £45 in connectivity costs. I don’t meter the power taken by the router, switch etc, so the total cost will be marginally higher.
Given the increases that have hit, I expect to see a 3x increase in power costs to run this blog on the current hardware – an unsustainable jump.

I’ve got a couple of Raspberry Pi’s floating around, at the moment uncommitted to any tasks, so I thought I’d use one of these. A Pi CM4 module in a support PCB with dual network links should so just fine. I ogirinally got this to build a Pi based router (what it was designed for) – nevermind!

I’m using the new 64-Bit version of Raspberry Pi OS, and a 128GB MicroSD card for storage, since this board doesn’t have an open PCIe lane for a NVMe drive.

A quick setup with Nginx, php-fpm & MariaDB & I’m in business! The site is now running on a Raspberry Pi! As a plus, everything does seem to be quicker than the old Apache setup too. I have for now dropped a few features of the site, just until I get a feel of how quick this new setup is going to be, but they should be back soon!

Posted on — 1 Comment

FTTP Has Arrived – With Some Caveats

For 90% of the time I’ve run this website, it’s been run from a server in my house, over a domestic-grade ADSL connection. The problem has always been the very asymmetric download/upload bandwidth provided by domestic ISPs. Normally it’s about a 10:1 ratio these days (the current connection is 350MBit DL / 20MBit UL, which is even worse). Going back a few years, this wasn’t much of an issue – 99.9% of general ISP customers didn’t need a high upload rate, as they were only requesting web addresses, etc. Nowadays especially with the rise of online video services & social media, upload rates are more important than ever, since a good proportion of the population is uploading their own content daily.

Despite these changes in how the networks are utilised by the end users, the upload rates haven’t really improved much, which I’ve always found baffling.

Considering the bandwidth limitations imposed by such a connection on upload rates, the site’s done pretty well over the years, even though it can be a touch slow at times. I’ve been looking at changing this situation for a long time, watching to see if any of the direct fibre ISPs in the UK were going to get around to serving my area, such as Gigafibre, CityFibre or HyperOptic, but they unfortunately haven’t yet appeared anywhere near my locale.

Cue the entry of a new ISP: Brsk. These guys are building an entirely new fibre network, and just so happen to be doing the network build in my direct area at the moment. For only £5 more a month than I’m paying for the service I have now, I can now obtain full gigabit symmetric connectivity!

There’s one small caveat to the service, which wouldn’t bother the average member of the public, is a serious block to how I use my internet connectivity – they don’t hand out globally addressable IPv4 addresses. This is ostensibly due to the global IPv4 address shortage. Instead, they use CGNAT to address their endpoints, which renders me unable to route anything out to the global internet from my connection. Luckily there are ways around this issue.

I do have access to some public IPv4 addresses that are currently unallocated, in another networking installation, which also runs a VPN into my home network. So it’s just a matter of hooking up my primary server to that VPN, and routing one of the spare IPv4s over the private link from the router at the other site. This will limit thing somewhat, as the other site link is a 100MBit symmetric leased-line, however that’s still an 80MBit improvement in upload rate overall.

This latter solution of routing a public IP over VPN to a private endpoint is something I’ve already done in preparation – and it works great. The firewall on the site router provides extra protection for the server, and even with the VPN overhead it’s just as quick as it was before.

Since they’re still building out the network, I’ve only been able to place a preorder, and the connection will be installed (allegedly) sometime in July. Once the new link is in, I can get rid of my existing provider.

Once the link’s installed I’ll provide some updates!

Posted on — 3 Comments

DIY Thermal Laser Power Sensor Build

TEC On Heatsink
TEC On Heatsink
TEC On Heatsink

I’ve been looking for ways to build a DIY Laser Power meter for some time now, but I had no way to calibrate anything. I have been aware of DIY thermopile sensors with TECs – but no way to verify any results until now. Since I have my Gentec meter to calibrate against, I can finally get on with the project.
In the photo above, is the Peltier/TEC module mounted to the heatsink with thermal compound. In this case it’s a TEC1-12706, pulled from a cheapo dehumidifier. The cold face (with the part number printing) is against the heatsink, and the hot face will serve as the beam target. This was cleaned with solvent, and roughed up a bit with silicon carbide abrasive paper – the abrasive needs to be harder than the Alumina ceramic the module is constructed from.

Optical Coating
Optical Coating

To be any good as an optical power sensor, the front face of the module needs to be coated with something to absorb as much energy from the laser beam as possible. In this case, black paint was used, as it’s completely matte when dried. Lampblack also works, and this can be coated onto a sensor face with just a wax candle, but this is far too fragile to be any practical use (being just carbon, it’s much more resistant to thermal damage from the laser beams though!).

Coated Sensor
Coated Sensor

After two coats of the paint are applied to the front face of the module, the sensor head is complete. Try to get this as smooth as possible for best results. I designed & 3D printed a retention bracket for the module, and matches up with the screws that hold the fan on the finned side. This also has a block on the bottom which I threaded 1/4-20 to fit a standard tripod thread.

Like all commercial laser power sensors, the beam should be expanded as much as possible to fill the full face of the sensor. A focused high-power beam will quickly destroy the coating!

After completion, the sensor needs to be characterised. For this I set a diode module to be as close to 1W as possible, according to my calibrated meter, and applied the beam to the constructed sensor. A load impedance of 68Ω was placed across the output leads as a load. For this unit, I obtained a reading of 83.5mV/W of applied power. Even for low power levels, the fan does need to be running on the back of the heatsink, as the cold side of the sensor heating up will skew the reading.

After calibration at 1W optical power, I then ran some more tests at higher powers – 2W gave exactly double the output voltage, and throughout the power range I am able to test, the sensor seems to be entirely linear in operation.

Posted on — Leave a comment

BMW 5 Series Battery Pack – Reconfiguring The Modules & Wiring

Mounting Board
Mounting Board

Now the cell modules have been removed from their original home, it’s time to get them repurposed! A custom mounting board has been constructed from timber, and the modules mounted on them. To say this assembly is heavy would be an understatement – it’s barely a two-man lift!
As assembled in the car, the pack as 96S1P, with every cell in series. As we need a low voltage bus, the modules have been reconfigured for 4S4P, in total this makes 4S24P with all 6 modules bussed together. As the cell interconnects are laser welded, some ingenuity was required here.
It turned out the best method (and the safest, to avoid any swarf shorting out cells!), was to use a grinder to cut off the top of the loop on the aluminium interconnects, separating them.

Battery Bus Links
Battery Bus Links

12 5-way bus bars have been installed on the board, and 25mm² cable links them together. To get the angry pixies from the cell modules, 8mm² flexible silicone cable has been used, 4 links to a bus bar. This setup should provide more than enough current capacity.

Cell Connections
Cell Connections

Here can be seen the cell interconnects – and the grinder cuts to separate them where required to break the module up into 4S strings. As the interconnects are Aluminium, special solder was required to get the copper cables soldered down, in my case I used Alusol 45D solder, which contains a very active flux capable of stripping the oxide from the Aluminium.

Batteries Wired
Batteries Wired

Finally, here is the new pack, all connected together. All that needs to be done now is the balance wiring loom, which will allow the BMS to sense each cell voltage individually, and connection of the BMS, Coloumeter & fuses, this will all be covered in a future post!

Posted on — 5 Comments

BMW Series 5 Hybrid Battery Pack Teardown

Here’s something I didn’t think I’d be doing! Here’s a teardown of a BMW 5 Series G30 530E Hybrid Battery pack – a monster 351V, 9.2kWh Lithium pack, obtained for it’s cells to replace the boat’s aging lead acids.

This is something I didn’t have the safety gear to do right of the bat – opening one of these packs is a potentially lethal exercise, with 6 unfused battery modules in series, quite capable of blowing pieces off a nice conductive sack of salt water like a person. Cue the purchase of high-voltage rated gloves for protection, just while I got the pack split into something more manageable.

Needless to say, the combination of current capacity & voltage present in EV or Hybrid vehicle battery packs is nothing short of lethal, and these units should be treated with considerable respect.

Hybrid Battery Pack
Hybrid Battery Pack

Here’s the beast of a battery. Enclosed in an aluminium cast housing, it’s very heavy, and definitely not a one-man lift!

Cover Removed
Cover Removed

After removing the top cover, secured by combination Torx/10mm hex bolts, the internals of the pack are visible. There’s no sealant on the cover, just a large rubber gasket, so this came off easily. There are 6 individual modules in this pack, all wired in series with massive links. There’s also a cooling system for each battery module, supplied with refrigerant from the car’s AC system – there’s a TXV mounted on the side of the battery pack. I didn’t see any heaters present, but I don’t know if BMW have done any neat reverse-cycle magic to also heat the modules if required using the AC system on the car.

Left Side Modules
Left Side Modules

The modules are arranged 3 to a side, double-stacked at the back, then a single module at the front. The pack would normally sit under the rear seats of the vehicle, hence the unusual shape. The refrigerant lines going to the evaporators on this side of the pack can be seen in the bottom right corner.

Output Cables & Contactor Pack
Output Cables & Contactor Pack

The main contactor pack is on the left side, just behind the massive DC output connector. I’ll dig into this in another post later on.

Right Side Modules
Right Side Modules

The right side of the pack is arranged much the same as the left, the main difference here being the battery ECU is tucked in at the top here, along with the interface connector to the car, and the refrigerant lines to the TXV on the outside, which I’ve already removed. Each module has a cell balance control unit, in this case one is mounted on the top of a module, and on the side of the module in the lower right corner.

Cooling Evaporator
Cooling Evaporator

Once all the modules have been removed, the evaporator matrix is visible on the bottom, a series of very thin aluminium tubes, designed for the best contact with the aluminium frame of the battery modules.

Module Cell Layout
Module Cell Layout

Popping the plastic insulating cover off the battery module reveals the internal construction. I’ve not been able to find exact data on these cells, but I’m assuming them to be a similar chemistry to the ones used in the BMW i3 packs, so 4.15v Max, 3.68v nominal, 2.7v Minimum. The alloy frame itself is of laser welded construction, and there are 16 cells in series per module, giving about 58.8v per module. These will need to be reconfigured as 4 sets of 4 cells in series for 14.72v.
All the individual cell taps are nicely loomed down the middle of the module to each cell, and there are 3 temperature sensors per module (the red epoxy blobs).

Cell Welded Links
Cell Welded Links

The individual cell links are laser welded to the terminals of the cells, so this does make life a little more difficult when it comes to reconfiguring them. The links appear to be made from Aluminium, so soldering is going to be a bit more tricky than usual.

Posted on — 1 Comment

Undercounter Fridge – Conversion for 12 Volt Use

Since I do a lot of camping, and several festivals per year (when there isn’t a pandemic on!), I identified the need for a proper fridge that can be powered from my solar setup. Such fridges & coolers already exist, that run from either mains AC, 12/24v DC, and some of them (absorption cycle) will run from bottled gas.

The last option is out, as they’re hideously inefficient, and this would require the carrying around of a flammable gas source. Ready-made units using the vapour-compression method used in all domestic & industrial refrigeration, but they are very expensive. For an upright fridge type unit that could store enough to feed a family of 4, I was looking at over £550+VAT. A cheaper option was definitely required.

Since I already have a couple of spare Danfoss BD35 DC refrigeration compressors, I decided to grab a cheap domestic mini-fridge, and perform a compressor-ectomy to make the unit operable on a low voltage supply.

Russel Hobbs Mini-Fridge
Russel Hobbs Mini-Fridge

Here’s the fridge I obtained from one of the many suppliers of domestic kit, this is a Russel Hobbs branded mini-fridge.

Factory Compressor
Factory Compressor

I was careful to select a unit with no Aluminium pipework – the stuff is damn near impossible to join onto with soldering. Brazing is impossible due to the temperatures involved. These units have copper & steel in their circuit, so this will be easy. Factory charge is 16g of R600a (Isobutane). This one isn’t even going to make it to the point of being plugged in before modification!

BD35 Fitted
BD35 Fitted

I evacuated the factory charge, and removed the original compressor. To avoid having to disturb the capillary tube, I ensured the system was in continual nitrogen purge to keep moisture out – this meant I could retain the factory filter-drier. The condenser in this fridge is skin-type, on both sides of the outer shell, and formed from steel tube. This connection required the use of silver braze to connect to the compressor.
The suction line from the evaporator is copper, so that’s an easy braze onto an extension to the compressor.

New System Charging
New System Charging

Once the new compressor was brazed into place, a full leak & pressure strength test is performed. I’m using isolation valves on the charging hoses here – they’re quite nifty. Backseat them all the way & the charging hose is isolated from the system. Front seat all the way & the hose valve is opened, and the Schrader valve core is depressed in the service port. They really cut losses when charging systems with Schraders!

Vacuum Stage
Vacuum Stage

Next step is applying a vacuum to the system. I aimed for a final vacuum of 250 microns. This by far takes the longest amount of time in a refrigeration job. For reliability & longevity of the system, it’s imperative that all contaminants such as water vapour & air are removed from the circuit.

Refrigerant Bottle
Refrigerant Bottle

The final step is a refrigerant charge. Since I’m not at all fond of flammable refrigerants in this use case (camping), I broke out the bottle of R-134a. This isn’t ideal, as the capillary tube will be sized for the original charge of R600a, but the effect on efficiency shouldn’t be too terrible. (There will be a drop in COP, but I haven’t yet measured the actual COP of the re-engineered system). Unfortunately, as this uses a plate evaporator with a built in capillary tube, there’s no way to resize this for another gas. The capillary tube is fed down the centre of the suction line in these systems, to increase efficiency of the cycle.

Evaporator
Evaporator

A few minutes after an initial charge of 45g R-134a, there’s frost on the plate evaporator! Since this is a gas change as well as the compressor, there’s no other way than to charge slowly, and wait for the system to stabilize at temperature. Then gas is added until there’s an even frost all over the evaporator surface. I would have measured the charge by suction line superheat, but I have no idea of the original system specifications.

Suction Line Icing
Suction Line Icing

In this case, when running the cabinet down to the minimum temperature possible, a slight overcharge became evident. Releasing a small amount of the refrigerant back into the charging bottle sorted this out.

I may yet make another modification to this unit, to remove the skin-condenser from the circuit. While cheap, and difficult to damage as they’re buried behind the outer case metal, they’re not very efficient. I have some small fan-cooled condenser coils that will probably end up in the back next to the compressor to improve efficiency. This will also take some of the heat load off the cabinet insulation, as there won’t be a coil of hot refrigerant next to it.

Posted on — Leave a comment

DIY 75Ω Matching Pads

Recently I found the need to do some measurements on 75Ω CATV equipment, only having 50Ω test equipment to use. For this, matching networks exist to convert 50Ω to 75Ω, but they’re fairly simple, so building them was a viable option.

Matching Pad Schematic
Matching Pad Schematic

Above is the very simple schematic to create the 75Ω match. To help keep any parasitics down, this circuit will be built directly onto the back of BNC connectors, that are soldered back-to-back, before being covered in shielding tape.

Resistors Soldered
Resistors Soldered

Here’s the first 50Ω BNC connector, with the resistor network soldered on. I’ve used 4x 360Ω resistors in parallel to create the 90Ω to ground, and a single 43Ω series resistor on the centre pin.

End View
End View

This end view of the arrangement shows the 4 resistors evenly spaced around the centre pin & soldered to the shell.

BNCs Soldered
BNCs Soldered

The centre pin of the 75Ω BNC connector is trimmed down to match the length needed to touch the end of the series resistor, and it’s soldered in place. It’s a bit tricky, soldering within the gap between 2 of the ground pins!

Completed Matching Pads
Completed Matching Pads

Finally, the internals are shielded with copper tape, soldered at the seams.

Posted on — Leave a comment

HP 8753C Display LCD Replacement

Completed Install
Completed Install

Since I inherited an old HP 8753C Network Analyser from work, I figured updating a few things to relatively modern standards would be good. The factory CRT, being 28 years old, is definitely getting a little tired, not to mention being slow to warm up. I read over on the EEVBlog forums about a DIY modification to integrate an LCD display into place instead. There was also the option of a ready-made kit for these instruments which would integrate an LCD, but the cost at over £300 was very prohibitive!

CRT Pinout
CRT Pinout

The CRT display unit is a self-contained Sony unit, taking RGBHV signalling from the graphics control card of the analyser. Power is 65v DC which will definitely come in handy for powering the new LCD & control gear, after some conversion.

Test Wiring
Test Wiring

Doing a quick test with some wiring stuck into the video connector from the graphics controller, proved that I could get a decent video signal out of the unit! The only signals used here are RGB, along with the vertical & horizontal sync.

GBS-8200 Converter Board
GBS-8200 Converter Board

The video is converted to VGA by way of a GBS-8200 arcade machine video conversion board, which will take many different video formats & spit out standard VGA signalling. The power supply to the left is a standard 100-240v to 12v PSU, which is happy to run at 6t5v DC input voltage, albeit with a ~5 second delay on output startup when power is applied. This is due to the massive 6.6MΩ resistance of the startup resistor chain, which I did reduce by 50% to 3.3MΩ with no effect. Since it does start OK even with the delay, I think I’ll not tinker with it any further. I doubt I could pull the full rated power from it with such a low input voltage, but all included, this mod draws less than 600mA at 12v.
A custom 20-pin IDC cable was made up to connect to the analyser’s graphics board, and this was then broken out into the required RGB & sync signals. Quite a few of the grounds are unused, I’ve not yet noticed any issues with EMC or instability.

Sync Combiner
Sync Combiner

There is a quad-XOR gate deadbugged to the PCB, which is taking the separate sync signals & combining them into a composite sync. The conversion board does have separate sync inputs, but for some reason doesn’t sync when they’re applied separately. This gate IC is powered from the 3.3v rail of the converter board, with the power lines tacked across one of the decoupling caps for the DRAM IC.

LCD Control PCBs
LCD Control PCBs

The donor 8.4″ LCD came from eBay in the form of a POS auxiliary display. I pulled the panel from the plastic casing, along with the control boards, and attached them all to the back. This LCD also had a sheet of toughened glass attached to the front, no doubt to protect against the Great Unwashed while in use! This was also removed.

Control Boards Mounted
Control Boards Mounted

A cut piece of plexiglas allows the boards to be mounted in the cavernous space the CRT once occupied, with some brass standoffs. 12v power & VGA are routed down to the LCD on the front of the analyser.

LCD Wiring
LCD Wiring

The LCD itself is tacked in place with cyanoacrylate glue to the securing clips for the glass front panel, which is more than enough to hold things in place. The input board which just has the VGA connector & power connector is glued edge-on to the metal back panel of the LCD, and is under little strain so this joint should survive OK.

Posted on — 2 Comments

Covid-19 Lockdown Projects – Extracting Diesel Heater Exhaust Waste Heat

Since I fitted the new “8kW” diesel heater to the camping power trolley, it has occurred to me that there is a lot of energy in the exhaust gas stream that ordinarily would be wasted into the atmosphere. Since we’re all still on lockdown here in the UK, I figured it would be good to run an experiment to see if it was worth recovering this energy – in the form of heating water.

Heat Exchanger
Heat Exchanger

Some time ago, I stripped an old gas combi boiler, and recovered some parts – most important here the HDW plate heat exchanger. This large chunk of stainless steel is a stack of formed plates, brazed together, that usually would heat Domestic Hot Water. In this instance it’s being repurposed to transfer heat from exhaust gas to water.

Brazed Connections
Brazed Connections

These heat exchangers are mounted in the boiler via a plate with O-Ring seals on, so they don’t really have fittings – just holes in the end plate. Solving this problem was simple – braze on some copper fittings with 55% silver brazing rod. The 22mm is the exhaust side, while the 15mm is the water side.

First Test
First Test

Cobbling together some random hose & fittings, along with a small water pump allowed me to run a first test. At this point there is no lagging at all on the exhaust system from the heater, so it’s going to shed a lot of exhaust heat into the air before it even gets to the heat exchanger. However I was able to get around 600W of heat into 15L of water, heating it up nicely. The heat exchanger is plumbed contra-flow here – exhaust comes in via the stainless tube on the bottom right, and water comes in through the speedfit elbow on the top left.

Lagged Heat Exchanger
Lagged Heat Exchanger

After the temperature of the water tank hit a plateau at around 45°C, I decided to insulate everything the best I could with what I currently have. I’ve wrapped the heat exchanger with some recycled PET insulation here, just to hold the heat inside. I’m not concerned about the exhaust outlet being in contact with the fluff – this system is so effective at pulling the heat out of the exhaust that the gas exiting the far end is totally cold!

Unlagged Exhaust
Unlagged Exhaust

Now it was time to get the exhaust system under the trolley insulated. This is the system removed from the unit entirely. This is constructed from copper pipe, brazed onto standard silencers. Deadening the sound from the unit is important, as this gets used on campsites!

Fibreglass Tape Insulation
Fibreglass Tape Insulation

An hour & some itchiness later, the exhaust is completely covered in fibreglass insulation, secured in place with stainless steel ties.

Exhaust Hanger
Exhaust Hanger

The exhaust originally passed through a close-fitting hole in the frame rail which would obviously not work now due to the thickness of the insulation layer, so this was modified with a grinder. Since there was now no support for this end of the exhaust, a pair of drilled holes & some stainless steel wire form a nice hanger!

With all this insulation in place (including around the tank & pump), the rig is now able to easily hit 65°C within a short time, so there has definitely been an improvement. At this point, it’s clear that waste heat recovery is worthwhile, so I’ll be building a proper rig to capture this energy for reuse!

Posted on — Leave a comment

R-134a Refrigerant Analyser Module Exploration

Main Board
Main Board

This unit recently appeared on eBay, as a spare part from a refrigerant charging machine, and I figured it would make a good explorational post. This unit analyses the purity of R-134a refrigerant gas, using an Infrared sensor cell, and communicates over RS-232.

The sensor cell itself is at the top right of the board, we’ll get to that later on. There’s a small diaphragm pump at the lower right, for purging the cell with air. The port with the red cap is the outlet, and the remaining open hose barb is the input of gas to be tested. This would connect to a flow regulator & solenoid valve that the board controls.

It’s pretty clear that this board has multiple applications from all the unpopulated components. There’s space for a keypad, indicator LEDs & an LCD on board, so maybe this can also be fitted to a handheld analyser?

Power Supply
Power Supply

From looking at the input circuitry, I can surmise that the input voltage is somewhere between 12-24v DC, as there is a 35v input electrolytic filter capacitor. There’s a couple of switching regulators which generate 5v & 3.3v rails for the board, with some input fusing.

Main Connector
Main Connector

There’s two serial links on this board, driven from the main microcontroller – the primary one is on the connector marked JMAIN, along with the power input & a couple of other unknown signals.

MSP430 Microcontroller
MSP430 Microcontroller

Over on the other side of the board is the brains of the operation – an MSP430 microcontroller, with an RS232 transceiver IC & another RS-232 port marked COM2. The remaining connector is a JTAG port for the micro.

Gas Analysis Cell
Gas Analysis Cell

Here’s the gas analyser cell itself, sandwiched under another board. There’s a temperature sensor on the side of the cell at the bottom, and even though there’s many pins on the header here, only a couple are actually used for the IR emitter.

IR Emitter
IR Emitter

Removing the screws from the top allows the board to be removed, which exposes the Mid-IR emitter component with an exposed element. This looks to be very fragile, so I won’t be messing with this much. From metering the connections, this appears to be driven at about 2v from the microcontroller.

Top Of Gas Cell
Top Of Gas Cell

The window into the gas cell looks to be made of something exotic – considering the IR application & the colour, this is probably Zinc Selenide.

Gas Test Cell
Gas Test Cell

Removing some more screws on the bottom exposes the bottom of the cell with another IR window, and an O-Ring where a pressure sensor sits.

Output Amplifiers
Output Amplifiers

There’s a couple of very accurate LT1884 Rail-To-Rail Precision Op-Amps next to the cell, most likely used to measure the output from the sensor itself.

Mid-IR Sensor & Pressure Sensor
Mid-IR Sensor & Pressure Sensor

Finally, there is a dual-window thermopile sensor, and a pressure sensor. I wasn’t able to get any information on either of these, but I did find some ranges of sensors for Mid-IR measurement operations, that mentions a wavelength around 10µm for R-134a spectroscopy.

I will try to get this module going & measuring some gases, if I can work out how to talk to it – I already know the serial lines so it’s just working out a command set. If anyone has any information on these, please do get in touch! A service manual for the refrigerant machine this came out of would be good!

Posted on — Leave a comment

Tau-Volantis Storage Server – Upgrades Part 2

Tau-Volantis Server
Tau-Volantis Server

So, it’s time to finish off the upgrades to the core storage server on my network. Now a new motherboard, CPU & RAM have been obtained (MSI GA-X58-USB3), Core i7 950, 12GB), along with new SAS/SATA HBAs for the disk rack I can get everything fitted into place.

Dell H200 SAS RAID Card
Dell H200 SAS RAID Card

Proper branded LSI HBA cards are expensive so I went with the cheaper option & obtained a pair of Dell H200 RAID cards. These have custom firmware flashed to them, but luckily can be crossflashed to a standard LSI firmware to become an LSI9211-8i card – providing 8 lanes of either SAS or SATA connectivity on a pair of SFF-8087 ports. Flashing these cards was very simple, once I managed to work my way into the EFI shell on my main machine, which I was using to do the flashing. Find all the firmware files & required software here:

One thing I left out from the flashing was a BIOS – this means that the boot process is speeded up, but also means the system BIOS cannot see the disks connected to the cards, so they’re not bootable. This isn’t a problem however, as I never plan on booting from the data storage disk array.

Active Cooling
Active Cooling

The SAS2008 RoC (RAID on Chip) on these cards runs at around 8.5W thermal power, so some active cooling is required to keep temperatures within check. I have attached a 40mm fan to each card’s factory heatsink, using M3x25mm screws. Getting the screws to grab the heatsink was the tricky bit – I needed to crimp the outer corners of the fins together slightly, so when the screws are driven in, the gap is forced to expand, which grabs the threads. The fans will be connected to spare headers on the motherboard for speed monitoring.

GA-X58-USB3
GA-X58-USB3

It was a struggle finding a motherboard with the required number of high-lane-count PCIe slots. Even on modern motherboards, there aren’t many about within a reasonable price range that have more than a single x16 slot, and since I’m going with the new HBAs, a single slot is no longer enough. The motherboard I managed to obtain has a pair of x16 slots, and a x4 slot (x16 physical), along with a 3 x1 slots. The only downside is there’s no onboard graphics on this motherboard, so an external card will be required. Another cheapie from eBay sorted this issue out.

ATI Radeon HD3450
ATI Radeon HD3450

Since I need to use the x16 ports for the disk controllers, this card will have to go into the x4 slot.

Motherboard Installed
Motherboard Installed

Here the board has been installed into the new chassis, along with it’s IO shield. Both HBA cards are jacked into the x16 slots, with the SAS/SATA loom cables attached. I did have to grab longer cables – the originals I had were only 500mm, definitely not long enough to reach the ports on these cards, so 1m cables are used. The fans are plugged in with extensions to a pair of the headers on the motherboard, but the MB doesn’t seem to want to read RPM from those fans. Nevermind. While the fans are a little close to the adjacent cards, the heatsinks run just about warm to the touch, so there’s definitely enough airflow – not forgetting the trio of 120mm fans in the bulkhead just out of shot, creating a breeze right through the chassis.

Boot SSD
Boot SSD

Since the onboard SATA ports are in a better position, I was able to attach the boot SSD to the caddy properly, which helps tidy things up a bit. These slot into the 5-¾” bays on the front of the chassis, above the disk cage.

Loom Closeup
Loom Closeup

To take up the excess cable length, and tidy things up, the data loom to the disk cage is cable-tied to self-adhesive saddles on the side of the chassis. This arrangement also helps cooling air flow.

Server Overview
Server Overview

With the new components, and the cabling tied up, things inside the chassis look much cleaner. I’ve rationalised the power cabling to the disk backplanes down to a a pair of SATA power looms.

Posted on — 1 Comment

Tau-Volantis Storage Server – Upgrades Part 1

So I figured it was time to get a hardware update sorted for my network’s core storage server, which I have posted about before. The way I had the drives anchored to steel rails really doesn’t make moving or replacing disks easy, so a proper case needed to be sourced.

4U 16-Bay Hotswap Chassis
4U 16-Bay Hotswap Chassis

ServerCaseUK stocked 16-bay 4U chassis units, so one of these was ordered. These have 4 internal backplanes, with SFF-8087 Mini-SAS connections, so hooking into my existing 16-channel HBA card would be simple. In the current setup, the multi-lane cables are routed out via SFF-8088 connectors to the drive array, so this will tidy things up considerably.

SFF-8087 Connections
SFF-8087 Connections

The main data links are via these SFF-8087 connectors, each carrying 4 lanes of SATA.

Backplane Power
Backplane Power

Power is provided by 4x Molex connections, via SATA power adaptors (the good kind, which don’t create fire). There’s a 5th Molex hidden down the size of the last fan, which powers all 3 120mm fans.

Disk Array Fan Wall
Disk Array Fan Wall

The disks are kept cool by 3x 120mm hot-swap fans on the dividing wall. These don’t create much noise, and are always at full speed.

Motherboard Bay
Motherboard Bay

Here’s the back of the case after transplanting the motherboard & HBA from the old chassis. There’s a new 750W EVGA modular power supply, since I’ll be expanding the disk array as well. The boot SSD is currently sat on the bottom of the case since I don’t have a data cable long enough to mount it in the proper place as yet.

Fan Controller
Fan Controller

Here’s the fan controller, which takes care of the dual high speed Delta fans on the back wall of the chassis. This has a pair of temperature sensors – one on the HBA card’s heatsink, and the other on the fan wall monitoring the exhaust air temp of the drive array, to control the speed of the two fans. Temperatures are kept at around 30°C at all times.

16-Port SATA HBA
16-Port SATA HBA

Since the HBA card’s fan failed a while back, it’s had a couple of fans attached. The centrifugal one here works a little better than a massive 80mm axial fan, and is a little quieter. This is always run at full speed from a spare motherboard header. The temperature sensor feeding the fan controller can be seen here bonded to the heatsink. The 4 SFF-8087 cables are going off to the disk backplanes.

Cooling
Cooling

As mentioned before, there are a pair of 80mm Delta high-speed fans on the back wall of the case, to provide some extra cooling air flow just in case overheating manages to set in. These are usually spooled down to low RPM to keep them quiet.

Since space was getting a little tight, and having some slots spare on the HBA, I decided to add some more disks to bring the active members up to 12, from 9 – increasing available disk space from 28TB to 40TB.

/dev/md0:
        Version : 1.2
  Creation Time : Wed Mar 11 16:01:01 2015
     Raid Level : raid6
     Array Size : 39068874880 (37258.98 GiB 40006.53 GB)
  Used Dev Size : 3906887488 (3725.90 GiB 4000.65 GB)
   Raid Devices : 12
  Total Devices : 13
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon Nov 18 14:13:35 2019
          State : active 
 Active Devices : 12
Working Devices : 13
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

           Name : Main-PC:0
           UUID : 266632b8:2a8a3dd3:33ce0366:0b35fad9
         Events : 1653174

    Number   Major   Minor   RaidDevice State
      16       8      144        0      active sync   /dev/sdj
      11       8      160        1      active sync   /dev/sdk
       9       8      176        2      active sync   /dev/sdl
      10       8      208        3      active sync   /dev/sdn
      15       8      192        4      active sync   /dev/sdm
       5       8      112        5      active sync   /dev/sdh
       6       8       80        6      active sync   /dev/sdf
      13       8       96        7      active sync   /dev/sdg
       8       8       64        8      active sync   /dev/sde
      19       8        0        9      active sync   /dev/sda
      18       8       48       10      active sync   /dev/sdd
      17       8      128       11      active sync   /dev/sdi

      12       8       16        -      spare   /dev/sdb

The space expansion improves things there, I will be adding a couple more spare disks to bring the number up to the full 16, just in case of any failures.

There are still a couple of issues with this setup:

  • The motherboard & CPU are ancient. Currently an Intel Core 2 Quad, running 8GB of RAM, limits data throughput, and critically, the speed of mdadm data checks & rebuilds. The Core 2 Quad also runs at roughly the same temperature as the Sun’s core when under high load.
  • The SATA HBA is running 4 controllers on an expander, through a PCIe x4 link, which is a little slow due to congestion on the expander itself. RAID6 does have some write-speed penalties though.

These are issues I will address shortly, with a replacement motherboard on the way!

Posted on — Leave a comment

Festival Power Trolley – Integrating a Fridge – The Condensing Unit

While we’re away camping, be it at a festival or general site, it would be nice to have a relatively efficient fridge to store food & drink in, so I set to work assessing the requirements. Commercially supplied 12/24v fridges are hella expensive – running between £450-£700 for even a small compressor coolbox! So in my general style, I figured it was time to get building my own, integrated into the current system. With the bonus of learning some things along the way. There are some refrigerants that aren’t covered under the F-Gas regulations, such as R600a (Isobutane) & R290 (Propane), so I can easily avoid those legal requirements, not to mention being a little more environmentally conscious. The common Danfoss BD35F compressors are designed for R-134a, but there are equivalent replacements available for the automotive AC market that operate within the same pressure ranges. As far as I can tell, these replacements are mixtures of R600a & R290 in various proportions.

I decided to keep most of the weight, control & wiring internal to the main power trolley which goes everywhere with us while camping, this obviously required the refrigerant lineset to be dis-connectable from the fridge cabinet for transport, and has the requirement for a flexible lineset between the two. The marine-grade refrigeration systems destined for custom builds are supplied with dis-connectable copper linesets with special fittings, so I went in search for a supplier of said fittings. The need for flexibility & re-usability means I can’t use the copper lines in this application – work hardening will quickly cause the copper pipe to crack.

Parker-Hannifin supply these special fittings, and seem to be the original manufacturer – considering they specialise in fluid power & control, this isn’t a surprise. However ordering from them is a nightmare, so no go there. I did find an Italian supplier of equivalent fittings, the Faster Coupling RF series:

Lineset Disconnect Fittings
Lineset Disconnect Fittings

Luckily, I was able to get a distributor in the UK to order these in for me, but they are a bit expensive, at ~£130 for two pairs. I sized these for 1/4″ to match the compressor lines. They’re quite well made from brass, with copper tails silver-soldered for braze connections.

Fitting Pair
Fitting Pair

The fittings screw together like hydraulic couplings, however these are designed for very low leakage, and very low admission of air when connecting them together.

Fitted To Frame
Fitted To Frame

Time to get these panel mounted on the back of the power trolley. The 1/2″ plywood frame is too thick to mount these directly, so I had to improvise a mounting plate:

Fitting Mounting Plate
Fitting Mounting Plate

A bit of FR4 PCB material with suitable holes bored makes for a nice thin panel with high strength. I will colour-code these to make sure the suction & discharge lines aren’t mixed up when connecting, considering they’re both the same size.

Refrigerant Fittings
Refrigerant Fittings

The fittings are secured with M20 conduit nuts & spring washers.

Compressor Unit
Compressor Unit

Here I’ve got the compressor fitted & all the lines brazed, and have the system under vacuum to get any traces of water out. Side note: the RF series couplings are *very* difficult to braze with just a single burner MAPP torch – keeping the end of the fitting with the rubber seals submerged in water to stop them melting just conducts too much heat away from the copper tail too quickly. I did successfully manage to get them brazed though, and this section of the system holds both vacuum & pressure to 12 bar. The condenser coil is mounted underneath next to the fuel tank:

Condensing Coil
Condensing Coil

There’s a 120mm fan behind for forced air cooling, with the refrigerant lines run up to the compressor above.

I’ll be back soon with the build of the fridge cabinet & evaporator section!

Posted on — 17 Comments

Afterburner Aftermarket Diesel Heater Controller Build

While I’m pretty happy with the Chinese diesel heater replacement for the old Eberspacher on the trolley, the stock controller leaves much to be desired. While functional, it’s fairly unresponsive, bulky & doesn’t allow external control as the Eberspacher did with it’s simple ON/OFF signal. So after a massive amount of searching on the web, (it wasn’t easy to find, damn SEO!), I discovered a project by a chap in Australia who has reverse engineered the communications protocol of these heaters, & built a fully custom controller. This is based around the ESP32 Wi-Fi microcontroller, and a Bluetooth HC-05 module. Display is by means of a 1.3″ OLED screen.

Bare PCB
Bare PCB

Here’s the bare PCB kindly sent to me by Ray down under to get this project kickstarted. No THT components here apart from headers, everything is minimum 0805 size.

Populated PCB
Populated PCB

Here’s the controller PCB fully populated, with the ESP32 & Bluetooth HC-05 modules on board. There was a slight issue with the 3.3v regulator not matching the pinout of the PCB, so some minor bodge work required there, but the current draw of the unit is so low that the regulator doesn’t get warm, even with the heatsink tab floating in mid-air. I’ll hot-snot this down to avoid any vibration issues. Incedentally, soldering the castellated connections of the modules is a real pain – problems with contacts not wetting first time were an issue here. Use plenty of flux!

The two JST connectors are for the main heater loom, and an external temperature probe that feeds back for the thermostat mode of the controller. Reset & Bootloader buttons are provided on the board for easy firmware loading. There’s also some spare GPIO broken out for other uses, along with the required UART port for firmware & debugging access. The clock is maintained by a DS3231 RTC IC, which oddly enough is expensive on it’s own – buying an Arduino RTC module & using a hot-air pencil to desolder the IC worked out £4 cheaper than buying the IC direct! The RTC is backed up by a lithium coin cell.

The communications interface is taken care of by a couple of single gates in SOT-23 packages, which interfaces the 3v3 ESP32 with the 5v signalling levels of the heater’s control bus. This unit is a little odd for a communications interface – it is standard serial, but at an odd baud rate of 25,000, and instead of separate TX & RX lines, the transmissions are gated for TX & RX down a single wire. I fail to understand the logic of doing this, since wire isn’t expensive & extra components just add complexity!

A couple of build notes on this controller:

  • R4 & R5 are swapped on the silkscreen. Bootloader issues ensued!
  • The blue PCB Bluetooth module doesn’t function correctly with the controller, allowing receive but no transmit. Odd.
  • There are two pinout variations on the 1.3″ OLEDs. This layout requires the GND pin leftmost.
PCB Front
PCB Front

The front of the PCB holds the OLED display panel, and the control button array. Not much else on this side of the board besides the RTC battery switching diodes & some passives.

Glow Plug Heating
Glow Plug Heating

The case to house the PCB is 3D printed, I didn’t bother with the matching buttons as I don’t currently have any flexible filament to print with, so long stem tact switches are used. The controller is running the heater through a cycle here on the Detailed Info screen, where the most comprehensive heater info is displayed. In this stage, the heater’s ECU is warming the glow plug up ready for ignition.

Igniting
Igniting

After the glow plug has heated, the ECU starts the fuel metering pump at it’s lowest rate to get a flame going. The glow plug is still active to vaporise the fuel.

Burner Running - Heatup
Burner Running – Heatup

Once the thermistor on the heat exchanger registers a temperature increase, the ECU detects the burner has lit, and starts increasing the fuelling rate & blower speed.

Warming Up
Warming Up

Once the temperature hits a threshold, around 55°C, the ECU switches off the glow plug & ramps the fuelling rate up to max to warm the heat exchanger to operating temperature. The detailed page displays both the room temperature (left), set temperature & heat exchanger temperature (right).

Heater Running
Heater Running

Once the heat exchanger has reached running temperature, the ECU switches control to the thermostat, in this case the new controller.

Shutdown
Shutdown

When a shutdown is initiated, the heater brings the glow plug back on & reduces the fuelling rate to minimum for a couple of minutes, before stopping fuel flow. The glow plug remains running for a while to burn off any remaining fuel residue.

Cooling Cycle
Cooling Cycle

Finally, the ECU cuts power to the glow plug, and keeps the fan running at medium speed until the heat exchanger cools to below 55°C.

Posted on — Leave a comment

Festival Power Trolley – Heater Upgrades

Eberspacher D1LCC
Eberspacher D1LCC

As part of the giant power bank that gets dragged to all my major camping trips & festivals, there is an old Eberspacher Diesel heater, a D1LCC from at first guess somewhere in the mid 90’s. At only 1.8kW heat output this is a little small for our current tent, and it struggles to keep the temperature comfortable at night, so with Chinese clones on the market these days much cheaper than the Eberspacher or Webasto units, a replacement was up! Still, the old Eberspacher is in working order, and will probably get used for some other project.

Diesel Heaters
Diesel Heaters

After removing the old D1LCC & placing it next to the new one, the size difference is obvious! The new heater is a Chinese clone of the Eberspacher D4 unit, allegedly uprated to 8kW. (In reality, it’s probably around 5kW heat output at full tilt). Luckily, it’s not that much larger than the old one, so it’ll go into the same space.

New Heater
New Heater

The port layout on the bottom of the heater is identical apart from intake port size, a quick attack of the baseplate with a grinder to remove the old hole pattern allowed the supplied mounting plate to fit correctly into place for the new heater. The duct size on this unit is also bigger than the old 60mm – 75mm duct is used on these large units. No modification to the vent hole was required, as the 75mm vent already fit perfectly. To clear the fittings on the top of the fuel tank, which is just underneath the hot air exhaust cowling of the heater, the mounting plate is fixed using 10mm nylon standoffs, this also helps get a bit more natural airflow around the base of the heater, as the mounting gets to 90°C in operation at full power!

These heaters don’t use the Eberspacher standard switch wire for control – there are only 3 pins in the loom to the controller, for 5v power & an odd UART which uses gated TX/RX to avoid having a separate line for each.

Stock Controller
Stock Controller

The stock controller has quite a nice looking LCD display, but it’s less than responsive & the backlight is always on at full tilt. It’s also much larger than the Eberspacher 701 controller so would require some rejigging of the control panel on the trolley. The built-in thermostat is also inaccurate, being almost 5°C high no matter what the room temperature. Ray Jones from Down Under has designed an open source ESP32 based controller for these heaters, and one of these is currently being built to control the unit. More to come on this bit!

Stock Controller PCB
Stock Controller PCB

A quick teardown of the controller reveals pretty simple internals, there’s a microcontroller, probably an STM8 device by looking at the programming header, but the markings have been scrubbed off the IC. There’s a standard LCD controller IC, a RTC which isn’t battery backed, and a 433MHz receiver IC with PCB trace antenna.
I wasn’t able to get the remote control function working with any of the remotes I have, any attempt at pairing a remote didn’t give any response from the controller unit. I also tried a 315MHz remote, but that didn’t work either. Not an issue since I’m building a much better open source controller.

Fuel & Exhaust
Fuel & Exhaust

Under the base is the exhaust system & the fuel dosing pump. There’s a small filter in the feed line from the tank to keep crap out of the pump, and nylon fuel line then runs the fuel to the heater inlet. The exhaust is made as gas-tight as possible with foil tape & exhaust paste, to keep the exhaust fumes contained in the pipework until they’re vented outside. The rest of the exhaust after the right hand silencer is done in brazed 22mm copper pipe, and a piece of Eberspacher exhaust duct is removable from the final exhaust tail for storage. The black pipe is the combustion air intake, which is simply fed into a silencer cable tied to the trolley frame.

Heater Ports
Heater Ports

The 3 ports are visible under the mounting plate, the square hole cut out of the trolley base to accommodate everything.

 

Posted on — 3 Comments

ECL82 Amplifier Build

Marking Out
Marking Out

I figured it was about time I built another valve amplifier, and since I already had most of the required parts in stock, here it is! Above is the lid of a cake tin sourced from a local shop as a case, marked out & drilled for the valve sockets, output transformers & speaker terminals.

The ECL82 valve is a Triode & Audio Output Pentode in a single envelope, requiring only a single valve per audio channel. There are a pair of extra holes drilled here for a couple of EM80 magic-eye valves wired as VU meters to give a bit of a lightshow.

Amplifier Schematic
Amplifier Schematic

Here’s the base schematic for the Class-A ECL82 amplifier sections, obtained from the interweb. It’s pretty basic, and doesn’t mention a value for the volume potentiometer, so I used a 100K audio taper for that. Power will be supplied from low-voltage DC, running through a high voltage DC-DC converter for the anode supply of 200v, and a 5A buck converter for the 6.3v filament supply.

EM80 Schematic
EM80 Schematic

The EM80 side is as the schematic above, the signal input being taken directly from the Pentode anode of the ECL82. I have removed the second 1N4148 diode down to ground, leaving only a single diode.

Valve Base
Valve Base

Most of the parts comprising the ECL82 amplifier stages are mounted directly on the back of the valve sockets, requiring only a 6.3v filament supply, 200v anode supply & audio I/O connections. Axial electrolytics have been used for ease of assembly, even though they’re getting a little expensive nowadays!

Point To Point Wiring
Point To Point Wiring

After fitting the components to the top lid, point-to-point wiring is used to connect up the valve socket assemblies. Some large electrolytics provide B+ smoothing, and all the filaments are daisy-chained in parallel. Audio is brought in on micro-coax from the I/O, and straight out to the output transformers on twisted pairs, keeping the audio wiring away from the B+ voltage.

Output Transformers
Output Transformers

The audio transformers, from a 1960’s Philips Radiogram, are mounted behind the valves, with the wiring emerging through holes in the case. I’ve already done the paint job here, in metallic copper.

I/O
I/O

Audio & power sockets are on the back of the tin, with both 3.5mm Stereo inputs & phono inputs. A DC barrel jack takes care of the power, accepting 12-24v.

Controls
Controls

Controls on the front provide volume, balance, bass & treble adjustments.

Amplifier Operational
Amplifier Operational

Here’s the amplifier with it’s valves glowing nicely. Total power consumption is roughly 30W, using NOS Svetlana ECL82s & EM80s. In operation there is no hum or noise in the background, with no audio input the connected speakers are entirely silent.

Posted on — 2 Comments

Goodmans Quadro 902 Composite Video Mod

CRT Module
CRT Module

Here’s the CRT & it’s drive board removed from the main chassis. Nicely modular this unit, all the individual modules (radio, tape, TV), are separate. This is effectively a TV itself, all the tuner & IF section are onboard, unlike in other vintage units I’ve modified, where the tuner & IF has been on a separate board. There’s a 3-pin header bottom centre for the tuning potentiometer, and external antenna input jack. The internal coax for the built in antenna has been desoldered from the board here. here a the usual controls on the back for adjusting brightness, contrast & V Hold, all the other adjustments are trimmers on the PCB.
Unfortunately after 30+ years of storage, this didn’t work on first power up, neither of the oscillators for vertical or horizontal deflection would lock onto the incoming signal, but a couple of hours running seemed to improve things greatly. The numerous electrolytic capacitors in this unit were probably in need of some reforming after all this time, although out of all of them, only 21 are anything to do with the CRT itself.

Anode Cap
Anode Cap

Here’s the anode side of the unit, with the small flyback transformer. The rubber anode cap has become very hard with age, so I’ll replace this with a decent silicone one from another dead TV. The Horizontal Output Transistor (a 2SC2233 NPN type) & linearity coil are visible at the bottom right corner of the board. Unfortunately, the disgusting yellow glue has been used to secure some of the wiring & large electrolytics, this stuff tends to turn brown with age & become conductive, so it has to be removed. Doing this is a bit of a pain though. It’s still a little bit flexible in places, and rock hard in others. Soaking in acetone softens it up a little & makes it easier to detach from the components.

Neck PCB
Neck PCB

There’s little on the neck board apart from a few resistors, forming the limiting components for the video signal, and the focus divider of 1MΩ & 470KΩ feeding G3. No adjustable focus on this unit. There’s also a spark gap between the cathode line & ground, to limit the filament to cathode voltage. The flyback transformer is nestled into the heatsink used by the horizontal output transistor & a voltage regulator transistor.

Tube Details
Tube Details

The CRT is a Samsung Electron Devices 4ADC4, with a really wide deflection angle. It’s a fair bit shorter than the Chinese CRT I have which is just a little larger, with a neck tube very thin indeed for the overall tube size.
Unusually, while the filament voltage is derived from the flyback transformer as usual, it’s rectified into DC in this unit, passing through a 1Ω resistor before the filament connection. I measured 5.3v here. The glow from the filament is barely visible even in the dark.

Electron Gun 1
Electron Gun 1

The electron gun is the usual for a monochrome tube, with 7 pins on the seal end.

Electron Gun 2
Electron Gun 2

The electrodes here from left are Final Anode, G3 (Focus Grid), Accelerating Anode, G2 (Screen Grid), G1 (Control Grid). The cathode & filament are hidden inside G1. In operation there’s about 250v on G2, and about 80v on G3.

Chipset
Chipset

The chipset used here is all NEC, starting with a µPC1366C Video IF Processor, which receives the IF signal from the tuner module to the left. This IC outputs the standard composite signal, and a modulated sound signal.
This then splits off to a µPC1382C Sound IF Processor & Attenuator IC, which feeds the resulting sound through the two pin header at the right bottom edge of the board to the audio amplifier in the chassis.
The composite video signal is fed through a discrete video amplifier with a single 2SC2229 transistor before going to the CRT cathode.
The remaining IC is a µPC1379C Sync Signal Processor, containing the sync separator, this is generating the required waveforms to drive the CRT deflection systems from another tap off the composite video line.
From this chip I can assume the unit was built around 1986, since this is the only date code on any of the semiconductors. Besides these 3 ICs, the rest of the circuit is all discrete components, which are well-crammed into the small board space.
There are 5 trimmer potentiometers on the board here, I’ve managed to work out the functions of nearly all of them:

  • SVR1: IF Gain Adjust
  • SVR2: H. Hold
  • SVR3: V. Size
  • SVR4: B+ Voltage Adjust
  • SVR5: Tuner Frequency Alignment? It’s in series with the tuning potentiometer in the chassis.
PCB Bottom
PCB Bottom

The PCB bottom shows the curved track layout typical of a hand taped out board. The soldermask is starting to flake off in places due to age, and there a couple of bodge wires completing a few ground traces. Respinning a board in those days was an expensive deal! Surprisingly, after all this time I’ve found no significant drift in the fixed resistors, but the carbon track potentiometers are drifiting significantly – 10KΩ pots are measuring as low as 8KΩ out of circuit. These will have to be replaced with modern versions, since there are a couple in timing-sensitive places, like the vertical & horizontal oscillator circuits.

Anode Cap Replaced
Anode Cap Replaced

Here the anode cap has been replaced with a better silicone one from another TV. This should help keep the 6kV on the CRT from making an escape. This was an easy fix – pulling the contact fork out of the cap with it’s HT lead, desoldering the fork & refitting with the new cap in place.

Here I’ve replaced the important trimmers with new ones. Should help stabilize things a little.

Composite Injection Mod
Composite Injection Mod

Injecting a video signal is as easy as the other units. Pin 3 of the µPC1366C Video IF Processor is it’s output, so the track to Pin 3 is cut and a coax is soldered into place to feed in an external signal.

CRT In Operation
CRT In Operation

After hooking up a Raspberry Pi, we have display! Not bad after having stood idle for 30+ years.

Datasheets for the important ICs are available below:



Posted on — Leave a comment

Camping Support Trolley – Finishing Touches

Now the trolley is pretty much built, some burn-in testing is currently underway before it’s first trial-by-fire at Download Festival. There are a couple of minor issues that have needed fixing, since everything is enclosed in a pretty tight box.

Lagging
Lagging

The Eberspacher was the first point that needed sorting – the radiant heat from the hot air ducting was pretty much cooking everything around it – the inverter’s heatsink got to temperatures in the 50°C range. I pullsed some ceramic fibre lagging from a decomissioned domestic oven, and entirely lagged the hot end of the heater, securing everything with cable ties. Good stuff this, the inside of the box now hardly even warms up, nearly all of the heat going out through the vent where it’s needed.

The intake was also an issue, since these heaters adjust their power levels based on inlet air temperature, pulling the air from inside the box was infeasible, since within a few minutes the heater thinks the ambient temperature has reached 30°C, resulting in a shutdown. Another vent has been fitted in the back panel, drawing cool air into the heater’s fan.

 

Finally, meet FrankenCompressor:

FrankenCompressor
FrankenCompressor

I didn’t have the time to order a proper 12v compressor in time for this year, so after some rummaging around in the parts bin & came up with a couple of 12v car tyre inflator type compressor units. These are pretty crap for proper air compressor use, but they should survive just long enough to get me through this year. They’re mounted to a wooden board, with a large 120mm high-speed server fan mounted on the end, blowing a huge amount of air over the cylinders to help cool them down. Output is channelled through a couple of modified 6mm push-fit pneumatic fittings epoxied onto the factory hose barbs. Copper tubing helps cool the compressed air before the transition to poly tube.

Posted on — Leave a comment

Camping Support Trolley – Some Build Progress

Since I do festivals every year, along with a couple of other camping trips if the weather is good enough, I’ve been taking equipment with me for years in flight cases to make things more comfortable. Things like a large battery to power lights & device charging, an old Eberspacher diesel heater for the times when the weather isn’t great, and an inverter to run the pumps built into airbeds.

Red Diesel / Heating Oil is my fuel of choice for camping purposes, as it’s about the safest fuel around, unlike Butane/LPG it is not explosive, will not burn very readily unless it’s atomized properly & it’s very cheap. Paraffin is an alternative fuel, but it’s expensive in the UK, at about £12 per 5L.
The Hexamine-based tablet fuels the UK festivals promote is nasty stuff, and the resulting combustion products are nastier still. (Things like Hydrogen Cyanide, Formaldehyde, Ammonia, NOX). They also leave a sticky black grok on every cooking pot that’s damn near impossible to remove. Meths / Trangia stoves are perfectly usable, but the flame is totally invisible, and the flammability of alcohol has always made me nervous when you’ve got a small pot of the stuff boiling while it’s in operation in the middle of a campsite filled with sloshed festival goers. A single well-placed kick could start a massive fire.

Previous System
Previous System

Over the years the gear has evolved and grown in size, so I decided building everything into one unit on wheels would be the best way forward. I’ve been working on this for some time, so it’s time to get some of the details on the blog! Above you can see the system used for last year’s camping, the heater is separate, with a 25L drum of heating oil, the battery is underneath the flight case containing all the power components, and it’s currently charging All The Things.

Overview
Overview

Above is the new unit almost finished, the bottom frame is a standard eBay-grade 4-wheel trolley with a few modifications of my own, with a new top box built from 12mm hardwood marine plywood. This top is secured in place with coach bolts through the 25mm angle iron of the trolley base. The essential carbon monoxide detector is fitted at the corner.

Internal View
Internal View

The inside gets a bit busy with everything crammed in. The large Yuasa 200Ah lead-acid battery is at the far end, with it’s isolation switch. Right in the middle is the Eberspacher heater with it’s hot air ducting. I’ve fitted my usual 12/24v dual voltage system here, with the 24v rail generated from a large 1200W DC-DC converter.

Heater Vent
Heater Vent

The hot air duct for the heater is fed out through a standard vent in the front. Very handy for drying out after a wet day.

Main Bus Bars & Solar Controller
Main Bus Bars & Solar Controller

Here’s a closeup of the distribution bus bars, with both negative rails tied together in the centre to keep the positives as far away from each other as possible, to reduce the possibility of a short circuit between the two when working on the wiring. The EpEver Tracer 4210A MPPT Solar Charge Controller is on the left, tucked into the corner. This controller implements the main circuit protection for the battery, having a 40A limit. Individual circuits are separately fused where required. Solar input on this unit is going to be initially provided by a pair of 100W flexible panels in series for a 48v solar bus, the flexible panels are essential here as most of the festivals I attend do not allow glass of any kind onsite, not to mention the weight of rigid panels is a pain.

DC Output Sockets
DC Output Sockets

I’ve stuck with the 3-pin XLR plugs for power in this design, giving both the 12v rail, 24v rail & ground.

Inverter Outputs
Inverter Outputs

Tucked under the DC outputs are a pair of panel sockets for the 600W inverter. This cheapo Maplin unit is only usually used to pump up air beds, so I’m not expecting anyone to pull anything near max output, but a warning label always helps.

Power Socket Wiring
Power Socket Wiring

Behind the front panel is the hardwiring for the power sockets. The DC jacks are connected together using 2mm solid copper wire, bent into bus bars.The mains wiring underneath is a simple radial circuit straight from the inverter. The large cylinder on the left is a hydraulic pump from a BMW Z3, which runs a hydraulic cylinder for lifting the lid of the top box, used simply because I had one in the box of junk.

Fuel Pump
Fuel Pump

External fuelling is dealt with by a small gear pump, this is used to fuel up the Optimus Stove & Petromax Lantern. This is in fact a car windscreen wash pump, but it has coped well with pumping hydrocarbons, it currently has a small leak on the hose connections, but the seals are still entirely intact.

Remote Relays
Remote Relays

There’s a small remote relay module here, for switching the DC output for lighting & the heater from afar. Very useful when it’s dark, since there’s no need to fumble around looking for a light switch. A car-style fob on my keyring instead.

Heater Timer
Heater Timer

Since the Eberspacher 701 controller I have is an ex-BT version, it’s very limited in it’s on time, a separate timeswitch is fitted to control the heater automatically. Being able to return to a nice warm tent is always a bonus.
Just to the left can be seen the top ball joint for the hydraulic cylinder that lifts the top of the box.

Battery Charger
Battery Charger

The final large component is the battery charger. This unit will maintain the battery when the trolley isn’t being used.

Router Motherboard
Router Motherboard

On the left side is the old Atom motherbaord used to provide a 4G router system. This unit gets it’s internet feed from a UMTS dongle & provides a local WiFi network for high speed connectivity. The bottom of the hydraulic cylinder is visible in the bottom right corner.

Fuel Tank Completed
Fuel Tank Completed

Since the Eberspacher obviously needs fuel, a tank was required. In previous years I’ve used jerry cans for this purpose, but this trolley is supposed to have everything onboard, for less setup time. The tank is constructed from 3mm steel plate, MIG welded together at the seams to create a ~40L capacity. The filler neck is an eBay purchase in Stainless Steel. No photos of the tank being welded together, as I was aiming to beat sunset & it’s very difficult to operate a camera with welding gauntlets on 😉

The tank is the same width as the trolley frame, so some modification was required, having the wheels welded directly to the sides of the tank. This makes the track wider at the rear, increasing stability.

Fuel Dip Tubes
Fuel Dip Tubes

A quick view inside the tank through the level sender port shows the copper dip tubes for fuel supply to the heater, and an external fuel hose for my other fuel-powered camping gear. These tubes stop about 10mm from the bottom of the tank to stop any moisture or dirt from being drawn into the pumps.

Fuel Feeds & Level Sender Port
Fuel Feeds & Level Sender Port

The top of the tank is drilled for the fuel fittings & the level sender and has already been painted here. The 1mm base plate has yet to be painted.

Level Sender Installed
Level Sender Installed

Touching up the paint & fitting the sender is the last job for this part. The mesh bottom of the trolley has been replaced by a 1mm steel sheet to support the other parts, mainly the heater. Fuel lines are run in polyurethane tubing to the fuel pumps.

All the instruments & controls are on a single panel, with the Eberspacher thermostat, external fuelling port & pump switch, inverter control, the solar controller monitor panel, cover buttons, router controls, compressed air & fuel gauges.

Panel Wiring
Panel Wiring

As is usual behind instrument panels, there’s a rat’s nest of wiring. There’s still the pressure gauge to connect up for the compressed air system, and the nut on one of the router buttons is such a tight fit I’ve not managed to get it into place yet.

Eberspacher Fuel Pump
Eberspacher Fuel Pump

The support components for the Eberspacher heater are mounted underneath the baseplate, with the fuel dosing pump secured to a rail with a pair of cable ties, and some foam tape around to isolate the constant clicking noise these pumps create in operation. The large black cylinder is the combustion air intake silencer, with the stainless steel exhaust pipe to the left of that. Silencing these heaters is essential – they sound like a jet engine without anything to deaden the noise. Most of this is generated from the side-channel blower used in the burner.

Eberspacher Exhaust
Eberspacher Exhaust

Bolted to the underside are a pair of exhaust silencers, one is an Eberspacher brand, the other is Webasto, since the latter type are better at reducing the exhaust noise. Connections are sealed with commerical exhaust assembly paste, the usual clamps supplied do not do a good enough job of stopping exhaust leaks.

Next update to come when I get the parts in for the air compressor system.

Posted on — 12 Comments

Sterling ProCharge Ultra PCU1210 Teardown & Repair

The Sterling charger we’ve had on board nb Tanya Louise since Feb 2014 has bitten the dust, with 31220 hours on it’s internal clock. Since we’re a liveaboard boat, this charger has had a lot of use while we’re on the mooring during winter, when the solar bank isn’t outputting it’s full rate. First, a bit of a teardown to explore the unit, then onto the repair:


Active PFC Section
Active PFC Section

There’s the usual mains input filtering on the left, with the bridge rectifier on it’s heatsink.
Underneath the centre massive heatsinks is the main transformer (not visible here) & active PFC circuit. The device peeking out from underneath is the huge inductor needed for PFC. It’s associated switching MOSFET is to the right.

Logic PSU Section
Logic PSU Section

On the other side of the PFC section is the main DC rail filter electrolytic, a 450v 150µF part. Here some evidence of long-term heating can be seen in the adhesive around the base, it’s nearly completely turned black! It’s not a decent brand either, a Chinese CapXon.
The PCB fuse just behind it is in the DC feed to the main switching supply, so the input fuse only protects the filter & Active PFC circuitry. Luckily this fuse didn’t blow during the failure, telling me the fault was earlier in the power chain.
The logic circuits are powered by an independent switching supply in the centre, providing a +5v rail to the microcontroller. The fan header & control components are not populated in this 10A model, but I may end up retrofitting a fan anyway as this unit has always run a little too warm. The entire board is heavily conformal coated on both sides, to help with water resistance associated with being in a marine environment. This has worked well, as there isn’t a single trace of moisture anywhere, only dust from years of use.
There is some thermal protection for the main SMPS switching MOSFETS with the Klixon thermal fuse clipped to the heatsink.

DC Output Section
DC Output Section

The DC output rectifiers are on the large heatsink in the centre, with a small bodge board fitted. Due to the heavy conformal coating on the board I can’t get the ID from this small 8-pin IC, but from the fact that the output rectifiers are in fact IRF1010E MOSFETS, rated at 84A a piece, this is an synchronous rectifier controller.
Oddly, the output filter electrolytics are a mix of Nichicon (nice), and CapXon (shite). A bit of penny pinching here, which if a little naff since these chargers are anything but cheap. (£244.80 at the time of writing).
Hiding just behind the electrolytics is a large choke, and a reverse-polarity protection diode, which is wired crowbar-style. Reversing the polarity here will blow the 15A DC bus fuse instantly, and may destroy this diode if it doesn’t blow quick enough.

DC Outputs
DC Outputs

Right on the output end are a pair of large Ixys DSSK38 TO220 Dual 20A dual Schottky diodes, isolating the two outputs from each other, a nice margin on these for a 10A charger, since the diodes are paralleled each channel is capable of 40A. This prevents one bank discharging into another & allows the charger logic to monitor the voltages individually. The only issue here is the 400mV drop of these diodes introduce a little bit of inefficiency. To increase current capacity of the PCB, the aluminium heatsink is being used as the main positive busbar. From the sizing of the power components here, I would think that the same PCB & component load is used for all the chargers up to 40A, since both the PFC inductor & main power transformer are massive for a 10A output. There are unpopulated output components on this low-end model, to reduce the cost since they aren’t needed.

Front Panel Control Connections
Front Panel Control Connections

A trio of headers connect all the control & sense signals to the front panel PCB, which contains all the control logic. This unit is sensing all output voltages, output current & PSU rail voltages.

Front Panel LEDs
Front Panel LEDs

The front panel is stuffed with LEDs & 7-segment displays to show the current mode, charging voltage & current. There’s 2 tactile switches for adjustments.

Front Panel Reverse
Front Panel Reverse

The reverse of the board has the main microcontroller – again identifying this is impossible due to the heavy conformal coat. The LEDs are being driven through a 74HC245D CMOS Octal Bus Transceiver.


Now on to the repair! I’m not particularly impressed with only getting 4 years from this unit, they are very expensive as already mentioned, so I would expect a longer lifespan. The input fuse had blown in this case, leaving me with a totally dead charger. A quick multimeter test on the input stage of the unit showed a dead short – the main AC input bridge rectifier has gone short circuit.

Bridge Rectifier Removed
Bridge Rectifier Removed

Here the defective bridge has been desoldered from the board. It’s a KBU1008 10A 800v part. Once this was removed I confirmed there was no longer an input short, on either the AC side or the DC output side to the PFC circuit.

Testing The Rectifier
Testing The Rectifier

Time to stick the desoldered bridge on the milliohm meter & see how badly it has failed.

Yep, Definitely Shorted
Yep, Definitely Shorted

I’d say 31mΩ would qualify as a short. It’s no wonder the 4A input fuse blew instantly. There is no sign of excessive heat around the rectifier, so I’m not sure why this would have failed, it’s certainly over-rated for the 10A charger.

Testing Without Rectifier
Testing Without Rectifier

Now the defective diode bridge has been removed from the circuit, it’s time to apply some controlled power to see if anything else has failed. For this I used a module from one of my previous teardowns – the inverter from a portable TV.

Test Inverter
Test Inverter

This neat little unit outputs 330v DC at a few dozen watts, plenty enough to power up the charger with a small load for testing purposes. The charger does pull the voltage of this converter down significantly, to about 100v, but it still provides just enough to get things going.

It's Alive!
It’s Alive!

After applying some direct DC power to the input, it’s ALIVE! Certainly makes a change from the usual SMPS failures I come across, where a single component causes a chain reaction that writes off everything.

Replacement Rectifier
Replacement Rectifier

Unfortunately I couldn’t find the exact same rectifier to replace the shorted one, so I had to go for the KBU1010, which is rated for 1000v instead of 800v, but the Vf rating (Forward Voltage), is the same, so it won’t dissipate any more power.

Soldered In
Soldered In

Here’s the new rectifier soldered into place on the PCB & bolted to it’s heatsink, with some decent thermal compound in between.

Input Board
Input Board

Here is the factory fuse, a soldered in device. I’ll be replacing this with standard clips for 20x5mm fuses to make replacement in the future easier, the required hole pattern in the PCB is already present. Most of the mains input filtering is also on this little daughterboard.

Fuse Replaced
Fuse Replaced

Now the fuse has been replaced with a standard one, which is much more easily replaceable. This fuse shouldn’t blow however, unless another fault develops.

Full Load Test
Full Load Test

Now everything is back together, a full load test charging a 200Ah 12v battery for a few hours will tell me if the fix is good. This charger won’t be going back into service onboard the boat, it’s being replaced anyway with a new 50A charger, to better suit the larger loads we have now. It won’t be a Sterling though, as they are far too expensive. I’ll report back if anything fails!