The Anatomy of a Half-Configured Package

Falco 0.45 installs cleanly on modern kernels and leaves Ubuntu 20.04 half-configured without printing a single error. I traced it to two lines that are harmless on their own, fixed it with two more, and checked the fix on 48 real VMs. Along the way: a shell rule most scripts get subtly wrong, a stepper for the real postinst, and the qualification grid itself. Budget ~12 minutes.
1. One red row on a release candidate
My release-qualification gate, Kestrel QA, installs Falco’s real .deb and .rpm packages on 48 VM cells: twelve distro and kernel pairs, each with a fresh install on the automatic driver, a fresh install forcing the kernel module, a fresh install forcing modern eBPF, and an upgrade from 0.44.1. Real VMs, not containers, because a container shares the host’s kernel and the whole point is to test against many kernels.
0.45.1-rc1 came back 34 pass, 13 fail, 1 blocked. Most of the red was old news: CentOS 7 is below the package’s glibc floor, and Ubuntu 18.04 has problems of its own. The new red was Ubuntu 20.04 on its stock 5.4 kernel, where every fresh install failed. The same cells failed on 0.45.0-rc4 and on 0.45.0, so this had already shipped in a stable release.
Two neighbours made it interesting:
• Ubuntu 20.04 on the 5.15 HWE kernel, same distro, same package: all green.
• The upgrade from 0.44.1 on the very same 5.4 host: green.
Same distro, same kernel, same package. The upgrade passes and the fresh install fails. That pair turned out to contain the whole bug.
2. What “half-configured” means
Here is the reproduction, one VM, nothing else installed:
dpkg installs a package in two phases. Unpack puts the files on disk. Configure runs the package’s postinst script, which for Falco picks a driver, writes falcoctl’s config, builds the kernel module if needed, and enables the service. If the postinst exits non-zero, dpkg records the package as half-configured: every file is present, and none of that setup happened. No driver configured, no service enabled, and later apt runs keep trying to configure it and failing the same way until someone steps in.
Now look at what the log does not contain. The script prints a progress line before each step. It printed the first one, and then it died. No error message, no hint which command failed. Whatever went wrong, it went wrong in a place where its output was not reaching the terminal.
3. Where set -e fires, and where it does not
The postinst starts with set -e: exit as soon as any command fails. That sounds absolute. It is not. POSIX lists places where a failure is deliberately ignored, and most shell scripts lean on them without knowing it. Each snippet below runs under dash, which is /bin/sh on Debian and Ubuntu and therefore what runs a .deb postinst. Guess before you press run:
What you just saw: a failure inside a command substitution kills the script only in one shape. It has to be a bare assignment, x=$(cmd), outside any if, || or &&, not behind local or export, and not the left side of a pipeline. A function called normally inherits the rule; a function called as f || true has it switched off for its entire body.
And the substitution swallows the failing command’s output. That is the missing error message from section 2: whatever the command said went into a variable that the script never got to read.
The postinst has exactly one line in that shape. It is in a function called package_driver_version:
4. Two harmless lines
Both lines arrived together in falco#3994, fix(packaging): restore services and drivers after upgrades. That PR fixed a real problem: upgrades were losing the enabled service and any custom driver version pin. To keep the pin, it reads the current driver version with falcoctl driver printenv before reconfiguring. And it added set -e at line 20 of the postinst.
The second part is the subtle one. 0.44.1’s postinst already had a set -e, at line 105, after the block that configures the driver. Whether or not that placement was deliberate, it meant the configure block ran with errexit off, and a falcoctl failure there was printed and survived. #3994 did not move that line. It added a second one at the top. The file now has two, and the first one quietly puts the configure block under errexit for the first time.
On its own, that is fine as long as nothing in the block fails on a supported host. So why does printenv fail? Compare what the two 5.4 cells had on disk when the postinst ran:
The package ships a falcoctl.yaml that allows only modern_ebpf. The postinst widens that to modern_ebpf plus kmod a few lines later. But package_driver_version runs first, and printenv checks the configured drivers against the running kernel. A 5.4 kernel cannot run modern eBPF, so with nothing else allowed, falcoctl exits 1.
On an upgrade, falcoctl.yaml is a conffile, so dpkg keeps the one already on disk. 0.44.1’s postinst had configured it with --type modern_ebpf --type kmod, so kmod is allowed before the new postinst touches anything. Same command, same kernel, a driver it can use, exit 0. That is why the upgrade cell was green, and why testing the upgrade path, the thing #3994 was about, would not show it.
Step through the script yourself. Change one control at a time:
What to try: 0.45.1-rc1, 5.4, fresh, any driver choice: the script dies on the third line, before it has even looked at your choice. Switch to upgrade and the same line succeeds. Switch to 0.44.1 with modern_ebpf and the driver config line fails too, but errexit is still off there, so the script prints the error and carries on to a configured package whose service cannot start.
That last behaviour is worth defending. A package that installs and whose service fails says something true and specific: systemctl status will tell you this kernel cannot run modern eBPF. A half-configured package says nothing, and it breaks every apt run after it until someone cleans it up by hand.
5. Fixing it twice
My first fix only touched printenv. On the grid, Ubuntu 20.04 auto and kmod went green, and the forced modern_ebpf cells were still half-configured. Their install log now got one line further:
falcoctl driver config --type modern_ebpf refuses a kernel that cannot run it, and that line is now under errexit too. Removing one exit just lets the script run far enough to reach the next one. That is a good reason to run the whole matrix after every fix rather than the one cell you were looking at.
The full fix is two lines in shared code, plus the same one-line change in the rpm script:
The first line did not need inventing. An empty driver_env already falls back to the package’s own driver version a few lines further down, and the function directly above it, package_save_previous_driver, already calls printenv exactly this way. The codebase had the answer next door. The second line restores what 0.44.1 did: configure the package, let the service fail where the failure belongs. A maintainer may prefer a different shape, such as checking whether the kernel can run modern eBPF before trying, and that is their call to make.
A test that fails first
The packaging scripts have a real test suite, scripts/packaging/test_package_scripts.py, which runs the postinst against fake falcoctl and systemctl commands. The fake falcoctl driver printenv printed a driver every time. It modelled a machine where every driver works, so no test could have caught this.
I taught the fake both refusals, behind a modern_ebpf_supported flag, and added two tests covering deb and rpm with all three driver choices. Against unpatched master, all six new subtests fail with the same no supported driver found error. With the fix, all 16 tests in the file pass. A test that has never failed has not yet shown it can.
6. 48 cells, two builds
Unit tests prove the logic. The grid proves the packages. Here is every cell for 0.45.1-rc1 and for the same release candidate rebuilt with the fix. Click any cell for what happened on that VM.
Four cells change verdict and none get worse. Ubuntu 20.04 on 5.4, auto and kmod, go from half-configured to passing. The two forced modern_ebpf cells on 18.04 and 20.04 go from fail to unsupported.
If “0 unsupported” looks better than “2 unsupported”, it is worth slowing down, because I had to as well. Unsupported is not a softer fail. The gate only reaches that verdict when three things are true: the scenario forced modern_ebpf, the package configured cleanly, and the kernel cannot run modern eBPF. On rc1 the package never configured, so those two cells could only ever be failures. No package can make modern eBPF run on a 4.15 or 5.4 kernel. The best possible result there is “installs, and tells you the driver cannot run”, and that is what the fix gets.
The nine failures left are identical in both builds, and none of them are this bug:
• CentOS 7: the rpm requires glibc 2.28 and CentOS 7 ships 2.17. Rejected before any script runs, and the upgrade cell is blocked because 0.44.1 never installed either.
• Ubuntu 18.04 auto, kmod and upgrade: with the fix the package now configures, but systemd 237 rejects ExecReload=kill -1 $MAINPID (“Executable path is not absolute”) and will not load the units. The bundled container plugin also needs glibc 2.28, which 18.04 does not have. That floor showed up in the undefined symbol post too.
• kmod on Debian 12, Ubuntu 22.04 and Rocky 9: the kernel module build fails inside falcoctl driver install. That step sat after 0.44.1’s set -e as well, so a failed module build has always failed the install. Whether it should is a separate question.
The grid checks install, driver, service, and one detection. For depth I also ran a 61-test sensor suite (detections, containers, outputs, load, lifecycle) on nine cells with the first, printenv-only build. Stock Ubuntu 20.04 went from 0 of 61 passing, nearly all skipped because there was no running sensor to test, to 58 of 61. The other eight cells gave identical results before and after.
7. Why it got through
Nobody did anything careless here. #3994 fixed a real upgrade bug, had tests, and was reviewed. Three ordinary things lined up:
• The fake modelled a kernel where everything works. The shim’s printenv could not fail, so the tests could not see what happens when it does. A fixture contains only what you thought to put in it.
• The natural test of an upgrade fix is an upgrade, and the upgrade passed, even on the exact host where fresh installs failed.
• The failing kernel is the one nobody develops on. Ubuntu 20.04 is still in production in a lot of places under extended support, but anyone running it on the HWE kernel is on 5.15, where modern eBPF works and the bug cannot appear.
The lesson is not about falcoctl. Adding set -e to an existing script is not adding a check. It changes the meaning of every line that was already there, including lines the diff does not show. The useful review question is not “is this line correct?” but “which commands are now under errexit that were not before, and can any of them fail on a host we support?”
8. Shipping it upstream
I opened falcosecurity/falco#4037 against master, with the reproduction above, the regression traced to #3994, the grid results, and a note that release/0.45.x carries the same line. The out of scope problems on 18.04 are in the PR as separate items, offered as their own issue rather than folded in. A fix that quietly changes three unrelated things is harder to review and harder to revert.
Four lines of diff, two new tests, and 48 VMs of evidence. The ratio is the job.
9. Check yourself
1 · Under set -e, which of these exits: x=$(false), local x=$(false), x=$(false | cat)?
local returns its own status, 0, which hides the substitution’s. A pipeline’s status is its last command’s, and cat succeeds. Without set -o pipefail, which this postinst does not set, the failure on the left is invisible.2 · Why did the upgrade from 0.44.1 pass on the same 5.4 host where fresh installs failed?
/etc/falcoctl/falcoctl.yaml is a conffile. On upgrade dpkg keeps the one on disk, which 0.44.1 had set to kmod, so printenv succeeds. A fresh install gets the shipped file, which allows only modern_ebpf.3 · The postinst failed with no error message. Where did the message go?
driver_env=$(falcoctl driver printenv) captures what the command prints, and errexit fires before anything reads driver_env. When a script dies without a word, look for a capture.