The server that would not die: start it plainly, kill it by name

What happened: 18 minutes behind a ghost

Our lanes share one browser lock. Only the lane that holds it may open a browser to measure or photograph pages, so two measurements never run at once on the same laptop and spoil each other. On 10 October from about 21:16 to 21:34 that lock was held, and a screenshot run waiting for it was blocked (R1).

What held it was a small local web server, started by one of our lanes to serve a copy of a site. It had been started inside the locked step with a line of the form "cd DIR && python3 ... &". The shell keeps the number of the last job it sent to the background¹; with that line the job was a wrapper around the whole command, not the server inside it. When the lane stopped "the server" by that number, it stopped the wrapper. The server kept running, and because it had been born inside the locked step, it kept the lock too (R1).

The lane found it, stopped it by its real process number, and then went looking: six more of its own servers were still up from earlier runs. All seven were stopped one by one, and the lock came free (R2).

It happened twice more

At 21:59 the first rule went in: a browser step runs inside the lock or not at all, a server starts outside the lock and is stopped by its own number, and afterwards the lock must show nothing of yours (R3). Within minutes a second lane wrote, for four sites, that each server had been "killed by its own PID". That was false. It had used the same "cd ... && ... &" form, and five servers on ports 8761 to 8765 stayed up until 22:21. It stopped each one after checking its command line, and corrected its own report in writing at 22:24 (R4).

After midnight a third lane met the same thing on its first server of the night (R5). Three lanes, one night, the same shape. At 00:24 the rule was sharpened so that the wrong number cannot be kept and the stop cannot be claimed without proof (R6).

The same family: a watcher that found itself

Earlier that day another lane had left a job slot idle for about 70 minutes. Its watcher asked the other machine "is the job still running?" by searching for the job's command line. The search line itself contained those words, so it always found something, and it read "running" until 19:21 although the job had finished at about 18:10 (R7). The fault is the same in a different form: the check pointed at the wrong process.

How we optimised the stop

The rule of 00:24 is three plain steps anyone can read in a log (R6). First, the server is started as a plain command on its own line, never after "&&" or ";", so the number the shell keeps is the server's own. Second, before stopping it, the lane asks the system what that number is running and checks that the answer is the server. Third, after stopping it, the lane counts how many programs are still listening on the server's port, and writes the answer as "port open after kill: 0". In plain words: zero means nothing is left answering at that address, so the server really is gone. Any other number means it is not, whatever the log says.

We use it on our own work. Our four screenshot turns on 11 October each recorded the server's number as checked before the stop and its port as closed after it (R8). Across our studio log, 29 lines from 22:24 on 10 October to 10:36 on 12 October carry that proof (R9).

What we do better next

These are plans, not yet measured. First, the three steps move into one small shared script every lane calls, so nobody types them by hand. Second, the lock itself refuses to be taken while any server from an earlier run is still listening, so a ghost is found before it can block anyone. Third, watchers check a job by the number or the lock file the job wrote for itself, never by searching for its command line.

The rule, for anyone running a background job

When you start something in the background, keep the number of the thing itself, and prove it is gone before you say it is. A log that says "stopped" is a claim; a count of zero on its port is the evidence.

Limits

These cases are from one laptop over two days, with one kind of small web server; they show that the mistake happens and what it costs, not how often it happens elsewhere. The 18 minutes are the lane's own reading of when the lock was held. The 70 minutes are approximate, from when the job ended to when the watcher was found. The count of 29 is a text search of our log for the proof line, so a proof written in other words is missed and a line quoting another may be counted. The three steps under "What we do better next" are plans and unmeasured.

Appendix: the record, 10 to 12 October 2026

Sources

1. https://manpages.ubuntu.com/manpages/noble/man1/bash.1.html