45 scheduled tasks. 25 were lying in crontab, 18 were hanging on user-level systemd timers, and 2 were pilot/split-tier tasks on the original cron. At 05:40 on August 24, 2026, I migrated them all into my self-hosted Windmill—45/45 smoke tests green, all original schedules removed.

If I’d done this sooner, I would’ve saved a lot of midnight log-diving. But the closer I got to actually doing it, the more I realized the migration itself had more pitfalls than cron ever did.

I. Why Clear This Mess

These 45 tasks didn’t have a “ledger” to begin with. They were scattered across three places:

• 25 lines in crontab, running naked—no history at all; wrong edits couldn’t be rolled back, wrong deletions left no recycle bin;
• 18 systemd timers, scattered under ~/.config/systemd/user/, where you’d have to check each one with list-timers just to see which were alive;
• Frequencies ranging from every minute to the first Sunday of the month, some of which nobody remembered who added or why.

The worst part was invisible failures: if a cron task crashed, there was no notification, no dashboard—maybe a month would pass before someone noticed that some data pipeline had gone stale for thirty days.

I had only three demands of this setup: visibility, editability, and failure alerts. Self-hosted Windmill CE nailed all three—web UI, editable flows, failure notifications. But before I could make it work, I had to solve a physical problem: Windmill flows run inside Docker containers, so they can’t touch the host’s crontab scripts or execute any commands on the host.

II. Core Architecture: A “Mailbox Protocol” Bridge

The fix was to decouple “execution” from “scheduling”: scheduling and flow definitions stay in Windmill, and the actual commands still run on the host. In between, Windmill’s built-in variables act as a mailbox, with a host-side daemon as the postman:

Here’s how it works:

1. A Windmill schedule fires the flow on cue, using crontab syntax;
2. The flow’s only step, wm_exec (a small module I wrote), POSTs the command to be executed as a mailbox variable, named u/admin/wm_cmd_<taskID>;
3. A bridge daemon on the host (a systemd user unit, resident at boot) polls the mailbox every 10 seconds, picks up commands FIFO, and executes them via the native shell according to a registry (command, working directory, lock file, timeout, whitelist);
4. After execution, the rc and output tail are written back into the receipt mailbox u/admin/wm_job_<taskID>; the step reads it, deletes the variable, and we’re done.

A few key details:

• The variable inbox is naturally atomic: creating a command variable has no “overwrite race condition,” so it naturally serves as a queue;
• Deleting a variable requires an HTTP DELETE with the /w/admins/ prefix; otherwise the command variable lingers and the bridge replays it every 10 seconds;
• The bridge only has 3 concurrent slots—when long tasks queue up, other tasks must wait. This limitation later forced me to redesign the approach for one special task (see Section V);
• Each script also has an ok_rc whitelist: for audit-scan scripts, an rc of 1 means “findings detected,” which is not a failure, so it’s normalized to 0 automatically.

III. The Iron Rule: Only “True Green” Counts as Migrated

During the migration I set myself one iron rule: no smoke test truly green, don’t touch the original cron/timer. “True green” has a strict definition—the flow must produce a result that is a dictionary with rc == 0, and the tail(output tail) must contain real script output.

Why so strict? Because if you relax the acceptance criteria even a little, your migration becomes an illusion. The incident described in the next section is proof.

IV. First Incident: Failures Can Be Judged as Successes

On the first night of migration, I was doing a full acceptance check at midnight and found that some flows showed as “migrated,” yet not a single task had actually executed on the host.

Tracing the issue revealed that the execution module had been broken in an earlier revision: the posting path in the flow pointed inside the container, so every command delivery would fail. But after failure, the flow’s failure-handling fallback module would exit with code 0, returning “alert dispatched” as a success signal. The acceptance driver saw “success” and checked the box—19 phantom green flows were born this way.

The fix was two things: rewrite the execution module in the mailbox-protocol version, and tighten the acceptance logic to result is dict and rc == 0. Then rerun all 19 flows and this time inspect the actual execution records in the job table.

The lesson is plain: success must be a single, strict signal—not a string. Any “roughly good enough” fallback will become a fake handover checklist in the middle of the night.

V. Second Incident: At 4 AM, My Identity Was Deleted

Worse than phantom greens was what happened at 04:08 the next morning. The bridge daemon suddenly started getting 401s across the board—my admin token and the admin user had both been deleted from the database’s token and user tables. The audit table was spotless, with no record of who deleted them or through what path.

The bridge was connected to a completely broken API at the time. The fix was a database surgery: rebuild the admin user row, create a new token row, and satisfy all the hidden rules of the token table—

• The table’s token_hash stores the sha256 hex digest of the raw token, not the token itself;
• The auth query requires the token row to satisfy both owner = 'u/admin' (the internal-ID form with the u/ prefix) and a non-empty email field—get either one wrong and you get a permanent 401.

I only figured these rules out by reading the auth source code line by line. After recovery, I persisted the token renewal date and backup paths into the handover doc and flagged this single point of failure for “needs audit.”

The lesson for self-hosting enthusiasts: never treat the scheduler’s own credentials as “something that could never go wrong.” It’s infrastructure just like your crontab, and it deserves the same backup and audit treatment.

VI. Third Incident: A Persistent Process Is Not a “Task”

The most stubborn one was ghwebfollow: a Playwright-driven GitHub follow engine, while True, launched at boot and running until the end of time—it never has a “completion” concept.

Migrating it as a regular task into Windmill would cause two deaths:

1. The bridge gives it a 900-second timeout, after which it’s judged failed—a healthy engine killed for “living too long”;
2. The scheduler fires every 10 minutes; the previous 15-minute task is still alive when the next one arrives, so new tasks queue up—the 3 bridge slots get eaten by this one task in no time, starving all other flows.

The correct fix is to think the semantics through first: the original systemd timer was never “run every 10 minutes”—it was “check every 10 minutes, relaunch if dead”—a watchdog gate. So I gave the engine a watchdog script: Windmill fires every 10 minutes, the watchdog checks—

• Engine is alive, heartbeat is fresh → do nothing, exit rc=0;
• Engine is dead or heartbeat is stale → relaunch a new engine, rc=0.

Each inspection completes within 5 seconds and never occupies a slot. Things that don’t finish running are services, not tasks; services need a watchdog-style inspection, not being shoved into a task queue.

VII. Fourth Incident: A 1G Memory Container Can Kill Itself

When migrating the WeChat draft task, the smoke test failed three times in a row, all with exit code 137. 137 means the process was killed by signal 9—inside a container, that’s almost always the cgroup memory limit choking it.

Checking the host kernel logs confirmed it: the task builds a static site from three articles in Hugo, and the process’s memory peak hit 1.16 GiB, while its container’s mem_limit was only 1g. The build itself needed more memory than the container had allowed itself. Bumping mem_limit to 3g made the smoke test turn green immediately; the three drafts were produced in 422 seconds of real runtime.

This lesson is the most actionable: when a migration smoke test returns rc=137, check the memory limit first—don’t jump to doubting the migration logic. 137 is telling you “I died, but not of natural causes.”

VIII. Fifth Incident (and a Bunch of Small Pitfalls)

The fifth incident was a three-in-one: none was dramatic, but all three cost me real time.

Dead endpoints in the proxy candidate list. An auto-update script picks available proxies from a local pool, and the first entry in the candidate list was a port that had been dead for a long time. Each execution would bang into that dead port first, then fall back—slow, but no error. During post-migration troubleshooting, I initially suspected a git problem, but the real issue was ordering: live ports should come first, dead ones sink to the bottom.

Parallel cron-line deletions stomped each other. Multiple parallel migrations each read the crontab, deleted their own line, and wrote back—later writers resurrected earlier writers’ deletions. Three cron lines came back to life this way. Lesson: crontab read-modify-write must be serialized.

The cron weekday-0 trap. Windmill’s crontab syntax starts with seconds and has 6 fields; Sunday must be written as 7, not 0. The migration tool doesn’t do this conversion, so a monthly task would be rejected outright with an error.

One pure troubleshooting anecdote: the bridge logs were full of errors for a long time, and after checking everything I finally realized that log file itself was a five-day-old fossil—after the bridge switched its output to journal, nobody deleted the old file. Always confirm you’re reading live logs before debugging; this one tip alone was worth half an hour.

IX. What It Looks Like Now

Post-migration state:

• All 45 tasks are green and enabled, spanning frequencies from every minute to the first Sunday of the month, all visible through the web UI and a single ledger;
• The original crontab retains only 6 entries that I chose not to migrate—watchdog tasks (self-healing probes tightly bound to the system, better left in place);
• Both incidents (phantom greens, token deletion) are fixed and documented in the handover doc;
• Two lingering items: 9 dead containers pending cleanup after confirmation, and the admin token expiring on 2026-09-22—renewal reminder is on file.

If you also have a scattered mess of cron jobs, whether you should tackle it is another question—but this migration gives you a reusable decision framework:

• Inventory before you migrate: all 45 tasks were catalogued before a single one was moved. Don’t head-fault in the mud when your tasks can swim on their own;
• Accept only rc==0 + real output as success—never accept a string as a success signal;
• Classify first, then act: finishes quickly? migrate as a task. Runs continuously? wrap in a watchdog inspection. Self-healing watchdogs? leave them alone;
• Removing the original schedule is the last step—only act once everything is green, and keep a rollback path for every single task.

Scheduled tasks are infrastructure, not desk-candyst. They deserve a web UI, a ledger, and an alerting system that actually rings.