Our internal projects hub holds every project card, Ahmed's task list, the hand-offs between him and our AI assistant, and an encrypted vault. It has no database. Neither does our command log, nor the register of hospital backups we run for a healthcare customer. Each one is a handful of JSON files on disk, written with a lock, an fsync and an atomic rename. This article explains when that is the right call, the exact write path we use, and the morning a single file owned by the wrong user made the hub refuse every request.
TL;DR
- For a tool with one writer process, a few hundred records and a human reading it, a JSON file is simpler to run, back up and inspect than any database, SQLite included.
- Safe writes need four things: an exclusive lock file, a re-read under the lock, a write to a temporary file followed by
fsync, and an atomicrename. - Audit trails go in append-only JSON Lines, never in the store they audit. A backup register kept inside the database it backs up disappears exactly when you need it.
- The trap is not corruption. It is permissions: a script run as
rootrewrote a file with mode600owned by root, and the service could no longer read its own data.
The problem: small tools, real data
Internal tools grow quietly. The projects hub started as a page of project cards. It now stores a task list that the owner, our AI coding agent and our WhatsApp assistant Kauthar AI all read and write, a vault of credentials released through single-use links, API tokens for the machine clients, and shared settings that other products read over an API. None of it is large, but all of it matters.
The default answer is "put it in Postgres". We run Postgres for every customer product. For these tools we chose not to, and we would choose the same again.
Why not a database
- Size. The whole hub data directory, screenshots and uploaded files included, is 60 MB. The task list is a single file of about 216 KB. There is nothing here a query planner helps with.
- Operations. A database is another service to start, upgrade, back up, restore and monitor. A file is backed up by copying it. Before a risky change we copied
tasks.jsonto a dated.bakfile next to it; restoring it would be onecp. - SQLite was the obvious middle ground. In Node it means
better-sqlite3, which needs a native build on the server and has to be rebuilt with every Node upgrade. For a few hundred rows that cost bought us nothing.
The trade-off: no ad-hoc queries, no concurrent writers at scale, and everything must fit in memory. If that changes, the tool moves to Postgres.
The write path
Node runs JavaScript on one thread, so inside a single process each mutation is already atomic. The danger comes from outside: a command-line admin tool that edits the same file, a crash in the middle of a write, or a power loss after the write but before the data reaches the disk. Our JsonDoc class handles all three in about 50 lines:
update(fn) {
this._acquire(); // O_EXCL lock file: open(lock, 'wx')
try {
this._load(false); // re-read if another process changed the file (mtime)
const out = fn(this.data); // mutate in memory
const tmp = this.file + '.tmp' + process.pid;
const fd = fs.openSync(tmp, 'w', 0o600);
fs.writeSync(fd, JSON.stringify(this.data, null, 1));
fs.fsyncSync(fd); fs.closeSync(fd); // data is on disk before the swap
fs.renameSync(tmp, this.file); // atomic replace: readers see old or new, never half
return out;
} finally { try { fs.unlinkSync(this.lock); } catch {} }
}
Details that matter:
- The lock is a file created with
O_EXCL, which the kernel guarantees only one process can create. A waiter retries every 10 ms, up to 400 times, so a stuck write fails in about four seconds instead of hanging a request. A lock older than ten seconds is treated as left behind by a crashed process and removed. - Re-read under the lock. Readers use an in-memory copy refreshed when the file's modification time changes. A writer re-checks that time after taking the lock, so it never overwrites a change the admin tool made a moment earlier.
- A corrupt file is an error, not an empty store. If the file exists but does not parse, loading throws. The worst bug a file store can have is to "recover" by starting from the defaults and then saving them over the real data.
- The temporary file sits in the same directory, so the rename never crosses file systems and stays atomic.
Our command log uses the same idea in Python: fcntl.flock on a lock file, tempfile.mkstemp in the same directory, then os.replace.
Audit trails: append-only, and somewhere else
Every change to shared settings, every vault link and every reveal is appended as one JSON object per line to a .jsonl file, recording field names and events, never values. Appending a short line is cheap, needs no lock for our volumes, and a partly written last line is simply skipped when the tail is read. The shared-settings audit has 933 lines today; the hub's own action log has 267.
The same reasoning shaped the backup register of a hospital system we operate. Each backup run appends one line to log.jsonl next to the dumps, and a small manifest.json holds the current state of each tier. We deliberately did not put the register in a database table: it would be stored inside the very database it describes, and would be lost exactly when a restore is needed. The register files are readable by the web application; the dumps themselves are not, so the admin screen can show what was backed up without ever being able to read patient data.
The mistake: a correct write by the wrong user
On 10 November we needed to change one field on one task, and the hub had no API route for that edit. We ran a short Node script over SSH that loaded the same JsonDoc class and called update(). It worked perfectly. The JSON was valid, the change was right, the rename was atomic.
The script ran as root. The temporary file was created by root with mode 600, and the rename replaced the service's file with it. The hub runs as its own unprivileged user, which could now neither read nor write tasks.json. Every request that touched tasks failed with EACCES. Kauthar AI, which reads the task list to remind the owner on WhatsApp, told him the projects board was not responding. It took about four minutes to notice and fix with chown.
Atomic rename protects you from half-written files. It does nothing for ownership: the new file belongs to whoever created it.
What we changed:
- Rule: anything that writes the hub's data runs as the service user (
sudo -u <service> node …), never as root, and we list the directory afterwards to confirm. - Prefer the API. Edits go through the hub's own HTTP API whenever a route exists, so the process that owns the files is the one writing them.
- Make the writer fix ownership itself. The command-log engine we wrote ten days later sets the mode and calls
chownto the site's user on the temporary file before the rename, so it is safe even when run as root.
Result
- The hub has run on JSON files since late October. The task list, added on 8 November and shared by the owner, our coding agent and Kauthar AI, has grown to 138 tasks without a format change.
- The command log has recorded 231 commands since 17 November in one 260 KB file read directly by its web page.
- The one outage we have recorded came from file ownership, not from a torn or lost write, and the writer we built after it handles ownership on its own.
Checklist for a JSON file store
- One writer process, data that fits in memory, a few thousand records at most. Otherwise use a database.
- Lock with
O_EXCLorflock, re-read under the lock, write a temporary file in the same directory,fsync, then rename. - Treat a file that does not parse as an error. Never fall back to defaults and save over it.
- Keep audit trails as append-only JSON Lines, outside the store they describe.
- Set owner and mode explicitly before the rename, or never write the files as any user but the service's.
Need this?
GeoTech builds internal tools, admin dashboards and SaaS back ends that stay simple to operate, with the right storage for each job, from a JSON file to Postgres. If you need an internal tool your team can run without a database administrator, talk to us.