Fixing SQLite "database is locked" When Many Scheduled Jobs Share One File

You have a self-hosted agent in one container. Several fetchers run on their own schedules, a scoring job reads what they collect, and everything lands in one SQLite file. It works in testing. Then two jobs fire at the same moment and one of them dies with:

sqlite3.OperationalError: database is locked

Here is the short answer. Four changes fix most cases:

1. Switch the database to WAL mode so readers stop blocking writers.

2. Open every write transaction with BEGIN IMMEDIATE, not the default deferred BEGIN.

3. Do all network I/O and LLM calls outside the transaction. Hold the write lock only for the inserts.

4. Make inserts idempotent with a UNIQUE constraint, so a retried or overlapping run can't create duplicates.

The rest of this article explains why each one matters, with code you can drop into a Python scheduler.

Why SQLite says "locked" at all

SQLite allows one writer at a time. Concurrent writes are fine as long as each writer waits its turn. The error shows up when a writer gives up waiting, or when SQLite refuses to let it wait at all.

In the default rollback-journal mode, a writer also blocks readers while it commits. With a scheduler firing jobs on independent intervals, collisions are guaranteed eventually. You just can't predict when.

Step 1: WAL mode and a busy timeout

import os
import sqlite3

DB_PATH = os.environ["DB_PATH"]
BUSY_TIMEOUT_S = float(os.environ["DB_BUSY_TIMEOUT_S"])

def connect() -> sqlite3.Connection:
 conn = sqlite3.connect(
 DB_PATH,
 timeout=BUSY_TIMEOUT_S, # how long to wait on a lock before raising
 isolation_level=None, # autocommit; we issue BEGIN ourselves
 )
 conn.execute("PRAGMA journal_mode=WAL")
 conn.execute("PRAGMA synchronous=NORMAL")
 return conn

WAL (write-ahead logging) lets readers keep reading a consistent snapshot while a writer appends to the log. journal_mode=WAL is stored in the database file, so setting it on every connect is harmless. synchronous=NORMAL is the usual pairing with WAL. You can lose the last commit on power loss, but the database won't corrupt.

The timeout argument is Python's way of setting SQLite's busy handler. Pick a value from your own job durations: long enough to outlast your longest write transaction.

Step 2: BEGIN IMMEDIATE, and the failure mode a timeout can't fix

The busy timeout alone does not cover the case where a deferred transaction has to upgrade from reader to writer. Here's why.

With isolation_level left at its legacy default, Python's sqlite3 quietly opens a deferred transaction before your first write. A deferred transaction starts as a reader. If it reads something first and then tries to write, it has to upgrade to a writer. In WAL mode, if another connection committed in between, the snapshot is stale. SQLite then returns SQLITE_BUSY right away, without calling the busy handler, because waiting can't make that snapshot current.

So your timeout is set correctly and still gets skipped. The fix is to take the write lock up front:

from contextlib import contextmanager

@contextmanager
def write_tx(conn: sqlite3.Connection):
 conn.execute("BEGIN IMMEDIATE") # acquire the write lock now, waiting if needed
 try:
 yield conn
 conn.execute("COMMIT")
 except BaseException:
 conn.execute("ROLLBACK")
 raise

BEGIN IMMEDIATE waits on the busy handler (respecting your timeout) until it gets the writer slot. Once inside, nothing can invalidate your snapshot halfway through.

Step 3: never hold the lock across I/O

The easiest way to starve every other job is to hold a write transaction open while you wait on the network. This is the shape you want for a fetcher:

def run_source(source_name: str, fetch) -> None:
 items = fetch() # HTTP happens here, no lock held
 if not items:
 return
 conn = connect()
 try:
 with write_tx(conn): # lock held only for the inserts
 conn.executemany(
 """
 INSERT INTO items (source, external_id, title, url, fetched_at)
 VALUES (?, ?, ?, ?, datetime('now'))
 ON CONFLICT (source, external_id) DO NOTHING
 """,
 [(source_name, i.id, i.title, i.url) for i in items],
 )
 finally:
 conn.close()

Jobs that call an LLM need the same discipline, because a model call is the slowest I/O in the whole process. Split the job into three phases:

def score_pending(score_fn) -> None:
 conn = connect()
 try:
 rows = conn.execute(
 "SELECT id, title, url FROM items WHERE scored_at IS NULL"
 ).fetchall() # plain read, no write lock
 finally:
 conn.close()

 results = [(score_fn(r), r[0]) for r in rows] # slow model calls, no lock

 conn = connect()
 try:
 with write_tx(conn):
 conn.executemany(
 "UPDATE items SET score = ?, scored_at = datetime('now') "
 "WHERE id = ? AND scored_at IS NULL",
 results,
 )
 finally:
 conn.close()

The AND scored_at IS NULL guard means that if two scoring runs ever overlap, the second write is a no-op instead of overwriting the first.

Step 4: idempotent inserts

Fetchers overlap with themselves too. A source that returns the same item on consecutive runs, or a run that retries after a crash, should not create duplicate rows. Push the deduplication into the schema:

CREATE TABLE IF NOT EXISTS items (
 id INTEGER PRIMARY KEY,
 source TEXT NOT NULL,
 external_id TEXT NOT NULL,
 title TEXT,
 url TEXT,
 fetched_at TEXT NOT NULL,
 score REAL,
 scored_at TEXT,
 UNIQUE (source, external_id)
);

Combined with ON CONFLICT ... DO NOTHING, every fetch can be retried safely. That also means you can be blunt about error handling: if a job fails, run it again.

Wiring the scheduler

With APScheduler you can give every source its own interval and stop a slow run from piling up behind itself:

from apscheduler.schedulers.blocking import BlockingScheduler

sched = BlockingScheduler()

for name, (fetch, interval_s) in SOURCES.items():
 sched.add_job(
 run_source,
 "interval",
 seconds=interval_s,
 args=[name, fetch],
 id=name,
 max_instances=1, # never run the same source twice at once
 coalesce=True, # missed runs collapse into one, not a burst
 )

sched.start()

I set max_instances and coalesce explicitly so the behavior is visible in the code instead of inherited from a default someone might change. Each job opens its own connection and closes it when done. Never share one sqlite3.Connection across scheduler threads.

The Docker detail that silently breaks WAL

In WAL mode, SQLite keeps two extra files next to the database: yourdb.sqlite-wal and yourdb.sqlite-shm. If you bind-mount only the database file into the container, those sidecar files end up somewhere they shouldn't, and you'll see odd locking or lost writes. Mount the directory:

services:
 agent:
 volumes:
 - ./data:/app/data # the directory, not ./data/app.sqlite
 environment:
 DB_PATH: /app/data/app.sqlite

Also keep that directory on a local disk. WAL relies on shared memory and file locking that network filesystems don't reliably provide.

What this does not solve

All of this assumes one host and one process tree. SQLite still has a single writer. These techniques make the writers take turns politely, but they don't make writes parallel. If your write volume keeps the lock busy most of the time, or if you need more than one container writing to the same data, you've outgrown this setup and should move to Postgres.

WAL has its own catch. A checkpoint folds the log back into the main file, and it can't finish while a long-lived reader is holding an old snapshot. A dashboard that keeps a read transaction open indefinitely will let the -wal file grow. Keep read transactions short, just like writes.

None of this helps if the database is on NFS or a similar network mount. Don't put it there.

Market Radar Kit is the self-hosted market-intelligence agent I built and run on exactly this shape: 11 sources and 14+ scheduled jobs in one Python container writing to one SQLite file. It needs a Docker host and a Claude subscription. https://fulcrumenterprises.tech/go/market-radar-kit/?c=hashnode&v=8bd7cd