web: keep the database open between requests, wait without the GVL (#1041)

CHANGELOG.md

@@ -196,6 +196,10 @@
 * Web: a database upgrade step that fails stops the boot with an error naming the step and leaves the database as it was, instead of logging a warning and serving a half-upgraded database (SPEC SU2)
 * Web: the boot recreates every index from `data/*.sql` that the database lacks, and stops, naming what is missing, when a table or column is still missing after the upgrade (SPEC SU3)
 * Web: a database created by v0.2.0 gains `messages.encrypted`; a database created before v0.6.0 gets the MeshCore text index and the duplicate purge on its first boot, not its second; the upgrade matches column names in any case (SPEC SU4)
+* Web: a single-record ingest POST takes about 1 ms instead of 10 ms and no longer syncs the disk four times; `mesh.db-wal` and `mesh.db-shm` stay beside `mesh.db` while the web app runs (SPEC SK1)
+* Web: each Puma worker (`WEB_CONCURRENCY`) gets the same, with a database connection of its own (SPEC SK2)
+* Web: `mesh.db-wal` shrinks back to 4 MiB after a checkpoint (SPEC SK3)
+* Web: a request that waits for the database write lock no longer stops the other requests of its web process (SPEC SK4)
 * Web: a process that starts while another one upgrades the same database no longer fails on a duplicate column or a locked database (SPEC SU5)
 * Data: the Reticulum ingestor posts interface names without peer addresses: a TCP, Backbone, UDP or AutoInterface peer name keeps its class and the operator-given name and drops the address and port, an outbound I2P peer drops its I2P destination, a Weave peer posts its class, and RNode names are unchanged (SPEC RI1)
 * Web: interface names are stored without peer addresses whatever the ingestor posts, and stored names that carry one lose it at the next boot, so `GET /api/destinations` serves none, with or without `PRIVATE=1` (SPEC RI2/RI3)

README.md

@@ -133,6 +133,17 @@ repository root when unset:
 - Well-known document: `$XDG_CONFIG_HOME/potato-mesh/well-known/potato-mesh`
 - Database: `$XDG_DATA_HOME/potato-mesh`
 
+While the web app runs, its newest writes sit in `mesh.db-wal` beside
+`mesh.db`. Copy the database with:
+
+```bash
+sqlite3 ~/.local/share/potato-mesh/mesh.db ".backup mesh-copy.db"
+```
+
+In Docker, prefix it with `docker compose exec web` and use
+`/app/.local/share/potato-mesh/` for both files. After a graceful stop of the
+web app, `mesh.db` alone holds every write.
+
 If the web app exits at boot with `Instance private key file cannot be parsed`,
 restore the file it names from a backup, or delete that file to start with a new
 key. A new key gives the instance a new id; peers replace its old record when it

web/lib/potato_mesh/application.rb

@@ -261,6 +261,7 @@ module PotatoMesh
       scrub_destination_interfaces # SPEC RI3: needs the destinations table, so after the upgrades
       init_db unless db_schema_present?
       ensure_schema_parity!
+      start_database_keeper! # SPEC SK1: on the checked schema, so after the parity check
 
       log_instance_domain_resolution
       log_instance_public_key

web/lib/potato_mesh/application/database.rb

@@ -15,6 +15,7 @@
 # frozen_string_literal: true
 
 require_relative "database/schema_guard"
+require_relative "database/keeper"
 
 module PotatoMesh
   module App
@@ -61,12 +62,24 @@ module PotatoMesh
 
       # Open a connection to the application database applying common pragmas.
       #
+      # A busy connection waits up to +db_busy_timeout_ms+ in a busy handler
+      # that sleeps in Ruby, without the GVL, so the thread of this process
+      # that holds the write lock runs on and releases it; +busy_timeout=+
+      # would sleep inside SQLite holding the GVL, and the waiter would give
+      # up first (SPEC SK4). A connection belongs to the thread that opened
+      # it: while it waits, its SQLite mutex is held, so another thread's
+      # statement on it blocks holding the GVL and neither thread wakes, and
+      # an interrupt that lands in the wait leaves the mutex held until that
+      # thread closes the connection. +journal_size_limit+ trims the WAL back
+      # to +db_journal_size_limit_bytes+ after a checkpoint (SPEC SK3).
+      #
       # @param readonly [Boolean] whether to open the database in read-only mode.
       # @return [SQLite3::Database] configured database handle.
       def open_database(readonly: false)
         SQLite3::Database.new(PotatoMesh::Config.db_path, readonly: readonly).tap do |db|
-          db.busy_timeout = PotatoMesh::Config.db_busy_timeout_ms
+          db.busy_handler_timeout = PotatoMesh::Config.db_busy_timeout_ms
           db.execute("PRAGMA foreign_keys = ON")
+          db.execute("PRAGMA journal_size_limit = #{Integer(PotatoMesh::Config.db_journal_size_limit_bytes)}")
         end
       end
 

web/lib/potato_mesh/application/database/keeper.rb

@@ -0,0 +1,206 @@
+# Copyright © 2025-26 l5yth & contributors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# frozen_string_literal: true
+
+# Loaded first, so the gem's own fork hook sits below KeeperForkHook and
+# discards inherited connections before the keeper opens its new one.
+require "sqlite3"
+
+module PotatoMesh
+  module App
+    module Database
+      # One idle, long-lived connection to the application database per
+      # process (SPEC SK1).
+      #
+      # SQLite checkpoints the WAL and deletes it when the last connection
+      # to the database closes. Each request opens and closes a connection
+      # of its own, so without another one open every request's close was
+      # the last, and a single-record POST paid a checkpoint and four
+      # fsyncs. While the keeper is open no request's connection is the
+      # last: the WAL stays in place, SQLite's automatic checkpoint (1000
+      # WAL pages) moves it into the database file, and +journal_size_limit+
+      # trims it afterwards (SPEC SK3).
+      #
+      # The keeper runs one statement as it opens. That puts it in WAL mode,
+      # in which a connection holds a shared lock on the database file for
+      # as long as it is open. The statement ends at once, so the keeper
+      # holds no read transaction and never stops a checkpoint. It is
+      # read-write so that, closed last at exit, it checkpoints the WAL into
+      # the database file and deletes it, as the last request connection did
+      # before. While it holds its lock, SQLite keeps the descriptor of each
+      # closed connection for the next one opened with the same flags, so a
+      # process holds about as many descriptors on the database as it ever
+      # had connections open at once.
+      #
+      # No connection crosses a fork: the keeper closes before every fork
+      # and opens again after it, in the parent and in the child, so each
+      # forked worker holds its own (SPEC SK2).
+      class Keeper
+        # The statement the keeper runs once as it opens.
+        OPEN_STATEMENT = "PRAGMA journal_mode"
+
+        # @return [String, nil] database file the keeper holds open; nil
+        #   before {#start} and after {#stop}.
+        attr_reader :path
+
+        # @return [SQLite3::Database, nil] the connection, nil while closed.
+        attr_reader :connection
+
+        # A stopped keeper: nothing is open until {#start}.
+        def initialize
+          @mutex = Mutex.new
+          @path = nil
+          @connection = nil
+          @owner_pid = nil
+        end
+
+        # Open the keeper on +path+, closing a connection it already holds.
+        #
+        # @param path [String] database file.
+        # @return [void]
+        # @raise [SQLite3::Exception] when the database cannot be opened;
+        #   the keeper stays closed.
+        def start(path)
+          @mutex.synchronize do
+            close_connection
+            @path = path
+            open_connection
+          end
+        end
+
+        # Close the connection and forget the path, so a later fork opens
+        # nothing.
+        #
+        # @return [void]
+        def stop
+          @mutex.synchronize do
+            @path = nil
+            close_connection
+          end
+        end
+
+        # @return [Boolean] whether this process holds a connection it opened.
+        def open?
+          @mutex.synchronize { open_here? }
+        end
+
+        # Close the connection before a fork and keep the path for
+        # {#after_fork}, so the child inherits no open connection.
+        #
+        # @return [void]
+        def before_fork
+          @mutex.synchronize { close_connection }
+        end
+
+        # Open a new connection after a fork, in the parent and in the
+        # child, unless the keeper is stopped. The fork has happened by then
+        # and must not raise in either process, so a failure logs a warning
+        # and leaves the keeper closed.
+        #
+        # @return [void]
+        def after_fork
+          @mutex.synchronize { open_connection if @path && !open_here? }
+        rescue SQLite3::Exception => e
+          PotatoMesh::Logging.log(
+            PotatoMesh::Logging.logger_for,
+            :warn,
+            "Database keeper did not reopen after fork",
+            context: "database.keeper",
+            error_class: e.class.name,
+            error_message: e.message,
+          )
+        end
+
+        private
+
+        # @return [Boolean] whether a connection opened by this process is
+        #   held; one opened before a fork belongs to the parent.
+        def open_here?
+          !@connection.nil? && @owner_pid == Process.pid
+        end
+
+        # Open the connection on the keeper's path and run
+        # {OPEN_STATEMENT} on it. The statement can wait for a lock, so
+        # whatever ends it early, an error, a +Timeout+ or a killed thread,
+        # closes the connection again, on this thread, before it propagates.
+        #
+        # @return [void]
+        # @raise [Exception] what opening or the statement raised.
+        def open_connection
+          connection = SQLite3::Database.new(@path)
+          opened = false
+          begin
+            connection.busy_handler_timeout = PotatoMesh::Config.db_busy_timeout_ms
+            connection.execute(OPEN_STATEMENT)
+            opened = true
+          ensure
+            connection.close unless opened
+          end
+          @connection = connection
+          @owner_pid = Process.pid
+        end
+
+        # Close the connection, if one is held.
+        #
+        # @return [void]
+        def close_connection
+          connection = @connection
+          @connection = nil
+          @owner_pid = nil
+          connection&.close
+        end
+      end
+
+      # Prepended to the singleton class of +Process+, whose +_fork+ Ruby
+      # calls for +Kernel#fork+, +Process.fork+ and +IO.popen("-")+, so a
+      # forked Puma worker opens a keeper of its own (SPEC SK2).
+      module KeeperForkHook
+        # Close the keeper, fork, then open it again in the parent and in the
+        # child.
+        #
+        # @return [Integer] the child's pid in the parent, 0 in the child.
+        def _fork
+          Database.keeper.before_fork
+          super
+        ensure
+          Database.keeper.after_fork
+        end
+      end
+
+      @keeper = Keeper.new
+
+      class << self
+        # @return [Keeper] this process's keeper.
+        attr_reader :keeper
+      end
+
+      Process.singleton_class.prepend(KeeperForkHook)
+
+      # Registered as the file loads, before the boot installs the
+      # federation and retention hooks, so the keeper closes after them.
+      at_exit { Database.keeper.stop }
+
+      # Open this process's keeper on the application database (SPEC SK1).
+      # The configure block calls this right after +ensure_schema_parity!+,
+      # so the keeper first opens on the upgraded and checked schema.
+      #
+      # @return [void]
+      # @raise [SQLite3::Exception] when the database cannot be opened.
+      def start_database_keeper!
+        Database.keeper.start(PotatoMesh::Config.db_path)
+      end
+    end
+  end
+end

web/lib/potato_mesh/config.rb

@@ -23,6 +23,9 @@ module PotatoMesh
     DEFAULT_DB_BUSY_TIMEOUT_MS = 5_000
     DEFAULT_DB_BUSY_MAX_RETRIES = 5
     DEFAULT_DB_BUSY_RETRY_DELAY = 0.05
+    # Size SQLite trims the WAL back to after a checkpoint (SPEC SK3): just
+    # above the 1000 pages of 4 KiB its automatic checkpoint lets it reach.
+    DEFAULT_DB_JOURNAL_SIZE_LIMIT_BYTES = 4 * 1024 * 1024
     DEFAULT_MAX_JSON_BODY_BYTES = 1_048_576
     DEFAULT_REFRESH_INTERVAL_SECONDS = 60
     # Slow safety-poll cadence used by the dashboard when live SSE updates are
@@ -226,6 +229,13 @@ module PotatoMesh
       DEFAULT_DB_BUSY_RETRY_DELAY
     end
 
+    # Retrieve the size SQLite trims the WAL back to after a checkpoint.
+    #
+    # @return [Integer] limit in bytes, set as +PRAGMA journal_size_limit+.
+    def db_journal_size_limit_bytes
+      DEFAULT_DB_JOURNAL_SIZE_LIMIT_BYTES
+    end
+
     # Default rolling retention window in seconds.
     #
     # Used as the freshness floor for every "general" bulk read endpoint —

web/spec/database_keeper_spec.rb

@@ -0,0 +1,490 @@
+# Copyright © 2025-26 l5yth & contributors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# frozen_string_literal: true
+
+require "spec_helper"
+require "sqlite3"
+require "json"
+require "rbconfig"
+require "stringio"
+require "timeout"
+require "tmpdir"
+
+# The keeper connection that holds the database open between requests and
+# never crosses a fork, the WAL size limit, and the busy handler that sleeps
+# without the GVL (SPEC SK1-SK4).
+RSpec.describe "database keeper" do
+  # The database methods the app calls, outside any app.
+  let(:harness) { Class.new { extend PotatoMesh::App::Database } }
+  let(:four_mib) { 4 * 1024 * 1024 }
+
+  # @return [Float] seconds on the monotonic clock.
+  def monotonic_now
+    Process.clock_gettime(Process::CLOCK_MONOTONIC)
+  end
+
+  # @param path [String] database path.
+  # @return [Integer, nil] size of the database's WAL file, nil when absent
+  #   or empty.
+  def wal_size(path)
+    File.size?("#{path}-wal")
+  end
+
+  # Run the block in a temporary directory whose +mesh.db+ the app opens,
+  # with +table t(v)+ created in WAL mode.
+  #
+  # @yieldparam path [String] the database path.
+  # @return [void]
+  def with_scratch_database
+    Dir.mktmpdir("database-keeper-spec-", "/var/tmp") do |dir|
+      path = File.join(dir, "mesh.db")
+      SQLite3::Database.new(path) do |db|
+        db.execute("PRAGMA journal_mode=WAL")
+        db.execute("CREATE TABLE t(v BLOB)")
+      end
+      RSpec::Mocks.with_temporary_scope do
+        allow(PotatoMesh::Config).to receive(:db_path).and_return(path)
+        yield path
+      end
+    end
+  end
+
+  # Insert one row through a connection +open_database+ opens, then close it.
+  #
+  # @param value [Object] value stored in +t.v+.
+  # @return [void]
+  def write_row(value = 1)
+    db = harness.open_database
+    db.execute("INSERT INTO t(v) VALUES (?)", [value])
+  ensure
+    db&.close
+  end
+
+  # @param path [String] database path.
+  # @return [Integer] rows of +t+, read through a new connection.
+  def row_count(path)
+    SQLite3::Database.open(path, readonly: true) { |db| db.get_first_value("SELECT COUNT(*) FROM t") }
+  end
+
+  describe "the boot's keeper" do
+    let(:app) { Sinatra::Application }
+    let(:api_token) { "database-keeper-token" }
+    let(:keeper) { PotatoMesh::App::Database.keeper }
+
+    around do |example|
+      original = ENV["API_TOKEN"]
+      ENV["API_TOKEN"] = api_token
+      example.run
+    ensure
+      original.nil? ? ENV.delete("API_TOKEN") : ENV["API_TOKEN"] = original
+    end
+
+    it "holds the application database open after the boot (SK1)" do
+      expect(keeper.open?).to be(true)
+      expect(keeper.path).to eq(PotatoMesh::Config.db_path)
+    end
+
+    it "keeps the WAL in place when a single-record POST closes its connection (SK1)" do
+      message = { "id" => 7_310_001, "rx_time" => Time.now.to_i, "from_id" => "!7310a001", "to_id" => "^all",
+                  "channel" => 0, "portnum" => "TEXT_MESSAGE_APP", "text" => "keeper" }
+      post "/api/messages", message.to_json, { "CONTENT_TYPE" => "application/json", "HTTP_AUTHORIZATION" => "Bearer #{api_token}" }
+
+      expect(last_response.status).to eq(201)
+      # The request's connection was not the last one, so SQLite neither
+      # checkpointed nor deleted the WAL as it closed.
+      expect(wal_size(PotatoMesh::Config.db_path)).to be > 0
+    end
+
+    it "holds no read transaction: a TRUNCATE checkpoint beside it completes (SK1)" do
+      expect(keeper.open?).to be(true)
+      SQLite3::Database.new(PotatoMesh::Config.db_path) do |db|
+        db.busy_timeout = 1_000
+        expect(db.execute("PRAGMA wal_checkpoint(TRUNCATE)")).to eq([[0, 0, 0]])
+      end
+      expect(wal_size(PotatoMesh::Config.db_path)).to be_nil
+    end
+
+    it "opens after the schema parity check, and closed last at exit leaves no WAL (SK1)" do
+      Dir.mktmpdir("database-keeper-boot-", "/var/tmp") do |dir|
+        result = File.join(dir, "result.json")
+        script = <<~RUBY
+          require "json"
+          web, out = ARGV
+          require File.join(web, "lib/potato_mesh/application/database")
+          order = []
+          PotatoMesh::App::Database.prepend(Module.new do
+            define_method(:ensure_schema_parity!) { |*args| super(*args).tap { order << "ensure_schema_parity!" } }
+            define_method(:start_database_keeper!) { |*args| order << "start_database_keeper!"; super(*args) }
+          end)
+          require File.join(web, "lib/potato_mesh/application")
+          keeper = PotatoMesh::App::Database.respond_to?(:keeper) ? PotatoMesh::App::Database.keeper : nil
+          db = PotatoMesh::Application.open_database
+          db.execute("INSERT INTO messages(id, rx_time, rx_iso, text) VALUES (1, 0, '1970-01-01T00:00:00Z', 'kept')")
+          db.close
+          File.write(out, JSON.generate(order: order, keeper_open: keeper&.open? || false,
+                                        wal_after_write: File.size?(PotatoMesh::Config.db_path + "-wal").to_i,
+                                        db_path: PotatoMesh::Config.db_path))
+        RUBY
+        env = { "XDG_DATA_HOME" => File.join(dir, "data"), "XDG_CONFIG_HOME" => File.join(dir, "config"),
+                "RACK_ENV" => "test", "FEDERATION" => "0", "INSTANCE_DOMAIN" => "keeper.mesh.test" }
+        web_root = File.expand_path("..", __dir__)
+        status = system(env, RbConfig.ruby, "-e", script, web_root, result, out: File::NULL, err: File.join(dir, "boot.err"))
+        expect(status).to be(true), File.read(File.join(dir, "boot.err"))
+
+        child = JSON.parse(File.read(result))
+        expect(child["order"]).to eq(%w[ensure_schema_parity! start_database_keeper!])
+        expect(child["keeper_open"]).to be(true)
+        expect(child["wal_after_write"]).to be > 0
+        # The keeper closed last, at exit: it checkpointed the WAL into the
+        # database file and deleted it.
+        expect(File.exist?("#{child["db_path"]}-wal")).to be(false)
+        SQLite3::Database.new(child["db_path"], readonly: true) do |db|
+          expect(db.get_first_value("SELECT text FROM messages WHERE id = 1")).to eq("kept")
+        end
+      end
+    end
+
+    it "closes before a fork and opens a connection of its own in the child and again in the parent (SK2)" do
+      before_fork = keeper.connection
+      reader, writer = IO.pipe
+      pid = fork do
+        reader.close
+        writer.write(JSON.generate(open: keeper.open?, own: !keeper.connection.equal?(before_fork),
+                                   inherited_closed: before_fork.closed?))
+        writer.close
+        exit!(0)
+      end
+      writer.close
+      child = JSON.parse(reader.read)
+      reader.close
+      Process.wait(pid)
+
+      expect(child).to eq("open" => true, "own" => true, "inherited_closed" => true)
+      expect(before_fork.closed?).to be(true)
+      expect(keeper.open?).to be(true)
+      expect(keeper.connection).not_to equal(before_fork)
+    end
+  end
+
+  # Referenced lazily, so each example names a missing class on its own.
+  describe "PotatoMesh::App::Database::Keeper" do
+    subject(:keeper) { PotatoMesh::App::Database::Keeper.new }
+
+    after { keeper.stop }
+
+    it "holds the database open: the WAL outlives the connection that wrote it (SK1)" do
+      with_scratch_database do |path|
+        keeper.start(path)
+        write_row
+        expect(wal_size(path)).to be > 0
+        expect(keeper.connection.readonly?).to be(false)
+      end
+    end
+
+    it "closed last, checkpoints the WAL into the database file and deletes it (SK1)" do
+      with_scratch_database do |path|
+        keeper.start(path)
+        write_row
+        keeper.stop
+
+        expect(File.exist?("#{path}-wal")).to be(false)
+        expect(File.exist?("#{path}-shm")).to be(false)
+        expect(row_count(path)).to eq(1)
+        expect(keeper.open?).to be(false)
+        expect(keeper.path).to be_nil
+      end
+    end
+
+    it "holds no read transaction: a TRUNCATE checkpoint beside it completes (SK1)" do
+      with_scratch_database do |path|
+        keeper.start(path)
+        write_row
+        SQLite3::Database.new(path) do |db|
+          expect(db.execute("PRAGMA wal_checkpoint(TRUNCATE)")).to eq([[0, 0, 0]])
+        end
+        expect(wal_size(path)).to be_nil
+      end
+    end
+
+    it "keeps the descriptors of closed connections at their peak, reused by the next ones (SK1)" do
+      skip "needs /proc/self/fd" unless File.directory?("/proc/self/fd")
+
+      with_scratch_database do |path|
+        descriptors = lambda do
+          Dir.glob("/proc/self/fd/*").count { |fd| (File.readlink(fd) rescue nil) == path }
+        end
+        keeper.start(path)
+        # While the keeper holds its lock, SQLite cannot close a closed
+        # connection's descriptor without dropping that lock, so it keeps it
+        # for the next connection opened with the same flags.
+        Array.new(4) { harness.open_database }.each(&:close)
+        100.times { write_row }
+
+        expect(descriptors.call).to eq(5)
+        keeper.stop
+        expect(descriptors.call).to eq(0)
+      end
+    end
+
+    it "starting again on another path closes the first connection (SK1)" do
+      with_scratch_database do |path|
+        keeper.start(path)
+        first = keeper.connection
+        other = File.join(File.dirname(path), "other.db")
+        keeper.start(other)
+
+        expect(first.closed?).to be(true)
+        expect(keeper.path).to eq(other)
+        expect(keeper.open?).to be(true)
+      end
+    end
+
+    it "raises when the database cannot be opened and stays closed (SK1)" do
+      expect { keeper.start("/nonexistent-potato-mesh-dir/mesh.db") }.to raise_error(SQLite3::CantOpenException)
+      expect(keeper.open?).to be(false)
+    end
+
+    # Make every connection +SQLite3::Database.new+ opens run +statement+
+    # instead of its first statement, and collect the connections.
+    #
+    # @param statement [Proc] stands in for +execute+.
+    # @return [Array<SQLite3::Database>] the connections opened so far.
+    def stub_first_statement(&statement)
+      opened = []
+      allow(SQLite3::Database).to receive(:new).and_wrap_original do |original, *args, **kwargs|
+        original.call(*args, **kwargs).tap do |db|
+          opened << db
+          allow(db).to receive(:execute, &statement)
+        end
+      end
+      opened
+    end
+
+    it "closes its connection when its first statement raises, whatever it raises (SK1)" do
+      with_scratch_database do |path|
+        [SQLite3::IOException.new("disk I/O error"), Timeout::Error.new("execution expired"), Interrupt.new].each do |error|
+          opened = stub_first_statement { raise error }
+
+          expect { keeper.start(path) }.to raise_error(error.class)
+          expect(opened.map(&:closed?)).to eq([true])
+          expect(keeper.open?).to be(false)
+        end
+      end
+    end
+
+    it "closes its connection when its thread is killed during the first statement (SK1)" do
+      with_scratch_database do |path|
+        reached = Queue.new
+        opened = stub_first_statement do
+          reached.push(true)
+          sleep
+        end
+        starting = keeper
+        starter = Thread.new { starting.start(path) }
+        expect(reached.pop(timeout: 5)).to be(true)
+        starter.kill
+        starter.join
+
+        expect(opened.map(&:closed?)).to eq([true])
+        expect(keeper.open?).to be(false)
+      ensure
+        starter&.kill
+        starter&.join
+      end
+    end
+
+    it "reopens after a fork, and a stopped keeper stays closed (SK2)" do
+      with_scratch_database do |path|
+        keeper.start(path)
+        first = keeper.connection
+        keeper.before_fork
+        expect(keeper.open?).to be(false)
+        expect(first.closed?).to be(true)
+        keeper.after_fork
+        expect(keeper.open?).to be(true)
+        expect(keeper.connection).not_to equal(first)
+
+        keeper.stop
+        keeper.before_fork
+        keeper.after_fork
+        expect(keeper.open?).to be(false)
+      end
+    end
+
+    it "logs a reopen that fails after a fork and leaves the keeper closed (SK2)" do
+      with_scratch_database do |path|
+        keeper.start(path)
+        keeper.before_fork
+        allow(SQLite3::Database).to receive(:new).and_raise(SQLite3::CantOpenException, "unable to open database file")
+        log = StringIO.new
+        allow(PotatoMesh::Logging).to receive(:logger_for).and_return(PotatoMesh::Logging.build_logger(log))
+
+        expect { keeper.after_fork }.not_to raise_error
+        expect(keeper.open?).to be(false)
+        expect(log.string).to include("context=database.keeper")
+        expect(log.string).to include('error_class="SQLite3::CantOpenException"')
+        expect(log.string).to include("Database keeper did not reopen after fork")
+      end
+    end
+
+    it "reports a connection another process opened as not open (SK2)" do
+      with_scratch_database do |path|
+        keeper.start(path)
+        allow(Process).to receive(:pid).and_return(Process.pid + 1)
+        expect(keeper.open?).to be(false)
+      end
+    end
+  end
+
+  describe "the WAL size limit" do
+    it "is 4 MiB (SK3)" do
+      expect(PotatoMesh::Config.db_journal_size_limit_bytes).to eq(four_mib)
+    end
+
+    it "is set on every connection open_database opens (SK3)" do
+      with_scratch_database do
+        [false, true].each do |readonly|
+          db = harness.open_database(readonly: readonly)
+          expect(db.get_first_value("PRAGMA journal_size_limit")).to eq(four_mib)
+        ensure
+          db&.close
+        end
+      end
+    end
+
+    it "trims a WAL that grew past it back to it after the next checkpoint (SK3)" do
+      with_scratch_database do |path|
+        # A second connection keeps the database open, as the keeper does,
+        # so no close deletes the WAL.
+        holder = SQLite3::Database.new(path)
+        holder.execute("PRAGMA journal_mode")
+        # A reader inside a read transaction stops each checkpoint from
+        # restarting the WAL, so the writes below grow it past the limit.
+        reader = SQLite3::Database.new(path)
+        statement = reader.prepare("SELECT COUNT(*) FROM t")
+        statement.step
+        80.times { write_row(SQLite3::Blob.new("w" * 100_000)) }
+        expect(wal_size(path)).to be > 2 * four_mib
+        statement.close
+        reader.close
+
+        # The first write checkpoints the whole WAL; the next restarts it and
+        # trims the file to the limit.
+        2.times { write_row }
+        expect(wal_size(path)).to eq(four_mib)
+      ensure
+        statement&.close unless statement&.closed?
+        reader&.close
+        holder&.close
+      end
+    end
+  end
+
+  describe "the busy handler" do
+    # Hold the write lock on a connection +open_database+ opens, in another
+    # thread, and commit once +release+ is pushed or +hold+ seconds pass.
+    #
+    # @param hold [Float] longest time the lock is held, in seconds.
+    # @param release [Queue] pushed to commit at once.
+    # @return [Thread] the holding thread.
+    def hold_write_lock(hold:, release: Queue.new)
+      holder = harness.open_database
+      holder.execute("BEGIN IMMEDIATE")
+      holder.execute("INSERT INTO t(v) VALUES (0)")
+      Thread.new do
+        release.pop(timeout: hold)
+        holder.execute("COMMIT")
+      ensure
+        holder.close
+      end
+    end
+
+    it "lets the thread that holds the write lock commit while another waits for it (SK4)" do
+      with_scratch_database do |path|
+        holding = hold_write_lock(hold: 0.1)
+        waiter = harness.open_database
+        started = monotonic_now
+        # The waiter sleeps between attempts without the GVL, so the holding
+        # thread runs its COMMIT; busy_timeout= slept inside SQLite holding
+        # it, and the waiter gave up after the whole 5 s timeout.
+        waiter.execute("INSERT INTO t(v) VALUES (1)")
+        waited = monotonic_now - started
+        holding.join
+
+        expect(waited).to be < 2.5
+        expect(row_count(path)).to eq(2)
+      ensure
+        waiter&.close
+        holding&.join
+      end
+    end
+
+    it "gives up after db_busy_timeout_ms (SK4)" do
+      with_scratch_database do
+        allow(PotatoMesh::Config).to receive(:db_busy_timeout_ms).and_return(100)
+        release = Queue.new
+        holding = hold_write_lock(hold: 10, release: release)
+        waiter = harness.open_database
+        started = monotonic_now
+
+        expect { waiter.execute("INSERT INTO t(v) VALUES (1)") }.to raise_error(SQLite3::BusyException)
+        expect(monotonic_now - started).to be_between(0.1, 2.5)
+      ensure
+        release&.push(true)
+        waiter&.close
+        holding&.join
+      end
+    end
+
+    it "lets an interrupt land during the wait, and the waiting thread keeps its connection (SK4)" do
+      with_scratch_database do |path|
+        release = Queue.new
+        holding = hold_write_lock(hold: 10, release: release)
+        waiter = harness.open_database
+        started = monotonic_now
+
+        # Inside the 5 s wait, not after it.
+        expect { Timeout.timeout(0.2) { waiter.execute("INSERT INTO t(v) VALUES (1)") } }.to raise_error(Timeout::Error)
+        expect(monotonic_now - started).to be < 2.5
+        release.push(true)
+        holding.join
+        waiter.execute("INSERT INTO t(v) VALUES (2)")
+        expect(row_count(path)).to eq(2)
+      ensure
+        release&.push(true)
+        waiter&.close
+        holding&.join
+      end
+    end
+
+    it "is installed on read-only connections too, never busy_timeout= (SK4)" do
+      with_scratch_database do
+        calls = []
+        allow_any_instance_of(SQLite3::Database).to receive(:busy_handler_timeout=).and_wrap_original do |original, ms|
+          calls << [:busy_handler_timeout=, ms]
+          original.call(ms)
+        end
+        allow_any_instance_of(SQLite3::Database).to receive(:busy_timeout=).and_wrap_original do |original, ms|
+          calls << [:busy_timeout=, ms]
+          original.call(ms)
+        end
+
+        harness.open_database(readonly: true).close
+        harness.open_database.close
+        expect(calls).to eq([[:busy_handler_timeout=, 5_000], [:busy_handler_timeout=, 5_000]])
+      end
+    end
+  end
+end