diff --git a/CHANGELOG.md b/CHANGELOG.md index bd469e54..080fd53f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -196,6 +196,10 @@ * Web: a database upgrade step that fails stops the boot with an error naming the step and leaves the database as it was, instead of logging a warning and serving a half-upgraded database (SPEC SU2) * Web: the boot recreates every index from `data/*.sql` that the database lacks, and stops, naming what is missing, when a table or column is still missing after the upgrade (SPEC SU3) * Web: a database created by v0.2.0 gains `messages.encrypted`; a database created before v0.6.0 gets the MeshCore text index and the duplicate purge on its first boot, not its second; the upgrade matches column names in any case (SPEC SU4) +* Web: a single-record ingest POST takes about 1 ms instead of 10 ms and no longer syncs the disk four times; `mesh.db-wal` and `mesh.db-shm` stay beside `mesh.db` while the web app runs (SPEC SK1) +* Web: each Puma worker (`WEB_CONCURRENCY`) gets the same, with a database connection of its own (SPEC SK2) +* Web: `mesh.db-wal` shrinks back to 4 MiB after a checkpoint (SPEC SK3) +* Web: a request that waits for the database write lock no longer stops the other requests of its web process (SPEC SK4) * Web: a process that starts while another one upgrades the same database no longer fails on a duplicate column or a locked database (SPEC SU5) * Data: the Reticulum ingestor posts interface names without peer addresses: a TCP, Backbone, UDP or AutoInterface peer name keeps its class and the operator-given name and drops the address and port, an outbound I2P peer drops its I2P destination, a Weave peer posts its class, and RNode names are unchanged (SPEC RI1) * Web: interface names are stored without peer addresses whatever the ingestor posts, and stored names that carry one lose it at the next boot, so `GET /api/destinations` serves none, with or without `PRIVATE=1` (SPEC RI2/RI3) diff --git a/README.md b/README.md index bb2c3032..b17945d7 100644 --- a/README.md +++ b/README.md @@ -133,6 +133,17 @@ repository root when unset: - Well-known document: `$XDG_CONFIG_HOME/potato-mesh/well-known/potato-mesh` - Database: `$XDG_DATA_HOME/potato-mesh` +While the web app runs, its newest writes sit in `mesh.db-wal` beside +`mesh.db`. Copy the database with: + +```bash +sqlite3 ~/.local/share/potato-mesh/mesh.db ".backup mesh-copy.db" +``` + +In Docker, prefix it with `docker compose exec web` and use +`/app/.local/share/potato-mesh/` for both files. After a graceful stop of the +web app, `mesh.db` alone holds every write. + If the web app exits at boot with `Instance private key file cannot be parsed`, restore the file it names from a backup, or delete that file to start with a new key. A new key gives the instance a new id; peers replace its old record when it diff --git a/web/lib/potato_mesh/application.rb b/web/lib/potato_mesh/application.rb index 096d4a12..ebb41126 100644 --- a/web/lib/potato_mesh/application.rb +++ b/web/lib/potato_mesh/application.rb @@ -261,6 +261,7 @@ module PotatoMesh scrub_destination_interfaces # SPEC RI3: needs the destinations table, so after the upgrades init_db unless db_schema_present? ensure_schema_parity! + start_database_keeper! # SPEC SK1: on the checked schema, so after the parity check log_instance_domain_resolution log_instance_public_key diff --git a/web/lib/potato_mesh/application/database.rb b/web/lib/potato_mesh/application/database.rb index 7b3ede82..2543e6a8 100644 --- a/web/lib/potato_mesh/application/database.rb +++ b/web/lib/potato_mesh/application/database.rb @@ -15,6 +15,7 @@ # frozen_string_literal: true require_relative "database/schema_guard" +require_relative "database/keeper" module PotatoMesh module App @@ -61,12 +62,24 @@ module PotatoMesh # Open a connection to the application database applying common pragmas. # + # A busy connection waits up to +db_busy_timeout_ms+ in a busy handler + # that sleeps in Ruby, without the GVL, so the thread of this process + # that holds the write lock runs on and releases it; +busy_timeout=+ + # would sleep inside SQLite holding the GVL, and the waiter would give + # up first (SPEC SK4). A connection belongs to the thread that opened + # it: while it waits, its SQLite mutex is held, so another thread's + # statement on it blocks holding the GVL and neither thread wakes, and + # an interrupt that lands in the wait leaves the mutex held until that + # thread closes the connection. +journal_size_limit+ trims the WAL back + # to +db_journal_size_limit_bytes+ after a checkpoint (SPEC SK3). + # # @param readonly [Boolean] whether to open the database in read-only mode. # @return [SQLite3::Database] configured database handle. def open_database(readonly: false) SQLite3::Database.new(PotatoMesh::Config.db_path, readonly: readonly).tap do |db| - db.busy_timeout = PotatoMesh::Config.db_busy_timeout_ms + db.busy_handler_timeout = PotatoMesh::Config.db_busy_timeout_ms db.execute("PRAGMA foreign_keys = ON") + db.execute("PRAGMA journal_size_limit = #{Integer(PotatoMesh::Config.db_journal_size_limit_bytes)}") end end diff --git a/web/lib/potato_mesh/application/database/keeper.rb b/web/lib/potato_mesh/application/database/keeper.rb new file mode 100644 index 00000000..8c74e744 --- /dev/null +++ b/web/lib/potato_mesh/application/database/keeper.rb @@ -0,0 +1,206 @@ +# Copyright © 2025-26 l5yth & contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# frozen_string_literal: true + +# Loaded first, so the gem's own fork hook sits below KeeperForkHook and +# discards inherited connections before the keeper opens its new one. +require "sqlite3" + +module PotatoMesh + module App + module Database + # One idle, long-lived connection to the application database per + # process (SPEC SK1). + # + # SQLite checkpoints the WAL and deletes it when the last connection + # to the database closes. Each request opens and closes a connection + # of its own, so without another one open every request's close was + # the last, and a single-record POST paid a checkpoint and four + # fsyncs. While the keeper is open no request's connection is the + # last: the WAL stays in place, SQLite's automatic checkpoint (1000 + # WAL pages) moves it into the database file, and +journal_size_limit+ + # trims it afterwards (SPEC SK3). + # + # The keeper runs one statement as it opens. That puts it in WAL mode, + # in which a connection holds a shared lock on the database file for + # as long as it is open. The statement ends at once, so the keeper + # holds no read transaction and never stops a checkpoint. It is + # read-write so that, closed last at exit, it checkpoints the WAL into + # the database file and deletes it, as the last request connection did + # before. While it holds its lock, SQLite keeps the descriptor of each + # closed connection for the next one opened with the same flags, so a + # process holds about as many descriptors on the database as it ever + # had connections open at once. + # + # No connection crosses a fork: the keeper closes before every fork + # and opens again after it, in the parent and in the child, so each + # forked worker holds its own (SPEC SK2). + class Keeper + # The statement the keeper runs once as it opens. + OPEN_STATEMENT = "PRAGMA journal_mode" + + # @return [String, nil] database file the keeper holds open; nil + # before {#start} and after {#stop}. + attr_reader :path + + # @return [SQLite3::Database, nil] the connection, nil while closed. + attr_reader :connection + + # A stopped keeper: nothing is open until {#start}. + def initialize + @mutex = Mutex.new + @path = nil + @connection = nil + @owner_pid = nil + end + + # Open the keeper on +path+, closing a connection it already holds. + # + # @param path [String] database file. + # @return [void] + # @raise [SQLite3::Exception] when the database cannot be opened; + # the keeper stays closed. + def start(path) + @mutex.synchronize do + close_connection + @path = path + open_connection + end + end + + # Close the connection and forget the path, so a later fork opens + # nothing. + # + # @return [void] + def stop + @mutex.synchronize do + @path = nil + close_connection + end + end + + # @return [Boolean] whether this process holds a connection it opened. + def open? + @mutex.synchronize { open_here? } + end + + # Close the connection before a fork and keep the path for + # {#after_fork}, so the child inherits no open connection. + # + # @return [void] + def before_fork + @mutex.synchronize { close_connection } + end + + # Open a new connection after a fork, in the parent and in the + # child, unless the keeper is stopped. The fork has happened by then + # and must not raise in either process, so a failure logs a warning + # and leaves the keeper closed. + # + # @return [void] + def after_fork + @mutex.synchronize { open_connection if @path && !open_here? } + rescue SQLite3::Exception => e + PotatoMesh::Logging.log( + PotatoMesh::Logging.logger_for, + :warn, + "Database keeper did not reopen after fork", + context: "database.keeper", + error_class: e.class.name, + error_message: e.message, + ) + end + + private + + # @return [Boolean] whether a connection opened by this process is + # held; one opened before a fork belongs to the parent. + def open_here? + !@connection.nil? && @owner_pid == Process.pid + end + + # Open the connection on the keeper's path and run + # {OPEN_STATEMENT} on it. The statement can wait for a lock, so + # whatever ends it early, an error, a +Timeout+ or a killed thread, + # closes the connection again, on this thread, before it propagates. + # + # @return [void] + # @raise [Exception] what opening or the statement raised. + def open_connection + connection = SQLite3::Database.new(@path) + opened = false + begin + connection.busy_handler_timeout = PotatoMesh::Config.db_busy_timeout_ms + connection.execute(OPEN_STATEMENT) + opened = true + ensure + connection.close unless opened + end + @connection = connection + @owner_pid = Process.pid + end + + # Close the connection, if one is held. + # + # @return [void] + def close_connection + connection = @connection + @connection = nil + @owner_pid = nil + connection&.close + end + end + + # Prepended to the singleton class of +Process+, whose +_fork+ Ruby + # calls for +Kernel#fork+, +Process.fork+ and +IO.popen("-")+, so a + # forked Puma worker opens a keeper of its own (SPEC SK2). + module KeeperForkHook + # Close the keeper, fork, then open it again in the parent and in the + # child. + # + # @return [Integer] the child's pid in the parent, 0 in the child. + def _fork + Database.keeper.before_fork + super + ensure + Database.keeper.after_fork + end + end + + @keeper = Keeper.new + + class << self + # @return [Keeper] this process's keeper. + attr_reader :keeper + end + + Process.singleton_class.prepend(KeeperForkHook) + + # Registered as the file loads, before the boot installs the + # federation and retention hooks, so the keeper closes after them. + at_exit { Database.keeper.stop } + + # Open this process's keeper on the application database (SPEC SK1). + # The configure block calls this right after +ensure_schema_parity!+, + # so the keeper first opens on the upgraded and checked schema. + # + # @return [void] + # @raise [SQLite3::Exception] when the database cannot be opened. + def start_database_keeper! + Database.keeper.start(PotatoMesh::Config.db_path) + end + end + end +end diff --git a/web/lib/potato_mesh/config.rb b/web/lib/potato_mesh/config.rb index 542c12b9..4f200716 100644 --- a/web/lib/potato_mesh/config.rb +++ b/web/lib/potato_mesh/config.rb @@ -23,6 +23,9 @@ module PotatoMesh DEFAULT_DB_BUSY_TIMEOUT_MS = 5_000 DEFAULT_DB_BUSY_MAX_RETRIES = 5 DEFAULT_DB_BUSY_RETRY_DELAY = 0.05 + # Size SQLite trims the WAL back to after a checkpoint (SPEC SK3): just + # above the 1000 pages of 4 KiB its automatic checkpoint lets it reach. + DEFAULT_DB_JOURNAL_SIZE_LIMIT_BYTES = 4 * 1024 * 1024 DEFAULT_MAX_JSON_BODY_BYTES = 1_048_576 DEFAULT_REFRESH_INTERVAL_SECONDS = 60 # Slow safety-poll cadence used by the dashboard when live SSE updates are @@ -226,6 +229,13 @@ module PotatoMesh DEFAULT_DB_BUSY_RETRY_DELAY end + # Retrieve the size SQLite trims the WAL back to after a checkpoint. + # + # @return [Integer] limit in bytes, set as +PRAGMA journal_size_limit+. + def db_journal_size_limit_bytes + DEFAULT_DB_JOURNAL_SIZE_LIMIT_BYTES + end + # Default rolling retention window in seconds. # # Used as the freshness floor for every "general" bulk read endpoint — diff --git a/web/spec/database_keeper_spec.rb b/web/spec/database_keeper_spec.rb new file mode 100644 index 00000000..9b8cb069 --- /dev/null +++ b/web/spec/database_keeper_spec.rb @@ -0,0 +1,490 @@ +# Copyright © 2025-26 l5yth & contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# frozen_string_literal: true + +require "spec_helper" +require "sqlite3" +require "json" +require "rbconfig" +require "stringio" +require "timeout" +require "tmpdir" + +# The keeper connection that holds the database open between requests and +# never crosses a fork, the WAL size limit, and the busy handler that sleeps +# without the GVL (SPEC SK1-SK4). +RSpec.describe "database keeper" do + # The database methods the app calls, outside any app. + let(:harness) { Class.new { extend PotatoMesh::App::Database } } + let(:four_mib) { 4 * 1024 * 1024 } + + # @return [Float] seconds on the monotonic clock. + def monotonic_now + Process.clock_gettime(Process::CLOCK_MONOTONIC) + end + + # @param path [String] database path. + # @return [Integer, nil] size of the database's WAL file, nil when absent + # or empty. + def wal_size(path) + File.size?("#{path}-wal") + end + + # Run the block in a temporary directory whose +mesh.db+ the app opens, + # with +table t(v)+ created in WAL mode. + # + # @yieldparam path [String] the database path. + # @return [void] + def with_scratch_database + Dir.mktmpdir("database-keeper-spec-", "/var/tmp") do |dir| + path = File.join(dir, "mesh.db") + SQLite3::Database.new(path) do |db| + db.execute("PRAGMA journal_mode=WAL") + db.execute("CREATE TABLE t(v BLOB)") + end + RSpec::Mocks.with_temporary_scope do + allow(PotatoMesh::Config).to receive(:db_path).and_return(path) + yield path + end + end + end + + # Insert one row through a connection +open_database+ opens, then close it. + # + # @param value [Object] value stored in +t.v+. + # @return [void] + def write_row(value = 1) + db = harness.open_database + db.execute("INSERT INTO t(v) VALUES (?)", [value]) + ensure + db&.close + end + + # @param path [String] database path. + # @return [Integer] rows of +t+, read through a new connection. + def row_count(path) + SQLite3::Database.open(path, readonly: true) { |db| db.get_first_value("SELECT COUNT(*) FROM t") } + end + + describe "the boot's keeper" do + let(:app) { Sinatra::Application } + let(:api_token) { "database-keeper-token" } + let(:keeper) { PotatoMesh::App::Database.keeper } + + around do |example| + original = ENV["API_TOKEN"] + ENV["API_TOKEN"] = api_token + example.run + ensure + original.nil? ? ENV.delete("API_TOKEN") : ENV["API_TOKEN"] = original + end + + it "holds the application database open after the boot (SK1)" do + expect(keeper.open?).to be(true) + expect(keeper.path).to eq(PotatoMesh::Config.db_path) + end + + it "keeps the WAL in place when a single-record POST closes its connection (SK1)" do + message = { "id" => 7_310_001, "rx_time" => Time.now.to_i, "from_id" => "!7310a001", "to_id" => "^all", + "channel" => 0, "portnum" => "TEXT_MESSAGE_APP", "text" => "keeper" } + post "/api/messages", message.to_json, { "CONTENT_TYPE" => "application/json", "HTTP_AUTHORIZATION" => "Bearer #{api_token}" } + + expect(last_response.status).to eq(201) + # The request's connection was not the last one, so SQLite neither + # checkpointed nor deleted the WAL as it closed. + expect(wal_size(PotatoMesh::Config.db_path)).to be > 0 + end + + it "holds no read transaction: a TRUNCATE checkpoint beside it completes (SK1)" do + expect(keeper.open?).to be(true) + SQLite3::Database.new(PotatoMesh::Config.db_path) do |db| + db.busy_timeout = 1_000 + expect(db.execute("PRAGMA wal_checkpoint(TRUNCATE)")).to eq([[0, 0, 0]]) + end + expect(wal_size(PotatoMesh::Config.db_path)).to be_nil + end + + it "opens after the schema parity check, and closed last at exit leaves no WAL (SK1)" do + Dir.mktmpdir("database-keeper-boot-", "/var/tmp") do |dir| + result = File.join(dir, "result.json") + script = <<~RUBY + require "json" + web, out = ARGV + require File.join(web, "lib/potato_mesh/application/database") + order = [] + PotatoMesh::App::Database.prepend(Module.new do + define_method(:ensure_schema_parity!) { |*args| super(*args).tap { order << "ensure_schema_parity!" } } + define_method(:start_database_keeper!) { |*args| order << "start_database_keeper!"; super(*args) } + end) + require File.join(web, "lib/potato_mesh/application") + keeper = PotatoMesh::App::Database.respond_to?(:keeper) ? PotatoMesh::App::Database.keeper : nil + db = PotatoMesh::Application.open_database + db.execute("INSERT INTO messages(id, rx_time, rx_iso, text) VALUES (1, 0, '1970-01-01T00:00:00Z', 'kept')") + db.close + File.write(out, JSON.generate(order: order, keeper_open: keeper&.open? || false, + wal_after_write: File.size?(PotatoMesh::Config.db_path + "-wal").to_i, + db_path: PotatoMesh::Config.db_path)) + RUBY + env = { "XDG_DATA_HOME" => File.join(dir, "data"), "XDG_CONFIG_HOME" => File.join(dir, "config"), + "RACK_ENV" => "test", "FEDERATION" => "0", "INSTANCE_DOMAIN" => "keeper.mesh.test" } + web_root = File.expand_path("..", __dir__) + status = system(env, RbConfig.ruby, "-e", script, web_root, result, out: File::NULL, err: File.join(dir, "boot.err")) + expect(status).to be(true), File.read(File.join(dir, "boot.err")) + + child = JSON.parse(File.read(result)) + expect(child["order"]).to eq(%w[ensure_schema_parity! start_database_keeper!]) + expect(child["keeper_open"]).to be(true) + expect(child["wal_after_write"]).to be > 0 + # The keeper closed last, at exit: it checkpointed the WAL into the + # database file and deleted it. + expect(File.exist?("#{child["db_path"]}-wal")).to be(false) + SQLite3::Database.new(child["db_path"], readonly: true) do |db| + expect(db.get_first_value("SELECT text FROM messages WHERE id = 1")).to eq("kept") + end + end + end + + it "closes before a fork and opens a connection of its own in the child and again in the parent (SK2)" do + before_fork = keeper.connection + reader, writer = IO.pipe + pid = fork do + reader.close + writer.write(JSON.generate(open: keeper.open?, own: !keeper.connection.equal?(before_fork), + inherited_closed: before_fork.closed?)) + writer.close + exit!(0) + end + writer.close + child = JSON.parse(reader.read) + reader.close + Process.wait(pid) + + expect(child).to eq("open" => true, "own" => true, "inherited_closed" => true) + expect(before_fork.closed?).to be(true) + expect(keeper.open?).to be(true) + expect(keeper.connection).not_to equal(before_fork) + end + end + + # Referenced lazily, so each example names a missing class on its own. + describe "PotatoMesh::App::Database::Keeper" do + subject(:keeper) { PotatoMesh::App::Database::Keeper.new } + + after { keeper.stop } + + it "holds the database open: the WAL outlives the connection that wrote it (SK1)" do + with_scratch_database do |path| + keeper.start(path) + write_row + expect(wal_size(path)).to be > 0 + expect(keeper.connection.readonly?).to be(false) + end + end + + it "closed last, checkpoints the WAL into the database file and deletes it (SK1)" do + with_scratch_database do |path| + keeper.start(path) + write_row + keeper.stop + + expect(File.exist?("#{path}-wal")).to be(false) + expect(File.exist?("#{path}-shm")).to be(false) + expect(row_count(path)).to eq(1) + expect(keeper.open?).to be(false) + expect(keeper.path).to be_nil + end + end + + it "holds no read transaction: a TRUNCATE checkpoint beside it completes (SK1)" do + with_scratch_database do |path| + keeper.start(path) + write_row + SQLite3::Database.new(path) do |db| + expect(db.execute("PRAGMA wal_checkpoint(TRUNCATE)")).to eq([[0, 0, 0]]) + end + expect(wal_size(path)).to be_nil + end + end + + it "keeps the descriptors of closed connections at their peak, reused by the next ones (SK1)" do + skip "needs /proc/self/fd" unless File.directory?("/proc/self/fd") + + with_scratch_database do |path| + descriptors = lambda do + Dir.glob("/proc/self/fd/*").count { |fd| (File.readlink(fd) rescue nil) == path } + end + keeper.start(path) + # While the keeper holds its lock, SQLite cannot close a closed + # connection's descriptor without dropping that lock, so it keeps it + # for the next connection opened with the same flags. + Array.new(4) { harness.open_database }.each(&:close) + 100.times { write_row } + + expect(descriptors.call).to eq(5) + keeper.stop + expect(descriptors.call).to eq(0) + end + end + + it "starting again on another path closes the first connection (SK1)" do + with_scratch_database do |path| + keeper.start(path) + first = keeper.connection + other = File.join(File.dirname(path), "other.db") + keeper.start(other) + + expect(first.closed?).to be(true) + expect(keeper.path).to eq(other) + expect(keeper.open?).to be(true) + end + end + + it "raises when the database cannot be opened and stays closed (SK1)" do + expect { keeper.start("/nonexistent-potato-mesh-dir/mesh.db") }.to raise_error(SQLite3::CantOpenException) + expect(keeper.open?).to be(false) + end + + # Make every connection +SQLite3::Database.new+ opens run +statement+ + # instead of its first statement, and collect the connections. + # + # @param statement [Proc] stands in for +execute+. + # @return [Array] the connections opened so far. + def stub_first_statement(&statement) + opened = [] + allow(SQLite3::Database).to receive(:new).and_wrap_original do |original, *args, **kwargs| + original.call(*args, **kwargs).tap do |db| + opened << db + allow(db).to receive(:execute, &statement) + end + end + opened + end + + it "closes its connection when its first statement raises, whatever it raises (SK1)" do + with_scratch_database do |path| + [SQLite3::IOException.new("disk I/O error"), Timeout::Error.new("execution expired"), Interrupt.new].each do |error| + opened = stub_first_statement { raise error } + + expect { keeper.start(path) }.to raise_error(error.class) + expect(opened.map(&:closed?)).to eq([true]) + expect(keeper.open?).to be(false) + end + end + end + + it "closes its connection when its thread is killed during the first statement (SK1)" do + with_scratch_database do |path| + reached = Queue.new + opened = stub_first_statement do + reached.push(true) + sleep + end + starting = keeper + starter = Thread.new { starting.start(path) } + expect(reached.pop(timeout: 5)).to be(true) + starter.kill + starter.join + + expect(opened.map(&:closed?)).to eq([true]) + expect(keeper.open?).to be(false) + ensure + starter&.kill + starter&.join + end + end + + it "reopens after a fork, and a stopped keeper stays closed (SK2)" do + with_scratch_database do |path| + keeper.start(path) + first = keeper.connection + keeper.before_fork + expect(keeper.open?).to be(false) + expect(first.closed?).to be(true) + keeper.after_fork + expect(keeper.open?).to be(true) + expect(keeper.connection).not_to equal(first) + + keeper.stop + keeper.before_fork + keeper.after_fork + expect(keeper.open?).to be(false) + end + end + + it "logs a reopen that fails after a fork and leaves the keeper closed (SK2)" do + with_scratch_database do |path| + keeper.start(path) + keeper.before_fork + allow(SQLite3::Database).to receive(:new).and_raise(SQLite3::CantOpenException, "unable to open database file") + log = StringIO.new + allow(PotatoMesh::Logging).to receive(:logger_for).and_return(PotatoMesh::Logging.build_logger(log)) + + expect { keeper.after_fork }.not_to raise_error + expect(keeper.open?).to be(false) + expect(log.string).to include("context=database.keeper") + expect(log.string).to include('error_class="SQLite3::CantOpenException"') + expect(log.string).to include("Database keeper did not reopen after fork") + end + end + + it "reports a connection another process opened as not open (SK2)" do + with_scratch_database do |path| + keeper.start(path) + allow(Process).to receive(:pid).and_return(Process.pid + 1) + expect(keeper.open?).to be(false) + end + end + end + + describe "the WAL size limit" do + it "is 4 MiB (SK3)" do + expect(PotatoMesh::Config.db_journal_size_limit_bytes).to eq(four_mib) + end + + it "is set on every connection open_database opens (SK3)" do + with_scratch_database do + [false, true].each do |readonly| + db = harness.open_database(readonly: readonly) + expect(db.get_first_value("PRAGMA journal_size_limit")).to eq(four_mib) + ensure + db&.close + end + end + end + + it "trims a WAL that grew past it back to it after the next checkpoint (SK3)" do + with_scratch_database do |path| + # A second connection keeps the database open, as the keeper does, + # so no close deletes the WAL. + holder = SQLite3::Database.new(path) + holder.execute("PRAGMA journal_mode") + # A reader inside a read transaction stops each checkpoint from + # restarting the WAL, so the writes below grow it past the limit. + reader = SQLite3::Database.new(path) + statement = reader.prepare("SELECT COUNT(*) FROM t") + statement.step + 80.times { write_row(SQLite3::Blob.new("w" * 100_000)) } + expect(wal_size(path)).to be > 2 * four_mib + statement.close + reader.close + + # The first write checkpoints the whole WAL; the next restarts it and + # trims the file to the limit. + 2.times { write_row } + expect(wal_size(path)).to eq(four_mib) + ensure + statement&.close unless statement&.closed? + reader&.close + holder&.close + end + end + end + + describe "the busy handler" do + # Hold the write lock on a connection +open_database+ opens, in another + # thread, and commit once +release+ is pushed or +hold+ seconds pass. + # + # @param hold [Float] longest time the lock is held, in seconds. + # @param release [Queue] pushed to commit at once. + # @return [Thread] the holding thread. + def hold_write_lock(hold:, release: Queue.new) + holder = harness.open_database + holder.execute("BEGIN IMMEDIATE") + holder.execute("INSERT INTO t(v) VALUES (0)") + Thread.new do + release.pop(timeout: hold) + holder.execute("COMMIT") + ensure + holder.close + end + end + + it "lets the thread that holds the write lock commit while another waits for it (SK4)" do + with_scratch_database do |path| + holding = hold_write_lock(hold: 0.1) + waiter = harness.open_database + started = monotonic_now + # The waiter sleeps between attempts without the GVL, so the holding + # thread runs its COMMIT; busy_timeout= slept inside SQLite holding + # it, and the waiter gave up after the whole 5 s timeout. + waiter.execute("INSERT INTO t(v) VALUES (1)") + waited = monotonic_now - started + holding.join + + expect(waited).to be < 2.5 + expect(row_count(path)).to eq(2) + ensure + waiter&.close + holding&.join + end + end + + it "gives up after db_busy_timeout_ms (SK4)" do + with_scratch_database do + allow(PotatoMesh::Config).to receive(:db_busy_timeout_ms).and_return(100) + release = Queue.new + holding = hold_write_lock(hold: 10, release: release) + waiter = harness.open_database + started = monotonic_now + + expect { waiter.execute("INSERT INTO t(v) VALUES (1)") }.to raise_error(SQLite3::BusyException) + expect(monotonic_now - started).to be_between(0.1, 2.5) + ensure + release&.push(true) + waiter&.close + holding&.join + end + end + + it "lets an interrupt land during the wait, and the waiting thread keeps its connection (SK4)" do + with_scratch_database do |path| + release = Queue.new + holding = hold_write_lock(hold: 10, release: release) + waiter = harness.open_database + started = monotonic_now + + # Inside the 5 s wait, not after it. + expect { Timeout.timeout(0.2) { waiter.execute("INSERT INTO t(v) VALUES (1)") } }.to raise_error(Timeout::Error) + expect(monotonic_now - started).to be < 2.5 + release.push(true) + holding.join + waiter.execute("INSERT INTO t(v) VALUES (2)") + expect(row_count(path)).to eq(2) + ensure + release&.push(true) + waiter&.close + holding&.join + end + end + + it "is installed on read-only connections too, never busy_timeout= (SK4)" do + with_scratch_database do + calls = [] + allow_any_instance_of(SQLite3::Database).to receive(:busy_handler_timeout=).and_wrap_original do |original, ms| + calls << [:busy_handler_timeout=, ms] + original.call(ms) + end + allow_any_instance_of(SQLite3::Database).to receive(:busy_timeout=).and_wrap_original do |original, ms| + calls << [:busy_timeout=, ms] + original.call(ms) + end + + harness.open_database(readonly: true).close + harness.open_database.close + expect(calls).to eq([[:busy_handler_timeout=, 5_000], [:busy_handler_timeout=, 5_000]]) + end + end + end +end