Kiwix/Wikipedia summaries pull in infobox/hatnote/citation clutter, and sometimes mis-decode UTF-8 as mojibake #143

New Issue

2026-10-05T01:25:42+02:00

adam commented

2026-10-05 01:25:42 +02:00

Why this matters

Mesh networks like Meshtastic are especially valuable specifically because they keep working when normal infrastructure doesn't - during power outages, storms, or other situations where cellular/internet service is down. Kiwix exists for exactly that same reason: it's widely used by preppers, off-grid communities, and disaster-response groups to keep Wikipedia, medical references, repair guides, and (relevant here) amateur radio references available with zero internet dependency.

That means in the exact scenario where this bot is most useful - internet actually down - the online Wikipedia API fallback isn't available at all, and Kiwix becomes the only source, not just a backup. A garbled or citation-cluttered response might be a minor annoyance day-to-day, but it's a real problem if someone's trying to read a first-aid procedure, a repair guide, or FT8 operating etiquette during an actual emergency and the answer comes back as scrambled infobox text. So I think it's worth getting the Kiwix output path solid, even though it's not the default/most commonly hit code path day-to-day.

Describe the bug

Scope note: this only affects the Kiwix/offline path (get_kiwix_summary()), not the default online Wikipedia REST API path (get_wikipedia_summary()). The online path just reads Wikipedia's own pre-built extract field from JSON, which is already clean - so this only shows up for users running their own local Kiwix server with useKiwixServer = True.

Two related issues with the text extracted by text_from_html() / get_kiwix_summary() in modules/wiki.py:

1. Non-prose content pollutes the summary

text_from_html() grabs every visible string inside the whole mw-parser-output div:

texts = main.find_all(string=True)
visible_texts = filter(tag_visible, texts)
return " ".join(t.strip() for t in visible_texts if t.strip())

tag_visible() only filters style/script/head/title/meta tags, so it doesn't exclude infobox tables, hatnotes/redirect notices, or citation reference markers. These all get concatenated in with the actual lead paragraph.

Real example (Kiwix-backed kiwix Japan or wiki Japan with a local Wikipedia ZIM):

"Nippon" redirects here. For other uses, see Japan (disambiguation) and Nippon
(disambiguation). Japan ... Nippon-koku or Nihon-koku ... Flag Imperial crest
Anthem: "Kimigayo" ... [3][4] Government crest of Japan State Seal of Japan
Area controlled by Japan shown in green-claimed, but uncontrolled shown in
light green Capital and largest city Tokyo [5] 35°41'N 139°46'E / ...

That's the article's hatnote + infobox sidebar table + citation markers, not the actual lead paragraph a reader would expect from a summary.

2. Encoding sometimes mis-detected, producing mojibake

article_response.encoding = article_response.apparent_encoding or 'utf-8'

apparent_encoding is a statistical guess and can be wrong, especially on pages that are mostly ASCII with just a few UTF-8 multi-byte characters. Real example (Kiwix-backed wiki day of pentecost):

MusÃ©e CondÃ©, Chantilly ... Â· Virgin birth Â· Crucifixion Â· Resurrection ...

That's classic UTF-8-decoded-as-Latin-1 mojibake (Musée -> MusÃ©e, · -> Â·). Since Kiwix ZIM content is consistently genuine UTF-8, guessing isn't necessary here.

Suggested fix

For #2, simplest fix - just force UTF-8 directly instead of guessing:

article_response.encoding = 'utf-8'

For #1, strip non-prose elements before extracting text, and prefer just the <p> tags (the actual lead paragraphs) over the entire div's text:

# Remove non-prose elements that pollute the summary: infoboxes,
# hatnotes/redirect notices, navigation boxes, citation markers, edit links.
for unwanted in main.select('table.infobox, table.sidebar, div.hatnote, div.navbox, div.metadata, sup.reference, .mw-editsection'):
    unwanted.decompose()

paragraphs = main.find_all('p')
if paragraphs:
    texts = [p.get_text(" ", strip=True) for p in paragraphs]
    texts = [t for t in texts if t]
    if texts:
        return " ".join(texts)

# fallback to existing full-text behavior if no <p> tags found
texts = main.find_all(string=True)
visible_texts = filter(tag_visible, texts)
return " ".join(t.strip() for t in visible_texts if t.strip())

I haven't been able to test this exact snippet against live Kiwix HTML myself (no direct access to run it here), so treat it as a starting point/one possible approach rather than a guaranteed drop-in - happy to test on my end once there's something to try.

Additional context

Both examples came from a Kiwix server running Wikipedia (Simple English) and other MediaWiki-based libraries, tested via both wiki and kiwix commands. The default online experience (no local Kiwix server) is unaffected by either issue.

**Why this matters** Mesh networks like Meshtastic are especially valuable specifically *because* they keep working when normal infrastructure doesn't - during power outages, storms, or other situations where cellular/internet service is down. Kiwix exists for exactly that same reason: it's widely used by preppers, off-grid communities, and disaster-response groups to keep Wikipedia, medical references, repair guides, and (relevant here) amateur radio references available with zero internet dependency. That means in the exact scenario where this bot is most useful - internet actually down - the online Wikipedia API fallback isn't available at all, and Kiwix becomes the *only* source, not just a backup. A garbled or citation-cluttered response might be a minor annoyance day-to-day, but it's a real problem if someone's trying to read a first-aid procedure, a repair guide, or FT8 operating etiquette during an actual emergency and the answer comes back as scrambled infobox text. So I think it's worth getting the Kiwix output path solid, even though it's not the default/most commonly hit code path day-to-day. **Describe the bug** Scope note: this only affects the **Kiwix/offline path** (`get_kiwix_summary()`), not the default online Wikipedia REST API path (`get_wikipedia_summary()`). The online path just reads Wikipedia's own pre-built `extract` field from JSON, which is already clean - so this only shows up for users running their own local Kiwix server with `useKiwixServer = True`. Two related issues with the text extracted by `text_from_html()` / `get_kiwix_summary()` in `modules/wiki.py`: **1. Non-prose content pollutes the summary** `text_from_html()` grabs every visible string inside the whole `mw-parser-output` div: ```python texts = main.find_all(string=True) visible_texts = filter(tag_visible, texts) return " ".join(t.strip() for t in visible_texts if t.strip()) ``` `tag_visible()` only filters `style/script/head/title/meta` tags, so it doesn't exclude infobox tables, hatnotes/redirect notices, or citation reference markers. These all get concatenated in with the actual lead paragraph. **Real example** (Kiwix-backed `kiwix Japan` or `wiki Japan` with a local Wikipedia ZIM): ``` "Nippon" redirects here. For other uses, see Japan (disambiguation) and Nippon (disambiguation). Japan ... Nippon-koku or Nihon-koku ... Flag Imperial crest Anthem: "Kimigayo" ... [3][4] Government crest of Japan State Seal of Japan Area controlled by Japan shown in green-claimed, but uncontrolled shown in light green Capital and largest city Tokyo [5] 35°41'N 139°46'E / ... ``` That's the article's hatnote + infobox sidebar table + citation markers, not the actual lead paragraph a reader would expect from a summary. **2. Encoding sometimes mis-detected, producing mojibake** ```python article_response.encoding = article_response.apparent_encoding or 'utf-8' ``` `apparent_encoding` is a statistical guess and can be wrong, especially on pages that are mostly ASCII with just a few UTF-8 multi-byte characters. Real example (Kiwix-backed `wiki day of pentecost`): ``` MusÃ©e CondÃ©, Chantilly ... Â· Virgin birth Â· Crucifixion Â· Resurrection ... ``` That's classic UTF-8-decoded-as-Latin-1 mojibake (`Musée` -> `MusÃ©e`, `·` -> `Â·`). Since Kiwix ZIM content is consistently genuine UTF-8, guessing isn't necessary here. **Suggested fix** For #2, simplest fix - just force UTF-8 directly instead of guessing: ```python article_response.encoding = 'utf-8' ``` For #1, strip non-prose elements before extracting text, and prefer just the `<p>` tags (the actual lead paragraphs) over the entire div's text: ```python # Remove non-prose elements that pollute the summary: infoboxes, # hatnotes/redirect notices, navigation boxes, citation markers, edit links. for unwanted in main.select('table.infobox, table.sidebar, div.hatnote, div.navbox, div.metadata, sup.reference, .mw-editsection'): unwanted.decompose() paragraphs = main.find_all('p') if paragraphs: texts = [p.get_text(" ", strip=True) for p in paragraphs] texts = [t for t in texts if t] if texts: return " ".join(texts) # fallback to existing full-text behavior if no <p> tags found texts = main.find_all(string=True) visible_texts = filter(tag_visible, texts) return " ".join(t.strip() for t in visible_texts if t.strip()) ``` I haven't been able to test this exact snippet against live Kiwix HTML myself (no direct access to run it here), so treat it as a starting point/one possible approach rather than a guaranteed drop-in - happy to test on my end once there's something to try. **Additional context** Both examples came from a Kiwix server running Wikipedia (Simple English) and other MediaWiki-based libraries, tested via both `wiki` and `kiwix` commands. The default online experience (no local Kiwix server) is unaffected by either issue.

adam closed this issue

2026-10-05 01:25:42 +02:00

adam commented

2026-10-05 01:25:43 +02:00

nice thanks, in place

nice thanks, in place

adam commented

2026-10-05 01:25:44 +02:00

Quick follow-up, and a mea culpa on my end: when I filed this, I didn't realize I'd already made some local, uncommitted changes to modules/wiki.py myself a while back (added a kiwix command with library_name support, so kiwix ham <term> can search a specific Kiwix library like Amateur Radio Stack Exchange). One of those local changes happened to include an apparent_encoding fallback I'd written trying to fix the same mojibake issue - so part of what I reported as "already attempted but guessing wrong" was actually my own code, not yours. Should have checked git diff/git status before writing the report - sorry for any confusion that caused.

Good news: your fix is exactly right, and I've now manually merged it together with my local additions (had a real merge conflict since both touched the same functions, resolved it by hand). Both work great together now - infobox/hatnote stripping, forced UTF-8, and the kiwix ham library routing all confirmed working after the merge.

Nothing further needed on your end for this one - just wanted to be upfront about the mix-up rather than leave you wondering. If you're ever interested in the kiwix <library> routing as a real feature (letting a query pick a specific Kiwix library by keyword, not just the default one), happy to clean it up and submit it properly as a PR sometime - it's been genuinely useful for offline ham-radio-specific lookups.

Thanks again for how quickly you turn these around!

Quick follow-up, and a mea culpa on my end: when I filed this, I didn't realize I'd already made some local, uncommitted changes to modules/wiki.py myself a while back (added a `kiwix` command with `library_name` support, so `kiwix ham <term>` can search a specific Kiwix library like Amateur Radio Stack Exchange). One of those local changes happened to include an `apparent_encoding` fallback I'd written trying to fix the same mojibake issue - so part of what I reported as "already attempted but guessing wrong" was actually my own code, not yours. Should have checked `git diff`/`git status` before writing the report - sorry for any confusion that caused. Good news: your fix is exactly right, and I've now manually merged it together with my local additions (had a real merge conflict since both touched the same functions, resolved it by hand). Both work great together now - infobox/hatnote stripping, forced UTF-8, and the `kiwix ham` library routing all confirmed working after the merge. Nothing further needed on your end for this one - just wanted to be upfront about the mix-up rather than leave you wondering. If you're ever interested in the `kiwix <library>` routing as a real feature (letting a query pick a specific Kiwix library by keyword, not just the default one), happy to clean it up and submit it properly as a PR sometime - it's been genuinely useful for offline ham-radio-specific lookups. Thanks again for how quickly you turn these around!

adam commented

2026-10-05 01:25:44 +02:00

Sure :)

Sure :)

adam commented

2026-10-05 01:25:45 +02:00

Here's the actual kiwix/library_name code, in case it's useful:

In mesh_bot.py, added to the command dictionary:

"kiwix": lambda: handle_kiwix(message),

And the handler itself:

def handle_kiwix(message):
    parts = message.split()
    if len(parts) < 2:
        return "Please add a search term example:\U0001F4F2kiwix ham ft8"

    search_term = " ".join(parts[1:])
    library_name = None

    # "ham" selects the Amateur Radio Q&A Kiwix library.
    if parts[1].lower() == "ham":
        if len(parts) < 3:
            return "Please add a search term example:\U0001F4F2kiwix ham ft8"
        search_term = " ".join(parts[2:])
        library_name = "ham.stackexchange.com_en_all_2026-02"

    return get_kiwix_summary(search_term, library_name=library_name)

In modules/wiki.py, get_kiwix_summary() just needed an optional library_name parameter that falls back to the configured default:

def get_kiwix_summary(search_term, truncate=True, library_name=None):
    ...
    library = library_name or kiwix_library_name
    search_url = f"{kiwix_url}/search?content={library}&pattern={search_encoded}"
    ...

Also added kiwix to the help text (modules/system.py, inside the existing if wikipedia_enabled or use_kiwix_server: block) so it's discoverable via cmd:

trap_list = trap_list + ("wiki", "kiwix",)
help_message = help_message + ", wiki"
help_message = help_message + ", kiwix"

Right now ham is the only hardcoded library shortcut, but the pattern's easy to extend if you ever wanted more (e.g. a repair/electronics shortcut for iFixit, for servers running a broader Kiwix library set) - just more if parts[1].lower() == "..." branches.

Here's the actual `kiwix`/`library_name` code, in case it's useful: **In `mesh_bot.py`**, added to the command dictionary: ```python "kiwix": lambda: handle_kiwix(message), ``` And the handler itself: ```python def handle_kiwix(message): parts = message.split() if len(parts) < 2: return "Please add a search term example:\U0001F4F2kiwix ham ft8" search_term = " ".join(parts[1:]) library_name = None # "ham" selects the Amateur Radio Q&A Kiwix library. if parts[1].lower() == "ham": if len(parts) < 3: return "Please add a search term example:\U0001F4F2kiwix ham ft8" search_term = " ".join(parts[2:]) library_name = "ham.stackexchange.com_en_all_2026-02" return get_kiwix_summary(search_term, library_name=library_name) ``` **In `modules/wiki.py`**, `get_kiwix_summary()` just needed an optional `library_name` parameter that falls back to the configured default: ```python def get_kiwix_summary(search_term, truncate=True, library_name=None): ... library = library_name or kiwix_library_name search_url = f"{kiwix_url}/search?content={library}&pattern={search_encoded}" ... ``` Also added `kiwix` to the help text (`modules/system.py`, inside the existing `if wikipedia_enabled or use_kiwix_server:` block) so it's discoverable via `cmd`: ```python trap_list = trap_list + ("wiki", "kiwix",) help_message = help_message + ", wiki" help_message = help_message + ", kiwix" ``` Right now `ham` is the only hardcoded library shortcut, but the pattern's easy to extend if you ever wanted more (e.g. a repair/electronics shortcut for iFixit, for servers running a broader Kiwix library set) - just more `if parts[1].lower() == "..."` branches.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adam/meshing-around#143