Feature request: Allow wiki search to local wiki. For example kiwix server #47

New Issue

Closed

opened 2026-10-05 01:17:46 +02:00 by adam · 9 comments

adam commented 2026-10-05 01:17:46 +02:00

Owner

Copy link

Copy source

Since the meshtastic project is meant to be offline it would be nice to be able to interact with a local wikipedia server(kiwix server) for wiki searches instead of using wikipedia. https://kiwix.org/en/
https://kiwix-tools.readthedocs.io/en/latest/kiwix-serve.html#http-api

Since the meshtastic project is meant to be offline it would be nice to be able to interact with a local wikipedia server(kiwix server) for wiki searches instead of using wikipedia. https://kiwix.org/en/ https://kiwix-tools.readthedocs.io/en/latest/kiwix-serve.html#http-api <!-- github-sync-id:2846612789 -->

adam closed this issue 2026-10-05 01:17:46 +02:00

adam commented 2026-10-05 01:17:47 +02:00

Author

Owner

Copy link

Copy source

I agree, I have looked into it a bit. I haven't found an awesome way to handle it all yet. I'm all ears for ideas here.

Currently my stumbling block to make it "turn key easy" is that the data presented is very unstructured.

Not to mention getting relevant search terms which match without user interaction.

Both items really need passed thru a LLM and this is costly obviously.

I'm looking for examples of deployment models which are not a "custom sd card image" but a handy example in like a single .py

Remeber we have very little room for verbose results

I agree, I have looked into it a bit. I haven't found an awesome way to handle it all yet. I'm all ears for ideas here. Currently my stumbling block to make it "turn key easy" is that the data presented is very unstructured. Not to mention getting relevant search terms which match without user interaction. Both items really need passed thru a LLM and this is costly obviously. I'm looking for examples of deployment models which are not a "custom sd card image" but a handy example in like a single .py Remeber we have very little room for verbose results <!-- github-comment-id:2652622899 -->

adam commented 2026-10-05 01:17:48 +02:00

Author

Owner

Copy link

Copy source

Both items really need passed thru a LLM and this is costly obviously.

But this could be ollama right?

I'm looking for examples of deployment models which are not a "custom sd card image" but a handy example in like a single .py

What do you mean by this? Are you looking for a way to automate deploying the Kiwix server itself? I would say default to the real Wikipedia and enabling local Wiki/Kiwix with a config option. Another option for the IP and port of the kiwix server. I think for actually deploying the whole kiwix server I think it's just a docker command docker run -v /docker/kiwixserve:/data -p 8080:80 -e 'DOWNLOAD'='https://download.kiwix.org/zim/wikipedia/wikipedia_en_all_maxi_2024-01.zim' kiwix/kiwix-serve *.zim

Remember we have very little room for verbose results

The current way with regular Wikipedia is sending multiple messages now, I think that could also work for this version right?

I'm all ears for ideas here.

Alright I started with a little script for interacting with the Kiwix server. Now I am in no way claiming to be anything more than just a noob so I am open to feedback. But what do you think? Going in the right direction?

import requests
from bs4 import BeautifulSoup
from urllib.parse import quote
from bs4.element import Comment

def tag_visible(element):
    if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:
        return False
    if isinstance(element, Comment):
        return False
    return True
def text_from_html(body):
    soup = BeautifulSoup(body, 'html.parser')
    texts = soup.findAll(string=True)
    visible_texts = filter(tag_visible, texts)  
    return u" ".join(t.strip() for t in visible_texts)

search = input("What would you like to search for?") # this would ofcourse come from the meshtastic message.
kiwix_url = "<IP FROM THE CONFIG.INI>" # needs to be replaced by the config ip
search_encoded = quote(search)
wiki = search_encoded.capitalize().replace("%20", "_")
exact_url = f"{kiwix_url}/raw/wikipedia_en_all_maxi_2024-01/content/A/{wiki}" # maybe the wiki name needs to be configurable?
alternatives = []

search_result = requests.get(exact_url)
if search_result.status_code == 200:
    print(text_from_html(search_result.text))
else:
    url = f"{kiwix_url}/search?content=wikipedia_en_all_maxi_2024-01&pattern={search_encoded}&books.filter.category=wikipedia"
    response = requests.get(url)

    if response.status_code == 200:
        if not response.text.__contains__("No results were found for"):
            soup = BeautifulSoup(response.text, 'html.parser')

            links = [a['href'] for a in soup.find_all('a', href=True)]

            for link in links:
                if not link.__contains__("start="):
                    wiki_article = link.split("/")[-1]
                    article_search = search_encoded.capitalize().replace("%20", "_")

                    if wiki_article == article_search:
                        article_found = True
                        search_result = (requests.get(f"{kiwix_url}{link}").text)
                            
                        print(text_from_html(search_result))
                        break
                    else:
                        article_found = False
                        alternatives.append(wiki_article)
            if article_found == False:
                print(f"Here is a list of alternatives {alternatives}")

        else:
            print("Article not found. No other suggestions found.")

    else:
        print("Failed to load the local kiwix server.")

> Both items really need passed thru a LLM and this is costly obviously. But this could be ollama right? > I'm looking for examples of deployment models which are not a "custom sd card image" but a handy example in like a single .py What do you mean by this? Are you looking for a way to automate deploying the Kiwix server itself? I would say default to the real Wikipedia and enabling local Wiki/Kiwix with a config option. Another option for the IP and port of the kiwix server. I think for actually deploying the whole kiwix server I think it's just a docker command ```docker run -v /docker/kiwixserve:/data -p 8080:80 -e 'DOWNLOAD'='https://download.kiwix.org/zim/wikipedia/wikipedia_en_all_maxi_2024-01.zim' kiwix/kiwix-serve *.zim``` > Remember we have very little room for verbose results The current way with regular Wikipedia is sending multiple messages now, I think that could also work for this version right? > I'm all ears for ideas here. Alright I started with a little script for interacting with the Kiwix server. Now I am in no way claiming to be anything more than just a noob so I am open to feedback. But what do you think? Going in the right direction? ```python import requests from bs4 import BeautifulSoup from urllib.parse import quote from bs4.element import Comment def tag_visible(element): if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']: return False if isinstance(element, Comment): return False return True def text_from_html(body): soup = BeautifulSoup(body, 'html.parser') texts = soup.findAll(string=True) visible_texts = filter(tag_visible, texts) return u" ".join(t.strip() for t in visible_texts) search = input("What would you like to search for?") # this would ofcourse come from the meshtastic message. kiwix_url = "<IP FROM THE CONFIG.INI>" # needs to be replaced by the config ip search_encoded = quote(search) wiki = search_encoded.capitalize().replace("%20", "_") exact_url = f"{kiwix_url}/raw/wikipedia_en_all_maxi_2024-01/content/A/{wiki}" # maybe the wiki name needs to be configurable? alternatives = [] search_result = requests.get(exact_url) if search_result.status_code == 200: print(text_from_html(search_result.text)) else: url = f"{kiwix_url}/search?content=wikipedia_en_all_maxi_2024-01&pattern={search_encoded}&books.filter.category=wikipedia" response = requests.get(url) if response.status_code == 200: if not response.text.__contains__("No results were found for"): soup = BeautifulSoup(response.text, 'html.parser') links = [a['href'] for a in soup.find_all('a', href=True)] for link in links: if not link.__contains__("start="): wiki_article = link.split("/")[-1] article_search = search_encoded.capitalize().replace("%20", "_") if wiki_article == article_search: article_found = True search_result = (requests.get(f"{kiwix_url}{link}").text) print(text_from_html(search_result)) break else: article_found = False alternatives.append(wiki_article) if article_found == False: print(f"Here is a list of alternatives {alternatives}") else: print("Article not found. No other suggestions found.") else: print("Failed to load the local kiwix server.") ``` <!-- github-comment-id:2654852202 -->

adam commented 2026-10-05 01:17:48 +02:00

Author

Owner

Copy link

Copy source

here is a little refactor of that same idea, I am still stuck on the data returned is really hard to manage my terminal buffer was exceeded searching Elvis haha

import requests
from bs4 import BeautifulSoup
from urllib.parse import quote
from bs4.element import Comment

# kiwxi server url and the title of the library item you want to search in
kiwix_url = "http://127.0.0.1:8080"
wiki_title = "wikipedia_en_100_nopic_2024-06"

search = input("What would you like to search for? ")
search_encoded = quote(search)
wiki = search_encoded.capitalize().replace("%20", "_")
exact_url = f"{kiwix_url}/raw/{wiki_title}/content/A/{wiki}"
alternatives = []

def tag_visible(element):
    if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:
        return False
    if isinstance(element, Comment):
        return False
    return True
    
def text_from_html(body):
    soup = BeautifulSoup(body, 'html.parser')
    texts = soup.findAll(string=True)
    visible_texts = filter(tag_visible, texts)  
    return u" ".join(t.strip() for t in visible_texts)

def fetch_url(url):
    try:
        response = requests.get(url)
        response.raise_for_status()
        return response
    except requests.RequestException as e:
        return None

def search_alternatives():
    url = f"{kiwix_url}/search?content={wiki_title}&pattern={search_encoded}&books.filter.category=wikipedia"
    response = fetch_url(url)
    if response and response.status_code == 200:
        if "No results were found for" not in response.text:
            soup = BeautifulSoup(response.text, 'html.parser')
            links = [a['href'] for a in soup.find_all('a', href=True)]
            for link in links:
                if "start=" not in link:
                    wiki_article = link.split("/")[-1]
                    article_search = search_encoded.capitalize().replace("%20", "_")
                    if wiki_article == article_search:
                        search_result = fetch_url(f"{kiwix_url}{link}")
                        if search_result:
                            print(text_from_html(search_result.text))
                        return
                    else:
                        # ignore blank links
                        if wiki_article == "":
                            continue
                        # ignore links that are not articles
                        if wiki_article[0].islower():
                            continue
                        alternatives.append(wiki_article)
            if not alternatives:
                print("Article not found. No other suggestions found.")
            else:
                print(f"Here is a list of alternatives {alternatives}")
        else:
            print("Article not found. No other suggestions found.")
    else:
        print("Failed to load the local kiwix server.")

search_result = fetch_url(exact_url)
if search_result and search_result.status_code == 200:
    print(text_from_html(search_result.text))
else:
    search_alternatives()

here is a little refactor of that same idea, I am still stuck on the data returned is really hard to manage my terminal buffer was exceeded searching Elvis haha ```python import requests from bs4 import BeautifulSoup from urllib.parse import quote from bs4.element import Comment # kiwxi server url and the title of the library item you want to search in kiwix_url = "http://127.0.0.1:8080" wiki_title = "wikipedia_en_100_nopic_2024-06" search = input("What would you like to search for? ") search_encoded = quote(search) wiki = search_encoded.capitalize().replace("%20", "_") exact_url = f"{kiwix_url}/raw/{wiki_title}/content/A/{wiki}" alternatives = [] def tag_visible(element): if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']: return False if isinstance(element, Comment): return False return True def text_from_html(body): soup = BeautifulSoup(body, 'html.parser') texts = soup.findAll(string=True) visible_texts = filter(tag_visible, texts) return u" ".join(t.strip() for t in visible_texts) def fetch_url(url): try: response = requests.get(url) response.raise_for_status() return response except requests.RequestException as e: return None def search_alternatives(): url = f"{kiwix_url}/search?content={wiki_title}&pattern={search_encoded}&books.filter.category=wikipedia" response = fetch_url(url) if response and response.status_code == 200: if "No results were found for" not in response.text: soup = BeautifulSoup(response.text, 'html.parser') links = [a['href'] for a in soup.find_all('a', href=True)] for link in links: if "start=" not in link: wiki_article = link.split("/")[-1] article_search = search_encoded.capitalize().replace("%20", "_") if wiki_article == article_search: search_result = fetch_url(f"{kiwix_url}{link}") if search_result: print(text_from_html(search_result.text)) return else: # ignore blank links if wiki_article == "": continue # ignore links that are not articles if wiki_article[0].islower(): continue alternatives.append(wiki_article) if not alternatives: print("Article not found. No other suggestions found.") else: print(f"Here is a list of alternatives {alternatives}") else: print("Article not found. No other suggestions found.") else: print("Failed to load the local kiwix server.") search_result = fetch_url(exact_url) if search_result and search_result.status_code == 200: print(text_from_html(search_result.text)) else: search_alternatives() ``` <!-- github-comment-id:2655414701 -->

adam commented 2026-10-05 01:17:49 +02:00

Author

Owner

Copy link

Copy source

print(text_from_html(search_result.text))

print(text_from_html(search_result.text)[:200])

What about changing it to this? It will limit the output to 200 characters.

> print(text_from_html(search_result.text)) print(text_from_html(search_result.text)[:200]) What about changing it to this? It will limit the output to 200 characters. <!-- github-comment-id:2658645807 -->

adam commented 2026-10-05 01:17:50 +02:00

Author

Owner

Copy link

Copy source

The problem with this look up some stuff on Wikipedia.zim

The first 200 characters are a lot of time is stuff which isn't like Wikipedia.com where it's a paragraph of the topic on point.

I seem to find a lot of cataloging and library data before the real context of value. how is the ocean boiled down to value of 200 characters with high confidence?

And we are only using Wikipedia, I feel like, at least that, and ifixit.zim and ideally some popular off grid book of knowledge like medical and plants.

I do think, it demands LLM So it would be a dependency and used as a offline query context. The problem of note here is a pi5 while it will work. If I ask "who is Elvis" the return from wiki for example is a lot of compute to chunk. And it's only as good as the LLM which can be crap a lot of times as well.

The problem with this look up some stuff on Wikipedia.zim The first 200 characters are a lot of time is stuff which isn't like Wikipedia.com where it's a paragraph of the topic on point. I seem to find a lot of cataloging and library data before the real context of value. how is the ocean boiled down to value of 200 characters with high confidence? And we are only using Wikipedia, I feel like, at least that, and ifixit.zim and ideally some popular off grid book of knowledge like medical and plants. I do think, it demands LLM So it would be a dependency and used as a offline query context. The problem of note here is a pi5 while it will work. If I ask "who is Elvis" the return from wiki for example is a lot of compute to chunk. And it's only as good as the LLM which can be crap a lot of times as well. <!-- github-comment-id:2660679919 -->

adam commented 2026-10-05 01:17:50 +02:00

Author

Owner

Copy link

Copy source

I would recommend using the ‘Wikipedia in simple English mini’ Kiwix image instead of the maxi file that I see discussed above. The articles are already pared down to include much less text which should make finding the most valuable content to present over Meshtastic much easier. This file is substantially smaller at 417.5MB and will allow for leaner resource consumption and potentially better performance.

I would recommend using the ‘Wikipedia in simple English mini’ Kiwix image instead of the maxi file that I see discussed above. The articles are already pared down to include much less text which should make finding the most valuable content to present over Meshtastic much easier. This file is substantially smaller at 417.5MB and will allow for leaner resource consumption and potentially better performance. <!-- github-comment-id:2843956084 -->

adam commented 2026-10-05 01:17:51 +02:00

Author

Owner

Copy link

Copy source

I have tried this file but the data returned is still incredibly verbose

And! Search terms need to be handled still.

So keeping this all in reasonable response time (under 45seconds ideally)

And also, keeping it under 300 characters or so for meshtastic

The existing bot wiki plugin is using the cloud for search handling and summary

I have tried this file but the data returned is still incredibly verbose And! Search terms need to be handled still. So keeping this all in reasonable response time (under 45seconds ideally) And also, keeping it under 300 characters or so for meshtastic The existing bot wiki plugin is using the cloud for search handling and summary <!-- github-comment-id:2844191069 -->

adam commented 2026-10-05 01:17:52 +02:00

Author

Owner

Copy link

Copy source

https://github.com/SpudGunMan/meshing-around/pull/203

This is a nice idea can it be refactored into a module/wiki.py?

https://github.com/SpudGunMan/meshing-around/pull/203 This is a nice idea can it be refactored into a module/wiki.py? <!-- github-comment-id:3383221288 -->

adam commented 2026-10-05 01:17:52 +02:00

Author

Owner

Copy link

Copy source

I am thrilled. I look forward to testing this functionality among other recently released enhancements.

I am thrilled. I look forward to testing this functionality among other recently released enhancements. <!-- github-comment-id:3385396404 -->

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

dependabot/github_actions/docker/login-action-4.7.0

copilot/refactor-tide-command-xtide

2.0.0.1

v2.0.0.0

v1.9.9.9

v1.9.9.8

v1.9.9.7

v1.9.9.6a

v1.9.9.6

v1.9.9.5

v1.9.9.4

v1.9.9.3

v1.9.9.1

v1.9.9.0

v1.9.8.9.1

v1.9.8.9

v1.9.8.8

v1.9.8.7

v1.9.8.6

v1.9.8.5f

v1.9.8.5d

v1.9.8.5

v1.9.8.4

v1.9.8.3

v1.9.8.2

v1.9.8.1

v1.9.8

V1.9.7.7.7.7.7

v1.9.7.7.7.7

v1.9.7.7.7

v1.9.7.4

v1.9.7.3

v1.9.7.2

v1.9.7.1

v1.9.7

v1.9.6

v1.9.5

v1.9

v1.8.0.1

v1.8.0

v1.7.9

v1.7.7

v1.7.6

v1.7.5

v1.7.4

v1.7.3

v1.7.2

v1.7.1

v1.7.0

v1.6.9

v1.6.7

v1.6.5

v1.6.2

v1.6.0

v1.5.9

v1.5.8

v1.5.0

v1.4.9

v1.4.8hotfix

v1.4.8

v1.4.7

v1.4.5

v1.4

v1.3.3

v1.2.3

v1.2

v1.1.4

v1.1.2

v1.1.1

v1.1.0

v1.0.0

v1.0.0-beta

v1.0.0-alpha

RC10

RC9

RC8

RC7

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

adam (Adam Wojcieszonek)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adam/meshing-around#47