messageChunker: preserve blank line separators as --- instead of discarding them #134

New Issue

2026-10-05T01:25:18+02:00

adam commented

2026-10-05 01:25:18 +02:00

When using multi-section news files with readnews, blank lines used to separate sections are silently discarded by messageChunker, causing section headers to run together with the preceding content in the same message chunk.

Proposed fix in modules/system.py:

Replace the blank line discard in the chunker:

if not part:
continue

with:

if not part:
if message_list and message_list[-1] != '---':
message_list.append('---')
continue

And prevent the consolidation loop from merging across separators and section headers:

if message_list[idx] == '---' or message_list[idx+1] == '---'
or message_list[idx].endswith(':') or message_list[idx+1].endswith(':'):
idx += 1

Result: Blank lines in source files become visible --- separators in mesh output, and section headers are kept with their content rather than merged with the preceding chunk.

Tested locally on an Ionos Ubuntu VPS running the bot in SimRadio/MQTT mode.

When using multi-section news files with readnews, blank lines used to separate sections are silently discarded by messageChunker, causing section headers to run together with the preceding content in the same message chunk. Proposed fix in modules/system.py: Replace the blank line discard in the chunker: if not part: continue with: if not part: if message_list and message_list[-1] != '---': message_list.append('---') continue And prevent the consolidation loop from merging across separators and section headers: if message_list[idx] == '---' or message_list[idx+1] == '---' \ or message_list[idx].endswith(':') or message_list[idx+1].endswith(':'): idx += 1 Result: Blank lines in source files become visible --- separators in mesh output, and section headers are kept with their content rather than merged with the preceding chunk. Tested locally on an Ionos Ubuntu VPS running the bot in SimRadio/MQTT mode.

adam commented

2026-10-05 01:25:19 +02:00

Here's the script that motivated this fix — it fetches Wikipedia's "In the News" and "On This Day" sections daily via cron and writes them to the bot's news file. Others might find it useful as a starting point for their own news sources.

Place it in your home directory as ~/fetch-wiki-daily.sh and make it executable with chmod +x ~/fetch-wiki-daily.sh. Requires beautifulsoup4 (pip install beautifulsoup4). Add to cron with crontab -e:

0 6 * * * /bin/bash /root/fetch-wiki-daily.sh

#!/bin/bash
# ~/fetch-wiki-daily.sh
# Fetches Wikipedia "In the News" and "On This Day" sections
# and writes them to the meshing-around news file.

python3 - <<'EOF'
import os
import re
import requests
from bs4 import BeautifulSoup
from datetime import date

def clean(text):
    # Remove citation markers like [ 2 ], [60], etc.
    text = re.sub(r'\s*\[\s*\d+\s*\]', '', text)
    # Remove spaces before punctuation
    text = re.sub(r'\s+([.,;:])', r'\1', text)
    return text.strip()

out = []
headers = {"User-Agent": "Mozilla/5.0"}
today = date.today()

# --- IN THE NEWS ---
r = requests.get(
    "https://en.wikipedia.org/w/api.php",
    params={
        "action": "parse",
        "page": "Portal:Current_events",
        "prop": "text",
        "format": "json"
    },
    headers=headers,
    timeout=10
)
html = r.json()["parse"]["text"]["*"]
soup = BeautifulSoup(html, 'html.parser')

out.append("IN THE NEWS:")
in_news = False
count = 0
for tag in soup.find_all(['h2', 'ul']):
    if tag.name == 'h2' and 'Topics in the news' in tag.get_text():
        in_news = True
        continue
    if tag.name == 'h2' and in_news:
        break
    if in_news and tag.name == 'ul':
        for li in tag.find_all('li', recursive=False):
            text = li.get_text(" ", strip=True)
            if len(text) > 40 and ('–' in text or '.' in text or ',' in text):
                out.append("- " + clean(text)[:220])
                count += 1
            if count >= 5:
                break
    if count >= 5:
        break

# --- ON THIS DAY ---
url = f"https://en.wikipedia.org/api/rest_v1/feed/featured/{today.year}/{today.month:02d}/{today.day:02d}"
r2 = requests.get(url, headers=headers, timeout=10)
onthisday = r2.json().get("onthisday", [])

out.append("")
out.append("ON THIS DAY:")
for item in onthisday[:5]:
    year = item.get("year", "")
    text = clean(item.get("text", ""))
    out.append(f"- {year} - {text[:220]}")

output_path = os.path.expanduser("~/meshing-around/data/news.txt")
with open(output_path, "w") as f:
    f.write("\n".join(out) + "\n")

print(f"Written to {output_path}")
EOF

<!-- github-comment-id:4433558254 -->

Here's the script that motivated this fix — it fetches Wikipedia's "In the News" and "On This Day" sections daily via cron and writes them to the bot's news file. Others might find it useful as a starting point for their own news sources. Place it in your home directory as ~/fetch-wiki-daily.sh and make it executable with chmod +x ~/fetch-wiki-daily.sh. Requires beautifulsoup4 (pip install beautifulsoup4). Add to cron with crontab -e: 0 6 * * * /bin/bash /root/fetch-wiki-daily.sh ``` #!/bin/bash # ~/fetch-wiki-daily.sh # Fetches Wikipedia "In the News" and "On This Day" sections # and writes them to the meshing-around news file. python3 - <<'EOF' import os import re import requests from bs4 import BeautifulSoup from datetime import date def clean(text): # Remove citation markers like [ 2 ], [60], etc. text = re.sub(r'\s*\[\s*\d+\s*\]', '', text) # Remove spaces before punctuation text = re.sub(r'\s+([.,;:])', r'\1', text) return text.strip() out = [] headers = {"User-Agent": "Mozilla/5.0"} today = date.today() # --- IN THE NEWS --- r = requests.get( "https://en.wikipedia.org/w/api.php", params={ "action": "parse", "page": "Portal:Current_events", "prop": "text", "format": "json" }, headers=headers, timeout=10 ) html = r.json()["parse"]["text"]["*"] soup = BeautifulSoup(html, 'html.parser') out.append("IN THE NEWS:") in_news = False count = 0 for tag in soup.find_all(['h2', 'ul']): if tag.name == 'h2' and 'Topics in the news' in tag.get_text(): in_news = True continue if tag.name == 'h2' and in_news: break if in_news and tag.name == 'ul': for li in tag.find_all('li', recursive=False): text = li.get_text(" ", strip=True) if len(text) > 40 and ('–' in text or '.' in text or ',' in text): out.append("- " + clean(text)[:220]) count += 1 if count >= 5: break if count >= 5: break # --- ON THIS DAY --- url = f"https://en.wikipedia.org/api/rest_v1/feed/featured/{today.year}/{today.month:02d}/{today.day:02d}" r2 = requests.get(url, headers=headers, timeout=10) onthisday = r2.json().get("onthisday", []) out.append("") out.append("ON THIS DAY:") for item in onthisday[:5]: year = item.get("year", "") text = clean(item.get("text", "")) out.append(f"- {year} - {text[:220]}") output_path = os.path.expanduser("~/meshing-around/data/news.txt") with open(output_path, "w") as f: f.write("\n".join(out) + "\n") print(f"Written to {output_path}") EOF

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adam/meshing-around#134