modpython string conversion issue #240

New Issue

2026-10-05T04:07:17+02:00

adam commented

2026-10-05 04:07:17 +02:00

When an extended ASCII character is sent and we process it through a method such as OnChanMsg(self, nick, channel, msg), a UnicodeDecodeError is thrown. Somewhere in the Swig process of converting the character from C++ to Python, a conversion from the byte string to unicode is made. The extended ascii characters not being valid unicode error out. You can repeat this easily by implementing a simple module and then sending characters such as:

¬ « € ÿ

An an example module:

def catchfail(fn):
    """Catch exceptions and get them onto the module channel."""
    def wrapper(*args, **kwargs):
        try:
            fn(*args, **kwargs)
        except Exception as e:
            s = _is_self(*args)
            if s:
                trace=traceback.format_exc().split("\n")
                for line in trace:
                    s.PutModule(str(line))
                s.PutModule("Failed with %s" % (e))
    return wrapper

class unicodeTest(znc.Module):
    @catchfail
    def OnChanMsg(self, nick, channel, msg):
        self.PutModule(str(dir(msg)))
        self.PutModule(str(msg.__class__))
        self.PutModule(str(msg.s))
        return znc.CONTINUE

The exception is:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

We should be prepared to decode the message as CP1252/ISO-8859-1.

Further information on IRC encoding can be found here: http://hinrik.net/irc-and-character-encoding/

When an extended ASCII character is sent and we process it through a method such as OnChanMsg(self, nick, channel, msg), a UnicodeDecodeError is thrown. Somewhere in the Swig process of converting the character from C++ to Python, a conversion from the byte string to unicode is made. The extended ascii characters not being valid unicode error out. You can repeat this easily by implementing a simple module and then sending characters such as: `¬ « € ÿ` An an example module: ``` python def catchfail(fn): """Catch exceptions and get them onto the module channel.""" def wrapper(*args, **kwargs): try: fn(*args, **kwargs) except Exception as e: s = _is_self(*args) if s: trace=traceback.format_exc().split("\n") for line in trace: s.PutModule(str(line)) s.PutModule("Failed with %s" % (e)) return wrapper class unicodeTest(znc.Module): @catchfail def OnChanMsg(self, nick, channel, msg): self.PutModule(str(dir(msg))) self.PutModule(str(msg.__class__)) self.PutModule(str(msg.s)) return znc.CONTINUE ``` The exception is: ``` UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte ``` We should be prepared to decode the message as CP1252/ISO-8859-1. Further information on IRC encoding can be found here: http://hinrik.net/irc-and-character-encoding/

adam closed this issue

2026-10-05 04:07:17 +02:00

adam commented

2026-10-05 04:07:18 +02:00

That should be handled not in modpython, but in ZNC core, see #151
Then modpython won't fail to convert proper UTF-8 string to python string

That should be handled not in modpython, but in ZNC core, see #151 Then modpython won't fail to convert proper UTF-8 string to python string

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adam/znc#240