adam/znc
1
0 Fork 0
mirror of https://github.com/znc/znc.git synced 2026-10-31 01:32:02 +01:00
Code Issues 362 Packages Projects Releases Wiki Activity

ZNC possibly not decoding ISO-2022-JP when used together with UTF-8 #804

New Issue
Open
opened 2026-10-05 04:45:33 +02:00 by adam · 8 comments
adam commented 2026-10-05 04:45:33 +02:00
Owner
Copy link
Copy source

Trying to connect to irc.friend-chat.jp:6664 with znc as a bouncer but could not set it up to decode ISO-2022-JP properly. I have searched for similar issues here and the most common mistake people make is the Network Encoding setup, and here is my setup.

<qria> getnetwork encoding qria friend-chat
*<*controlpanel> Encoding = ^iso-2022-jp

Client side encoding is set up to UTF-8

<qria> get clientencoding qria
*<*controlpanel> ClientEncoding = UTF-8

ZNC Version:

ZNC 1.6.5 - http://znc.in
IPv6: yes, SSL: yes, DNS: threads, charset: yes

Here is the MOTD with current ZNC setup

- irc.xseed.ne.jp Message of the Day -
- 13/6/2017 10:55
- -------------------------------------------------------
- Welcome to IRC server of irc.nrc.xseed.ne.jp
-
- $B$3$N(BIRC$B%5!<%P!<$O(B XSEED $B1?MQIt$K$h$j1?1D$5$l$F$$$^$9!%(B
- $B$3$N%5!<%P!<$O(B irc.friend-chat.jp $B7O$N(BIRC$B%5!<%P!<72$H(B
- $B%j%l!<$7$F$$$^$9!%(B
-
- $B7R$,$j$K$/$$>l9g$O%]!<%H$r(B6660-6669$B$N4V$GJQ$($F$/$@$5$$!%(B
- ($B%G%U%)%k%H%]!<%H(B6667$B$OHr$1$F$/$@$5$$(B)
-
- $B4IM}>e$NH=CG$K$h$C$F<o!9$N@\B3@)8B$r9T$J$&$3$H$,$"$j$^$9!%(B
-
- $B3$30$+$i$NBgNL@\B3$KBP=h$9$k$?$a$K!$3$30$N(BIP$B%"%I%l%9$G(B
- $B@\B3$7$?>l9g$O!$@)8B$5$l$?@\B3(B(umode +R)$B$K$J$j$^$9!%(B
- $B$3$N@)8B$G$*:$$j$NJ}$O(B #$B4IM}<T(B $B$K$F$4AjCL$/$@$5$$!%(B
-
- 2017/04/05
-  $B%5!<%P@_CV8D=j$rBg:e$+$iEl5~$KJQ99$7$^$7$?!#(B
-
- $B;29M(B :
-  XSEED IRC
-    http://www.xseed.ne.jp/irc/
-  Friend Chat IRC
-    http://www.friend-chat.jp/
-
-                                    Since 2006/10/05
-                              Last Update 2017/04/05
- -------------------------------------------------------

With irssi /set recode_fallback iso-2022-jp option

19:16 -!- - irc.saiin.net Message of the Day -
19:16 -!- - 7/2/2014 9:10
19:16 -!- - ◆Friend Chat IRC ドメイン変更のお知らせ◆
19:16 -!- - 接続先を irc.friend.td.nu から irc.friend-chat.jp にご変更下さい
19:16 -!- -
19:16 -!- -
19:16 -!- - ********** [ irc.friend.td.nu ==> irc.friend-chat.jp ] **********
19:16 -!- - Please change your setting!
19:16 -!- - We changed the hostname from "irc.friend.td.nu" to "irc.friend-chat.jp".
19:16 -!- -
19:16 -!- - ============================================
19:16 -!- - ようこそ、Friend Chat IRC/西院ネットワークスIRCサーバーへ！
19:16 -!- - irc.saiin.net 6660-6669で稼動しています。
19:16 -!- - このサーバーは、Friend Chat IRCを構成するサーバーのうちの1台です
19:16 -!- -
19:16 -!- -
19:16 -!- - 負荷分散のため、以下のサーバー名を指定して接続下さい。
19:16 -!- - $B!J<+F0E*$K(BFriend Chat IRC$B$N%5!<%P!<72$N(B1$BBf$K?6$jJ,$1$i$l$^$9!K(
19:16 -!- - BB
19:16 -!- -   irc.friend-chat.jp (port 6660-6669)
19:16 -!- - ●Friend Chat IRC
19:16 -!- -   http://www.friend-chat.jp/
19:16 -!- - ●西院ネットワークス
19:16 -!- -   http://www.saiin.net/
19:16 -!- - それでは、楽しい時間をお過ごし下さい。
19:16 -!- - ============================================
19:16 -!- - Welcome to the Friend Chat IRC!
19:16 -!- - Please connect to irc.friend-chat.jp (port 6660-6669).
19:16 -!- -   http://www.friend-chat.jp/
19:16 -!- - ============================================
19:16 -!- End of MOTD command.
19:16 -!- Mode change [+ir] for user qria
19:16 -!- Irssi: Disconnecting from server znc.qria.net: [leaving]
19:16 -!- Irssi: Connection lost to znc.qria.net

Any way I can further debug this?

Trying to connect to `irc.friend-chat.jp:6664` with znc as a bouncer but could not set it up to decode `ISO-2022-JP` properly. I have searched for similar issues here and the most common mistake people make is the Network Encoding setup, and here is my setup. > <<markupdoesntworksothisisaworkaround>qria> getnetwork encoding qria friend-chat > *<*controlpanel> Encoding = ^iso-2022-jp Client side encoding is set up to `UTF-8` ><<loremipsum>qria> get clientencoding qria > *<*controlpanel> ClientEncoding = UTF-8 ZNC Version: ZNC 1.6.5 - http://znc.in IPv6: yes, SSL: yes, DNS: threads, charset: yes Here is the MOTD with current ZNC setup ``` - irc.xseed.ne.jp Message of the Day - - 13/6/2017 10:55 - ------------------------------------------------------- - Welcome to IRC server of irc.nrc.xseed.ne.jp - - $B$3$N(BIRC$B%5!<%P!<$O(B XSEED $B1?MQIt$K$h$j1?1D$5$l$F$$$^$9!%(B - $B$3$N%5!<%P!<$O(B irc.friend-chat.jp $B7O$N(BIRC$B%5!<%P!<72$H(B - $B%j%l!<$7$F$$$^$9!%(B - - $B7R$,$j$K$/$$>l9g$O%]!<%H$r(B6660-6669$B$N4V$GJQ$($F$/$@$5$$!%(B - ($B%G%U%)%k%H%]!<%H(B6667$B$OHr$1$F$/$@$5$$(B) - - $B4IM}>e$NH=CG$K$h$C$F<o!9$N@\B3@)8B$r9T$J$&$3$H$,$"$j$^$9!%(B - - $B3$30$+$i$NBgNL@\B3$KBP=h$9$k$?$a$K!$3$30$N(BIP$B%"%I%l%9$G(B - $B@\B3$7$?>l9g$O!$@)8B$5$l$?@\B3(B(umode +R)$B$K$J$j$^$9!%(B - $B$3$N@)8B$G$*:$$j$NJ}$O(B #$B4IM}<T(B $B$K$F$4AjCL$/$@$5$$!%(B - - 2017/04/05 - $B%5!<%P@_CV8D=j$rBg:e$+$iEl5~$KJQ99$7$^$7$?!#(B - - $B;29M(B : - XSEED IRC - http://www.xseed.ne.jp/irc/ - Friend Chat IRC - http://www.friend-chat.jp/ - - Since 2006/10/05 - Last Update 2017/04/05 - ------------------------------------------------------- ``` With irssi `/set recode_fallback iso-2022-jp` option ``` 19:16 -!- - irc.saiin.net Message of the Day - 19:16 -!- - 7/2/2014 9:10 19:16 -!- - ◆Friend Chat IRC ドメイン変更のお知らせ◆ 19:16 -!- - 接続先を irc.friend.td.nu から irc.friend-chat.jp にご変更下さい 19:16 -!- - 19:16 -!- - 19:16 -!- - ********** [ irc.friend.td.nu ==> irc.friend-chat.jp ] ********** 19:16 -!- - Please change your setting! 19:16 -!- - We changed the hostname from "irc.friend.td.nu" to "irc.friend-chat.jp". 19:16 -!- - 19:16 -!- - ============================================ 19:16 -!- - ようこそ、Friend Chat IRC/西院ネットワークスIRCサーバーへ！ 19:16 -!- - irc.saiin.net 6660-6669で稼動しています。 19:16 -!- - このサーバーは、Friend Chat IRCを構成するサーバーのうちの1台です 19:16 -!- - 19:16 -!- - 19:16 -!- - 負荷分散のため、以下のサーバー名を指定して接続下さい。 19:16 -!- - $B!J<+F0E*$K(BFriend Chat IRC$B$N%5!<%P!<72$N(B1$BBf$K?6$jJ,$1$i$l$^$9!K( 19:16 -!- - BB 19:16 -!- - irc.friend-chat.jp (port 6660-6669) 19:16 -!- - ●Friend Chat IRC 19:16 -!- - http://www.friend-chat.jp/ 19:16 -!- - ●西院ネットワークス 19:16 -!- - http://www.saiin.net/ 19:16 -!- - それでは、楽しい時間をお過ごし下さい。 19:16 -!- - ============================================ 19:16 -!- - Welcome to the Friend Chat IRC! 19:16 -!- - Please connect to irc.friend-chat.jp (port 6660-6669). 19:16 -!- - http://www.friend-chat.jp/ 19:16 -!- - ============================================ 19:16 -!- End of MOTD command. 19:16 -!- Mode change [+ir] for user qria 19:16 -!- Irssi: Disconnecting from server znc.qria.net: [leaving] 19:16 -!- Irssi: Connection lost to znc.qria.net ``` Any way I can further debug this? <!-- github-sync-id:285404183 -->
adam commented 2026-10-05 04:45:34 +02:00
Author
Owner
Copy link
Copy source

Yep, ^iso-2022-jp does not work, but iso-2022-jp does, and even provides better results than your paste from irssi - one of strings there is malformed: （自動的にFriend Chat IRCのサーバー群の1台に振り分けられます）��] - note the � at the end.

^ option ("Send as UTF-8, receive as UTF-8 and ____") in ZNC assumes that the non-utf8 encoding is 8-bit, and uses characters outside of ASCII range. E.g. ISO-8859-15 and KOI8-R work that way.
^ follows a simple logic: if the received string can be parsed as correct UTF-8, it assumes that it is UTF-8, otherwise it falls back to the other encoding.

ISO-2022-JP is different, because it doesn't use characters outside of ASCII range, so the strings looks like it's UTF-8, and ZNC parser doesn't even attempt ISO-2022-JP on it.

It look like irssi attempts ISO-2022-JP first, and that's why it shows the malformed string as $B!J<+F0E*$K(BFriend - because it was failed to parse as ISO-2022-JP, but succeeded as UTF-8.

But parsing it this way won't work encodings like ISO-8859-15. Does irssi hardcode list of encodings somewhere, to know what to attempt first - UTF-8 or the other one?
I didn't look at irssi implementation yet, but here's a random idea: what about another set of options which mean "Send as __, receive as ___ and UTF-8" and "Send as UTF-8, receive as ___ and UTF-8" in addition to existing ^ and *.

WDYT?

Yep, `^iso-2022-jp` does not work, but `iso-2022-jp` does, and even provides better results than your paste from irssi - one of strings there is malformed: `（自動的にFriend Chat IRCのサーバー群の1台に振り分けられます）��]` - note the � at the end. `^` option ("Send as UTF-8, receive as UTF-8 and ____") in ZNC assumes that the non-utf8 encoding is 8-bit, and uses characters outside of ASCII range. E.g. `ISO-8859-15` and `KOI8-R` work that way. `^` follows a simple logic: if the received string can be parsed as correct UTF-8, it assumes that it is UTF-8, otherwise it falls back to the other encoding. `ISO-2022-JP` is different, because it doesn't use characters outside of ASCII range, so the strings looks like it's UTF-8, and ZNC parser doesn't even attempt `ISO-2022-JP` on it. It look like irssi attempts `ISO-2022-JP` first, and that's why it shows the malformed string as `$B!J<+F0E*$K(BFriend` - because it was failed to parse as `ISO-2022-JP`, but succeeded as UTF-8. But parsing it this way won't work encodings like `ISO-8859-15`. Does irssi hardcode list of encodings somewhere, to know what to attempt first - UTF-8 or the other one? I didn't look at irssi implementation yet, but here's a random idea: what about another set of options which mean "Send as __, receive as ___ and UTF-8" and "Send as UTF-8, receive as ___ and UTF-8" in addition to existing `^` and `*`. WDYT? <!-- github-comment-id:354875372 -->
adam commented 2026-10-05 04:45:35 +02:00
Author
Owner
Copy link
Copy source

Thanks for your quick and thoughtful response.

I think that's a great idea and I believe it will solve the issue quite well.

Now I see how decoding works in znc and understand clearly why this happens.

And if I understood correctly the malformed string in MOTD (utf-8 string inside of iso-2022-jp) is a fault of friend-chat server admins, so I would have to contact them if I want to change it, correct?

Thanks for your quick and thoughtful response. I think that's a great idea and I believe it will solve the issue quite well. Now I see how decoding works in znc and understand clearly why this happens. And if I understood correctly the malformed string in MOTD (utf-8 string inside of iso-2022-jp) is a fault of friend-chat server admins, so I would have to contact them if I want to change it, correct? <!-- github-comment-id:354919807 -->
adam commented 2026-10-05 04:45:36 +02:00
Author
Owner
Copy link
Copy source

Your immediate issue can already be solved on your side, by removing ^ character, or by using webadmin to configure encoding, which makes it more clear what ^ means. Also see this wiki page.
The rest is only about mixing UTF-8 and ISO-2022-JP in a single connection.

I would have to contact them if I want to change it, correct?

Yes

the malformed string in MOTD (utf-8 string inside of iso-2022-jp)

No. That string is not UTF-8, because it looks gibberish when parsed as UTF-8. But:

  • it is "valid" UTF-8, as in, it is parseable as UTF-8 string at all. This is a property of ISO-2022-JP which uses 7-bit characters. It's possible for it to use 8th bit, but it's not necessary, and the string server sends is not using the 8th bit, which is fine. But all ASCII characters are valid UTF-8, which makes all those strings parseable as UTF-8, even if its meaning for humans is lost.
  • that exact line is not valid ISO-2022-JP, because it ends in middle of an escape sequence. I believe they meant BB from the next line to be continuation of the same line. That would make it valid. The same split applies to 2 other lines in that MOTD, but there breakage is not that visible, because it happened right before start of next escape sequence.
Your immediate issue can already be solved on your side, by removing `^` character, or by using webadmin to configure encoding, which makes it more clear what `^` means. Also see [this wiki page](https://wiki.znc.in/Charset). The rest is only about mixing UTF-8 and ISO-2022-JP in a single connection. > I would have to contact them if I want to change it, correct? Yes > the malformed string in MOTD (utf-8 string inside of iso-2022-jp) No. That string is not UTF-8, because it looks gibberish when parsed as UTF-8. But: * it is "valid" UTF-8, as in, it is parseable as UTF-8 string at all. This is a property of ISO-2022-JP which uses 7-bit characters. It's possible for it to use 8th bit, but it's not necessary, and the string server sends is not using the 8th bit, which is fine. But all ASCII characters are valid UTF-8, which makes all those strings parseable as UTF-8, even if its meaning for humans is lost. * that exact line is not valid ISO-2022-JP, because it ends in middle of an escape sequence. I believe they meant `BB` from the next line to be continuation of the same line. That would make it valid. The same split applies to 2 other lines in that MOTD, but there breakage is not that visible, because it happened right before start of next escape sequence. <!-- github-comment-id:354957559 -->
adam commented 2026-10-05 04:45:36 +02:00
Author
Owner
Copy link
Copy source

Ah, setting network encoding setting to iso-2022-jp and restarting znc solved the issue. Thanks.

I see now that znc not decoding malformed iso-2022-jp in MOTD as a iso-2022-jp is a separate issue so I will leave this issue open. My original issue is solved now so you can close this issue if you want.

Thanks again for all the help!

Ah, setting network encoding setting to `iso-2022-jp` and restarting znc solved the issue. Thanks. I see now that znc not decoding malformed iso-2022-jp in MOTD as a iso-2022-jp is a separate issue so I will leave this issue open. My original issue is solved now so you can close this issue if you want. Thanks again for all the help! <!-- github-comment-id:354974379 -->
adam commented 2026-10-05 04:45:37 +02:00
Author
Owner
Copy link
Copy source

Restarting is not necessary for that.

2018-01-03 10:00 GMT+00:00 qria notifications@github.com:

Ah, setting network encoding setting to iso-2022-jp and restarting znc
solved the issue. Thanks.

I see now that znc not decoding malformed iso-2022-jp in MOTD as a
iso-2022-jp is a separate issue so I will leave this issue open. My
original issue is solved now so you can close this issue if you want.

Thanks again for all the help!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/znc/znc/issues/1480#issuecomment-354974379, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAT15GCESnS29SNuK-NZ0AUH69hseINZks5tG0_OgaJpZM4RQgDA
.

Restarting is not necessary for that. 2018-01-03 10:00 GMT+00:00 qria <notifications@github.com>: > Ah, setting network encoding setting to iso-2022-jp and restarting znc > solved the issue. Thanks. > > I see now that znc not decoding malformed iso-2022-jp in MOTD as a > iso-2022-jp is a separate issue so I will leave this issue open. My > original issue is solved now so you can close this issue if you want. > > Thanks again for all the help! > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <https://github.com/znc/znc/issues/1480#issuecomment-354974379>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/AAT15GCESnS29SNuK-NZ0AUH69hseINZks5tG0_OgaJpZM4RQgDA> > . > <!-- github-comment-id:354985951 -->
adam commented 2026-10-05 04:45:38 +02:00
Author
Owner
Copy link
Copy source

Restarting is not necessary for that.

But reconnecting the network is.

> Restarting is not necessary for that. But reconnecting the network is. <!-- github-comment-id:355080416 -->
adam commented 2026-10-05 04:45:38 +02:00
Author
Owner
Copy link
Copy source

@dgw nope.

@dgw [nope](https://wiki.znc.in/ChangeLog/1.6.3). <!-- github-comment-id:355086875 -->
adam commented 2026-10-05 04:45:39 +02:00
Author
Owner
Copy link
Copy source

@DarthGandalf Well, now I have a compelling reason to update my instance. Thanks. Guess I've been lazily sitting on 1.6.0 for too long.

@DarthGandalf Well, now I have a compelling reason to update my instance. Thanks. Guess I've been lazily sitting on 1.6.0 for too long. <!-- github-comment-id:355105996 -->
Sign in to join this conversation.
No Branch/Tag Specified
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
adam (Adam Wojcieszonek)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: adam/znc#804