Feature Request: File based compressed buffers #794

New Issue

2026-10-05T04:44:45+02:00

adam commented

2026-10-05 04:44:45 +02:00

Hi, in #znc I learned that buffers are always in memory only.

Initial question there was, what is the maximal buffer size selectable. And I selected 500000 as the server admin does not seem to limit that.

With that large buffer sizes, many networks and channel and many users, this would not scale very well on bigger installations. It consumes useless memory. It never needs to be fast. But it's taking so much space, useless space.

Even syslog daemons or databases are file based. If performance is critical they can be work synchronously but still flushing to disk every size, count or time threshold.
That is what would be needed for ZNC as well.

Additonally using the Clientbuffer and using some of the clients only rarely, say every few months, I like to have very large buffer sizes. Example: I'm daily in the office, using my IRC there. In the weekend I may use my home office client, that has no history of the last 2 or 3 weeks, so I like that to be synced with my daily client. That's one reason why using ZNC.
Having Clientbuffer it is advised to disable auto clearing buffers, so every buffer won't be cleared because there could be some client id still needing the replay.

So feature request would be having file based buffers with flush thresholds flushing memory buffers to files based on size, count or time policies (I wondered that it is not that way already).

This could be a job for the Savebuff module. Currently this periodically flushes to files, but never clears the buffers. Buffers memory usage is not eased by savebuff. It's just a backup for unconditional crashes. So the functionality could also be added there.

One additional feature request is adding compression for every buffer and file. I wonder that all the plain text buffers are not compressed. Syslogs and man pages are compressed, project docs, kernels, kernel modules. Not compressing plain text isn't optimal. AFAIK currently there is no compression for memory buffers, Savebuff files (only encrypted) or logs.

Hi, in #znc I learned that buffers are always in memory only. > _Initial question there was, what is the maximal buffer size selectable. And I selected 500000 as the server admin does not seem to limit that._ With that large buffer sizes, many networks and channel and many users, this would not scale very well on bigger installations. It consumes useless memory. It never needs to be fast. But it's taking so much space, useless space. Even syslog daemons or databases are file based. If performance is critical they can be work synchronously but still flushing to disk every size, count or time threshold. That is what would be needed for ZNC as well. Additonally using the [Clientbuffer](https://wiki.znc.in/Clientbuffer) and using some of the clients only rarely, say every few months, I like to have very large buffer sizes. Example: I'm daily in the office, using my IRC there. In the weekend I may use my home office client, that has no history of the last 2 or 3 weeks, so I like that to be synced with my daily client. That's one reason why using ZNC. Having Clientbuffer it is advised to disable auto clearing buffers, so every buffer won't be cleared because there could be some client id still needing the replay. So feature request would be having file based buffers with flush thresholds flushing memory buffers to files based on size, count or time policies (I wondered that it is not that way already). This could be a job for the [Savebuff](https://wiki.znc.in/Savebuff) module. Currently this periodically flushes to files, but never clears the buffers. Buffers memory usage is not eased by savebuff. It's just a backup for unconditional crashes. So the functionality could also be added there. One additional feature request is adding compression for every buffer and file. I wonder that all the plain text buffers are not compressed. Syslogs and man pages are compressed, project docs, kernels, kernel modules. Not compressing plain text isn't optimal. AFAIK currently there is no compression for memory buffers, Savebuff files (only encrypted) or [logs](https://wiki.znc.in/log).

adam closed this issue

2026-10-05 04:44:45 +02:00

adam commented

2026-10-05 04:44:46 +02:00

There is a limit, and it's 500 by default. But it applies only to non-admins.

There is a limit, and it's 500 by default. But it applies only to non-admins.

adam commented

2026-10-05 04:44:47 +02:00

And I selected 500000 as the server admin does not seem to limit that.

Let's just pretend the average IRC line is 100 bytes large (I just made that number up). Let's further pretend that there are 20 channels (number made up as well), each of which uses your limit of 500k lines of scrollback. That's already 20*500k = 10m lines of scrollback. Each one takes 100 bytes, so we have 1GB (yes, GB not GiB) of scrollback.
How long do you want to wait for your irc client to connect to the IRC server? What if I told you that things won't work out since your IRC client will likely run in some kind of ping timeout before the connection is synchronized up? When are you going to read that gigabyte of scrollback? What if I told you that currently ZNC will have quite a hard time sending you all that scrollback (the full gigabyte of scrollback will be formatted and placed into the "send this over TCP" output queue all at once, so ZNC will still have all that data in memory).

So:

a) I don't think this is a reasonable idea.
b) Just because the data is stored in a file doesn't mean that all problems are solved.

However, since I'm not much involved with ZNC any more, feel free to ignore me. I guess patches are welcome.

So let's move to the design stage: What exactly is the idea for the file format? Something resembling raw irc lines with things like timestamps formatted as message tags? So, when a new line comes in, you just append it to the file, I guess. Is there some limit on the file size or number of lines in the file? If the latter: How is the number of lines tracked? And in general: What happens when that limit is exceeded? The whole file is rewritten, just to remove the first line from it that it contains?

syslogs and man pages are compressed, project docs, kernels, kernel modules.

Syslogs: But they are not compressed by the daemon writing the file. Appending stuff to a compressed file is quite expensive (or the compression is not good). Also, compression means that "remove the first line of the file" just became so much more expensive.

Project docs: Ha :-)

> And I selected 500000 as the server admin does not seem to limit that. Let's just pretend the average IRC line is 100 bytes large (I just made that number up). Let's further pretend that there are 20 channels (number made up as well), each of which uses your limit of 500k lines of scrollback. That's already 20*500k = 10m lines of scrollback. Each one takes 100 bytes, so we have 1GB (yes, GB not GiB) of scrollback. How long do you want to wait for your irc client to connect to the IRC server? What if I told you that things won't work out since your IRC client will likely run in some kind of ping timeout before the connection is synchronized up? When are you going to read that gigabyte of scrollback? What if I told you that currently ZNC will have quite a hard time sending you all that scrollback (the full gigabyte of scrollback will be formatted and placed into the "send this over TCP" output queue all at once, so ZNC will still have all that data in memory). So: a) I don't think this is a reasonable idea. b) Just because the data is stored in a file doesn't mean that all problems are solved. However, since I'm not much involved with ZNC any more, feel free to ignore me. I guess patches are welcome. So let's move to the design stage: What exactly is the idea for the file format? Something resembling raw irc lines with things like timestamps formatted as message tags? So, when a new line comes in, you just append it to the file, I guess. Is there some limit on the file size or number of lines in the file? If the latter: How is the number of lines tracked? And in general: What happens when that limit is exceeded? The *whole* file is rewritten, just to remove the first line from it that it contains? > syslogs and man pages are compressed, project docs, kernels, kernel modules. Syslogs: But they are not compressed by the daemon writing the file. Appending stuff to a compressed file is quite expensive (or the compression is not good). Also, compression means that "remove the first line of the file" just became so much more expensive. Project docs: Ha :-)

adam commented

2026-10-05 04:44:47 +02:00

With project docs I meant /usr/share/doc/* which my package manager compresses according to settings.

Right, I'm not involved in the tcp issues you mentioned. I see, this is leading to further issues anyway.

About the questions of file format, appending to files and so on... as you said compressing is not the job of log daemons but could be the job of the syslog or some external logrotate. Mapping this to ZNC, flushing to files according to a policy would need to create new files or recompress. Doing that on a daily base would mean to have one file per day...

But still that all could be failing on the tcp transfer. Big replays would take time, that's clear, but if IRC clients could deal with that in general? The IRC protocol itself cold miss some compression features like http already has, and coming to large replays that could be even more important. Ok, we just touched the tip of the iceberg... Thanks for the outline.

With project docs I meant /usr/share/doc/* which my package manager compresses according to settings. Right, I'm not involved in the tcp issues you mentioned. I see, this is leading to further issues anyway. About the questions of file format, appending to files and so on... as you said compressing is not the job of log daemons but could be the job of the syslog or some external logrotate. Mapping this to ZNC, flushing to files according to a policy would need to create new files or recompress. Doing that on a daily base would mean to have one file per day... But still that all could be failing on the tcp transfer. Big replays would take time, that's clear, but if IRC clients could deal with that in general? The IRC protocol itself cold miss some compression features like http already has, and coming to large replays that could be even more important. Ok, we just touched the tip of the iceberg... Thanks for the outline.

adam commented

2026-10-05 04:44:48 +02:00

I'm declaring this as duplicate of #269

I'm declaring this as duplicate of #269

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adam/znc#794