[LBBS-160] Improve NNTP functionality

Following the major overhaul mentioned in [~~LBBS-30~~], LBBS has a compliant and functional news server implementation that is capable of handling real world Usenet traffic, with ~full support for RFC 3977 and related extensions.

However, there is more we can do. For one, LBBS + nntp-web would benefit from implementation of the extensions mentioned in [LBBS-159].

Even before that, there is existing stuff supported by common news packages like INN (InterNetNews) that we should support:

Administrative:

Using results from nntp_junk.log, fine tune the exclusions for peering groups
After ~mid/late July, start running expiration (and add cron job) and monitor behavior and performance

New functionality:

Should be able to easily delete a message by Message-ID (underlying functionality needed for honoring cancels, or manual deletions)
Finish implementing the logic for control message processing. Local creation/deletion e.g. "news newgroup" may need to generate the appropriate control messages to cascade (and then we process it)
- RFC 5537 Section 5
- PGP verification is needed to automatically process control messages, e.g. Big 8 groups (auto-processing should still remain disabled by default)
  - PGPMOOSE / pgpcontrol: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
  - Likewise for accepting moderated messages w/ Approved header (in addition to control messages)
    - allow/moderate/reject, e.g. allow newgroup, moderate rmgroup, or moderate newgroup, reject rmgroup
  - Likewise for RFC 8315 Cancel-Lock support
    - INN: Julien Elie has implemented Cancel-Lock support in innd and nnrpd, based on RFC 8315 and libcanlock. A new inn-secrets.conf configuration file has been added in pathetc wherein you can set the secrets to use for Cancel-Lock. See the inn-secrets.conf(5) man page for more details.
Supersedes header
Archive/X-No-Archive: https://en.wikipedia.org/wiki/X-No-Archive
Finish implementing channel and program feeds e.g. WP option so we can generate the data needed to report to TOP1000
Support for Perl filters, like cleanfeed: https://www.mixmin.net/cleanfeed/
Allow also storing junk counts by peer (instead of just in aggregate)
Conversely, add the ability to log counts for "non junk" so we can see where the most articles are coming from, to tune retention appropriately
Collect and publish daily stats, e.g.
Support enhanced Path appending - RFC 5537 3.2.1 (verify peer, etc.)
Posting restrictions
- Add rate limits for posting and transit
- Reader only:
  - Reject MIME/multipart messages (any Content-Type besides text/plain)
  - Reject purely top-posted articles
  - Reject articles with too much quoted content (INN has an option for this, see function CheckIncludedText)
Allow peers to read articles rather than just send them

Non-functional improvements:

The final check for history existence needs to use locking for the check + adding to history so a race condition isn't possible (no need to skip the Bloom filter though, since it should be up to date)
Add support for additional headers to nntp-web, then publicize in news.software.nntp / news.software.readers
Keep a history of rejected articles (like INN does with its remembertrash setting), so we can refuse articles that we rejected for another peer already, without having to receive the article first
- Use another Bloom filter for this so we don't have to scan another potentially large file for each "query"
- Also helps with "news suckordered suckfeed" since if it has to re-request a bunch of articles we don't have (if many were rejected), only to immediately reject them again - avoidable if we keep track of rejections (and make the rejection file easy to flush)
test_nntp_stress to identify (and improve) performance bottlenecks for large news servers

Implementation resources:

Historical resources:

https://giganews.com/usenet-history/lapsley/
Personal notes from Phil Lapsley

User resources:

Comments

You must be logged in to leave a comment.