Contents
Newsgroups Server
Hosting Internal Forums
Linking to External Newsgroups
Maintenance
Conclusion
Resources

Newsgroups Server

Extremely pleased by the benefits brought about by the Intranet Server, Frodo asked Sam if he could better it. Kicked by his idea of an active community of the hobbits working for him, Frodo wanted them to be able to air their views, discuss issues, share news, etc. in a forum accessible to all.

Sam found the equivalent of forums in “newsgroups”. With a suitable server, he could host his own newsgroups and even connect to external servers on the network called USENET and get newsgroups hosted by them. For example, the hobbits regularly checked out a few of these newsgroups, and without adequate caching of these groups, the bandwidth was unnecessarily wasted as each hobbit individually read a newsgroup.

After some research, Sam decided to use the excellent INN (InterNetNews) server. He downloaded the source code for version 2.3.2 of this software and compiled it himself after customising it a bit. Since he was quite new to this, he decided to take help from the INN FAQ and Elena's INN pages, and of course, the manual pages that came with the software itself.

Hosting Internal Forums

After INN was compiled and installed, Sam followed the excellent step-by-step configuration document (a text file called “INSTALL”) that came with it to set up the forums. Instead of using the default location, he chose to install it in “/dump/news”, since he had a separate high-capacity hard disk mounted at “/dump”.

As recommended by the document, Sam verified that there was a user named “news” belonging to a group called “news” on the system. He didn't have to do anything special, as his Linux distribution already came with this user set up appropriately. He just made the home directory for this user the same as the folder where he installed INN. This was purely for convenience, as he had to do everything as this user rather than the root super-user. All of INN's configuration files were inside a folder called “etc” within the main installation folder.

Sam chose the “CNFS” format for article storage for the various advantages it had over the other methods and because it matched his requirements closely. The document clearly explained each method and its advantages as well as disadvantages.

Sam first had to edit the “inn.conf” file. This file was already filled in with reasonable defaults for most parameters and Sam just had to enter information like the organisation name, his email address to redirect complaints to, the domain for the server, etc. He also increased the expiry period for articles to 30 days instead of the default 10 days.

Sam skipped over the next few recommended configuration steps as he first wanted to set up an internal newsgroups server. He updated the “incoming.conf” file to list all the names of the Intranet Server for the self-referential peer entry named “ME”.

Since he was using CNFS, Sam next had to edit the “cycbuff.conf” file. He configured this appropriately, setting aside two storage areas - one for hosting internal forums and another for caching external newsgroups. He updated the “storage.conf” file to reflect this configuration.

Sam then had to update the “expire.ctl” file to reflect the fact that he wanted articles to be retained for 30 days instead of the default 10 days.

The next important configuration file to update was the “readers.conf” file that told INN how to respond to the newsreader clients that the hobbits used. Apart from the default entry, Sam added the following entry:

auth “Hobbitware_Employees” {
hosts: “192.168.1.*,*.internal.hobbitware.com”
default: “<Hobbit>”
default-domain: “internal.hobbitware.com”
}
access “Hobbitware_Employees” {
users: “<Hobbit>@internal.hobbitware.com”
newsgroups: “hobbitware.*,ibm.*,comp.*,alt.*,rec.*,weblogic.*”
newsmaster: “sam@internal.hobbitware.com”
access: RPA
}

This effectively allowed all hobbits accessing the server to read and post articles to the newsgroups. Apart from the default “junk” and “control.*” newsgroups, Sam wanted the server to host all internal newsgroups under the “hobbitware.*” hierarchy, while importing a few other newsgroups relevant to the hobbits' work. Since he had not listed the “junk” and “control.*” newsgroups explicitly in the entry above, these newsgroups were not visible to the newsreaders used by the hobbits - these were administrative groups used by INN, not meant for general use.

After this, Sam created the actual files to be used by the CNFS article storage format, using the “dd” command as explained by the document. Then he initialised the INN history database by using the “makedbz” command as recommended by the document. He also created the separate syslog files used by INN, as explained by the document.

Finally, Sam set up scripts to automate certain jobs for INN. For example, he had to set up a cron job that ran the “news.daily” script once in the middle of the night every day, that performed daily server maintenance. He added a call to the “rc.news” script to the system startup scripts, so that the news server was automatically started when the machine came up. All of this was clearly explained in the document.

Sam started the server manually by invoking the “rc.news” script. He then used the “ctlinnd” command to add some locally hosted newsgroups to the server.

The hobbits could then use their favourite newsreader software, usually the same as their e-mail clients, to connect to these groups and read and post articles.

Linking to External Newsgroups

After he had successfully configured internal forums, Sam set about configuring local caches of externally hosted newsgroups. To optimally use the available bandwidth, and to avoid unnecessary connections to external news servers, Sam decided to use a separate newsgroups updater called SUCK, rather than configuring INN to directly retrieve the newsfeed from these servers. Besides, some of these servers only accepted a “POST” from a newsreader and rejected incoming direct newsfeeds. For some reason, Sam could not connect to the “official” site and had to get SUCK from sunsite.unc.edu. (He could have also retrieved it from tsx-11.mit.edu.) Sam used version 4.2.5 of this software that worked perfectly with the version of INN he was using. He downloaded the source code and compiled it himself.

Sam decided to initially have local caches of the following newsgroups, based on the interests of the hobbits:

  1. comp.lang.java.programmer
  2. comp.lang.javascript
  3. comp.text.xml
  4. ibm.software.websphere.application-server
  5. weblogic.developer.interest

Sam first added these groups to the INN server by using the “ctlinnd” command with the “newgroup” parameter. The only problem was that these groups were served by three different newsgroup servers, one of them being the newsgroup server provided by their ISP. After reading the manual pages that came with SUCK and after a bit of experimentation, Sam had the answer.

Sam created a file named “myisp.cfg” that contained options for SUCK to connect to their ISP's newsgroup server and download newsgroups. This file looked like this:

#
# SUCK config file for downloading from news.myisp.com
#
# Contact specified local host for the active groups list.
-A
-hl 192.168.1.1
#
# Clean up after exit.
-c
#
# Fresh connection after every so-many messages
-C 99
#
# Directories for storing stuff
-dd /dump/news/ext_feed/data
-dm /dump/news/ext_feed/multifile
-dt /dump/news/ext_feed/tmp
#
# Explicitly locate the history file for INN.
-HF /dump/news/db/history
#
# Extension to be used for files related to this server.
-p .myisp
#
# Batch processing of posts and feed it to the localhost.
-bp
#
# Be quiet.
-s -q
#
# END.
#

news.myisp.com” was the newsgroup server provided by the ISP. Sam designated the folder “/dump/news/ext_feed” and sub-folders within it to serve as working folders for SUCK, as given in the file above. In the “data” sub-folder, Sam created a file called “active-ignore.myisp” that contained the groups that SUCK should avoid downloading while connecting to this server. This file looked something like this:

junk
control.*
hobbitware.*
ibm.*
weblogic.*

It effectively told SUCK to get all groups, except those internal to the company and those hosted by IBM and BEA. Sam then created similar pairs of these files for the newsgroup servers of IBM and BEA, that respectively excluded all but the groups that were to be retrieved from these servers.

SUCK came with a script named “getnews.inn” that downloaded articles from a remote server and posted articles, if any, to the server using the rpost program that came with the SUCK distribution. Sam modified this script a bit to reflect his setup and to accept the remote server name and the server configuration file as parameters. The script had code that recovered gracefully in the event that the remote server was down for some reason. An important modification that Sam made to this script was to make it use the “put.news.sm” output filter that came with it, as he was using the CNFS method of storage for articles - the default filter “put.news” assumed that each article was stored in a separate file, which was not a valid assumption for CNFS.

Sam then updated the crontab for the news user so that the “getnews.inn” script was called with these configuration files twice every day, for each of the remote servers. All of this was explained properly in the documentation that accompanied SUCK.

Sam had to update the “newsfeeds” INN configuration file so that the articles locally posted by the hobbits to external newsgroups were uploaded to the appropriate remote servers whenever SUCK connected to them, rather than immediately, as would have been the default behaviour for INN. After consulting the relevant manual pages, Sam added the following lines at the end of this configuration file:

# IBM's news.software.ibm.com for the WebSphere and IBM HTTP
# Server groups
ibm/news.software.ibm.com\
:ibm.*\
:Ae,Tf,Wnm:
# BEA's newsgroups.bea.com for the WebLogic groups
bea/newsgroups.bea.com\
:weblogic.*\
:Ae,Tf,Wnm:
# MyISP's news.myisp.com for other groups
myisp/news.myisp.com\
:comp.*\
:Ae,Tf,Wnm:

As a result, INN updated files named “ibm”, “bea” and “myisp” respectively in the “spool/outgoing” sub-folder of the INN installation folder, with the indices of the articles to be uploaded. Whenever SUCK connected to a remote server, it posted the articles indicated in the respective file and “flushed” the file after successfully posting them. The file names indicated the “server codes”, that were also used to name the respective SUCK configuration files (with a “.cfg” extension) mentioned earlier - this (artificial) relation enabled the “getnews.inn” script to correctly locate all the files needed to synchronise with a remote news server.

The effect of this setup was that SUCK automatically synchronised the local caches of these newsgroups twice every day, connecting to each newsgroup server in turn. The hobbits could now directly connect to the Intranet Server to get cached copies of these groups with near instant connectivity and even post articles to them.

Maintenance

Sam regularly checked the log files created by INN to proactively weed out any potential problems with the server. Since Sam was using the CNFS article storage format, that rotated articles within a fixed-size buffer file, he did not have to worry about space taken up by the articles themselves.

The ctlinnd program allowed Sam to control various parameters related to the INN server while it was running, as well as add or remove newsgroups. Sam also regularly checked the mails for the user news as INN sent problem reports and general reports to this user's mailbox.

SUCK did not require much by way of maintenance, except for occasional errors that were reported when his ISP's newsgroup server went down.

Conclusion

The newsgroups proved immensely popular within the company and soon had a very active community taking part in discussions, debates, gossip, etc. Frodo was immensely pleased with the results and heaped praises on Sam.

The caching of the external newsgroups allowed the hobbits to quickly find solutions to their problems and benefit from the experiences of the people from around the world.

Resources

The following resources proved immensely useful to Sam as he set up the newsgroup server:

  1. INN: The InterNetNews server software that hosted the newsgroups. Sam used version 2.3.2 of this software. The manual pages, and the installation document in particular, accompanying this software proved really useful in setting it up properly.
  2. INN FAQ: This document helped Sam find answers to some of the questions he had as well as find solutions to some of the problems he was facing.
  3. Elena's INN Pages: A set of documents that proved immensely helpful to Sam in understanding concepts related to USENET newsgroups and setting up a server to host them.
  4. SUCK: The set of tools that allowed Sam to optimally download and synchronise local caches of external newsgroups. Since he could not connect to the main site, he had to get it from sunsite.unc.edu, though he could have also got it from tsx-11.mit.edu. Sam used version 4.2.5 of this software.