Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
DIABLO-FILES(5)		      File Formats Manual	       DIABLO-FILES(5)

NAME
       diablo-files - Synopsis of configuration	and control files for Diablo

GENERAL
       This  manual  page provides a synopsis for various diablo configuration
       and control files.  It does not necessarily describe  all  commands  or
       the specific file format.  You can obtain most of the latter by looking
       at the example config/control files in  the  sample/  directory.	  What
       this manual page	does is	describe specific featurisms that would	other-
       wise not	be readily apparent.

       The location of all diablo files	is controlled by  /news/diablo.config.
       The  location  of  diablo.config	 itself	can be specified with the DIA-
       BLO_CONFIG_PATH environment variable or with the	'-C  diabloconfigpath'
       option to any command.  Diablo files are	typically rooted at either the
       news root, the library root, or the database root (path_news, path_lib,
       or  path_db  in diablo.config).	See samples/diablo.config for more in-
       formation.

DNEWSFEEDS
       The dnewsfeeds file specifies how diablo	generates  outbound  feeds  as
       well  as	specifies how diablo filters incoming feeds.  You specify each
       outbound	feed with a label keyword, followed by various other  keywords
       and ending with an end keyword.

       alias  specifies	 an outbound newspath alias, which may contain '*' and
       '?' wildcards.  An inbound article whos path element matches any	 given
       alias  for  any	given  outbound	 feed is not requeued to that outbound
       feed.

       addgroup	and delgroup filters articles to outbound feeds.  All add/del-
       group commands are run against each newsgroup in	the Newsgroups:	header
       for the article and the last add/delgroup command effecting  any	 given
       newsgroup is applied to the article.  Normally this means you start out
       more general at the beginning (i.e. '*')	and become  more  specific  at
       the  end	 of  your  list.   When	an article is cross posted to multiple
       newsgroups, success with	any of the groups will cause the article to be
       propogated.

       NOTE!  If you 'addgroup *', then	use delgroup and delgroupany to	remove
       groups that you do not want,  beware  that  control  messages  for  ALL
       groups  will  still  be	propogated unless you also do a	'delgroup con-
       trol.*'.

       delgroupany filters articles in a manner	similar	to  delgroup,  but  if
       the specified groups exist in the Newsgroups: line *AT ALL*, no distri-
       bution to *any* of the other newsgroups will occur.  i.e.  if  you  say
       'delgroupany  alt.warez*',  it  means  that  if an article is posted to
       comp.sys.amiga.misc and alt.warez*, the article will NOT	be propogated.

       requiregroup An extremely exclusive filter that is the inverse of  del-
       groupany.   The	article's Newsgroups MUST contain a group matching the
       wildcard	or it will not be propogated.  This  is	 generally  only  used
       when  splitting	off  control feeds.  Please see	samples/dnewsfeeds for
       typical usage.

       groupref	allows a feed to recursively include a group access  list  de-
       fined  by a groupdef command.  The groupdef command may occur before OR
       after the newsfeed that references it, and groupdef's can be recursive.
       see groupdef for	more information.

       filter effects INBOUND feeds.  Articles coming FROM the specified label
       (see diablo.hosts on how	to tie labels to  incomming  connections)  are
       rejected	 if any	listed newsgroup matches the wildcard specified	in any
       of the filter commands.	You normally use this to reject	articles cross
       posted  to your local newsgroups	that are propogated from outside enti-
       ties.  Note that	the history file is not	updated, just in case the  ar-
       ticle is	also brought in	from a valid 'local' newsfeed.

       maxpath	(1.08 or higher) effects OUTBOUND feeds.  Diablo will not send
       an article to an	outbound feed if the  article's	 Path:	line  contains
       more then the specified number of path elements.

       maxcross	(1.04 or higher) effects OUTBOUND feeds.  Diablo will not send
       an article to an	outbound feed if the article's Newsgroups:  line  con-
       tains more then the specified number of groups.

       maxsize	(1.04 or higher) effects OUTBOUND feeds.  Diablo will not send
       an article to an	outbound feed if the article is	larger then the	speci-
       fied  number  of	 bytes.	 You may use 'k', 'm', and 'g' to denote kilo-
       bytes, megabytes, and gigabytes.	 For example, <I>maxsize 100k</I> .

       rtflush (1.12 or	higher)	effects	OUTBOUND feeds.	 Diablo	will write the
       queue  file for this feed unbuffered.  Generally	used along with	'real-
       time' in	dnntpspool.ctl.

       There is	an (unimposed) arbitrary limit of 256 add/delgroup entries per
       feed.  Currently	this is	due to the fact	that Diablo scans the wildcard
       list linearly and cannot	really support group-specific wildcards	(where
       a  feed	wants  10,000  specific	 groups	 rather	 then fewer wildcarded
       groups).

       A special label,	called ME may be specified.  This label	generally only
       contains	filter commands.  Diablo will revert to	this label for any in-
       coming connections that have not	been associated	with a specific	label.

       There is	also the groupdef command, which must occur outside  any  feed
       commands.   The	groupdef  command  is  followed	 by a set of groupadd,
       groupdel, groupdelany, and/or groupref commands to create a  grouplist.
       The  groupdef command is	terminated by an end command.  Grouplist defi-
       nitions may be referenced by feeds via the groupref command, and	may be
       referenced by other grouplists.

DEXPIRE.CTL
       The  dexpire.ctl	 file  controls	the length of time articles are	stored
       for each	group in the reader header database and	the maximum number  of
       articles	 that  can be stored in	the article index per group for	dread-
       erd.  This index	value is adjusted by dexpireover based on  the	number
       of articles in the newsgroup.

       The  various  options  are separated by a colon and appear in any order
       after a wildmat specification of	the newsgroup names. The options are:

       x to specify the	number of days to keep articles	matching this wildmat.
       Not  specifying	this option means that articles	are not	removed. Frac-
       tions of	a day may also be used.

       a to override the default 512 for  the  starting	 'maxarts'  value  per
       group  matching	this wildmat. The starting value is only used when the
       group index is first created.

       i to specify the	minimum	'maxarts' that dexpireover may adjust to.

       j to specify the	maximum	'maxarts' that dexpireover may adjust to.

DEXPIRE.FACTOR
       The dexpire.factor file is no longer used by Diablo and can be deleted.
       It may be used again in the future.

DNNTPSPOOL.CTL
       The  dnntpspool.ctl file	is used	by the dspoolout program to manage the
       outbound	queues.	 Diablo	will maintain a	single outbound	queue file for
       each  feed.   dspoolout	renames	 this file to a	proper sequenced queue
       file, properly flushes Diablo's file descriptors, starts	 up  dnewslink
       programs,  and  removes any sequenced queue files backlogged beyond the
       specified limit.

       This file allows	you to specify the  remote  host,  maximum  number  of
       queue  files, and maximum number	of dnewslink processes to run for each
       outbound	feed, and other	options.  Please refer to  the	samples/dnntp-
       spool.ctl file.

DISTRIB.PATS
       The  distrib.pats  file	is  read  and  returned	by the NNTP 'list dis-
       trib.pats' command.  It is also used to generate	a Distribution:	header
       internally  when	 the  POST  command does not supply it (or supplies an
       empty Distributions header).

DISTRIBUTIONS
       The distributions file is read and returned by the NNTP 'list distribu-
       tions' command.

DIABLO.HOSTS
       The diablo.hosts	file controls authentication for incoming connections.
       Each line consists of a domain name or IP address wildcard and a	 label
       identifying which feed in dnewsfeeds the	incoming connection is associ-
       ated with.  THE LABEL IS	NO LONGER OPTIONAL.  You must supply a label.

       Diablo normally requires	that the  reverse  lookup  match  the  forward
       lookup  for  security  purposes,	but many sites set up their reverse to
       point to	a CNAME	or set up their	reverse	to point  to  an  unassociated
       host  yet  still	 request  that you put a common	hostname in your hosts
       file which 'resolves' to	all the	IP addresses of	their news machines.

       Diablo will attempt to wildcard match the last two domain  elements  of
       the  reverse domain name	with non-wildcard domain names in diablo.hosts
       then issue a forward lookup of the name in diablo.hosts and attempt  to
       match  the  IP.	 In  otherwords,  if  you  have	 an  entry  for	'news-
       feeds.fubar.com'	in diablo.hosts	and an incoming	 connection's  reverse
       lookup	 comes	  back	 'news55.fubar.com',   diablo	will   convert
       news55.fubar.com	to *.fubar.com and attempt to match that  against  en-
       tries  in  diablo.hosts.	  When	a  match occurs, it performs a forward
       lookup (in this case against 'newsfeeds.fubar.com') and tries to	 match
       up the IP address that way.

       This  methodology has the advantage of not requiring diablo to do a se-
       quential	forward	lookup of all the entries in diablo.hosts.  Each  con-
       nection's DNS load is consistent	only with the domain/IP	the connection
       is coming from which is very important for stability in the face	 of  a
       large number of feeds.

DREADER.HOSTS
       The  dreaderd.hosts  file  contains access permissions for NNTP readers
       for the dreaderd	server.	 This file also	 contains  access  permissions
       for header-only feeds coming into the dreaderd server.

DSERVER.HOSTS
       The  dserver.hosts  file	contains the outgoing spool and	posting	server
       configuration which  dreaderd  uses  to	make  connections  to  outside
       servers for message retrieval and outgoing POSTed message propogation.

DQUEUE (directory)
       The  dqueue directory contains the queue	files, both the	ones generated
       by the diablo server and	the ones maintained by	dspoolout.   Files  in
       this  directory are generally named as the label	(diablo	outbound queue
       file for	a label), as the label.Snnnnn (sequenced queue file maintained
       by dspoolout), or .label.seq (sequence number information maintained by
       dspoolout).

       Note that the diablo queue format has changed as	of V1.08.  Older  ver-
       sions  of  diablo  dumped the filename and message id.  As of V1.08, an
       third field formatted as	OFFSET,BYTES  is  added	 to  support  the  new
       multi-article spool files.  DNewslink understands both formats.

FEEDS (directory)
       The feeds directory contains automatic group add/del information	as re-
       quested by a feed through the feedrset and other	Diablo commands.  If a
       file  exists for	any given feed,	it overrides the addgroup and delgroup
       commands	in the dnewsfeeds file for that	feed.

NEWS SPOOL (/news/spool/news) (directory)
       Diablo implements a two-level news spool.   A  directory	 of  the  form
       D.xxxxxxxx  is  created on the first level every	10 minutes.  Each dis-
       crete fork creates a distinct file in one or more of these  directories
       when  it	receives an article.  The directory is chosen based on the ex-
       piration	and the	filename is chosen starting with a hash	of the	incom-
       ing IP.	The diablo process then	exclusively locks the file(s) in ques-
       tion.  In the case of contention, the loser will	generate another file-
       name  and loop until it finds or	creates	one.  The files	or of the form
       B.xxxx where xxxx is basically random.  Multiple	articles may be	stored
       in each file.  An ascii NUL (code 0x00) is used as an out-of-band arti-
       cle separator.  The history and outgoing	queue files reference articles
       by relative file	path, offset, and size.

       It should be noted that dnewslink explicitly looks for the separator as
       a double-check against corruption.

DACTIVE.KP (/news/dactive.kp)
       The dactive.kp file is a	key-token-pair	database  (see	diablo-kp(5)).
       It is NOT compatible with the INN active	file.  The dsyncgroups program
       with -G and -F flags may	be used	to create dactive.kp  from  an	active
       file  in	 "list active" format (see dsyncgroups(8)).  A dactive.kp file
       may also	be created based on an active file on a	remote host (also  see
       dsyncgroups(8)).

       Diablo  KP database files are human readable but	should only be manipu-
       lated with the dkp program while	Diablo is active.  If Diablo is	 inac-
       tive,  KP  files	 may be	manipulated with dkp OR	can be edited by hand.
       Diablo's	dactive.kp file	serves the same	purpose	as INN's  active  file
       but,  being  a general token=value database, may	contain	additional in-
       formation.  The intention is to use this	database to track article num-
       ber assigns and to store	additional group description, moderator	email,
       and PGP keys for	the reader portion of Diablo.

       In Diablo, groups missing from dactive.kp do not	 normally  effect  the
       feeder  side of the system.   However, Xref: headers are	only generated
       for those newsgroups  listed  in	 dactive.kp.   This  behavior  can  be
       changed through the 'activedrop'	option in diablo.config

       The  dactive.kp	database is keyed off the group	name and uses the fol-
       lowing tokens, some of which are	optional: NE (last stored article num-
       ber,  %010d  format),  NB (first	stored article number %010d format), S
       (status), M (moderator email), and CPGP (pgp key, hierarchically	recur-
       sive).	The  status  may contain multiple characters.  'n' indicates a
       disabled	group, 'y' indicates an	enabled	group,	'm'  indicates	moder-
       ated.   itself  is  entirely optional (defaults to 'y').	 The group de-
       scription, if any, is stored with the GD	key.

       When using the feeder to	generate Xref: headers,	the feeder  creates  a
       copy  of	 the  NE field called NX and uses that to track	article	number
       assignments.  This way both the reader and  feeder  can	use  the  same
       physical	active file when both the reader and feeder are	running	on the
       same box.  The dsyncgroups command has options to manipulate NE and NX.
       WARNING!	  If NX	is present, the	reader will reset NE if	NX < NE	and an
       incoming	article	has been assigned a number less	then NE.

       if a group is marked moderated, the moderator email is obtained via the
       M key.  If control messages related to the group	(hierarchically	speak-
       ing), the CPGP key contains the public key.  For	example,  there	 might
       be  an  entry for 'comp'	with a CPGP key	but since 'comp' is not	a real
       newsgroup, the status would be blank.

SEE ALSO
       diablo(8),    dsyncgroups(8),	dicmd(8),    didump(8),	    diload(8),
       dnewslink(8),   doutq(8),  dexpire(8),  dexpireover(8),	diconvhist(8),
       dilookup(8), dspoolout(8), dkp(8), dpath(8),  diablo-files(5),  diablo-
       kp(5)

							       DIABLO-FILES(5)

NAME | GENERAL | DNEWSFEEDS | DEXPIRE.CTL | DEXPIRE.FACTOR | DNNTPSPOOL.CTL | DISTRIB.PATS | DISTRIBUTIONS | DIABLO.HOSTS | DREADER.HOSTS | DSERVER.HOSTS | DQUEUE (directory) | FEEDS (directory) | NEWS SPOOL (/news/spool/news) (directory) | DACTIVE.KP (/news/dactive.kp) | SEE ALSO

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=diablo-files&sektion=5&manpath=FreeBSD+12.1-RELEASE+and+Ports>

home | help