Notes from a session with Herveus d’Ormonde at the Known World Heralds and Scribal Symposium, June 9, 2018.

Who serves as Morsulus Herald?

Herveus d’Ormonde has held this position since Apr 2000.

His predecessors in the role were:

• Master Renfield Wanderscribe (1980–1985) created first society armorial and ordinary.
• Alban St. Albans (Jul 1985–) and Iathus set up a searchable database.
• Iulstan Sigewealding (early 90s– Mar 2000) wrote most of the current codebase.

What is the job of the Morsulus Herald?

The primary task of the Morsulus Herald is maintaining the database of heraldic registrations, including indexing new entries, making corrections, and maintaining the software that powers the oanda.sca.org web interface.

Also responsible for publishing the LoARs to the heraldry.sca.org/loar web site, which  only takes a few minutes each month. When the LoAR is published, Morsulus downloads these in HTML format and uploads them to the web site.

Where does the master database and code live?

All of the data processing happens on Herveus’s local workstation, a Mac laptop. This machine is archived on a external backup device in the same location.

The code is written in Perl and run from the command line. There are multiple scripts which perform different parts of the process. Morsulus has a text file that shows examples of the various script invocations required and works through each of the steps as required every month, figuring out which commands need to be run and making minor changes to the file names or options as needed.

The code is tracked in Git, and exposed to the public at github.com/herveus, although the GitHub version lags somewhat behind the current working version used by Herveus.

The master database is maintained in a SQLite database file. The headings used internally correspond to the ones in the mike.cat file described below.

How is the database updated for new LoARs?

Every month, the Laurel team sends Morsulus a batch of XML files corresponding to the new LoAR. Actually, this happens three times, the first two times corresponding to the two proof passes, and then with the final version. The XML files are generated by OSCAR, but then undergo manual revision during the editing process. If problems with the XML files interfere with processing the data, Morsulus reports this to the Laurel team which makes corrections in the next pass.

A Perl script converts the XML file to a delimited-text “action” file which includes all of the changes for each name or armory as a single line.

Another Perl script applies those actions to the master database, after doing some logical consistency checks — eg, confirming that you don’t add armory without a name, that you don’t add two pieces of armory with identical blazons, that you can’t do a name change unless the old name is already registered, etc.

Lines in the action file which can’t be successfully applied to the database are flagged as errors, and Morsulus comments them out and hand-writes a revised version of the line called a “temporary edit” which can be applied, and notifies Laurel about the issue so it can be fixed in the next revision of the XML.

Then Morsulus runs a program in which he can perform the indexing of armorial headings and features for each blazon. This is a Perl TK app which runs in the X11 environment. It has a point-and-click interface that allows you to add multiple armorial descriptions for each blazon, picking from lists of headings and features. The descriptions are then copied to the local batch and the master database, indexed by blazon. Because the descriptions are indexed by blazon, this process can be done when the first proof pass is received, and does not need to be repeated for the next two passes except for any items whose blazons are changed during editing.

In the course of processing a month’s batch, Morsulus may need to add items to the category file. It’s fairly common to add new category cross-references (eg, “bay leaf – see leaf”) and less common to add a new category with a heading (eg “manacle|MANACLE”). When this happens, related changes must be made to several files in parallel.

• The my.cat file is exposed to the public and corresponds to values visible in the oanda.db.
• There is a separate mike.cat file which is used for indexing and which contains some additional headings and in some places uses different heading names. The mike.cat file also includes a list of which feature sets are applied to a given heading.
• There is a tprint.cat file which lists all of the categories in the order they should be listed to the public. (They used to be sorted alphabetically by heading, which is why some of the heading names are strange.) It also specifies which categories to break down into smaller sections by tincture, number, or group; this is used to generate the web ordinary interface as well as the print ordinary.

Then Morsulus runs a consistency-check script that scans the SQLite database and summarizes various kinds of totals, and compare the output to a copy saved from the previous month, to see what’s changed and confirm that the number of changes match the expectation based on the number of items in the LoAR.

A Perl script exports a copy of the database from SQLite to a flat file format called “classic format.” This includes converting some internal headings from mike.cat to the public equivalents used in my.cat; for example, “BIRD-PELICAN” gets exported as “BIRD:pelican.” These changes cause the exported file to line up with the combined categories used for the ordinary display and for the complex search form.

A set of “DB Diffs” are created that compare the new oanda.db file with the one that was generated last month. This simplifies the process of loading new data into another search tool built by another developer, as they only have to import the changes rather than the entire file. (I am not sure which of the other search tools use this, but I think it’s Hirsch von Henford’s Golden Stag app.)

The updated my.cat and oanda.db are then uploaded to the oanda server and the search daemon is restarted so that it will re-index the file; this only takes a couple of seconds.

How is the database accessed by others?

Most people use the oanda web site. Among other things, log files show that some local kingdom OP sites crawl the oanda web site to automatically collect armorial blazons for people in their region.

You can get a copy of the software needed to run the oanda web site by downloading and running the config.web script, which unpacks a copy of the Morsulus-search package.

Some folks, especially in the West Kingdom, are heavily invested in using a “print” version of the ordinary for lookups and conflict checks. Rather than actually being printed on paper, this is a set of documents accessible from oanda.gigo.com which are built from the oanda.db in formats such as epub and PDF using some third-party software. That process uses a file named templ.cat which is similar to my.cat but has print-specific notes such as “do not print” added to some categories.

(There was an old print version which was built by Morsulus using some custom C code which used the database to generate PostScript, but the last time this was used was for Pennsic 2011 and it has now been retired.)

(Although not discussed in detail during this meeting, there are also a couple of other search tools which can import the oanda.db file, including two Windows apps and a Lotus Notes interface.)

How are errors corrected?

People write in to Morsulus when they notice a problem with an armorial listing.

Morsulus checks the reports against the original LoARs and against an archive of scans of the original registration files to figure out at what stage of the process the error appeared.

In some cases the error originates in the LoAR; for those, people need to go through an LoAR errata process, get Laurel signoff, and then the correction is processed as part of the corresponding LoAR update.

In other cases the problem is due to a transcription error, in which the LoAR entry was garbled when it was transferred to the database or the indexing of categories and features was done incorrectly. In those cases Morsulus can simply correct the problem to accurately reflect the LoAR and push that change out in the next update cycle.

Those changes are made via a command-line tool that can search and edit the SQLite master database. This has various internal-use-only functions like looking up records by regid (a synthetic primary key) and directly editing raw database fields.

What are some random factoids about the database?

The database contains approximately 110K registrations. It takes up 120 MB in SQLite with the various indexes, but the flat-file version is only 16 MB.

There are a few cases of where there are two name registrations for the same name.

Jointly owned items have a primary name that they’re attached to, plus a second entry that just references that they’re a co-owner.