On December 14, 2020, Master Herveus d’Ormonde led a few interested heralds through an online session in which we were able to observe a portion of his workflow as Morsulus Herald, watching as he added several new cross-references to the armorial category files and published the changes to the public O&A web site.
I am attaching my notes from this session below in hopes that they might be of interest to other members of the community, although this is admittedly a fairly-obscure topic with a limited audience.
See also my notes from last summer’s session on Updating the O-and-A Database.
Much of the information laid out below is summarized more sucinctly in the UpdateMyDotCat Morsulus Wiki entry.
What Do The Category Files Store?
This information wasn’t covered during the “ride along” session, but I’m restating it here for folks who’d like a refresher.
The contents of the category files can be seen by browsing through the my.cat file:
- Headings
- These are the primary codes used for indexing armory, such as “GU,” “CAT” and “FIELD DIV.-BENDY*3”
- Categories
- These are user-readable names, such as “gules field” and “beast, cat,” and “beast, lion.”
- They come in two types: some are associated with headings, and some are only cross-references.
- Features
- These are attributes that further refine the headings, such as “argent” and “embattled” and “rampant.”
The headings and features are then used in the armorial database by tagging each individual armory entry with several “armory descriptions,” separated by pipe characters. Each armory description contains a heading and optionally several associated features, separated by colons.
As an overly simplified example, a device of “Gules, a lion argent” might be coded “GU|CAT:primary:1:rampant:argent”. (In practice, the actual descriptions would be slightly more complicated.)
When Are Category File Updates Needed?
By far the most common source of updates to the category files is new cross-references.
Every month, when Herveus processes new armory submissions, he looks for terms which aren’t already referenced in the cross references — for example, if a new type of tree is registered for the first time, he might add a cross reference that says “Hawthorn trees are indexed under tree, rounded shape.”
The same thing is done when alternate spellings of an existing term are first used.
These additions are not urgent and can be held in a worksheet allowed to build up over the course of several months until they’re all updated in a batch.
On the other hand, when new headings are added, or features added, those changes need to be defined in advance so that they can be applied to the new blazons from within the visual armory indexing tool.
If the equivalences between features are going to be changed, such as if a ruling indicates that invected and indented lines conflict, the modified feature definitions need to be pushed out to the public database server in order to modify the search behavior.
Where Are Categories Recorded?
Herveus has started working towards combining all of the category files into the SQL database, but in the meantime there are currently several different heading files which each need to be carefully updated by hand.
- ~/aux/mike.cat:
- Master copy of headings, features, and cross references.
- Uses “classic-style” headings
- ~/aux/mike.hdg:
- Specifies how headings are grouped together in the indexing tool.
- Only needs to be updated if adding or removing headings
- ~/aux/old_new.desc:
- Specifies mappings from “classic-style” to “web-style” headings.
- Only needs to be updated if adding or removing headings.
- Used by xlate.pl to convert the SQLite data to the oanda.db flat file.
- {working directory}/my.cat:
- Copied to web server at http://oanda.sca.org/my.cat
- Referenced by the public to look up categories and features.
- Used by the database server to figure out which features are compatible.
- Includes all headings and cross-references.
- Morsulus-tools/Morsulus-Search/scripts/tprint.cat:
- Copied to web server at http://oanda.sca.org/tprint.cat
- Used to generate the Ordinary pages.
- Specifies which items get broken out into separate lines in the Ordinary like “annulet, 1, argent”.
- Morsulus-tools/Morsulus-Search/scripts/temp.cat:
- Copied to web server at http://oanda.sca.org/temp.cat
- Used to generate the Ordinary pages.
- Specifies lines which shouldn’t be printed by flagging them with “do not print”
There are also a couple of files which are rebuilt by scripts based on the above:
- ~/aux/mikecat:
- Pre-compiled Perl data structures for better performance.
- Produced by the mk_cat_file script based on ~/aux/mike.cat and ~/aux/mike.hdg.
- {WebDocumentRoot}/mycat.pl
- Pre-compiled Perl data structures for better performance.
- Produced by the mk_cat_file script based on my.cat.
- Unlike the above step, this is currently run on the web server.
The Update Process
When it’s time to update the category files, Herveus follows the below process.
Each file is edited manually, to enable double-checking for duplicates.
- Gather new headings and features, if any, along with the new and updated cross references in a workbook file. Sort them by type and then alphabetically.
- Edit mike.cat and merge in the new lines.
- Run mk_cat_file against mike.cat. This tool was built for the visual indexer tool, and has since been leveraged to also be used for the new complex search tool.
mk_cat_file -i ~/aux/mike.cat -h ~/aux/mike.hdg -o ~/aux/mikecat
- Update “bird, <something>” to simply “bird, whole.” Fix up any other targets that differ.
- Edit Morsulus::Search/scripts/temp.cat and merge in the new lines. Temp.cat is a version of my.cat used for ordinary generation.
- Edit Morsulus::Search/scripts/tprint.cat and merge in the new lines. Do not include any features. For cross-references, only include the part before the dash, not what they are references to.
Do not add cross-references to tprint.cat if you don’t want them to show up in the ordinary. (For example, in our recent session he added “flower, arnica” to tprint.cat, but did not add “cross columnar.”) I am not confident I know which categories should get which treatment.
- Edit ~/aux/old_new.desc and merge in any relevant lines if needed. Note that these are descriptions, which can include features. No changes needed unless categories are being added or deleted.
This file is used by xlate.pl to convert the SQLite data to the “export format” oanda.db flat file. Or perhaps it’s used to confirm that the changes applied by the tempmap script haven’t broken anything?
- Edit {DocumentRoot}/my.cat and merge in the new lines. These changes may be made against the live server. This needs to be done in “export format”. This version does include all of the features and categories. (For example, in our recent session this did include “cross columnar”.)
- Commit any code changes that you’ve made so far and push them to git.
- Make a new config.web and execute it to refresh the Ordinary pages and apply any other changes to the search tools.
perl assemble_here_docs.pl
If this command dies with an error specifying a line in PRINT_FILE, check your tprint.cat for typos.
- Upload temp.cat and tprint.cat from Morsulus::Search/scripts to {DocumentRoot}
scp config.web scripts/tprint.cat scripts/temp.cat herveus@oanda.sca.org:{DocumentRoot}
- Use ssh to open a remote shell to the server.
ssh herveus@oanda.sca.org
- Stop the dbserver process.
ps -fu herveus | grep oanda_server.pl kill <process-number>
- Copy my.cat from {DocumentRoot} to {DBServer}
cp oanda.sca.org/my.cat oanda-data
- Run the script to recreate mycat.pl.
./mk_cat_file -i my.cat -o mycat.pl
- By this point the database server should be quiescent, with no more active connections to port 1066.
- Restart dbserver process.
Congratulations! When you reach this point you’ve successfully updated the category files and published them to the Ordinary and Armorial web site.