Improving the O&A Website

I’ve recently been named the incoming Morsulus herald, and over the next few months I’m going to have my hands full getting up to speed on the internals of that position — but once that is out of the way, I hope to turn my attention to the development of a more modern web interface for the O&A site.

I’m not a magician, and the new interface isn’t going to miraculously answer every heraldic question with the push of a single button, but I think we can make some modest improvements that make the O&A site easier to use for a bunch of people.

As part of that effort, I’m seeking to gather feedback from folks with a range of different experiences and focuses as to what aspects of the current site are most annoying, or require a lot of extra work on your part, or frequently lead to mistakes or roadblocks.

So, if you’ve got a big feature idea, or a petty gripe, or fantasy wishlist, please send it my way! If there’s a sharp corner that you’ve frequently bumped into, or a process that requires you to open up five separate tabs to track down one answer, or some feature for which you need to look up the instructions every single time — send them my way and I’ll see what I can do to improve those in the next round.

You can comment here, or email me at math@digitalherald.org, or message me on Discord (I’m on the Known World Heraults server, among other places).

Existing Interface Will Remain as “Classic Mode”

For the folks who love the current O&A site, and have built up muscle memory using it in its current configuration, don’t worry — I don’t want to disrupt your workflow, and you’ll still be able to copy-n-paste codes out of my.cat just like folks did thirty years ago.

My intention is to leave the current web interface running exactly as it is today, and simply offer an additional interface as an alternative for folks who might prefer it.

O&A Website Traffic Statistics

The O&A website run by the SCA College of Arms is a crucial resource for the Society’s heraldic community, serving as the canonical reference for registered names and armory.

In August 2024, I was given the opportunity to review and analyze five weeks of server access logs in order to get a sense of current usage patterns.

As with all web traffic analysis based purely on log files, this exercise had some significant limitations: in the absence of Google Analytics or an equivalent web analytics tool, I had to rely on IP addresses and user agent strings to classify traffic and reconstruct visitor behaviors.

All of the numbers below should be taken as estimates, and it’s possible that some of them are off by a significant factor — but for the time being, this is the best source of data we have to go on, so let’s dive in.

Visitors

About 3,000 browsers visited the site in the 35-day period from July 14 through August 17, 2024. (Cases where a person visited the site from multiple locations, such as on their home computer and also from their mobile device on the road, would be counted as two browsers in this analysis.) This traffic does not include bots and crawlers, discussed separately below.

Of these, one thousand of the browsers visited just one page and then left forever; many of these are likely folks who wandered in via search engine links and either had their question answered immediately, or realized that they were on the wrong site and left.

Another thousand of the browsers viewed two to four pages and then left, and the final thousand viewed five or more pages, some many more — of those thousand heavier users, about 350 browsers viewed 5-10 pages, another 300 browsers viewed 10-20 pages, and the final 350 browsers viewed over 20 pages.

At the high end, 35 browsers viewed more than a hundred pages each over the course of the month, including one outlier who viewed an average of almost a hundred pages per day! (And yes, I double-checked and it did appear to be a real person.)

All told, those 3,000 browsers issued just over 30,000 requests for pages, including the home page, search forms, and search result listings.

Search Modes

The key function of the O&A website is searching the armorial database in one of a handful of ways. This breakdown shows roughly how much each search mode is used:

  • 4,840 views / 1,100 browsers — Name pattern search and results
  • 2,080 views / 660 browsers — Ordinary pages and armory listing
  • 2,440 views / 480 browsers — Blazon pattern search and results
  • 4,150 views / 590 browsers — Complex search search and results

After people view the results of a search, they often click through to the individual registrant names, and sometimes also to the year/month/kingdom.

  • 2,424 views / 1,020 browsers — Individual name view
  • 637 views / 350 browsers — Letter year/month/kingdom view

Search Options

Most of the armorial search forms support a common set of display options, including armory descriptions and glossary links. Of the roughly 11,420 searches using such forms:

  • 1,804 had armory descriptions enabled
  • 34 had the glossary enabled
  • 25 had dates shown in SCA format

There were also 13 requests to the standalone glossary viewer; this feature clearly is not used much.

Of the 4,840 name-pattern searches:

  • 285 had case-sensitivity enabled
  • 28 were in “narrow” mode

Of the 2,440 blazon pattern searches:

  • 13 had case-sensitivity enabled

Of the 4,150 complex searches:

  • 45 had “only show registered results” enabled
  • 2 had “only show unregistered results” enabled
  • 15 had raw mode output enabled

Some of these features are used so rarely that these occasional invocations might just be accidents.

Complex Search Criteria

Of the 4,150 complex searches, here’s the number that used at least one of the indicated type of criteria:

  • 3,713 searches for armory
  • 251 searches for blazon
  • 177 searches for name
  • 75 searches for broad name
  • 37 searches for date/kingdom
  • 21 searches for record type

Notably, the complex search form allows people to combine multiple criteria, either of the same or different type, in any number of combinations. Unsurprisingly, various combinations of two or more armory criteria does constitute over 85% of the usage, but the variety of other search types speaks to the flexibility of this tool.

The frequency for various combinations of criteria is set forth below.

  • 904 armory (3x)
  • 784 armory (4x)
  • 626 armory (5x)
  • 543 armory (2x)
  • 362 armory (1x)
  • 307 armory (6x)
  • 131 name
  • 100 blazon
  • 77 armory (7x)
  • 68 broad name
  • 43 blazon (2x)
  • 43 — (no search criteria entered)
  • 22 armory AND blazon
  • 19 armory (8x)
  • 16 blazon AND armory
  • 15 name AND date/kingdom
  • 15 armory (3x) AND blazon
  • 14 name (2x)
  • 14 blazon (2x) AND date/kingdom
  • 7 name AND record type
  • 7 blazon AND armory (2x)
  • 6 armory AND name
  • 6 armory (2x) AND blazon
  • 5 broad name AND date/kingdom
  • 5 blazon AND record type
  • 5 blazon (3x)
  • 4 blazon (4x)
  • 3 record type
  • 3 blazon AND armory (3x)
  • 6 armory (4x) AND blazon
  • 3 armory (9x)
  • 2 record type AND broad name
  • 2 date/kingdom
  • 2 blazon (2x) AND armory
  • 1 record type (2x)
  • 1 record type AND name
  • 1 name (3x)
  • 1 name AND date/kingdom AND record type
  • 1 armory AND record type
  • 1 armory (2x) AND blazon (2x)
  • 1 armory (2x) AND name
  • 1 armory (5x) AND blazon
  • 1 armory (6x) AND blazon (2x)

Crawlers and Bots

That human browser traffic is dwarfed by automated crawlers and bots: nearly 600,000 requests were received from devices other than people using a browser, making up over 90% of the total traffic.

This traffic was split between traditional search engines (Google, Bing, Yandex, DuckDuckGo), “AI” crawlers (ChatGPT, Alexa, and numerous unlabeled LLM crawlers), Facebook’s crawler, and a smaller number of malicious bots probing for common security vulnerabilities like insecure PHP or ASP scripts.

Some of this traffic is useful and some is not, but it is challenging to efficiently filter out the malicious and wasteful requests without incorrectly rejecting some real requests from actual people, and so for now we’re stuck with it.

Morsulus Software Agenda

I’m gearing up to take on the Morsulus office in the next couple of months, and will be posting here about software development activity in three broad areas:

  1. The toolchain that Morsulus uses to maintain the master version of the O&A database currently runs on the officer’s personal computer; I’d like to migrate this so it runs in the cloud.
  2. The O&A website mostly looks and works the same way it did when it was first written thirty years ago; I’d like to add a modern web interface while retaining the existing UI as a “classic mode” option for folks who are accustomed to it.
  3. The LoARs are authored as structured data but then published as very simple HTML files; in the long term, I’d like to see a modern presentation that made it easier to browse and search, and cross-linking to related entries, to the O&A, and to emblazons from OSCAR and the legacy archives.

Those are listed in intended sequence; I’ve already started work with the back-end Morsulus toolchain in order to enable the office transition, and hope to be able to turn my attention to the O&A website in a few months, and to the LoARs within a year, if things go well.

Publishing the LoARs to the Web

My thanks to Herveus Gnome for letting me watch over his shoulder as he published the November LoAR so that I could document this process here.

In addition to maintaining the O&A website, the Morsulus office is also responsible for publishing the LoARs to the College’s website.

Production of the LoARs in multiple formats (HTML, PDF, XML) is handled by Silver Staple, the post-meeting clerk. The final LoARs arrive as an email with attached zip files, distributed to the Sovereigns and relevant staff.

After unpacking the final-FORMAT.zip package of HTML and PDF files, the resulting folder is renamed to numeric form of the month — in the case of November that is 11.

Then an index page is created by copying welcome.html from the prior month’s folder, and the month name is updated in both written and numeric form — s/October/November, s/24-10/24-11. The errata link might need to be commented or uncommented, if last month’s letter included one and this one doesn’t, or vice-versa. The copyright date also needs to be updated for the first letter posted each year. Other material in the index file, such as a line of PHP code that was added recently, can mostly be ignored.

Make sure that the files are all set to world-readable, or Unix-style 644 permissions. Then copy the files to the production server with scp -r 11 charlemagne.sca.org:loar/2024. (This pathname works because Herveus has a symlink at ~/loar that points to the web directory that holds the LoAR files.)

Lastly, the loar/welcome.html file is edited to uncomment the link to the current month’s directory. When publishing the January letter each year, a new line of month links has to be made by copying the previous year’s and then commenting out most of them.

At this point, the newly-published LoAR should be available on the http://heraldry.sca.org/loar/ page, and this process is completed for the month.

Running the O&A via Docker

More than a decade ago, Tanczos Istvan put together “HeraldStick,” a software package for heralds running consultation tables at events with poor connectivity. The package let a herald with a Windows laptop run a pocket web server with a copy of the O&A web application and a bunch of sites full of names and armory articles and resources, so they could do the same kinds of documentation lookups and conflict checks that they would normally do online even though they were sitting in a tent beyond reach of wireless links.

Continue reading “Running the O&A via Docker”

Prototyping a New Look for the O&A

As part of my ongoing efforts to prototype possible changes to the College of Arms website, I’ve applied a related set of stylistic changes to my local mirror of the SCA’s armorial database.

Under the hood, this site is running a fork of the main O&A codebase (along with a custom local configuration file), so the core functionality is the same, but a number of changes have been made to the user interface throughout.

Site-Wide Visual StylE

The most obvious changes are to the overall appearance of the site.

Continue reading “Prototyping a New Look for the O&A”

O&A Search for Unregistered OSCAR Submissions

Here’s a nifty trick for the folks who might be running their own O&A server — which is admittedly a very, very small audience.

The OSCAR software can generate a supplementary data file in the same format as oanda.db which contains the names and armory currently in-process on LoIs which have not yet made it to an LoAR.

Continue reading “O&A Search for Unregistered OSCAR Submissions”

Descriptions for Field Division Directions

Following the 2021 rules change, SENA A5F1b now says that that changing the direction of partition lines is considered a Substantial Change, as is the difference between divided and undivided fields.

As a result, when using the Complex Search form to do conflict checking for fielded armory, we can add a second line for the field that matches anything with a similar direction. Continue reading “Descriptions for Field Division Directions”

A Roadmap to Morsulus’s Monthly Updates

Last year I wrote up a summary of Morsulus’s process for applying updates from the monthly LoARs to the Society’s O&A database, and then more recently I put together a high-level visual overview of the context in which the Morsulus herald does his work.

More recently, I thought it might be useful to use a similar visual style to summarize the monthly update process, as a way of giving people a graphical roadmap to the data flow before they dive into the step-by-step technical nitty-gritty. Continue reading “A Roadmap to Morsulus’s Monthly Updates”