Notes from the Heraldic Technology Roundtable

I’m pleased to report that this weekend’s roundtable session on heraldic technology at KWHSS was productive. More than a dozen people gathered for an hour-long conversation, and we talked through most of the items that had been on my draft agenda while sharing a lot of information and turning up a bunch of new ideas that I hadn’t thought of.

I’ve attempted to summarize the conversation below for reference by folks who weren’t able to attend the session in person.

With that out of the way, let’s dive in — the notes below review the major technology systems the College is responsible for, and then talk about some general challenges that all of our projects face.

---

Review of Systems and Projects

Main College Website

• Mathghamhain: There’s a bunch of stuff on the website, but let’s start by talking about the basic infrastructure, navigation, interface, and general “about us” information.
• Codex is responsible for maintaining this, but isn’t necessarily responsible for coming up with new visual designs or content improvements.
• The look and feel and navigation are a decade old, and the content is older.
• There have been multiple efforts to improve this over the last decade, all unsuccessful.

• Mathghamhain: As an effort to break the logjam, I’ve made a few sandbox sites which have copies of the top-level pages set up with a CMS so that we can easily try out changes, and then when we have a set of changes nailed down we can ask Codex to deploy them on the actual platform that hosts the real site. One sandbox looks exactly like the live site, one has some small style changes for better readability and mobile support, one has navigation and organizational changes, one has a significant visual redesign.
• Emma: Reis has started implementing the style changes for responsive layout. The code and content for the production site and staging sandbox is in a Git repository.

Articles

• Mathghamhain: Clarion and the sovereigns control the content and Codex handles publication.
• There’s a project underway to provide much-needed updates to this content, as discussed in the Education roundtable earlier today.

Rules

• Mathghamhain: Palimpsest updates the rules (SENA, the admin handbook, etc) on a regular basis; there were challenges in this area years ago, but there’s now a process in place for keeping them up to date which seems to be working smoothly.

OSCAR

• Mathghamhain: There were two rounds of effort over the last decade to replace OSCAR, but both fizzled.
• Istvan: The new plan is to incrementally clean up and modularize and improve the existing system rather than to replace it all at once, and I now have someone who has volunteered to help with the code.
• Malyss: In order to tackle my first small task, I’m completely refactoring multiple files of code, and I expect that more of that will need to be done in the future, but that investment will make future improvements easier.

• Istvan: OSCAR is built in PHP, JavaScript/jQuery, and MariaDB.
• I’ve added Selenium regression testing, which made it possible to refactor a bunch of database code and then run the tests and see what had broken. Other recent changes included better image support in comments and new support for uploading PDFs.
• In the longer run, I hope to be able to expose a REST backend paired with an Angular front-end, but that’s going to be a very long-term process.

Heralds Point Forms

• Mathghamhain: The Forms tool handles submissions workflow for events.
• Istvan: Elsbeth wrote the first version of this in Ruby, which I then rewrote in PHP, and significantly extended it.
• Emma: During Covid there was an overhaul so it now supports virtual events such as VHP as well as in-person events like Pennsic. There’s also now support for separate exchequers for each kingdom, although that hasn’t been tested.
• Istvan: Conceivably that system could also be extended to handle online submissions, but it would require a lot of additional work.

• Istvan: I’ve recently been updating it to work with the new version of the submission forms. I could use some volunteers to test heraldry.sca.org/forms; contact me if you need an account, and then try testing all of the options and let me know what issues you find.
• There’s an issue-tracker on gitlab.org but free projects are limited in how many users can access them so only Malyss and I can see it.

Post-Meeting

• Emma: Reis is working to replace the current Jade implementation with a new XSLT system; it’s now generating HTML and RTF which can be easily converted to PDF. It’s been run against a bunch of XML but still needs to tested against more to find edge cases.
• Elsbeth: The first version of this was implemented twenty-five years ago, and a bunch of new tools have been built since then. Eric wrote part of it that generated XML, I worked on the DSSL during my tenure in 1999.
• Istvan: Daniel made updates around the time of Shauna’s tenure.
• Emma: And the tools the current system depends on are old enough that they’re becoming increasingly fragile.

• Emma: After we replace the current toolchain, our next goal is to get it to where sovereigns don’t have to hand-edit XML.
• Mathghamhain: Someone in the East mentioned that it would be great if the new system could help convert internal letters into kingdom LODs.
• Emma: We’d need to figure out how that should work; I’ll pass the idea along to Reis.

LoARs and Precedents

• Mathghamhain: The letters are published reliably every month — but then to search them, we rely on Google; it would be great to change that.
• Elsbeth: We could pull out individual entries from the XML and feed them into a tool that let us search within entries, in chronological order. Tools like Solr or ElasticSearch have smart text search which knows about nearby words and word stemming, etc. I expect to have more time available in the months ahead and could work on this if I had some support.
• Herveus: Related to that is the precedents database, which I think is no longer functioning; when it was working, you could feed in the monthly XML and use it for text searches on individual cover letter items and so forth.
• Elsbeth: A good text search system could be used for precedent diving without requiring tagging.
• Istvan: A decade ago I wrote a text search tool for LoAR items and people said it wasn’t useful.
• Elsbeth: At one point I had code that I used to create the old precedent indexes (as well as ordinaries) from decisions during my tenure, but that’s lost now.
• Emma: The Google search tool isn’t reliable as it sometimes overlooks matches that are in the letters.

O&A

• Mathghamhain: Herveus has kept this code running well for 25 years, but under the hood it is showing its age.
• Herveus: The public-facing code is 90% still as I inherited from Iulstan in 2000; it was written around 1994 and to this day still has a lot of Perl 4 idioms, but thankfully it still works fine on a modern Perl.
• Elsbeth: The command-line C predecessor to that code was still working on my laptop until my last hardware upgrade.
• Herveus: Although the public web interface is mostly unchanged, I did change a lot of the backend tools, which I converted to use a SQLite relational database, which I do upload publicly although it’s not widely publicized. Also the my.cat files on the backend are slightly different. And I wrote the point-and-click armory-indexing tool which runs on X-Windows on my Mac.
• Herveus: I’ve told Laurel that I’m looking to lay down the job of Morsulus early next year, but I’ll be sticking around to back up my successor.

• Mathghamhain: One of my two software project goals for the O&A in the course of the next year or so is to build an alternative to Herveus’s armory indexing tool. The armory indexing consumes more than half of the hours of the Morsulus role every month, and it’s the part that is heraldically challenging as opposed to finicky technical procedure. If that tool was available in a web interface, it would make it easier to split up the work of Morsulus and let multiple people contribute to the indexing, or let someone do an indexing pass and then have someone else do a quality-inspection pass to look for minor things that were overlooked.
• The other O&A-related project I’d like to see happen is to make the public web interface a little prettier — not a huge change, but add a bit of navigation and visual styling.

• Herveus: There are also two alternate ways of getting to the O&A that a number of heralds have relied on for a long time: Hirsch’s O&A app for Windows, and the O&A-to-ebook converter.

HeraldStick

• Mathghamhain: HeraldStick was a way of mirroring the O&A search tools, as well as a whole bunch of websites including heraldry.sca.org, the PicDic, the MNA, and others, and getting them to run on your laptop while you’re at an event without web connectivity.
• Sisuile: I used to rely on HeraldStick for consultations at events.
• Istvan: The HeraldStick project is dead; I would have to rebuild it from scratch, likely as a Docker container.
• Emma: That would be a great thing for a volunteer to work on.
• Wu: I can do some recruiting to help you find volunteers for that.

Scanned Legacy Archives

• Emma: For several years now, all of the old scanned pre-OSCAR paper archives have been available online in full, but restricted to password-protected access for the sovereigns and a couple of key staff; we’re no longer mailing these around on hard drives.
• The project to extract the armory images and make those publicly accessible is underway. I’ve been handling the technical part, which uses Istvan’s automated image trimmer in batch mode, and then Helena and Shauna go through and look for items which should be exclude, or which didn’t trimmed automatically, or various kinds of data quality problems.
• That’s also turning up various things that need to be corrected in the O&A, and places where the archives are wrong while the O&A and LoARs are correct.

• Herveus: I have a hard copy of the first armorial from 1983 and I’ve been thinking I could go through and cross-check it against the O&A and LoARs.
• Emma: We ought to scan that, because yes, there’s a lot of entries from 1983/82/81 where Wilhelm says “we registered this armory in the 1970s” but when you go look at the LoARs from that time the items don’t appear there. There are clearly items that were added to the files and published in an armorial but not included in an LoAR, or only mentioned in a letter retroactively.

• Mathghamhain: When the trimmed armory images are publicly available, I would like it if someday when you did a search in the O&A, for there to be a way to click through and see the corresponding images from the archive and from OSCAR.
• Elsbeth: I’d also like to be able to click through and see the decision text from the individual LoAR entry, at least for the later letters which are in XML format.
• Emma: We should be able to make a table of associations between O&A unique IDs and the images and letter items.
• Istvan: OSCAR will link you from a submission to the LoAR that letter was scheduled to be published on (which is usually correct) but not to the individual item.

---

Big Questions / Challenges

Recruiting Technical Contributors

Challenge: How can we get more people involved?

• Mathghamhain: This is always going to be challenging, because successful technology projects generally require a small number of volunteers to put in a lot of sustained effort.
• Coming up with an idea is easy, but building out and launching a useful resource takes work, and keeping it running for a decade is a big commitment.
• Most volunteer tech projects (not just in the SCA) are implemented by a single person; it’s difficult to recruit other volunteers who’re familiar with the same tools and techniques, and coordinating multiple people requires a lot of additional overhead.

• Given that, how do we get a few more people into a position where they contribute to the big projects discussed earlier?
• None of the projects we’ve discussed are ones where it would be helpful to add twenty well-meaning random folks without specialized skills, but on the other hand, it would be good to have a couple more really savvy folks.
• We don’t want to be in a situation where there’s a single person who’s the only one who can do this job, and they can’t ever quit, and if something happened to them we’d be in trouble. For a couple of our key projects (but not all of them) we’ve gotten that “bus factor” number up from one to two, but I’d be happier if it went from two to three… how can we find a couple more dedicated people with the right skills?

• Istvan: I would not mind more people but they have to be the right people.
• Wu: I’d like to help with recruiting.

• Emma: One thing that we do need more of is folks who are comfortable hand-editing HTML who can help to convert articles that are written in Google Docs or Word or whatever into clean HTML for use on the website. For example, last year’s proceedings are in a Google Drive and need to be converted.
• Wu: Yes, that conversion is what has delayed their publication.
• Bróccín: The HTML export from Google Docs should be a good starting point.
• Emma: Yes, but then you have to go back in and clean it up.
• Wu: I can definitely help recruit people for this, including from among non-heralds.
• Sisuile: I will happily help with that process.

IT Project Visibility

Challenge: How do we make it easier for technical people to see what projects are underway and figure out how to get involved?

• Mathghamhain: Both in the SCA in general, and specifically in the SCA heraldry world, it’s difficult for folks to learn about IT systems and projects unless they make a significant effort to connect to the inner circle. We’ve been slow to adopt the “working in public” methodologies that are common in other volunteer tech communities; for example, few projects have a publicly-discoverable source code repository or issue tracker.
• Emma: A lot of projects do have public source-code repository but there’s no way to find them; even if we’re not all using the same platforms we should at least have a web page that lists projects and their associated resources.
• Sisuile: And there definitely are systems for which the code is not public.
• Wu: People might be reluctant to share information because being the only person who knows something makes you important.
• Istvan: And a possible drawback to making things public is that it might open us up to having them be used by people whose goals we don’t support.

Cataloging Unofficial Resources

Challenge: How do we make it easier for heralds to discover non-College sites and tools?

• Mathghamhain: Laurel and the College are understandably reluctant to undertake new official projects, because ensuring they’re authoritative and keeping them running for decades is a huge commitment.
• It’s easier to build things as non-College projects, and lots of things get built that way — but the resulting resources are not easily discoverable; there’s no centralized directory so people only find out about those things by word of mouth, and if only a few people ever use something, it’s very discouraging for the developer.

• Mathghamhain: I’d like to brainstorm ways to make it easier for heralds to discover cool things that individuals have built.
• Emma: We should have a page on the website of “cool toys” that includes all of those useful things that aren’t official, like the Traceable Art.
• Mathghamhain: And Xavid’s blazon parser, and the SCA heraldry wiki, and so many great blog posts by Cormac and Marie and Alys, and so on.
• Emma: I don’t want to make additional work for Wu, but we should publish links to those.
• Wu: Yes, please give those to me for inclusion on the site.

• Bruni: Would it make sense to occasionally have a cover-letter item that mentions “this is a cool thing you could go look at?”
• Wu: Yes, I plan to talk to Birgita about doing more “update from Clarion” items on the cover letters.

• Andrewe: If you set up a repository of unofficial resources, you need a process for deciding what to include. Some volunteer efforts might be well-intentioned but not actually high-quality.
• And especially if you say “there’s a need for this kind of resource,” and then someone goes out and builds something, but then you say “oh, actually, that’s not great, and we’re not going to include it on our list or encourage people to use it,” that will be very discouraging, so you need to make the acceptance criteria clear ahead of time.
• Wu: A list of non-College resources is always going to be a gray area compared to our official resources.