Category Archives: Technical

Scraping XML from CAMENA

CAMENA – Latin Texts of Early Modern Europe was a great project at the University of Mannheim. The project ended in 2011, making available many thousands of pages of Neo-Latin texts, not only by German authors. Some of the texts are available in a machine-readable XML form, though (as the project carefully warns) the texts were not fully corrected, and are therefore to be considered as “raw material”. All material produced by CAMENA is available under a CC-BY-SA Open License.

CAMENA Thesaurus logo

An opportunity to experiment with a large collection of Neo-Latin XML is for us immensely attractive; we can use everything we had learned and developed for the Croatica et Tyrolensia project on other material.

The first step is, however, to collect machine-readable files of CAMENA. As its interface was designed more like a library than a database — perfect for reading, but less suited for other analytical approaches — finding all XML files (or, as will be seen, at least all that we could get) was a time-consuming operation.

We solved it by writing two Python scripts to scrape CAMENA’s index pages for links to their XML. The scripts can be seen as Github gists: Script 1 (which downloads XML files) and Script 2 (which only writes XML links to a file; the links are downloaded by a Linux Bash command such as wget -nd -i ../listthes2.txt).

The other approach — just collecting the links — enabled us to remove duplicates (CAMENA texts sometimes come from the same book, so the links point to thesame document).

There were some links which had to be corrected by hand, however, and some links which we were not able to retrieve at all. We also did not try to download page images (this time).

The textual part of CAMENA which is retrievable in XML consists of four collections (CERA, Historica et politica, Poemata, Thesaurus; the ITALI collection seems to lack any machine-readable XML), with ca. 1751 XML files.

We plan to publish here the full list of CAMENA XML links, and the files themselves.

Analysing XML with BaseX

TEI XML, often used for encoding texts in the humanities, actually carries in its elements certain semantic load. This means that just knowing which elements are used for encoding a file could provide some basic information about the file and its salient points.

We decided to experiment with two XQuery functions. One, currently called croala-analist, lists all documents in a database, e. g. for the database latty: latty. At the same time, as an orientational help, the function lists number of distinct elements used for encoding; e. g. there are eight different elements in the text part of the document putsch-c-brevis.xml.

The other function provides information on distinct elements: in XML/XPath parlance, who are their parents and their children, and how many occurrences of a certain element there are in the document. Here is the information for a latty document mentioned above: putsch-c-brevis.xml.

Joe Futrelle, "Guernica" (from Flickr)
Joe Futrelle, “Guernica” (from Flickr)

An additional challenge for our coding skills was making these functions as general as possible, i. e. applicable to all databases on our server. The two functions can be found inside the croala.xqm module in our Bitbucket repository (croalatransform / plutonbasex / repo /); functions are croala:analist and croala:anatomia.

Index of saints in Croatian Latin school drama

Saints often appear as characters in Croatian Latin school drama (which is mostly Jesuit). Here is how we index them in our digital bibliography. (A more technical note here.)

St Marianus, hermit and confessor of Berry, has until now received no identifier on the internet (though Fr Labbé wrote a page on him, available as a digital facsimile on BSB). Therefore his record has a digital ID / URL base exclusively in CroALa: croala.s.marianus.

On the other hand, St Margaret of Cortona is in Wikipedia, so we can use her Wikidata URL: www.wikidata.org/wiki/Q254632.

St Margaret of Cortona

Index personarum in CroALaBib

The bibliography branch of the Croatica et Tyrolensia project collects data on people and works connected with Croatian Latin. Following recommendations from the Cookbook of the SNAP:DRGN project, we have started putting our digital data online in a slightly more usable form (currently it exists in an XML database). The SNAP:DRGN Cookbook says:

The first task is to put your prosopography online, with stable URIs for as many entities as you can identify (person, name, source etc.). You need to be able to point (…) and reliably/persistently find (…) your record. This may be no more than an HTML page intended for human consumption.

For CroALaBib persons, this is now possible through the Index personarum on croala.ffzg.unizg.hr. Now the URL croala.ffzg.unizg.hr/basex/persona/jamometicd1e720 leads to data we have collected on Andrija Jamometić. The links and pages were created with a BaseX XML database instance, using the Bootstrap HTML framework.

Among the 1468 persons listed, there are some Italians and some other authors (e. g. Aristotle). They are included either because they were the audience of Croatian authors, or Croatian writers translated their works. Additions to the CroALaBib prosopography are expected soon — we will publish a list of persons involved with the Latin school drama.

A handwritten prosopography, Berkeley.

Technical note: installing PhiloLogic 4 on localhost

PhiloLogic, developed by the ARTFL Project and the Digital Library Development Center at the University of Chicago, is an excellent system for serving and querying XML texts. CroALa and LatTy are currently deployed using PhiloLogic 3, but version 4 is almost ready to be published; try it out on Github (github.com/ARTFL-Project/PhiloLogic4).

Some notes on how we tried — and, thanks to PhiloLogic 4 developers, also succeeded — to get PhiloLogic 4 working and serving Croatian Latin texts on a local machine (localhost in computer networking parlance) are published on the Croatica et Tyrolensia wiki: solr.ffzg.hr/dokuwiki/doku.php/z:philologic4.

Philologia Latina

Technical note: XSLT to number elements consecutively across parent elements

What follows is a solution to an XSLT problem.

The problem: in an XML document, we want to number consecutively elements which share a grandparent element, but not parent. The document contains multiple grandparents. In each we want to restart the counter (imagine a book with chapters which have section; we want to number paragraphs of each chapter consecutively, but across sections: Chapter 1 / Section 1 / Para 1 , Para 2 / Section 2 / Para 3, Para 4 / Chapter 2 / Section 1 / Para 1, Para 2 / Section 2 / Para 3 , Para 4 /).

We want to accomplish this with an XSLT stylesheet.

Short answer: add counts of all preceding paras in sections and preceding paras in current section; subtract the count of all preceding paras in other chapters, and add 1. Like this:

count(preceding::section/para) +
count(preceding-sibling::para) -
count(preceding::chapter/section/para) + 1

For a long answer, read more here: z:xslt-number.