Tag Archives: xml

Scraping XML from CAMENA

CAMENA – Latin Texts of Early Modern Europe was a great project at the University of Mannheim. The project ended in 2011, making available many thousands of pages of Neo-Latin texts, not only by German authors. Some of the texts are available in a machine-readable XML form, though (as the project carefully warns) the texts were not fully corrected, and are therefore to be considered as “raw material”. All material produced by CAMENA is available under a CC-BY-SA Open License.

CAMENA Thesaurus logo

An opportunity to experiment with a large collection of Neo-Latin XML is for us immensely attractive; we can use everything we had learned and developed for the Croatica et Tyrolensia project on other material.

The first step is, however, to collect machine-readable files of CAMENA. As its interface was designed more like a library than a database — perfect for reading, but less suited for other analytical approaches — finding all XML files (or, as will be seen, at least all that we could get) was a time-consuming operation.

We solved it by writing two Python scripts to scrape CAMENA’s index pages for links to their XML. The scripts can be seen as Github gists: Script 1 (which downloads XML files) and Script 2 (which only writes XML links to a file; the links are downloaded by a Linux Bash command such as wget -nd -i ../listthes2.txt).

The other approach — just collecting the links — enabled us to remove duplicates (CAMENA texts sometimes come from the same book, so the links point to thesame document).

There were some links which had to be corrected by hand, however, and some links which we were not able to retrieve at all. We also did not try to download page images (this time).

The textual part of CAMENA which is retrievable in XML consists of four collections (CERA, Historica et politica, Poemata, Thesaurus; the ITALI collection seems to lack any machine-readable XML), with ca. 1751 XML files.

We plan to publish here the full list of CAMENA XML links, and the files themselves.

Analysing XML with BaseX

TEI XML, often used for encoding texts in the humanities, actually carries in its elements certain semantic load. This means that just knowing which elements are used for encoding a file could provide some basic information about the file and its salient points.

We decided to experiment with two XQuery functions. One, currently called croala-analist, lists all documents in a database, e. g. for the database latty: latty. At the same time, as an orientational help, the function lists number of distinct elements used for encoding; e. g. there are eight different elements in the text part of the document putsch-c-brevis.xml.

The other function provides information on distinct elements: in XML/XPath parlance, who are their parents and their children, and how many occurrences of a certain element there are in the document. Here is the information for a latty document mentioned above: putsch-c-brevis.xml.

Joe Futrelle, "Guernica" (from Flickr)
Joe Futrelle, “Guernica” (from Flickr)

An additional challenge for our coding skills was making these functions as general as possible, i. e. applicable to all databases on our server. The two functions can be found inside the croala.xqm module in our Bitbucket repository (croalatransform / plutonbasex / repo /); functions are croala:analist and croala:anatomia.

Technical note: installing PhiloLogic 4 on localhost

PhiloLogic, developed by the ARTFL Project and the Digital Library Development Center at the University of Chicago, is an excellent system for serving and querying XML texts. CroALa and LatTy are currently deployed using PhiloLogic 3, but version 4 is almost ready to be published; try it out on Github (github.com/ARTFL-Project/PhiloLogic4).

Some notes on how we tried — and, thanks to PhiloLogic 4 developers, also succeeded — to get PhiloLogic 4 working and serving Croatian Latin texts on a local machine (localhost in computer networking parlance) are published on the Croatica et Tyrolensia wiki: solr.ffzg.hr/dokuwiki/doku.php/z:philologic4.

Philologia Latina

Technical note: XSLT to number elements consecutively across parent elements

What follows is a solution to an XSLT problem.

The problem: in an XML document, we want to number consecutively elements which share a grandparent element, but not parent. The document contains multiple grandparents. In each we want to restart the counter (imagine a book with chapters which have section; we want to number paragraphs of each chapter consecutively, but across sections: Chapter 1 / Section 1 / Para 1 , Para 2 / Section 2 / Para 3, Para 4 / Chapter 2 / Section 1 / Para 1, Para 2 / Section 2 / Para 3 , Para 4 /).

We want to accomplish this with an XSLT stylesheet.

Short answer: add counts of all preceding paras in sections and preceding paras in current section; subtract the count of all preceding paras in other chapters, and add 1. Like this:

count(preceding::section/para) +
count(preceding-sibling::para) -
count(preceding::chapter/section/para) + 1

For a long answer, read more here: z:xslt-number.