Monthly Archives: April 2016

Scraping XML from CAMENA

CAMENA – Latin Texts of Early Modern Europe was a great project at the University of Mannheim. The project ended in 2011, making available many thousands of pages of Neo-Latin texts, not only by German authors. Some of the texts are available in a machine-readable XML form, though (as the project carefully warns) the texts were not fully corrected, and are therefore to be considered as “raw material”. All material produced by CAMENA is available under a CC-BY-SA Open License.

CAMENA Thesaurus logo

An opportunity to experiment with a large collection of Neo-Latin XML is for us immensely attractive; we can use everything we had learned and developed for the Croatica et Tyrolensia project on other material.

The first step is, however, to collect machine-readable files of CAMENA. As its interface was designed more like a library than a database — perfect for reading, but less suited for other analytical approaches — finding all XML files (or, as will be seen, at least all that we could get) was a time-consuming operation.

We solved it by writing two Python scripts to scrape CAMENA’s index pages for links to their XML. The scripts can be seen as Github gists: Script 1 (which downloads XML files) and Script 2 (which only writes XML links to a file; the links are downloaded by a Linux Bash command such as wget -nd -i ../listthes2.txt).

The other approach — just collecting the links — enabled us to remove duplicates (CAMENA texts sometimes come from the same book, so the links point to thesame document).

There were some links which had to be corrected by hand, however, and some links which we were not able to retrieve at all. We also did not try to download page images (this time).

The textual part of CAMENA which is retrievable in XML consists of four collections (CERA, Historica et politica, Poemata, Thesaurus; the ITALI collection seems to lack any machine-readable XML), with ca. 1751 XML files.

We plan to publish here the full list of CAMENA XML links, and the files themselves.