Features

• Generative models support 105 languages! See here for all of them
• For backend, one may choose OpenAI/ChatGPT, Mistral, or any model that Ollama runs (e.g., Llama). I have added support for Ollama’s (paid) cloud service (though have not tested this yet).
• The project keeps the Stanza backend see all 11 languages supported with the Stanza backend here
• The public API remains nearly unchanged from previous v1.x.
• Likewise the architecture is nearly the same, too, though certain processes have dropped and others added in this new, v2.. See page 22-25 of our publication: https://aclanthology.org/2021.acl-demo.3.pdf
• We still map the string labels into valid UD POS tags, morphological features, and dependency syntax labels. Arguably the data types are easier to use. It is certainly easier to make corrections when incorrectly named outputs come from a model see here for examples.
• The architecture is clean, it builds fast, and has extensive logging; all of which mean that debugging and writing tested patches will be much faster than before.

Installation

$ pip install "cltk[openai,stanza,ollama]"

See more at: https://docs.cltk.org/quickstart/#install.

Use

ChatGPT

Create a file .env and put in it your ChatGPT key (OPENAI_API_KEY), i.e.:

OPENAI_API_KEY=YOURSECRETKEYDONTSHARE

Then run:

from cltk import NLP
nlp = NLP("lati1261", backend="openai", suppress_banner=True)
doc = nlp.analyze("Gallia est omnis divisa in partes tres.")
print(doc.words)

Ollama

Install (Ollama)[https://ollama.com/], start the application, and find a model to download. In my testing I have used llama3.1:8b.

from cltk import NLP
nlp = NLP("lati1261", backend="ollama", suppress_banner=True)
doc = nlp.analyze("Gallia est omnis divisa in partes tres.")
print(doc.words)

Should anyone try Ollama’s cloud GPU service, add OLLAMA_CLOUD_API_KEY and your key to the .env.

Stanza

from cltk import NLP
nlp = NLP("lati1261", backend="stanza", suppress_banner=True)
doc = nlp.analyze("Gallia est omnis divisa in partes tres.")
print(doc.words)

Weaknesses and gaps

• I should note that this is closer to “beta” than “production” software. I have decided to promote it to master anyways due to my limited time and limited number of test users.
• I have not added Spacy integrations yet, but hope to do so. For the past several years, I have had consistent difficulties getting it to install correctly on users’ machines.
• I have no benchmarks for the generative LLM, but hope to do so! When inspecting the generative LLM, it seems they have been trained on the entirety of the Universal Dependencies treebanks, and if so then new benchmarks ought be created, even if small.
• Calls to ChatGPT are slow! I am using async calls (per sentence) to speed things up, but this only goes so far. It’s relatively expensive, too, to label large amounts of text, but it’s reasonable to expect that costs will go down over time.
• The mapping of non-UD (but correct) labels is not complete. I have put a significant number of hours correcting those that come from ChatGPT 5 for Ancient Greek and Latin. I observe that other languages emit a number of these mistakes, but I do not feel confident mapping them. This will take a number of experts if it’s to ever be complete.
• There is some valuable code that did not (or has not yet) made its way over, including some wrappers around Collatinus, TLGU, and more. I have not made any firm decisions about these yet.

Legacy support

I insistent in stating version 1.x will not receive any more support. Yes, I should have given better warning.

As we did in the transition of v0 to v1, I have preserved the code, docs, and packages on PyPI. So if you do not want to upgrade to v2, you will always have available the last 1.x version with pip install cltk==1.5.0. Its docs are now at https://v1.cltk.org and the code is at branch v1. Likewise the initial generation of the CLTK is at branch v0, https://v0.cltk.org/, and pip install cltk==0.1.121.

• Kyle’s slides on the CLTK, generally
• Clément’s notebook on making a custom Process and examples using Old Norse
• Todd’s slides on MLOps and large language models

Some time prior to this, we also officially promoted our version 1.0 to “production”.

Preferably in a new virtual environment, in either Python 3.7 or 3.8, pull the latest “alpha” with:

$ pip install –pre cltk

The docs should be enough to begin stress-testing the new code:

• Docs: https://alpha.cltk.org
• Source: https://github.com/cltk/cltkv1 (Note: Not our main repo!)

For now, please open issues in this new development repo. In the coming weeks, we will merge the two Git trees together.

Languages which lack such digitized data, I’ll call “under-resourced.” To help explain, allow me to borrow a definition of a “language resource”:¹

The term Language Resource refers to a set of speech or language data and descriptions in machine readable form, used for building, improving or evaluating natural language … for language studies, electronic publishing, international transactions, subject-area specialists and end users. Examples of Language Resources are written and spoken corpora, computational lexica, terminology databases, speech collection, etc.

For the CLTK, language resources normally take the form of static data sets, such as text corpora (annotated or plaintext), treebanks, and lexica (dictionaries of various sort). Resources also take the form of algorithms, which are usually rules-based and optionally rely upon data sets. For an example of the former sort of algorithm, see Luke Hollis’s stemmer for Latin, which is itself a Python-language implementation of rules defined by previous peer-reviewed scholarship (Schinke et al., 1996). For an example of rules-plus-data, Todd Cook’s Latin syllabifier first defines a variety of character types in ScansionConstants.py and then uses these in Syllabifier.syllabify(). Examples of well-resourced languages, in both data and algorithm, include Ancient Greek and Latin, for which not coincidentally the CLTK has excellent coverage (look for Greek or Latin data sets on GitHub, to get an idea).

The CLTK is an NLP project, and it eschews ventures outside of this mission. Simply put, this means that the CLTK is neither a data annotation nor a (non-technical) user-facing project. In reference to the classic three-tier software architecture, the CLTK is exclusively restricted to the middle presentation tier, relying upon the data storage tier as upstream dependencies and adopted downstream by the presentation tier. The CLTK has a vested interest in the health of data storage and presentation tiers, however our project’s core NLP task, to write NLP algorithms by leveraging already available data, is formidable enough. To satisfy the needs of downstream makers of applications, we write what we hope are sensible and well documented APIs.

To illustrate briefly the challenges of even relatively simple data creation, I am reminded of several students whose natives tongues were, respectively, Telugu and Kannada. They approached with proposals to do NLP in their languages, and when I pointed out they’d need data, each came up with the idea of doing OCR to obtain plaintext corpora. After all, plenty of digitized book images could be found online. However, preliminary experiments demonstrated that OCR for these particular non-Latin characters was of very low accuracy. Simply making an OCR model would have constituted a summer project itself (one outside our bounds, nevertheless). The creation of annotated texts is a laborious in the extreme, requiring equal degrees of passion and technical expertise.

Having explained what a language resources are and why they are so important to the CLTK, I will next explain the states in which under-resourced (dead) languages exist and how one can decide whether a given language is under-resourced. Languages generally fail to meet the minimum bar of “resourced” due to one or more of the following attributes:

1. resources are not digitized;
2. resources are only available under non-free license;
3. resources are available but do not amount to a “critical mass” around which serious NLP tooling may be developed.

First, simply enough, if data has not been digitized, then it is not available for computational processing. As mentioned above, book page images are quite a distance from even plaintext files.

Second, non-free data poses a significant problem. My use of “free” here corresponds generally to that expounded by Richard Stallman as something more particular than no-cost and open source:

When we call software “free,” we mean that it respects the users’ essential freedoms: the freedom to run it, to study and change it, and to redistribute copies with or without changes. This is a matter of freedom, not price, so think of “free speech,” not “free beer.” (“Why Open Source misses the point of Free Software”)

Intended to support rigorous quantitative scholarship, users of the CLTK simply must have the ability to manipulate and redistribute data used to create results. Contrary to the understanding of many humanists I have met, “open access” and “open source” resources that have proprietary licenses are unstable foundations for projects like ours, the legacy of which (we hope) will be measured in decades, not years. It is undeniably practical for a scholar to quickly publish an article which uses proprietary data or software, however his results will likely not be reproducible, frozen in time, and lost to history.

Third, The line between a resourced and under-resourced language is not always clear-cut. For example, there could be annotations available, however they be of a rather small in number (the case with Tibetan POS); or some data sets be very robust for some tasks (e.g., a lexicon and word-lookup) however be completely lacking treebanks (Pali); or in the absence of any digitized data, certain within-reach algorithms may be written. For such in-between languages, if a potential contributor will first do diligent research on data sets, project mentors will be delighted to discuss algorithmic possibilities.

Is there some limited data creation that could fall within scope of the CLTK? We have had some success, for example, in generating stopword lists either according to specific algorithms minimizing manual curation (e.g., run tf-idf on a corpus and removing nouns) or with very narrow scope (e.g., writing every possible inflection of a definite article). I don’t want to preclude other ideas, so I would generalize that this issue may be evaluated on a case-by-case basis.

All the above is intentionally discouraging of those who might want to disembark on an ill-fated proposal to add new language support to the CLTK. On the flip side, we ought to highlight that there are several shining examples of languages I might consider well-resourced and not currently covered by the CLTK. Those are, in no particular order: Hebrew, Arabic, Sanskrit, Chinese, and Old English.² (There are likely others, too.) To conclude, I encourage those who would like to work with a mentor from our team, to first consider and have preliminary answers to the following questions:

1. What free data sets are available? If any non-free or ambiguously licensed language resources, are they so important that we would need to use them?
2. What NLP algorithms can I write with this data?
3. What free NLP algorithms have already been written? Do I have the skills, approximately, to re-implement them?
4. What data am I missing and is it reasonable to create this data within the project? Define very precise scoping and make an effort estimate.
5. Do you have the language skills to validate your own research? If not, have you identified another (say, a professor) who would be able to help? Please remember that the CLTK is exclusively interested in the pre-modern form of a language; so for example even though Hindi may be considered an “classical” language, its form as spoken today differs greatly (or so I am told) from how it was written 1,000 years ago.
6. How do you rate your skills in programming in Python, machine learning, and NLP? We have and do work with various types of specialists (some more human language, some more computer), however knowing your particular background helps to pair you with the right mentor.

With these questions answered, even if not perfectly, the core CLTK will be able to respond with concrete advice, criticism, and recommendations to further develop your proposal.

1: From the Under-resourced Languages group of the European Language Resources Association (ELRA).

2: Old and Middle English have good footing, due to the major pre-modern Germanic contributions by Eleftheria Chatziargyriou and Clément Besnier, however relative to the huge amounts of source data, lots of valuable work remains.

The Classical Language Toolkit is happy to present our Google Summer of Code projects for 2018:

• Eleftheria Chatziargyriou (Aristotle University of Thessaloniki): Extending NLP functionality for Germanic Languages.
• Andrew Deloucas (Harvard University): The Road to CDLI’s Corpora Integration into CLTK: An Undertaking
• James Gawley (University of Buffalo): Expanding the CLTK with Synonyms, Translations and Word Embeddings

As we see it, this year’s projects each have the potential to move the CLTK forward in its own way.

Eleftheria’s project takes on one of the CLTK’s primary development paths, extending the core tools to a new language and/or language group. Her project will be the first substantial introduction of resources for Germanic languages in the project, a welcome addition.

Andrew’s project moves in a similar direction, but with a different focus. The incorporation of cuneiform languages in the CLTK is a clear desideratum. This project not only needs to extend and develop existing corpora, resources, and tools but needs to rethink how these tools work at a basic level with consideration of encoding and multiple transliteration schemes, among other things. A challenge to be sure, but an opportunity to open up future development for other pictographic/iconographic text systems.

James’s project capitalizes on an emerging priority among CLTK developers and contributors – the use of the project as the basis for cutting-edge research in natural language processing for historical languages. He will develop the CLTK’s ML models and datasets for the discovery of intra-language synonyms and inter-language translations. Another clear benefit of James’s work will be that by formalizing model production for Latin and Greek, he will also provide a path forward for the development of similar resources for other CLTK languages.

Congratulations again to this year’s CLTK Summer of Code participants. To the participants: the organizers and mentors can’t wait to start working with you on taking your projects from proposal to working code to commits to the codebase. To our development community: look forward to upcoming blog posts from each of the participants with more detailed introductions. And to everyone whose project was not selected this year, we want to acknowledge your hard work on the proposals and encourage you to try again next year. The simple fact is that there are more great ideas for NLP work on historical languages than GSoC spots. We look forward to seeing how your work develops going forward and encourage you to submit again next year.

We’re happy to announce two exceptional students, Charles Pletcher and Natasha Voake, whom the CLTK will mentor for the 2017 Google Summer of Code. (See official GSoC page here.)

By the end of the summer, if not before, we will offer updates on what Charles and Natasha accomplish.

Charles Pletcher

Proposal:

I plan to build a flexible platform for teachers to annotate texts in the CLTK Archive. The purpose of the annotations is twofold: first, to guide the students in their (re)readings of the text; and second, to provide a ready-to-hand reference to which students and teachers can return.

Teachers will be able to enter annotations either by uploading a list of references and notes or by entering their annotations directly through a WYSIWYG interface built on top of Draft.js. Annotations will be searchable via Elasticsearch, taggable, and, if time allows, versioned.

About Charles:

Charles is a second-year PhD student in Classics and Comparative Literature at Columbia. His academic focus centers on Western Hemisphere receptions of tragedy, with particular attention paid to messenger speech. Having previously worked as a software engineer, he maintains a keen interest in digital humanities.

Natasha Voake

Proposal:

Old and Middle French are hardly studied outside of a limited network of French universities. Implementing NLP functionality to these languages would make it easier to study them and access the rich literature and culture expressed in them, famous examples of which include the “Chanson de Roland”, Chrétien de Troyes’ Arthurian legends, Marie de France’s “Lais”, Christine de Pizan’s writings, etc.

This project aims to extend basic CLTK functionality to Old and Middle French texts between c.900 and c.1500 CE, by implementing a tokenizer, stopwords, named entity recognition, a PoS tagger, and a lemmatizer with English translations for as many words as possible. Data from which the above will be sourced will be from texts licensed under creative commons licenses, which have been transcribed and digitized. For example, a number of Old French texts from the BNF’s 19th century editions have been digitized and made available at gallica.fr. Lemmas will be sourced from Godefroy’s 1901 “Lexique de l’Ancien Français” and the “Dictionnaire Electronique de Chrétien de Troyes”, which has the advantage of English-language definitions.

About Natasha:

Natasha is a final-year Linguistics student at Peterhouse, Cambridge with a particular interest in computational linguistics. She is a French-English bilingual and previously lived in New York and Paris, France.

We are thrilled to announce that the CLTK will once again be participating in the Google Summer of Code. The four of us will be acting as mentors to the forthcoming students.

See our organization’s official GSoC page for more information, but here is our call for projects:

See our Project ideas page for a list of GSOC tasks that are suited to three months’ work for a beginning–to–intermediate programmer or language student. What follows is a high–level overview of these projects and a few tips in applying. Most work will be done in the Python and JavaScript languages, of which a beginner’s or intermediate knowledge is expected.

If you know a Classical language that is not yet supported well by the CLTK (e.g., Hebrew, Sanskrit, Chinese), you may follow the pattern set by the current Greek and Latin libraries. See the Projects page on the wiki for ideas of what a good application will include. See also the CLTK docs for what we can already do for a given language.

The other family of tasks regards the CLTK’s in–development website, the Classical Language Archive (in Meteor and React) and API (in Python’s Flask framework). See our description of the frontend, experiment with the live demo, and explain in your application which functionality you would most like to see added, why, and how you will do so over the course of three months.

The CLTK’s GSoC mentors may not be able to reply to all enquiries, however the best way to ensure a good response is to submit, as early as possible, a draft application essay, in which you explain the resources you will use, the order in which you will do your work, and a timeline. For the website project, email Luke Hollis; for the add-a-language project, email Kyle P. Johnson, Ph.D..

This summer I’m working on a commentary on Descartes’ Meditationes de prima philosophia, usually known in English as his Meditations on First Philosophy. (Though actually a better translation is the less literal Metaphysical Meditations, which is how it’s usually translated into French.) In addition to providing a text and commentary, I plan to produce a vocabulary. This is a time-consuming and error-prone job, so naturally I want to offload at least some of the grunt work to a computer. (See laziness as virtue for programmers.)

As a start, I took at look at The Classical Language Toolkit. CLTK provides a set of natural language processing utilities for ancient texts. Although the Latin tools in CLTK are aimed primarily at classical material, the neo-Latin that Descartes wrote is largely classical in style and vocabulary. The following short Python script tokenizes and lemmatizes Descartes’ first meditation. This gets us well on our way since the script reduces its input text to a list of unique words reduced to their dictionary entry. That is, suppose that Descartes wrote scīre and sciēns (an infinitive meaning to know and a participle meaning knowing), the script would output only sciō, the shared dictionary entry of those two forms.

from cltk.stem.lemma import LemmaReplacer
from cltk.stem.latin.j_v import JVReplacer
from cltk.tokenize.word import WordTokenizer

meditatio_prima = open("./meditatio-prima.txt", "r").read()

jv = JVReplacer()
meditatio_prima = jv.replace(meditatio_prima)

t = WordTokenizer("latin")
l = LemmaReplacer("latin")

words = l.lemmatize(t.tokenize(meditatio_prima))
words = sorted(list(set(words)))
print("\n".join(words))

It would be even better if we could take the dictionary entries from this first script and feed them to a program that would give us definitions. So that’s what I worked on next. The Perseus Digital Library has opensourced the XML version of A Latin Dictionary edited by Lewis and Short. I’d prefer the modern Oxford Latin Dictionary, but Lewis and Short is still an outstanding Latin dictionary. Once again, a relatively short Python script will parse the XML and match the words from Descartes’ text with their Lewis and Short dictionary entries.

from sys import stderr
from lxml import etree as ET

# Helper functions
def warn(*args, **kwargs):
    '''Print message to stderr'''
    print(*args, file=stderr, **kwargs)

def inner_text(node):
   '''Return all inner text of an XML node'''
   return ''.join([text for text in node.itertext()])

xml = ET.parse('no-entities-ls.xml')
entries = xml.xpath('//entryFree')
lewis_short = {}
for item in entries:
    lewis_short[item.attrib['key']] = inner_text(item)

# Load the vocabulary words that we're searching for
words_file = open('meditatio-words.txt', 'r')
wanted_words = words_file.read().splitlines()
words_file.close()

# Work through the words we want, trying to match their meanings
for wanted in wanted_words:
    if wanted in lewis_short:
        print('%s' % (lewis_short[wanted]))
    else:
        warn('%s has no entry.' % (wanted))

This is worth walking through. After defining two small helper functions, the next block of code parses the Lewis and Short XML file and stores the entries in a Python dictionary. (Since dictionary is now very ambiguous, I’m going to call the data structure a hash from here on in.) The keys of the hash are lookup words, and the values are full entries from Lewis and Short. Since lookup words are just what our first script gave us, we load those into a list and iterate over them. For each word in Descartes, we test whether it’s in the hash. If it is, we print out the entry from Lewis and Short. If it’s not in the hash, we tell the user that no matching entry was found. (The “no item found” messages are printed to stderr instead of stdout, so that if the user wants they can print the two output streams to different places. E.g., python3 vocabulary-builder.py 1>descartes-vocabulary.txt 2>missing-words.txt)

This is a terrific amount of progress for two short Python scripts. In addition, I extracted the textual data from the XML file and saved that in plain text format. This allows for lots of other potential uses of Lewis and Short without having to deal with the XML. (The plain text is available on GitHub, under a CC license that allows for further changes.)

What’s next?

There’s a lot of room for improvement. Here’s a quick list of things I’m working on or would like to see.

• Ambiguous words in Lewis and Short have numbers in their XML keys to distinguish them. E.g., there’s a verb adeō and an adverb adeō, and their XML keys are adeo1 and adeo2 respectively. The CLTK lemmatizer also numbers ambiguous words, but I’m not sure whether the two systems are the same. For what I’m doing here, it would be ideal if they were, but there may be other considerations at play. Still, it would be worth investigating how much the two numerations overlap and whether CLTK could mimic Lewis and Short.
• The vocabulary builder spends most of its time parsing the XML file, which is very large and rather complex. As an alternative, it would be simpler to store Lewis and Short in a simple CSV file. (This would make reading in the data trivial.)
• On the other hand, the way that I extracted the textual data from the XML loses some information. In particular, the details of the section breakdowns in Lewis and Short are not stored as text, but only in the structure and attributes of the XML tags. My method of brute-force extraction didn’t preserve those structural signposts. For my purposes, that’s fine, but other people may prefer to try to get more structure out of the XML.
• Lewis and Short have very odd conventions for macrons. First, they very frequently use the brevis to show where vowels are short. E.g., ăgō. That’s nearly never useful for students. Second, and much worse, they don’t provide macrons in several cases where macrons can be assumed as the norm. E.g., the first principal part of verbs or third declension nouns ending in -tiō. These cases, and many others, never have macrons. So in fact, the entry for that verb actually starts with the form ăgo: an unhelpful brevis and a missing macron. Both would likely confuse contemporary students. That’s enormously frustrating, though luckily there is an excellent Latin macronizer available on GitHub that might help with this problem, and it’s easy enough to remove all the brevis marks via simple scripts.
• Finally, as helpful as all of this may be, it’s not what I really need. To build a student glossary, I need to strike a balance: (i) I should provide general and basic meanings for the words I gloss, but (ii) I have to make sure to include any specific meanings necessary for the text at hand. (Student glossaries that only provide the meanings for the text at hand are the worst. They should be illegal.) But what these programs provide is far, far more than that. The result now is the entire dictionary entry for each word. Lewis and Short is a detailed, scholarly dictionary. The entries for core words in the language (e.g., ferō or et) can go on for column after column of small, single-spaced writing. So what I’d need to do is trim these entries down to something more manageable. That’s fine, but it would still require a fair amount of work—though nowhere near as much as doing it all from scratch.
• These command-line scripts are great for me, but for most people they would be a pain. Eventually, it would probably be good to wrap this in a web application or a desktop application. A user could upload the text they wanted to gloss and the application would give them a vocabulary list. (This would be something like Haverford’s terrific Bridge, but for any Latin text. It’s obviously a tall order.)

As usual, trying to make a computer do your work for you is fun, helpful, and exhausting. I got to learn some Python, and the tokenizer and lemmatizer results alone will save me hours compared to doing it by hand. At the same time, I’ve only scratched the surface and there’s far more work to do. But I’m excited to continue learning Python and to improve the glossary-maker scripts that I’ve started here.

The CLTK now includes a clausulae analysis module that – in conjunction with the existing prosody module – can produce a rhythmic profile of a text.

from cltk.prosody.latin.scanner import Scansion
from cltk.prosody.latin.clausulae_analysis import Clausulae

# A sample analysis of Cicero's First Catilinarian
# NB: The text must be macronized before scanning
with open("~/CiceroCat1.txt") as file_open: 
    text = file_open.read()

scansion = Scansion().scan_text(text)
clausulae = Clausulae().clausulae_analysis(scansion)
print(clausulae)

This profile comes in the form of a dictionary in which each key is a clausulae type and each value is the frequency of that clausulae in the text.

{
    'molossus + double trochee': 12, 
    'cretic + double trochee': 6, 
    'choriamb + double trochee': 1, 
    'cretic + double spondee': 1, 
    'heroic': 9, 
    '1st paeon + trochee': 12, 
    'double cretic': 9, 
    'molossus + cretic': 6, 
    'double spondee': 8, 
    'molossus + iamb': 11, 
    '1st paeon + anapest': 0, 
    'dactyl + double trochee': 2, 
    'cretic + trochee': 27, 
    'cretic + iamb': 13, 
    'substituted cretic + trochee': 0, 
    'double trochee': 27, 
    '4th paeon + trochee': 2, 
    '4th paeon + cretic': 3
}

The specific clausulae that the module searches for is derived from a list provided in the introduction of John Ramsey’s Cambridge commentary on Cicero’s Philippics I-II. See the table below for the relation between the name of a clausulae, the ‘type’ which Ramsey assigns to it (note that the heroic clausulae, type 6, is my own addition to the list), and its metrical construction. Short syllables are denoted with ‘u’, long with ‘-‘, and substitution with parentheses.

For more information, see Clausulae analyses docs.