TransTools for Visio

Not terribly Perl-specific, obviously, but here's the first tool I've ever found to help with translating Visio documents and embeds: TransTools for Visio. It's not perfect (what in the Microsoft ecosystem can ever be?) but it's a damn sight better than slogging through by hand, and a point of departure should I run into more Visio in the future. For some reason, it's sadly rare.

As things stand, with Word 2003 and Visio 2007, translating in TRADOS 2011, the workflow is:

• Open the TransTools for Visio file in Visio.
• Find a Visio drawing. If it opens directly, great, otherwise:
• Cut and paste the drawing into a new document and open it directly there.
• Select All and copy out everything in the drawing.
• Paste into a new unnamed drawing in the same process as the open TTV drawing with the macros.
• Run the magical macro to search the unnamed drawing and build a Word table.
• Copy that table into a new document because the macro doesn't open the table in a full Word instance with menus (this could be fixed, obviously)
• Drag that new document into the source language side of SDL.
• Prepare that for translation.
• Move to target language and translate. Don't use tabs, as they'll mess us up three steps from now.
• Save the target document.
• Open that document, select the left column, and paste into the original interim document, then copy the whole table.
• Run the magical macro to search-and-replace the unnamed drawing. This uses a tab-delimited paste of the table, so tabs in your text will be Very Bad Indeed.
• Select All in the unnamed drawing, and copy.
• Delete everything in the embedded drawing, and paste the translated bits.
• Jockey around to align properly. This is pretty manual. Sometimes you need to adjust text sizes as well, so actually this probably has to be manual.
• If you copied the drawing out into a separate document, now copy it back in. Adjust for location and size if that step screwed things up, which they appear always to do (I don't have enough experience yet).

That's a pretty lengthy process, and yet it's still far superior in terms of mental capacity and quality control than not translating using a real tool. Everything except the translation and the last two steps could be automated, some during job preparation (which is obviously a good thing, because otherwise your word counts will be off).

So. Very nice tool.

Marpa or ParZu?

I spent most of May working through my old natural-language tokenizer, adding a vocabulary-driven lexer/lexicon for German, all in preparation for undertaking a Marpa-based German parser. That's looking halfway decent at this point (except I need to do much better stemming), and then I decided to do a general search on German parsers and found ParZu.

The unusual thing about ParZu, among parsers especially, is that it's fully open source. That is, it has a free license, not a free-for-academics-only license - and it's hosted on GitHub. Also, I can try it online. So I fed it some more-or-less hairy sentences from my current translation in progress - and it parsed them perfectly.

So here's the thing. I kind of want to do my own work and come to terms with the hairiness of things myself. And then on the other hand, parsing German by any means would allow me to jump ahead and maybe start doing translation-related tasks directly....

It's a dilemma.

Update 2014-09-26: Maybe not such a dilemma. ParZu is written in Prolog and I'm just not sure I'm up for that. It honestly seems it would be easier to do it in Marpa...

This is probably incorrect. But I think I'm going to start finding out, this week.

Compiling corpi

So there's this German news corpus obtained between 1996 and 2000 from online retrieval that I intend to use for some of my NLP work, and it occurred to me that I could build a similar corpus (well, the monolingual side of it, anyway) by doing my own periodic retrievals.

To that end, here's the RSS feed pages for the Süddeutsche Zeitung, the Népszabaság Online, and the Népszava (published in New York for Hungarian-Americans).

Analysis of chemical names

Turns out the linguistic structure of chemical names is non-trivial. Unfortunately, as it's also quite profitable, it all seems to be behind paywalls, but I'm visiting Bloomington this summer and will have the opportunity to spend some time in the library, so this is one of the things I hope to make some headway on.

In the meantime, here's a paywalled article from the promisingly named Journal of Chemical Information and Modeling, which describes an early version of Name>Struct, a closed-source interpreter for chemical names that strives to understand them in a way similar to a human chemist - that is, they attempt to model actual usage, not just reflect the official definitions of usage. Descriptive, not prescriptive, chemical linguistics.

Anyway, the folks at CambridgeSoft who make Name>Struct have also highlighted some of the pitfalls of chemical linguistics here.

Ah - silly me. A search on "Name>Struct open source" quickly returns OPSIN, an open-source algorithm that I could probably adapt pretty easily. It's here at BitBucket, and written in (shudder) Java. Nifty Web interface here.

TRADOS 2007 has an OLE dispatch API

I wish I'd known this years ago! TRADOS TagEditor 2007 exposes an OLE API. Workbench doesn't appear to. SDL doesn't support it any more, either. But here are some references to it I ran across the other day. I'm going to try to dump the typelib soon (I have no time at all until Thursday).

• This - from Perl
• This from VB
• This from VB again.

GlobalSight and Microsoft Translator

Two links to software resources: GlobalSight is an open-source translation manager, and this Microsoft Translator page is probably a pretty decent definitive list of who to watch in the industry.

Vocabulary in English

I've been trying to find resources pertaining to the frequency of words in English - surely there must be some kind of graded scale of "commonness" or something? But so far I can't find anything that organized.

Instead, I've got two interesting links here:

• English for Academic Purposes
• 1000 most common SAT words

If we consider each of those and their ilk to be "general vocabulary words", then we'll have that much more luck in identifying technical content in a given document.

TaaS

Terminology as a Service, as envisaged by the Europeans.

Terminology resources

A bit of a linkdump here, first:

• WordNet is now at 3.0 on Unix, still 2.1 on Windows. The database from Linux is probably more useful. Interestingly, it's also available in Prolog. The licensing is pretty open these days. I don't think it used to be. That's welcome news.
• Here's something called CoreLex.
• A good overview of the OLIF format.

I think I could do worse for termbase storage in Perl than simply a database schema that mirrors OLIF (at least partly). That could be part of a general OLIF-handling set of modules. OLIF is attractive because it's model-agnostic in terms of how terms are conceptualized, so an OLIF-based module should be able to do something reasonable with essentially any terminological source.

Lingua::OLIF?

TreeTagger

The Perl module Lingua::TreeTagger provides an interface to the TreeTagger program (Win installation here).  The only problem with TreeTagger is that it's got a commercial-license requirement. That, and the approach isn't good for Hungarian - but you can't have everything.  This would probably be the best possible intermediate structure for frame-based translation, which I still think should be a valid approach.

Word lists

I really need some kind of principled way to keep track of word lists and terminology.  Ideally this would be a full-blown terminology management system with an online component and everything, but it would also be a word list source.

Here are some good places to start with word lists:

• http://wordlist.sourceforge.net/
• http://www.puzzlers.org/dokuwiki/doku.php?id=solving:wordlists:start

File::TMX

I just started scratching the surface for TMX files.  I'm going to end up with some generalized useful tools for XML files.

Terminology source: EMA

The European Medicines Agency publishes patient information about European-approved (not nationally approved) drugs in all the European languages.  This would be a useful corpus for terminological (and syntactic) analysis.

OLIF

Open Lexicon Interchange Format (OLIF) is an XML terminology format that SDL Multiterm 2009 can import.  (In other news, unlike the first time I bought them, TRADOS 2007 and Multiterm 2009 are now interoperable.  Must have been an upgrade between then and when I bought this laptop. Bodes well for my everyday work!)

So ... building on OLIF and my new SQP tool, maybe it's time to consider writing that terminology database thing with a nice Perly wrapper.

By the way, OLIF was initially supported by SAP (and it's an SAP-related job I'm working on right now!) and the OLIF Consortium is like a who's-who of the big players in the translation industry.  So it's probably worth grokking.

Translation of chemical names

Here's a pretty fascinating survey of chemical name translation (I've been doing a lot of pharma translation this month).  Turns out it's pretty tricky - looking at it, I'm not 100% sure it's as tricky as people make it out to be, because it's typical of language people that they find software magical, and typical of programmers to find natural language unreasonably hairy.  But still - I think there's probably a (small) market for this kind of tool.

Cross-posted.

Terminology from patent databases

It should be relatively easy to automate a crawl of any patent database and extract terminology from the abstracts and translations of abstracts.

Just a thought.

Terminology identification

On term extraction: I checked Okapi; they're just doing the same up-to-n-gram extraction I've always done and found wanting. Their English tokenizer is also comparable to mine (better tested, to be sure). So all in all, maybe I'm capable of doing competent work here. I need to do better testing, though.

But anyway, I figured maybe NLTK might have something more suited to my needs, so I searched on "NLTK term extraction" and came upon this miscategorized post on the topic at nltk-dev. That post led me to the Wikipedia page for named-entity recognition (not so fantastically relevant but interesting nonetheless) and - gold - a suggestion to check Chapter 6 of Manning, Ragavan, and Schütze (which is one of the texts for the upcoming Stanford online NLP course, actually).

Chapter 6 addresses term weighting. Finding relevant terms for indexing of documents is equivalent to finding interesting terms for terminology searches, and it turns out that the best way to weight terms is by inverse document frequency. (Which makes sense; clustering of terms in documents indicates that they're low-entropy and contain more information than, say, "the", whose inverse document frequency is 1.)

Long story short, a term in a document is interesting in proportion to the number of times it appears in the document and in inverse proportion to the number of documents in which it appears in a given relevant corpus. Given that my corpus is about five million words of translation memories, I have a good corpus, provided that I organize it into something document-like.

I'm going to consider a "document" to be each group of entries in the TM clustered by date. Since not all my translation goes through one TM, I can pretty much guarantee that all my work will be easy to cluster; from that I can derive an overall set of terminology for the entire corpus and calculate inverse document frequency for each term. From that, I can score each term found in a new document. If it's a known term for which I already have a gloss, I'm happy. If it's a new term with high relevance for which I have no gloss, I can research it. And if it's a known term for which I have no gloss, I can study my own corpus to try to extract a likely candidate for translation.

(This leads to a terminology extraction tool, too - working on both target and source languages and trying to correlate presence in segments, I'll bet I can come up with some pretty good guesses at glosses. Make that service a free one and you ought to do well.)

So I've got a pretty good plan for terminology identification at this point. Just have to find the time to implement it. Here's kind of the list of subtasks:

• Tool to convert Trados TM to TMX without my getting my hands dirty. I love Win32 scripting anyway - and this can run on the laptop for a day or two to chew its way through my five million words.
• Do that TMX module for Perl.
• For all my TMs, cluster the segments into "documents".
• Polish up a terminology extraction tool (e.g. the same n-grams between stop words strategy I've used in the past).
• Run said tool on the five million. Might want to do some kind of proper database and index at this point so I never have to do this again.
• Calculate inverse document frequencies for everything.
• Take a target TTX, extract terms, and classify them. This is the actual task.

Okapi

Okapi (Java) is a pretty comprehensive set of open-source tools to facilitate the translation process - including a simple workflow manager. (You can group sets of steps together to define your own processes, a technique I'm going to steal.)

One more thing to take note of.

Incidentally, its token types are rather similar to the ones I've proposed.

Task: concordance-to-glossary tool

I want to be able to look up one or more terms in a TM in the same way that concordances work now, then make a decision for a given document or customer, then have that decision checked globally. I'm most of the way to having this ready to go.

Task: find "actionable" terms in a given source

This is probably solved by NLTK somehow, but given a source text I want to be able to find probable glossary items to be researched and to be checked against a TM or glossary.