Vocabulary in English

I've been trying to find resources pertaining to the frequency of words in English - surely there must be some kind of graded scale of "commonness" or something? But so far I can't find anything that organized.

Instead, I've got two interesting links here:

• English for Academic Purposes
• 1000 most common SAT words

If we consider each of those and their ilk to be "general vocabulary words", then we'll have that much more luck in identifying technical content in a given document.

Terminology resources

A bit of a linkdump here, first:

• WordNet is now at 3.0 on Unix, still 2.1 on Windows. The database from Linux is probably more useful. Interestingly, it's also available in Prolog. The licensing is pretty open these days. I don't think it used to be. That's welcome news.
• Here's something called CoreLex.
• A good overview of the OLIF format.

I think I could do worse for termbase storage in Perl than simply a database schema that mirrors OLIF (at least partly). That could be part of a general OLIF-handling set of modules. OLIF is attractive because it's model-agnostic in terms of how terms are conceptualized, so an OLIF-based module should be able to do something reasonable with essentially any terminological source.

Lingua::OLIF?

OLIF

Open Lexicon Interchange Format (OLIF) is an XML terminology format that SDL Multiterm 2009 can import.  (In other news, unlike the first time I bought them, TRADOS 2007 and Multiterm 2009 are now interoperable.  Must have been an upgrade between then and when I bought this laptop. Bodes well for my everyday work!)

So ... building on OLIF and my new SQP tool, maybe it's time to consider writing that terminology database thing with a nice Perly wrapper.

By the way, OLIF was initially supported by SAP (and it's an SAP-related job I'm working on right now!) and the OLIF Consortium is like a who's-who of the big players in the translation industry.  So it's probably worth grokking.

Terminology from patent databases

It should be relatively easy to automate a crawl of any patent database and extract terminology from the abstracts and translations of abstracts.

Just a thought.

Task: concordance-to-glossary tool

I want to be able to look up one or more terms in a TM in the same way that concordances work now, then make a decision for a given document or customer, then have that decision checked globally. I'm most of the way to having this ready to go.

Automotive engineering glossary HU>EN

Wow! This is a great find!

Terminology

Another non-Xlat post!

Automotive terminology is kind of tricky and I'm finding it hard to find good references - although I'm seeing more demand. Here are a couple of links not to forget.

• A list of glossaries from ATS Group.
• Translation Journal has a short glossary.

Second topic: I really want to mine the SAP help site for accounting terminology. Here's just a teaser link that's been open on my browser for a couple of weeks now - the technique is simple. Google "site:help.sap.com xxx" for a likely term, then replace the language in the link with "en". Then align your results. It works! A list of likely terms (from a tagger, perhaps) is the right place to start.

A generalized terminology research framework would be useful.

Terminology

This has been done to death, of course, but I need to start thinking about a terminology engine, and also about specific terminology - right now I'd like a database of titles of industrial standards in various languages. They come up rather a lot.

tf-idf weights

Quoth Wikipedia, "The tf–idf weight (term frequency–inverse document frequency) is a weight often used in information retrieval and text mining."

The idea is that you determine the weights of terms based on their frequency in both the current document and in your overall corpus. This lets you find documents based on terms they use that are less frequent overall, and thus that are likely to indicate what the document is about.

Terminology mining is a technique by means of which "interesting" terms can be found in a document. The interesting terms can then be researched in advance of the translation process, so that the translation itself can be both consistent and quick.

There are lots of links I want to save that are tangentially related to this sort of textual analysis.

• Gensim is a textual analysis library in Python.
• An earlier paper on term weighting.
• tdidf library in Python at Google Code.
• And another at Github.

Terminology checking v0.01

So. I just posted v0.01 of a terminology checker script to the Wiki. It is painfully naive in its structure and coding, but it got the job done tonight for some terminology checking I wanted to do, and it illustrates just how simple these basic tools can be. The key of it is this:

foreach $s ($ttx->segments()) {
   my $c = $t->check ($s->source, $s->translated);
   if ($c) {
      foreach my $missing (keys %$c) {
         $terms->{$missing} = $c->{$missing};
         $bad->{$missing} = [] unless defined $bad->{$missing};
         push @{$bad->{$missing}}, $s;
      }
   }
}

Now, note that it's using a termbase module I haven't published yet (because it's even more terribly naive), but the key here is that this loop is really, really simple.

This is what translation tools should look like. I'm pretty happy with this.