File::TTX 0.03 released

I haven't been moving very fast on this project, have I?

That whole MT project

OK, so the post-editing project I foolishly agreed to help with consisted of:

• OCR with Able2Extract
• MT with a mixture of (I think) Google Translate and Systran
• First-pass proofreading
• Second-pass post-editing

So let's talk about that. A far, far better workflow would have been:

• OCR with whatever
• Source-language spell checking and correction
• Identification of key phrases and terminology as cues for MT
• TRADOS or similar to avoid rework of existing sentences
• MT with whatever
• Target-language spell checking, feeding results back through MT until at least everything is English
• First-pass post-editing
• Second-pass proofreading

This workflow uses (or at least could use) the exact same tools as above, but without the introduction of errors at each step that make later steps impossible to manage. First-pass post-editing should be done by a bilingual translator, using specialized post-editing tools (not yet written) plus a normal translation memory (and of course the TM should also be used before passing text off to the MT stage). Systematic errors should be documented and recycled through the MT process.

One key insight: terminology research really starts to get a lot more important in this workflow than in normal CAT.

More thoughts on a non-stupid text editor

I'm doing some post-editing for Portuguese today (I know, I know, never do MT post-editing, but this customer is a good one and I just couldn't say no). As usual with post-Systran work, there is a lot of dragging and dropping involved, and frankly? Word freaking sucks at dragging and dropping. Why should that be? Why can't I drag a word from the end of a punctuated sentence into its middle and have Word get the spacing right?

The mind boggles.

So it looks like I'm just going to have to break down and address non-stupid text editing again.

Another workflow with PDFs

I have a set of documents that consist of PDFs that have been highlighted and scanned. That is, each PDF consists of a set of documents. The text to be translated has been highlighted - with a physical marker, I mean - and the documents scanned. The PDFs were unfortunately not encoded as allowing comments (this is unfortunately a flag of the digital signature, not a flag in the PDF standard, and Adobe has not provided the key for digital signature from what I'm reading - thus there is no tool in the world that can flag a PDF to allow comments from Adobe Reader except for the full paid version of Adobe Acrobat.)

So my workflow is to go through the documents and use the snapshot tool to copy the highlighted bits. I put each bit into one column of a Word file, and the translation in the other column. It's nearly as good as comments in the PDF.

It seems to me that this would be a simple tool to implement: create the Word file, create the table, then every time I select something that's graphical, put it into the Word file for me and bring Word to the top. It's not a huge help, but it's the principle of the thing.

Working with source text

There are open-source ways to break text into sentences and to find terms. Those need to be part of the toolkit.

The splitta library is a sentence boundary finder. This I have to incorporate, as segmentation is an extremely important function of any translation system. So that should be Perl-ized here.

The Topia term extractor is the other thing I wanted to point out here.

Also, the fact that both of these libraries are in Python. An awful lot of natural language work ends up in Python. That's kind of interesting, actually.

Terminology

This has been done to death, of course, but I need to start thinking about a terminology engine, and also about specific terminology - right now I'd like a database of titles of industrial standards in various languages. They come up rather a lot.

PDF reading

There are a couple of workflows where PDFs are needed.

First is where a series of pages have been scanned and need to be translated starting from the graphics. OCR can come in handy here (if it works, which it usually doesn't), but I want to highlight the fact that (1) the pages are very often disjoint (think medical records) and (2) sometimes have Bates numbers (legal annotations identifying each individual page in a set of documents). This overall structure could do with some software support. I'm thinking something that takes individual document segments and ties them back into a structured overall document with, say, the Bates numbers.

[With respect to that OCR: it would be nice to have a pre-OCR stage that finds and identifies pages that are similar - this could simplify finding letterhead, headings, and so on.]

The second workflow of interest is text PDFs. See, PDFs don't have document structure like Word documents. If a header appears on every page, well then it will be reproduced on every page in text. So it would be nice to be able to impose - to recognize - this sort of structure in order to take PDFs and translate them. (You could argue that a TM tool would do this for you - but I would prefer to abstract out the different document parts in order to translate them separately, when we start thinking about machine translation. The MT tool will need as much help as it can get.)

Anyway. Just a thought I'm too busy to follow up on right at the moment.