harry 's Network

Shared by harry , harry added annotation, like and 1 save total

Shared by harry , harry added annotation, like and 1 save total

- Chart parse-decoding: to support hierarchical models and syntax-based translation models
- Depth-first decoding: to provide anytime algorithms for decoding
- Forced decoding: to compute scores for provided output
- Suffix-array translation models: an alternative way to store large rule-sets without the need to translate them
- Maximum entropy translation models: translation models that incorporate additional source-side and context information for scoring translation rules.

Shared by harry , harry added annotation, like and 1 save total

The training data (a parallel corpus) has to be annotated with the additional factors. For instance, if we want to add part-of-speech information on the input and output side, we need to obtain part-of-speech tagged training data. Typically this involves running automatic tools on the corpus, since manually annotated corpora are rare and expensive to produce.

I’m doing word and bigram counts on a corpus of tweets. I want to store and rapidly retrieve them later for language model purposes. So there’s a big table of counts that get incremented many times. The easiest way to get something running is to use an open-source key/value store; but which? There’s recently been some development in this area so I thought it would be good to revisit and evaluate some options.

Shared by harry , harry added annotation, like and 1 save total

Typically, LM estimation starts with the collection of n-grams and their frequency counters. Then, smoothing parameters
are estimated for each n-gram level; infrequent n-grams are possibly pruned and, finally, a LM file is
created containing n-grams with probabilities and back-off weights. This procedure can be very demanding
in terms of memory and time if applied to huge corpora. IRSTLM provides a simple way to split LM training
into smaller and independent steps, which can be distributed among independent processes.

Shared by harry , harry added annotation, like and 1 save total

Warning: This script runs GIZA++'s snt2cooc.out in the background. Use the following command if you have 1.5gigs of RAM or more. Otherwise skip it and try the alternative command below: