B ²ô`I6ã@s„dZddlZddlmZddlmZddlmZddlmZddlm Z ddlm Z dd l m Z Gd d „d e ƒZ Gd d „d e ƒZdS)aà Translation model that considers how a word can be aligned to multiple words in another language. IBM Model 3 improves on Model 2 by directly modeling the phenomenon where a word in one language may be translated into zero or more words in another. This is expressed by the fertility probability, n(phi | source word). If a source word translates into more than one word, it is possible to generate sentences that have the same alignment in multiple ways. This is modeled by a distortion step. The distortion probability, d(j|i,l,m), predicts a target word position, given its aligned source word's position. The distortion probability replaces the alignment probability of Model 2. The fertility probability is not applicable for NULL. Target words that align to NULL are assumed to be distributed uniformly in the target sentence. The existence of these words is modeled by p1, the probability that a target word produced by a real source word requires another target word that is produced by NULL. The EM algorithm used in Model 3 is: E step - In the training data, collect counts, weighted by prior probabilities. (a) count how many times a source language word is translated into a target language word (b) count how many times a particular position in the target sentence is aligned to a particular position in the source sentence (c) count how many times a source word is aligned to phi number of target words (d) count how many times NULL is aligned to a target word M step - Estimate new probabilities based on the counts from the E step Because there are too many possible alignments, only the most probable ones are considered. First, the best alignment is determined using prior probabilities. Then, a hill climbing approach is used to find other good candidates. Notations: i: Position in the source sentence Valid values are 0 (for NULL), 1, 2, ..., length of source sentence j: Position in the target sentence Valid values are 1, 2, ..., length of target sentence l: Number of words in the source sentence, excluding NULL m: Number of words in the target sentence s: A word in the source language t: A word in the target language phi: Fertility, the number of target words produced by a source word p1: Probability that a target word produced by a source word is accompanied by another target word that is aligned to NULL p0: 1 - p1 References: Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York. Peter E Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19 (2), 263-311. éN)Ú defaultdict)Ú factorial)Ú AlignedSent)Ú Alignment)ÚIBMModel)Ú IBMModel2)ÚCountscsNeZdZdZd‡fdd„ Z‡fdd„Zdd„Zd d „Zd d „Zd d„Z ‡Z S)Ú IBMModel3u Translation model that considers how a word can be aligned to multiple words in another language >>> bitext = [] >>> bitext.append(AlignedSent(['klein', 'ist', 'das', 'haus'], ['the', 'house', 'is', 'small'])) >>> bitext.append(AlignedSent(['das', 'haus', 'war', 'ja', 'groß'], ['the', 'house', 'was', 'big'])) >>> bitext.append(AlignedSent(['das', 'buch', 'ist', 'ja', 'klein'], ['the', 'book', 'is', 'small'])) >>> bitext.append(AlignedSent(['ein', 'haus', 'ist', 'klein'], ['a', 'house', 'is', 'small'])) >>> bitext.append(AlignedSent(['das', 'haus'], ['the', 'house'])) >>> bitext.append(AlignedSent(['das', 'buch'], ['the', 'book'])) >>> bitext.append(AlignedSent(['ein', 'buch'], ['a', 'book'])) >>> bitext.append(AlignedSent(['ich', 'fasse', 'das', 'buch', 'zusammen'], ['i', 'summarize', 'the', 'book'])) >>> bitext.append(AlignedSent(['fasse', 'zusammen'], ['summarize'])) >>> ibm3 = IBMModel3(bitext, 5) >>> print(round(ibm3.translation_table['buch']['book'], 3)) 1.0 >>> print(round(ibm3.translation_table['das']['book'], 3)) 0.0 >>> print(round(ibm3.translation_table['ja'][None], 3)) 1.0 >>> print(round(ibm3.distortion_table[1][1][2][2], 3)) 1.0 >>> print(round(ibm3.distortion_table[1][2][2][2], 3)) 0.0 >>> print(round(ibm3.distortion_table[2][2][4][5], 3)) 0.75 >>> print(round(ibm3.fertility_table[2]['summarize'], 3)) 1.0 >>> print(round(ibm3.fertility_table[1]['book'], 3)) 1.0 >>> print(ibm3.p1) 0.054... >>> test_sentence = bitext[2] >>> test_sentence.words ['das', 'buch', 'ist', 'ja', 'klein'] >>> test_sentence.mots ['the', 'book', 'is', 'small'] >>> test_sentence.alignment Alignment([(0, 0), (1, 1), (2, 2), (3, None), (4, 3)]) Ncsštt|ƒ |¡| ¡|dkrFt||ƒ}|j|_|j|_| |¡n2|d|_|d|_|d|_|d|_ |d|_ xt d|ƒD]}|  |¡q„WdS)a Train on ``sentence_aligned_corpus`` and create a lexical translation model, a distortion model, a fertility model, and a model for generating NULL-aligned words. Translation direction is from ``AlignedSent.mots`` to ``AlignedSent.words``. :param sentence_aligned_corpus: Sentence-aligned parallel corpus :type sentence_aligned_corpus: list(AlignedSent) :param iterations: Number of iterations to run training algorithm :type iterations: int :param probability_tables: Optional. Use this to pass in custom probability values. If not specified, probabilities will be set to a uniform distribution, or some other sensible value. If specified, all the following entries must be present: ``translation_table``, ``alignment_table``, ``fertility_table``, ``p1``, ``distortion_table``. See ``IBMModel`` for the type and purpose of these tables. :type probability_tables: dict[str]: object NÚtranslation_tableÚalignment_tableÚfertility_tableÚp1Údistortion_tabler) Úsuperr Ú__init__Úreset_probabilitiesrr r Úset_uniform_probabilitiesr r rÚrangeÚtrain)ÚselfÚsentence_aligned_corpusZ iterationsZprobability_tablesZibm2Ún)Ú __class__©úB/home/dcms/DCMS/lib/python3.7/site-packages/nltk/translate/ibm3.pyr‰s       zIBMModel3.__init__cs$ttˆƒ ¡t‡fdd„ƒˆ_dS)Ncst‡fdd„ƒS)Ncst‡fdd„ƒS)Ncst‡fdd„ƒS)NcsˆjS)N)ÚMIN_PROBr)rrrÚ¹ózeIBMModel3.reset_probabilities........)rr)rrrr¹rzSIBMModel3.reset_probabilities......)rr)rrrr¹rzAIBMModel3.reset_probabilities....)rr)rrrr¸sz/IBMModel3.reset_probabilities..)rr rrr)r)r)rrrµszIBMModel3.reset_probabilitiesc s>tƒ}x¨|D] }t|jƒ}t|jƒ}||f|kr | ||f¡d|}|tjkrft dt |ƒd¡xDt d|dƒD]2}x,t d|dƒD]}||j ||||<qŠWqvWq Wt dd„ƒ|j d<t dd„ƒ|j d<t dd„ƒ|j d <t d d„ƒ|j d <d } d | d‰x*t d| ƒD]} t ‡fdd„ƒ|j | <qWd|_dS)NézA target sentence is too long (z& words). Results may be less accurate.rcSsdS)Ngš™™™™™É?rrrrrrÕrz5IBMModel3.set_uniform_probabilities..cSsdS)NgÍÌÌÌÌÌä?rrrrrrÖrcSsdS)Ngš™™™™™¹?rrrrrr×récSsdS)Ng{®Gáz¤?rrrrrrØréé g{®Gáz„?écsˆS)Nrr)Úinitial_fert_probrrrÜrgà?)ÚsetÚlenÚmotsÚwordsÚaddrrÚwarningsÚwarnÚstrrrrr r ) rrZl_m_combinationsÚaligned_sentenceÚlÚmZ initial_probÚjÚiZ MAX_FERTILITYÚphir)r#rrÁs,     " z#IBMModel3.set_uniform_probabilitiesc Csþtƒ}x¶|D]®}t|jƒ}t|jƒ}| |¡\}}t| ¡ƒ|_| |¡}xn|D]f} |  | ¡} | |} x6t d|dƒD]$} |  | | | ¡|  | | | ||¡qvW|  | | ¡| | | ¡qPWq W|j} | ¡| |_| |¡| |¡| |¡| |¡dS)Nr)Ú Model3Countsr%r&r'ÚsamplerZzero_indexed_alignmentÚ alignmentZprob_of_alignmentsÚprob_t_a_given_srZupdate_lexical_translationÚupdate_distortionZupdate_null_generationZupdate_fertilityr rZ*maximize_lexical_translation_probabilitiesÚ!maximize_distortion_probabilitiesZ maximize_fertility_probabilitiesZ&maximize_null_generation_probabilities)rZparallel_corpusÚcountsr,r-r.Zsampled_alignmentsZbest_alignmentZ total_countÚalignment_infoÚcountZnormalized_countr/Zexisting_alignment_tablerrrràs0            zIBMModel3.trainc Cs tj}x”|j ¡D]†\}}x|| ¡D]p\}}xf| ¡D]Z\}}xP|D]H} |j|||| |j||| } t| |ƒ|j|||| <qDWq6Wq$WqWdS)N)rrÚ distortionÚitemsÚdistortion_for_any_jÚmaxr) rr8rr/Zi_sr0Zsrc_sentence_lengthsr-Ztrg_sentence_lengthsr.Zestimaterrrr7 s z+IBMModel3.maximize_distortion_probabilitiescCsh|j}|j}t|ƒd}t|ƒd}|j}d|}d}tj} | d¡} |t|| ƒt||d| ƒ9}|| krr| Sx:td| dƒD](} ||| | d| 9}|| kr‚| Sq‚WxJtd|dƒD]8} | | ¡} |t | ƒ|j | || 9}|| kr¾| Sq¾Wxhtd|dƒD]V} || }|j | } || }||j |||j | | ||9}|| kr | Sq W|S)zc Probability of target sentence and an alignment given the source sentence rgð?rr)Ú src_sentenceÚ trg_sentencer%r rrZfertility_of_iÚpowrrr r4r r)rr9r?r@r-r.r Zp0Z probabilityrZnull_fertilityr0Z fertilityr/ÚtÚsrrrr5s>      &  zIBMModel3.prob_t_a_given_s)N) Ú__name__Ú __module__Ú __qualname__Ú__doc__rrrrr7r5Ú __classcell__rr)rrr Ws0, ) r cs(eZdZdZ‡fdd„Zdd„Z‡ZS)r2zp Data object to store counts of various parameters during training. Includes counts for distortion. cs.tt|ƒ ¡tdd„ƒ|_tdd„ƒ|_dS)NcSs tdd„ƒS)NcSs tdd„ƒS)NcSs tdd„ƒS)NcSsdS)NgrrrrrrQrz]Model3Counts.__init__........)rrrrrrQrzKModel3Counts.__init__......)rrrrrrQrz9Model3Counts.__init__....)rrrrrrQrz'Model3Counts.__init__..cSs tdd„ƒS)NcSs tdd„ƒS)NcSsdS)NgrrrrrrTrzKModel3Counts.__init__......)rrrrrrTrz9Model3Counts.__init__....)rrrrrrTr)rr2rrr;r=)r)rrrrNs  zModel3Counts.__init__cCsF|j|}|j|||||7<|j||||7<dS)N)r4r;r=)rr:r9r/r-r.r0rrrr6Ws zModel3Counts.update_distortion)rDrErFrGrr6rHrr)rrr2Hs r2)rGr)Ú collectionsrÚmathrZnltk.translaterrrrZnltk.translate.ibm_modelrr r2rrrrÚJs       r