# 机器翻译之 - IBM Model 1的介绍

By [zekeer.eth](https://paragraph.com/@zekeer) · 2022-05-30

---

本文利用贝叶斯chain rule 对IBM model1模型进行了目标函数的推导与代码层面的一些实现，仅为学习时记录，理解不到位情况还请批评指正

> 本文作为自然语言处理训练营的学习材料，原文来自于课程助教陈老师的专栏：[https://zhuanlan.zhihu.com/p/72160554](https://zhuanlan.zhihu.com/p/72160554) 。

一.重要概念说明
--------

1.**alignment**:在平行文本中，我们将一种语言中的单词与另一种语言中的单词的对齐叫做alignment(eg:das->the,Haus->house,ist->is,klein->small)，alignment只是对应关系，并不是概率值

![a : {1 → 1, 2 → 2, 3 → 3, 4 → 4}](https://storage.googleapis.com/papyrus_images/dd3c6576b74756c2523d7bd285262bbec6412926ade286a4ee8e72a50fc0f601.svg)

a : {1 → 1, 2 → 2, 3 → 3, 4 → 4}

3.**IBM model1**:该模型是以统计alinment下语言1翻译为语言2的概率。IBM Model 1 采用最简单的假设：假设每个alignment所发生的概率相同。

![I^J](https://storage.googleapis.com/papyrus_images/1ba5b6a5080b7a25b5119bf6c5c9ec1dbfc04c4edb884bce2b7678f4da29c77e.svg)

I^J

![f=(f_1,f_2,...f_{l_f})](https://storage.googleapis.com/papyrus_images/899ebc0ea8f447a31bd420843eb768db840a68ffea176dfcef30560cde8bff3b.svg)

f=(f\_1,f\_2,...f\_{l\_f})

![e=(e1,e2,…,e_{l_e})](https://storage.googleapis.com/papyrus_images/72f24634781a6bbfad8bf77c747a775f30036d1c49ef9c7ff14e118687473629.svg)

e=(e1,e2,…,e\_{l\_e})

![P(a|e,f)](https://storage.googleapis.com/papyrus_images/181c4f33e24157fa9f87e690f8aa3efc6dc87fe48a306a3b78c89d7d0ae51804.svg)

P(a|e,f)

d) 参数估计：EM算法，最大化期望算法，通过概率模型中寻找参数最大似然估计或者最大 后验估计的算法，其中概率模型依赖于无法观测的隐性变量。最大期望算法经过两个步骤交替进行计算，第一步是计算期望（E），利用对隐藏变量的现有估计值，计算其最大似然估计值；第二步是最大化（M），最大化在E步上求得的最大似然值来计算参数的值。M步上找到的参数估计值被用于下一个E步计算中，这个过程不断交替进行，使参数不断接近最大似然值。

在IBM model中，因为引入了alignment这个概念，因此，整个翻译过程都是通过在alignment关系下，最大化外语单词翻译为目标语言单词的概率。

![p(a|e, f)](https://storage.googleapis.com/papyrus_images/d3c5b68b53a8c39d914d2352b7b5cf4f1ceeb05aa4ed822dc5873bc8d5cedc2b.svg)

p(a|e, f)

![P(e|f)](https://storage.googleapis.com/papyrus_images/9edc6cee7bbaf3577a7c8283d096b92de29f02c8dfef7a6ef2e44fa0f1a9c005.svg)

P(e|f)

![\\begin{aligned}  p(e|f) &= \\sum_{a}^{}{p(e, a|f)} \\\\ &=\\sum_{a(1)=0}^{l_f}{} ... \\sum_{a(l_e)=0}^{l_f}{p(e,a|f)} \\\\  &=\\sum_{a(1)=0}^{l_f}{} ... \\sum_{a(l_e)=0}^{l_f}{\\frac{\\epsilon}{(l_f + 1)^{l_e}}\\prod_{j=1}^{l_e}{t(e_j|f_{a(j)})}} \\\\  &=\\frac{\\epsilon}{(l_f + 1)^{l_e}}\\sum_{a(1)=0}^{l_f}{} ... \\sum_{a(l_e)=0}^{l_f}{} \\prod_{j=1}^{l_e}{t(e_j|f_{a(j)})} \\\\  &=\\frac{\\epsilon}{(l_f + 1)^{l_e}}\\prod_{j=1}^{l_e}\\sum_{i=0}^{l_f}{t(e_j|f_{i})}  \\end{aligned}](https://storage.googleapis.com/papyrus_images/7262d2ef509aa6ebef6d013ad996165a52a514567171ee928b802ec0d17a21fe.svg)

\\\\begin{aligned} p(e|f) &= \\\\sum\_{a}^{}{p(e, a|f)} \\\\\\\\ &=\\\\sum\_{a(1)=0}^{l\_f}{} ... \\\\sum\_{a(l\_e)=0}^{l\_f}{p(e,a|f)} \\\\\\\\ &=\\\\sum\_{a(1)=0}^{l\_f}{} ... \\\\sum\_{a(l\_e)=0}^{l\_f}{\\\\frac{\\\\epsilon}{(l\_f + 1)^{l\_e}}\\\\prod\_{j=1}^{l\_e}{t(e\_j|f\_{a(j)})}} \\\\\\\\ &=\\\\frac{\\\\epsilon}{(l\_f + 1)^{l\_e}}\\\\sum\_{a(1)=0}^{l\_f}{} ... \\\\sum\_{a(l\_e)=0}^{l\_f}{} \\\\prod\_{j=1}^{l\_e}{t(e\_j|f\_{a(j)})} \\\\\\\\ &=\\\\frac{\\\\epsilon}{(l\_f + 1)^{l\_e}}\\\\prod\_{j=1}^{l\_e}\\\\sum\_{i=0}^{l\_f}{t(e\_j|f\_{i})} \\\\end{aligned}

![P(a|e_j,f_i)=\\frac{\\epsilon}{l_f+1}](https://storage.googleapis.com/papyrus_images/e67e3376f12fbde1a4891daed3aa5c818cdcb192e6c1448e1bebf4258ec0dfd6.svg)

P(a|e\_j,f\_i)=\\\\frac{\\\\epsilon}{l\_f+1}

（b）计算目标函数，如下：

![\\begin{aligned} p(a|e, f) &= \\frac{p(e, a|f)}{p(e|f)} \\\\ &=\\frac{\\frac{\\epsilon}{(l_f + 1)^{l_e}}\\prod_{j=1}^{l_e}{t(e_j|f_{a(j)})}} {\\frac{\\epsilon}{(l_f + 1)^{l_e}}\\prod_{j=1}^{l_e}\\sum_{i=0}^{l_f}{t(e_j|f_{i})}} \\\\ &=\\prod_{j=1}^{l_e}{\\frac{t(e_j|f_{a(j)})}{\\sum_{i=0}^{l_f}{t(e_j|f_{i})}}} \\end{aligned}](https://storage.googleapis.com/papyrus_images/78dbb9684a273723938026dc8c89e51db8e46fd53db1df581b63416049b65a64.svg)

\\\\begin{aligned} p(a|e, f) &= \\\\frac{p(e, a|f)}{p(e|f)} \\\\\\\\ &=\\\\frac{\\\\frac{\\\\epsilon}{(l\_f + 1)^{l\_e}}\\\\prod\_{j=1}^{l\_e}{t(e\_j|f\_{a(j)})}} {\\\\frac{\\\\epsilon}{(l\_f + 1)^{l\_e}}\\\\prod\_{j=1}^{l\_e}\\\\sum\_{i=0}^{l\_f}{t(e\_j|f\_{i})}} \\\\\\\\ &=\\\\prod\_{j=1}^{l\_e}{\\\\frac{t(e\_j|f\_{a(j)})}{\\\\sum\_{i=0}^{l\_f}{t(e\_j|f\_{i})}}} \\\\end{aligned}

![t(e_j|f_i)](https://storage.googleapis.com/papyrus_images/a2e3fe89c1c86e6d0f38ce814f5b92c781c3f88a731361ea05a54989268cdc10.svg)

t(e\_j|f\_i)

![c(e, f)](https://storage.googleapis.com/papyrus_images/4a4e03e3b78605442122f0407e49a9b2a9e2afbd96002ca6c6037038d7264e1a.svg)

c(e, f)

![c(e|f; e, f) = \\sum_{a}^{}{p(e, a|f)}\\sum_{j=1}^{l_e}\\delta(e,e_j)\\delta(f,f_{a(i)})](https://storage.googleapis.com/papyrus_images/0ffe49f20f2007c25b6bc2df98dafe3e5c4e443ef7faa0d5f28ae1f3ee2561db.svg)

c(e|f; e, f) = \\\\sum\_{a}^{}{p(e, a|f)}\\\\sum\_{j=1}^{l\_e}\\\\delta(e,e\_j)\\\\delta(f,f\_{a(i)})

![c(e|f; e, f) = \\frac{t(e|f{})}{\\sum_{i=0}^{l_f}{t(e|f_i)}}\\sum_{j=1}^{l_e}\\delta(e,e_j)\\delta(f,f_{a(i)})](https://storage.googleapis.com/papyrus_images/b685f7ef19cd81e53fcd32613792f43942fc4bf3cea1aeed72e885736fbd1ca0.svg)

c(e|f; e, f) = \\\\frac{t(e|f{})}{\\\\sum\_{i=0}^{l\_f}{t(e|f\_i)}}\\\\sum\_{j=1}^{l\_e}\\\\delta(e,e\_j)\\\\delta(f,f\_{a(i)})

![\\delta(e,e_j)](https://storage.googleapis.com/papyrus_images/23e7f7a8deda6d126c2f3ed2499512253ddf0e98ca809e1d3977be8017c961e7.svg)

\\\\delta(e,e\_j)

最终可按照如下式子估计整个模型的参数：

![t(e|f ; e, f) =\\frac{\\sum_{(e,f)}{c(e|f ; e, f)}}{\\sum_{(e)}\\sum_{(e,f)}c(e|f ; e, f)}](https://storage.googleapis.com/papyrus_images/b8db24b1c75e18bbaac1649f655c215fb5f83c4d8dbe1b8dd94442c497b91f81.svg)

t(e|f ; e, f) =\\\\frac{\\\\sum\_{(e,f)}{c(e|f ; e, f)}}{\\\\sum\_{(e)}\\\\sum\_{(e,f)}c(e|f ; e, f)}

三.利用EM模型估计整个模型参数
----------------

![t(e|f)](https://storage.googleapis.com/papyrus_images/7a928c5b88df836d4e316c3bd63e3c7eaa35730ac94c11a446142591a2a8b62c.svg)

t(e|f)

![](https://storage.googleapis.com/papyrus_images/9e1dd448f82ba71637a4631490452c035703910896e52b9e85eacbf3027c1a4d.jpg)

![t(f|e)](https://storage.googleapis.com/papyrus_images/89853439cb2d0fd6c4bdefa716610417832e061b06719166ead170d69c10fac7.svg)

t(f|e)

![\\sum_{(e,f)}{c(e|f ; e, f)}](https://storage.googleapis.com/papyrus_images/e9b68e6bad14fc3e062e173026aa57cb34ee94692b8d62894f4eb0a2855aad9c.svg)

\\\\sum\_{(e,f)}{c(e|f ; e, f)}

![\\frac{\\sum_{(e,f)}{c(e|f ; e, f)}}{\\sum_{(e)}\\sum_{(e,f)}c(e|f ; e, f)}](https://storage.googleapis.com/papyrus_images/d42791cb6eb92a6c942cb490474c0566d8b1b47a4318bd787d39d31eccb204f4.svg)

\\\\frac{\\\\sum\_{(e,f)}{c(e|f ; e, f)}}{\\\\sum\_{(e)}\\\\sum\_{(e,f)}c(e|f ; e, f)}

---

*Originally published on [zekeer.eth](https://paragraph.com/@zekeer/ibm-model-1)*