在构建深度学习自然语言处理(NLP)分类模型时，存在两个主要困难.

1. Data collection (getting thousands or millions of classified data points)
2. 深度学习架构和训练

我们构建能够理解语言复杂性的复杂深度学习模型的能力通常需要在这些领域积累多年的经验. 你的问题越难, 你的产出就越多样化, the more time you need to spend on each of these steps.

数据收集是一项繁重、耗时、昂贵的工作，也是* * * * * * * * * * * * * * * * * * * * * * *的头号限制因素 成功的NLP项目. 准备数据, 建设弹性管道, making choices amongst hundreds of potential preparation options, and getting “model ready” can easily take months of effort even with 优秀的机器学习工程师. 最后, 训练和优化深度学习模型需要直觉理解的结合, 专业技术, 以及坚持解决问题的能力.

在本文中，我们将介绍

1. NLP深度学习的趋势:迁移学习如何使世界级模型开源
2. 伯特简介:介绍一种更强大的NLP解决方案——来自变压器的双向编码器表示(伯特)
3. How 伯特 works and why it will change the way companies execute on NLP projects

深度学习的趋势

Naturally, the optimization of this process started with increasing accuracy. LSTM (long-short term memory) networks revolutionized many NLP tasks, 但他们曾经(现在也是)极度渴求数据. Optimizing and training those models can take days or weeks on large and expensive machines. 最后, deploying those large models in production is costly and cumbersome.

以减少这些复杂的创造因素, the field of computer vision has long made use of transfer learning. 迁移学习是一种使用为不同但相似的任务训练的模型来加速解决新任务的能力. 重新训练一个已经可以对树木进行分类的模型比从头开始训练一个新模型来识别灌木丛要少得多的努力.

想象一下这样一个场景:一个人一生中从未见过灌木，但见过很多树. 你会发现，根据他们对树木的了解向他们解释灌木丛的样子要比从头开始描述灌木丛容易得多. 迁移学习是一种非常人性化的学习方式, so it makes intuitive sense that this would work in deep learning tasks.

伯特 means you need less data, less training time, and you get more business value. The quality of NLP products that any business can build has become world-class.

伯特进来了

伯特使用了所谓的转换器，并被设计用来产生句子编码. Essentially, 伯特 is a language model based on a specific deep learning model. 它是专门用来提供上下文的, 数字, representation of a sentence or a string of sentences. That digital representation is the input to a shallow and uncomplicated model. 不仅如此, 但结果通常是优越的，并且需要一小部分输入数据来完成尚未解决的任务.

想象一下，用一天的时间收集数据，而不是一年的时间，并且能够围绕数据集构建模型，否则您永远不会有足够的数据来创建LSTM模型. The number of NLP tasks that would be opened up for a business that, 之前, could not afford the development time and expertise required is staggering.

伯特是如何工作的

In traditional NLP, the starting point for model training is word vectors. 词向量是一个数字列表[0].55, 0.24, 0.90, …] that attempt to 数字ally represent what that word means. 用数字表示, we can use those words in training complex models, 用大的词向量, we can embed information about words into our models.

伯特做了类似的事情(事实上, 它的起点是词向量), but it creates a 数字 representation of an entire input sentence (or sentences).

Compared to LSTM models, 伯特 does many things differently.

1. It reads all the words at once rather than left-to-right or right-to-left
2. 在训练期间，随机选择15%的单词进行“屏蔽”(字面上用[MASK]标记替换)
   • 10% of the randomly selected words are left unchanged
   • 10% of the masked words are replaced with random words
   • (a)和(b)共同作用迫使模型预测句子中的每个单词(模型是懒惰的)
3. 伯特 then attempts to predict all the words in the sentence, 只有被掩盖的单词才会对损失函数有贡献——包括未改变的和随机替换的单词
4. 该模型对下一个句子的预测进行了微调. 在这一步中，模型试图确定给定的句子是否是文本中的下一个句子

Convergence is slow, and 伯特 takes a long time to train. 然而, it learns the contextual relationships in text far better. 词向量是非常肤浅的表示，这限制了它们可以建模的复杂性——bert没有这种限制.

许多企业使用预先训练的LSTM模型，这些模型利用多个gpu，需要花费数天时间来训练它们的应用程序. 但与这些旧的预训练模型相比, 伯特 allows a team to accelerate solutions by tenfold. 可以移动到确定业务解决方案, 建立一个概念验证, and finally moving that concept into production in a fraction of the time. 也就是说, 在很少的情况下，现有的伯特模型不能在适当的地方使用，或者不能调优到特定的用例.

实现伯特和比较业务价值

由于本文关注的是构建真实产品的业务和工程应用, we will create and train two models to better understand the comparative value.

1. 伯特:最直接的伯特管道. 我们以标准的方式处理文本, 我们产生伯特句子编码, and we feed those sentence encodings into a shallow neural network
2. LSTM:标准 嵌入-编码-参加-预测架构 (见上图)

这个任务? Predicting the origin of movies based on their plot from IMDB. 我们的数据集涵盖了美国, 澳大利亚, 英国, 加拿大, 日本, 中国人, 韩国, and Russian films in addition to sixteen other films for a total of 24 origins. We have just under 35,000 total training examples.

下面是一个情节的示例片段.

几千年前, Steppenwolf和他的超灵军团试图用三个母盒的联合能量接管地球. They are foiled by a unified army that includes the Olympian Gods, 亚马逊女战士, 亚特兰提斯岛, 人类, 还有绿灯军团. 在击退了斯特彭沃夫的军队之后, the Mother Boxes are separated and hidden in locations on the planet. 在现在, 人类在悼念超人, 他的死触发了母盒的重新激活，而草原狼为了重获主人的宠爱而回到了地球, Darkseid. Steppenwolf aims to gather the artifacts to form “The Unity”, which will destroy Earth ‘s ecology and terraform it in the image of…

If you had not guessed, this is the plot of the Justice League — an American film.

结果

我们训练了一组不同的参数，以了解结果如何响应不同数量的数据和模型大小. As we said, the most significant value add from 伯特 is the need for far less data.

对于LSTM模型, 我们在GPU上训练了最大的模型，并改变了词汇表的大小和单词长度，以找到表现最好的模型. For the 伯特 model, we had nothing more than a single layer.

We fixed our testing set across all these samples, so we are consistently scoring the same training set.

在这个任务中，使用伯特句子编码训练的模型达到了令人印象深刻的f1得分0.仅仅1000个样本. LSTM网络不会超过0.60. 更令人印象深刻的是 training the 伯特 models took on average 1/20th of the time 而不是准备LSTM模型.

结论

By any metric, these results point to a revolution in NLP. Using 100x less data and 20x less training time, we achieved world-class results. 在几秒钟或几分钟内训练高质量模型的能力，而不是几小时或几天，在以前无法提供的领域开辟了NLP.

伯特 has many more uses than the one in this post. 有多语言模式. 它可以用来解决许多不同的NLP任务, either individually as in this post or simultaneously using multiple outputs. 伯特 sentence encodings are set to become a cornerstone of many NLP projects going forward.

这篇文章背后的代码可以在 Github. 我也鼓励读者去看看 Bert-as-a-service, which was a cornerstone of building the 伯特 sentence encodings for this post.