You pause after a line of code, and an inline suggestion appears in your editor. GitHub documents that interaction for Copilot. The next word depends on what came earlier, but the system that serves today's suggestion is not the 2017 model in this paper. That paper translated English into German and French. Its question was whether a model could move information across a sentence without processing the positions through a recurrent chain.
This paper is hosted by its original publisher.
Read on proceedings.neurips.ccThe translation problem
A token is one item in a text sequence, sometimes a whole word and sometimes a smaller piece. The original Transformer maps each input token to an embedding, a learned list of numbers. Its base model uses 512 numbers at each position. Picture the illustrative sequence the cat sat: three positions, each with its own numeric vector.
In a recurrent layer, each position's running summary depends on the previous position's summary. The state for sat waits for cat, which waits for the. That dependency limits parallel work inside one training example. Information from the also reaches sat through the middle position. The paper's proposal removes recurrent and convolutional sequence-mixing layers and uses attention instead.
