79. The Attention Mechanism: Focus on Important Parts

The word "it" in this sentence: "The animal didn't cross the street because it was too tired." What does "it" refer to? The animal or the street? A human reads this and immediately knows: the animal was tired. The street cannot be tired. For a model processing this word by word, the word "it" arrives as just a token. Nothing in the token itself says what it refers to. The reference is determined by context: every other word in the sentence contributes to understanding what "it" means here. A...