GPT-3.5-Turbo drops from 90% accuracy to 50% when the answer sits in the middle of a 20k-token prompt instead of the sta

GPT-3.5-Turbo drops from 90% accuracy to 50% when the answer sits in the middle of a 20k-token prompt instead of the start or end. Liu et al. (2023) documented this in "Lost in the Middle: How Language Models Use Long Contexts" at ACL. The edges of your context window are prime real estate. The middle is a graveyard. This is not a retrieval bug. It is an attention pattern. Transformers use soft attention across the full sequence, but positional encodings and training distributions bias the mode...