The "Context Window" of Modern AI: Why Does AI "Forget" Previous Conversations?

When you engage in long conversations with Large Language Models (LLMs) or ask them to analyze PDF documents spanning dozens of pages, you may encounter a strange phenomenon: as the conversation progresses, the AI starts talking nonsense or completely forgets the requirements you set at the beginning.

This isn't because the AI is in a "bad mood," but rather because it has hit a hard physical limit—the Context Window.

What is a Context Window?

Simply put, the context window is the maximum amount of information an AI can "see at once" while processing a current task.

If we compare an AI to a student writing a thesis, the context window is the size of their desk. They can spread reference books, notes, and draft papers on the desk for easy access. However, the desk space is limited. As new materials keep arriving and the desk fills up, the student must throw away the oldest piece of paper to make room for a new one.

Technically, this "desk size" is measured in Tokens. One token is roughly equivalent to 0.75 English words or 0.5 to 1 Chinese character. If a model has a context window of 128K tokens, it means it can process approximately 100,000 Chinese characters simultaneously.

Why Can't We Make the Window Infinitely Large?

Since a larger window seems better, why not just make it 10 billion tokens? There are two core technical bottlenecks: computational cost and attention dilution.

1. "Quadratic" Growth in Computational Cost

Most modern AIs are based on the Transformer architecture, whose core is the Attention Mechanism.
The attention mechanism requires the model to calculate the relationship between the current word and all previous words every time it generates a new word. This means the computational load increases quadratically ($\text{O}(n^2)$) as the input length grows.
- If the input length increases by 2 times $\rightarrow$ the computational load increases by 4 times.
- If the input length increases by 10 times $\rightarrow$ the computational load increases by 100 times.
This causes Video RAM (VRAM) to be exhausted rapidly and drastically reduces inference speed.

2. "Lost in the Middle"

Even if the hardware can handle it, the AI's "attention" becomes scattered. Research has found that models remember information at the two ends of the input (the beginning and the end) most vividly, while retrieval accuracy drops significantly for information located in the middle of the text. This is known as the Lost in the Middle phenomenon—even if the information is within the window, the AI may "overlook" it because the context is too long.

How to Achieve "Infinite Memory" Within a Limited Window?

To break through this limitation, engineers have developed several clever solutions:

RAG (Retrieval-Augmented Generation) — Giving AI an External Library

Instead of spreading all materials on the desk (the context window), give the AI a library (a vector database).
When a user asks a question, the system first searches the library for the most relevant paragraphs and then places only those few paragraphs on the desk for the AI to process. This saves space while ensuring information precision.

Sliding Window — Remembering Only Recent Events

For simple chatbots, the most common method is to "discard the earliest messages." When the conversation exceeds the limit, the system automatically deletes the oldest record. This is why some AIs suddenly forget your name after a long chat.

Long-Context Optimization Techniques

Models like Gemini or Claude optimize attention algorithms (such as FlashAttention or Ring Attention) to reduce computational complexity from quadratic to linear or near-linear levels. This enables ultra-long context windows ranging from millions to tens of millions of tokens.

Practical Advice for Users: How to Use Context Efficiently?

Since the window is limited and subject to the "lost in the middle" phenomenon, you can try the following techniques:
1. Place Key Instructions at the End: If you send a long article to an AI and ask for a summary, place your specific summary requirements (e.g., "Please summarize in three sentences") at the very bottom of the text, not the top.
2. Regularly Clean/Summarize Context: In extremely long conversations, you can ask the AI to "please summarize the key points of our discussion so far," then start a new conversation and paste the summary into it. This is equivalent to manually "cleaning the desk" for the AI.
3. Structured Input: Use clear headings and separators (such as --- or ###) to help the model better locate information blocks.

Understanding the context window helps you recognize the limitations of AI, allowing you to maximize its potential through more scientific Prompt Engineering.