Fit the context window
A local model with an 8K context cannot take the whole history. Keep the system message and the most recent messages that fit.
Write two functions. estimateTokens(text) returns Math.ceil(text.length / 4), a common estimate for text in European languages.
fitContext(messages, budget) takes { role, content } messages and a token budget. Messages with role: 'system' are always kept. Then add the others from the most recent backwards while the total does not exceed budget; as soon as one does not fit, stop (do not skip to older messages). Return the messages in their original order, without changing the input array. If the system messages alone exceed the budget, throw an error.
CHAT holds an example history of 168 tokens.
Challenges 0/5
- estimateTokens rounds up
- With 70 tokens keeps system and the last three
- Stops at the first message that does not fit (no skipping to older ones)
- Returns everything when it fits and leaves the input unchanged
- Throws when the system message does not fit
function estimateTokens(text) {
// about 4 characters per token, rounded up
return text.length;
}
function fitContext(messages, budget) {
// keep system messages, then add the newest messages while they fit
return messages.slice(-3);
}
console.log(fitContext(CHAT, 70).map((m) => m.role));Go deeper: the related guide →
This in production, with your data? Let's talk for 15 minutes →