Add token limit handling to prevent 8k token overflow#8
Open
YosefHayim wants to merge 1 commit intohazadus:mainfrom
Open
Add token limit handling to prevent 8k token overflow#8YosefHayim wants to merge 1 commit intohazadus:mainfrom
YosefHayim wants to merge 1 commit intohazadus:mainfrom
Conversation
Problem: - GitHub Models API has an 8k token limit for entire requests - Large git diffs can exceed this limit, causing API failures - Users experience failures when staging large changes Solution: - Added token estimation using character-based heuristic (1 token ≈ 4 chars) - Implemented truncation logic that preserves UTF-8 boundaries - Added intelligent content prioritization when over limit Implementation Details: - estimateTokens(): Approximates tokens for any text content - truncateToTokenLimit(): Safely truncates text with ellipsis indicator - Modified GenerateCommitMessage() to: * Estimate tokens for prompt templates + changes + examples * Reserve tokens for templates (with buffer) * Prioritize examples (20% of remaining tokens) when present * Truncate changes to fit remaining budget * Display warning when truncation occurs Benefits: - Prevents API failures from token overflow - Maintains functionality by preserving maximum content - User-friendly with clear truncation warnings - No external dependencies, follows existing code style - Gracefully handles both changes-only and changes+examples scenarios
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds token limit handling to prevent API failures when large git diffs exceed the 8k token limit.
Problem
Solution
Implementation Details
estimateTokens(): Approximates tokens for any text contenttruncateToTokenLimit(): Safely truncates text with ellipsis indicatorGenerateCommitMessage()to:Benefits
Testing