My LLM extraction dropped listings at chunk boundaries

While working on a lead-generation agent using the OpenAI Agents SDK and ZenRows, I encountered a data-loss bug during chunked LLM extraction.

A Clutch directory page returned about 543K characters of Markdown, so extraction ran over 40K-character chunks. The prompt told the model to skip partial listings because a chunk could begin or end in the middle of a record.

That instruction caused a problem at the boundaries. A listing split between two chunks looked partial in both, so both chunks skipped it. There was no error, and the final count still looked plausible.

The fix was a 2K-character overlap between chunks. That ensured each boundary listing appeared complete in at least one chunk.

The overlap introduced duplicates, so deduplication needed two checks:

• Normalize URLs before comparing them because the model returned domains with and without www and trailing slashes.
• Fall back to the company name when no website is available. Otherwise, all records with an empty website field collapse into a single record.

Unwrapping the directory’s tracking URLs also reduced the page by about 86K characters and stopped the model from treating directory URLs as company websites.

The pipeline extracted 78 companies from a single page, with no websites missing in that run. Counts still vary because the directory changes and the model segments the page differently between runs.

How are you handling chunk boundaries in LLM extraction: overlap and deduplication, structural splitting, or a separate boundary-detection pass?