Define your data record before you build the collectorBody

Many data collection problems are actually data-modeling problems.

If the record structure is not defined first, it becomes difficult to compare results across regions, detect changes, or explain why two observations disagree.

A useful starting record might include:

• source_url

• product_id or another stable identifier

• region

• currency

• listed_price

• availability

• collected_at

• http_status

• error_type

The exact fields depend on the project, but each field should answer a practical question:

• Can this item be matched to the same item later?

• Which regional context produced the result?

• Was the value collected successfully?

• Can an unexpected result be reproduced?

It is also useful to keep raw observations separate from normalized values. For example, store the original price and currency before converting everything into a reporting currency. This makes exchange-rate decisions auditable and prevents the normalized value from hiding useful context.

For failures, avoid storing only “request failed.” Distinguish timeout, DNS error, connection error, access denial, rate limiting, parsing failure, and missing content. Different causes require different fixes.

What fields have been essential in your own web data projects?