Changelog

0.29.0 (2025-10-16)

Allowed passing None as the url field value to the item classes.
Explicitly re-export public names.

0.28.0 (2025-09-02)

Added official Python 3.14 support.
Switched attribute documentation from Sphinx comments (#:) to docstrings.

This allows IDEs to show them when hovering attributes, and allows ItemAdapter.get_json_schema() to include them as descriptions.
Added an "llmHint" JSON Schema metadata field to some item types and fields.

LLMs should have an easier time writing extraction code when given the corresponding JSON schema (generated with ItemAdapter.get_json_schema()).
We now guarantee that all types importable from zyte_common_items have type hints retrievable at run time with get_type_hints(), i.e. not hidden with TYPE_CHECKING.

This ensures they can be used with scrapy-poet.

0.27.1 (2025-06-26)

Added displayedUrlText to SerpOrganicResult.
Importing DropLowProbabilityItemPipeline no longer triggers a warning about the deprecation of the zyte_common_items.ae module.

0.27.0 (2025-01-16)

The DropLowProbabilityItemPipeline now supports nested items, i.e. dict objects with items as values.
Added an add-on to make Scrapy configuration easier.
Metadata now also has all fields from SerpMetadata.
Messages about dropped items, e.g. due to low probability, are now logged as information and not as warnings.

0.26.2 (2024-11-12)

Fixed the package build missing all nested packages:

zyte_common_items.components

zyte_common_items.items

zyte_common_items.pages

0.26.1 (2024-11-12)

Note

This version was yanked, see 0.26.2 (2024-11-12).

Migrated from setup.py to pyproject.toml.
Fixed Serp.from_dict returning an instance where organicResults list items were dict instead of instances of SerpOrganicResult.

0.26.0 (2024-11-11)

Added ForumThread and related classes.

0.25.0 (2024-11-11)

Removed Python 3.8 support, added Python 3.13 support.
Backward-incompatible change: SearchRequestTemplatePage now subclasses Page, adding a dependency on HttpResponse. A new BaseSearchRequestTemplatePage that subclasses BasePage has been added as well.

Tip

Where a dependency on HttpResponse is not needed, BaseSearchRequestTemplatePage is a better replacement for the SearchRequestTemplatePage class from zyte-common-items 0.24.0 and lower, as it only depends on web_poet.page_inputs.http.RequestUrl.
The keyword parameter of SearchRequestTemplate.request() has been deprecated in favor of query. As a result, Jinja templates in SearchRequestTemplate field values should now use the query variable (e.g. {{ query|quote_plus }}) instead of the keyword variable.
Unexpected variables in Jinja templates of SearchRequestTemplate field values (e.g. {{ foo }}), which used to be silently removed, will now trigger an UndefinedError exception when calling SearchRequestTemplate.request().
Fixed coverage data generation during tests.

0.24.0 (2024-10-02)

Added JobPostingNavigation and related classes.

0.23.0 (2024-09-19)

Added CustomAttributes and related classes.

0.22.0 (2024-09-09)

Added Serp and related classes.

0.21.0 (2024-08-27)

The new images_processor(), used by default in images fields, can convert a string, a list of strings or a list of dicts into an Image list. Strings become Image.url. Dicts get their url key mapped as Image.url.
brand_processor() now converts strings into Brand objects with the input string as Brand.name.
price_processor() and simple_price_processor() now convert numeric values into strings with 2 decimal positions.
metadata_processor() no longer assumes that the input metadata is not None.

0.20.0 (2024-06-19)

Now fields of auto page object classes have auto_field set to True in their field metadata, to make it easier to check if a page object subclass is overriding a given field.

0.19.0 (2024-04-24)

Now requires attrs >= 22.2.0.
New deprecations:
- zyte_common_items.components.request_list_processor (use zyte_common_items.processors.probability_request_list_processor)
- zyte_common_items.items.RequestListCaster (use zyte_common_items.converters.to_probability_request_list)
- zyte_common_items.util.metadata_processor (use zyte_common_items.processors.metadata_processor)
Added DropLowProbabilityItemPipeline that drops items with the probability value lower than a set threshold.
Added the BaseMetadata, ListMetadata, and DetailsMetadata classes (they were previously private).
Added the ListMetadata.validationMessages attribute.
Added the ListMetadata.get_date_downloaded_parsed() method.
Added the zyte_common_items.converters module with useful attrs converters.
Reorganized the module structure.
Documentation improvements.
Test and CI fixes and improvements.

0.18.0 (2024-03-15)

Initial support for request templates, starting with search requests.

0.17.1 (2024-03-13)

Added Python 3.12 support.
description_processor() and description_html_processor() now raise an exception when they receive an unsupported input value such as a non-HtmlElement node.
Documentation improvements.

0.17.0 (2024-02-14)

Implement the zyte_common_items.ae module and the zyte_common_items.pipelines.AEPipeline item pipeline to make it easier to migrate from Zyte Automatic Extraction to Zyte API automatic extraction.

0.16.0 (2024-02-06)

Auto-prefixed versions of page objects, such as AutoProductPage(), now have all their fields defined as synchronous instead of asynchronous.

0.15.0 (2024-01-30)

Now requires zyte-parsers >= 0.5.0.
Added SocialMediaPost and related classes.
Added ProductFromListExtractor, ProductFromListSelectorExtractor, ProductVariantExtractor and ProductVariantSelectorExtractor.
Added zyte_common_items.processors.rating_processor() and enabled it for the aggregateRating fields in the page classes for BusinessPlace and Product.
Improved the documentation about the processors.

0.14.0 (2024-01-16)

Now requires zyte-parsers >= 0.4.0.
Added zyte_common_items.processors.gtin_processor() and enabled it for the gtin fields in the page classes for Product.
Improved the API documentation.

0.13.0 (2023-11-09)

Added Auto-prefixed versions of page objects, such as AutoProductPage(), that return data from Zyte API automatic extraction from their fields by default, and can be used to more easily override that data with custom parsing logic.

0.12.0 (2023-10-27)

Added get_probability() helper method in item classes (e.g. Product, Article) and ProbabilityRequest.

0.11.0 (2023-09-08)

Now requires clear-html >= 0.4.0.
Added zyte_common_items.processors.description_processor() and enabled it for the description fields in the page classes for BusinessPlace, JobPosting, Product and RealEstate.
Added zyte_common_items.processors.description_html_processor() and enabled it for the descriptionHtml fields in the page classes for JobPosting and Product.
Added default implementations for the description (in the page classes for BusinessPlace, JobPosting, Product and RealEstate) and descriptionHtml (in the page classes for JobPosting and Product) fields: if one of these fields is user-defined, another one will use it.
price_processor() and simple_price_processor() now keep at least two decimal places when formatting the result.

0.10.0 (2023-08-24)

Now requires price-parser >= 0.3.4 (a new dependency) and zyte-parsers >= 0.3.0 (a version increase).
Added zyte_common_items.processors.price_processor() and enabled it for the price fields.
Added zyte_common_items.processors.simple_price_processor() and enabled it for the regularPrice fields.
Added default implementations for the currency (uses the CURRENCY attribute on the page class) and currencyRaw (uses the data extracted by the price field) fields.

0.9.0 (2023-08-03)

Now requires web-poet >= 0.14.0.
Fixed detection of the HasMetadata base class.

0.8.0 (2023-07-27)

Updated minimum versions for the following requirements:
- attrs >= 22.1.0
- web-poet >= 0.9.0
- zyte-parsers >= 0.2.0
Added JobPosting and related classes.
Added zyte_common_items.processors.brand_processor() and enabled it for the brand fields.
Added zyte_common_items.Request.to_scrapy() to convert zyte_common_items.Request instances to scrapy.Request instances.

0.7.0 (2023-07-11)

Now requires zyte-parsers.
Added navigation classes: ArticleNavigation, ProductNavigation, the page classes that produce them, and other related classes.
Improved the metadata field handling, also fixing some bugs:
- Added item-specific metadata classes. The metadata item fields were changed to use them.
- Backward-incompatible change: the DateDownloadedMetadata class was removed. The item-specific ones are now used instead.
- Backward-incompatible change: ArticleFromList no longer has a probability field and instead has a metadata field like all other similar classes.
- Backward-incompatible change: while in most items the old and the new type of the metadata field have the same fields, the one in Article now has probability, the one in ProductList no longer has probability, and the one in ProductFromList no longer has dateDownloaded.
- The default probability value is now 1.0 instead of None.
- Added the HasMetadata mixin which is used similarly to Returns to set the page metadata class.
- Metadata objects assigned to the metadata fields of the items or returned from the metadata() methods of the pages are now converted to suitable classes.
Added zyte_common_items.processors.breadcrumbs_processor() and enabled it for the breadcrumbs fields.

0.6.0 (2023-07-05)

Added Article and ArticleList.
Added support for Python 3.11 and dropped support for Python 3.7.

0.5.0 (2023-05-10)

Now requires itemadapter >= 0.8.0.
Added RealEstate.
Added the zyte_common_items.BasePage.no_item_found() and zyte_common_items.Page.no_item_found() methods.
Improved the error message for invalid input.
Added ZyteItemKeepEmptyAdapter and documented how to use it and ZyteItemAdapter in custom subclasses of itemadapter.ItemAdapter.

0.4.0 (2023-03-27)

Added support for business places.

0.3.1 (2023-03-17)

Fixed fields from BasePage subclasses leaking across subclasses. (#29, #30)
Improved how the from_dict() and from_list() methods report issues in the input data. (#25)

0.3.0 (2023-02-03)

Added page object classes for e-commerce product detail and product list pages.

0.2.0 (2022-09-22)

Supports web_poet.RequestUrl and web_poet.ResponseUrl and automatically convert them into a string on URL fields like Product.url.
Bumps the web_poet dependency version from 0.4.0 to 0.5.0 which fully supports type hints using the py.typed marker.
This package now also supports type hints using the py.typed marker. This means mypy would properly use the type annotations in the items when using it in your project.
Minor improvements in tests and annotations.

0.1.0 (2022-07-29)

Initial release.