Changelog

0.19.0 (2024-04-24)

Now requires attrs >= 22.2.0.
New deprecations:
- zyte_common_items.components.request_list_processor (use zyte_common_items.processors.probability_request_list_processor)
- zyte_common_items.items.RequestListCaster (use zyte_common_items.converters.to_probability_request_list)
- zyte_common_items.util.metadata_processor (use zyte_common_items.processors.metadata_processor)
Added DropLowProbabilityItemPipeline that drops items with the probability value lower than a set threshold.
Added the BaseMetadata, ListMetadata, and DetailMetadata classes (they were previously private).
Added the ListMetadata.validationMessages attribute.
Added the ListMetadata.get_date_downloaded_parsed() method.
Added the zyte_common_items.converters module with useful attrs converters.
Reorganized the module structure.
Documentation improvements.
Test and CI fixes and improvements.

Added Python 3.12 support.
description_processor() and description_html_processor() now raise an exception when they receive an unsupported input value such as a non-HtmlElement node.
Documentation improvements.

Implement the zyte_common_items.ae module and the zyte_common_items.pipelines.AEPipeline item pipeline to make it easier to migrate from Zyte Automatic Extraction to Zyte API automatic extraction.

Auto-prefixed versions of page objects, such as AutoProductPage(), now have all their fields defined as synchronous instead of asynchronous.

Now requires zyte-parsers >= 0.5.0.
Added SocialMediaPost and related classes.
Added ProductFromListExtractor, ProductFromListSelectorExtractor, ProductVariantExtractor and ProductVariantSelectorExtractor.
Added zyte_common_items.processors.rating_processor() and enabled it for the aggregateRating fields in the page classes for BusinessPlace and Product.
Improved the documentation about the processors.

Now requires zyte-parsers >= 0.4.0.
Added zyte_common_items.processors.gtin_processor() and enabled it for the gtin fields in the page classes for Product.
Improved the API documentation.

Added Auto-prefixed versions of page objects, such as AutoProductPage(), that return data from Zyte API automatic extraction from their fields by default, and can be used to more easily override that data with custom parsing logic.

Now requires clear-html >= 0.4.0.
Added zyte_common_items.processors.description_processor() and enabled it for the description fields in the page classes for BusinessPlace, JobPosting, Product and RealEstate.
Added zyte_common_items.processors.description_html_processor() and enabled it for the descriptionHtml fields in the page classes for JobPosting and Product.
Added default implementations for the description (in the page classes for BusinessPlace, JobPosting, Product and RealEstate) and descriptionHtml (in the page classes for JobPosting and Product) fields: if one of these fields is user-defined, another one will use it.
price_processor() and simple_price_processor() now keep at least two decimal places when formatting the result.

Now requires price-parser >= 0.3.4 (a new dependency) and zyte-parsers >= 0.3.0 (a version increase).
Added zyte_common_items.processors.price_processor() and enabled it for the price fields.
Added zyte_common_items.processors.simple_price_processor() and enabled it for the regularPrice fields.
Added default implementations for the currency (uses the CURRENCY attribute on the page class) and currencyRaw (uses the data extracted by the price field) fields.

Updated minimum versions for the following requirements:
- attrs >= 22.1.0
- web-poet >= 0.9.0
- zyte-parsers >= 0.2.0
Added JobPosting and related classes.
Added zyte_common_items.processors.brand_processor() and enabled it for the brand fields.
Added zyte_common_items.Request.to_scrapy() to convert zyte_common_items.Request instances to scrapy.http.Request instances.

Now requires zyte-parsers.
Added navigation classes: ArticleNavigation, ProductNavigation, the page classes that produce them, and other related classes.
Improved the metadata field handling, also fixing some bugs:
- Added item-specific metadata classes. The metadata item fields were changed to use them.
- Backwards incompatible change: the DateDownloadedMetadata class was removed. The item-specific ones are now used instead.
- Backwards incompatible change: ArticleFromList no longer has a probability field and instead has a metadata field like all other similar classes.
- Backwards incompatible change: while in most items the old and the new type of the metadata field have the same fields, the one in Article now has probability, the one in ProductList no longer has probability, and the one in ProductFromList no longer has dateDownloaded.
- The default probability value is now 1.0 instead of None.
- Added the HasMetadata mixin which is used similarly to Returns to set the page metadata class.
- Metadata objects assigned to the metadata fields of the items or returned from the metadata() methods of the pages are now converted to suitable classes.
Added zyte_common_items.processors.breadcrumbs_processor() and enabled it for the breadcrumbs fields.

Now requires itemadapter >= 0.8.0.
Added RealEstate.
Added the zyte_common_items.BasePage.no_item_found() and zyte_common_items.Page.no_item_found() methods.
Improved the error message for invalid input.
Added ZyteItemKeepEmptyAdapter and documented how to use it and ZyteItemAdapter in custom subclasses of itemadapter.ItemAdapter.

Fixed fields from BasePage subclasses leaking across subclasses. (#29, #30)
Improved how the from_dict() and from_list() methods report issues in the input data. (#25)

Supports web_poet.RequestUrl and web_poet.ResponseUrl and automatically convert them into a string on URL fields like Product.url.
Bumps the web_poet dependency version from 0.4.0 to 0.5.0 which fully supports type hints using the py.typed marker.
This package now also supports type hints using the py.typed marker. This means mypy would properly use the type annotations in the items when using it in your project.
Minor improvements in tests and annotations.

Initial release.