Page object API

Product

class zyte_common_items.BaseProductPage(**kwargs)

Bases: BasePage, DescriptionMixin, PriceMixin, Returns[Product], HasMetadata[ProductMetadata]

BasePage subclass for Product.

class zyte_common_items.ProductPage(**kwargs)

Bases: Page, DescriptionMixin, PriceMixin, Returns[Product], HasMetadata[ProductMetadata]

Page subclass for Product.

class zyte_common_items.AutoProductPage(**kwargs): Bases: BaseProductPage

Product list

class zyte_common_items.BaseProductListPage(**kwargs)

Bases: BasePage, Returns[ProductList], HasMetadata[ProductListMetadata]

BasePage subclass for ProductList.

class zyte_common_items.ProductListPage(**kwargs)

Bases: Page, Returns[ProductList], HasMetadata[ProductListMetadata]

Page subclass for ProductList.

class zyte_common_items.AutoProductListPage(**kwargs): Bases: BaseProductListPage

Article

class zyte_common_items.BaseArticlePage(**kwargs)

Bases: BasePage, Returns[Article], HasMetadata[ArticleMetadata]

BasePage subclass for Article.

class zyte_common_items.ArticlePage(**kwargs)

Bases: Page, Returns[Article], HasMetadata[ArticleMetadata]

Page subclass for Article.

class zyte_common_items.AutoArticlePage(**kwargs): Bases: BaseArticlePage

Article list

class zyte_common_items.BaseArticleListPage(**kwargs)

Bases: BasePage, Returns[ArticleList], HasMetadata[ArticleListMetadata]

BasePage subclass for ArticleList.

class zyte_common_items.ArticleListPage(**kwargs)

Bases: Page, Returns[ArticleList], HasMetadata[ArticleListMetadata]

Page subclass for ArticleList.

class zyte_common_items.AutoArticleListPage(**kwargs): Bases: BaseArticleListPage

Business place

class zyte_common_items.BaseBusinessPlacePage(**kwargs)

Bases: BasePage, Returns[BusinessPlace], HasMetadata[BusinessPlaceMetadata]

BasePage subclass for BusinessPlace.

class zyte_common_items.BusinessPlacePage(**kwargs)

Bases: Page, Returns[BusinessPlace], HasMetadata[BusinessPlaceMetadata]

Page subclass for BusinessPlace.

class zyte_common_items.AutoBusinessPlacePage(**kwargs): Bases: BaseBusinessPlacePage

Real estate

class zyte_common_items.BaseRealEstatePage(**kwargs)

Bases: BasePage, Returns[RealEstate], HasMetadata[RealEstateMetadata]

BasePage subclass for RealEstate.

class zyte_common_items.RealEstatePage(**kwargs)

Bases: Page, Returns[RealEstate], HasMetadata[RealEstateMetadata]

Page subclass for RealEstate.

class zyte_common_items.AutoRealEstatePage(**kwargs): Bases: BaseRealEstatePage

Job posting

class zyte_common_items.BaseJobPostingPage(**kwargs)

Bases: BasePage, DescriptionMixin, Returns[JobPosting], HasMetadata[JobPostingMetadata]

BasePage subclass for JobPosting.

class zyte_common_items.JobPostingPage(**kwargs)

Bases: Page, DescriptionMixin, Returns[JobPosting], HasMetadata[JobPostingMetadata]

Page subclass for JobPosting.

class zyte_common_items.AutoJobPostingPage(**kwargs): Bases: BaseJobPostingPage

Job posting navigation

class zyte_common_items.BaseJobPostingNavigationPage(**kwargs)

Bases: BasePage, Returns[JobPostingNavigation], HasMetadata[JobPostingNavigationMetadata]

BasePage subclass for JobPostingNavigation.

class zyte_common_items.JobPostingNavigationPage(**kwargs)

Bases: Page, Returns[JobPostingNavigation], HasMetadata[JobPostingNavigationMetadata]

Page subclass for JobPostingNavigation.

class zyte_common_items.AutoJobPostingNavigationPage(**kwargs): Bases: BaseJobPostingNavigationPage

Search engine results

class zyte_common_items.BaseSerpPage(**kwargs): Bases: BasePage, Returns[Serp], HasMetadata[SerpMetadata]

class zyte_common_items.SerpPage(**kwargs): Bases: Page, Returns[Serp], HasMetadata[SerpMetadata]

class zyte_common_items.AutoSerpPage(**kwargs): Bases: BaseSerpPage

Forum thread

class zyte_common_items.BaseForumThreadPage(**kwargs)

Bases: BasePage, Returns[ForumThread], HasMetadata[ForumThreadMetadata]

BasePage subclass for ForumThread.

class zyte_common_items.ForumThreadPage(**kwargs)

Bases: Page, Returns[ForumThread], HasMetadata[ForumThreadMetadata]

Page subclass for ForumThread.

class zyte_common_items.AutoForumThreadPage(**kwargs): Bases: BaseForumThreadPage

Request templates

class zyte_common_items.BaseSearchRequestTemplatePage(**kwargs)

Bases: BasePage, Returns[SearchRequestTemplate], HasMetadata[SearchRequestTemplateMetadata]

BasePage subclass for SearchRequestTemplate.

class zyte_common_items.SearchRequestTemplatePage(**kwargs)

Bases: Page, Returns[SearchRequestTemplate], HasMetadata[SearchRequestTemplateMetadata]

Page subclass for SearchRequestTemplate.

Mixins

class zyte_common_items.pages.DescriptionMixin

Provides description and descriptionHtml field implementations.

description: str: Plain-text description. The default implementation makes it from the descriptionHtml field if that is user-defined.

descriptionHtml: str: HTML description. The default implementation makes it from the description field if that is user-defined.

class zyte_common_items.pages.PriceMixin

Provides price-related field implementations.

currency: str: Price currency ISO 4217 alphabetic code (e.g. "USD"). The default implementation returns self.CURRENCY if this attribute is defined.

currencyRaw: str: Price currency as it appears on the webpage (no post-processing), e.g. "$". The default implementation uses the data extracted by price_processor() from the price field.

Custom page objects

Subclass Page to create your own page object classes that rely on HttpResponse.

If you do not want HttpResponse as input, you can inherit from BasePage instead.

Your subclasses should also inherit generic classes web_poet.pages.Returns and zyte_common_items.HasMetadata to indicate their item and metadata classes.

class zyte_common_items.pages.base._BasePage(**kwargs)

class zyte_common_items.BasePage(**kwargs)

Bases: _BasePage

Base class for page object classes that has web_poet.page_inputs.http.RequestUrl as a dependency.

metadata

Data extraction process metadata.

dateDownloaded is set to the current UTC date and time.

probability is set to 1.0.

url: str: Main URL from which the data has been extracted.

no_item_found() → ItemT

Return an item with the current url and probability=0, indicating that the passed URL doesn’t contain the expected item.

Use it in your .validate_input implementation.

class zyte_common_items.Page(**kwargs)

Bases: _BasePage, WebPage

Base class for page object classes that has HttpResponse as a dependency.

metadata: zyte_common_items.Metadata

Data extraction process metadata.

dateDownloaded is set to the current UTC date and time.

probability is set to 1.0.

url: str: Main URL from which the data has been extracted.

no_item_found() → ItemT

Return an item with the current url and probability=0, indicating that the passed URL doesn’t contain the expected item.

Use it in your .validate_input implementation.

class zyte_common_items.HasMetadata: Inherit from this generic mixin to set the metadata class used by a page class.