Item API
Product
- class zyte_common_items.Product(**kwargs)
Product from an e-commerce website.
url
is the only required attribute.- classmethod from_list(items: Optional[List[Dict]], *, trail: Optional[str] = None) List
Read items from a list.
- additionalProperties: Optional[List[AdditionalProperty]]
List of name-value pais of data about a specific, otherwise unmapped feature.
Additional properties usually appear in product pages in the form of a specification table or a free-form specification list.
Additional properties that require 1 or more extra requests may not be extracted.
See also
features
.
- aggregateRating: Optional[AggregateRating]
Aggregate data about reviews and ratings.
- availability: Optional[str]
Availability status.
The value is expected to be one of:
"InStock"
,"OutOfStock"
.
- breadcrumbs: Optional[List[Breadcrumb]]
Webpage breadcrumb trail.
- currency: Optional[str]
Price currency ISO 4217 alphabetic code (e.g.
"USD"
).See also
currencyRaw
.
- currencyRaw: Optional[str]
Price currency as it appears on the webpage (no post-processing), e.g.
"$"
.See also
currency
.
- description: Optional[str]
Plain-text description.
If the description is split across different parts of the source webpage, only the main part, containing the most useful pieces of information, should be extracted into this attribute.
It may contain data found in other attributes (
features
,additionalProperties
).Format-wise:
Line breaks and non-ASCII characters are allowed.
There is no length limit for this attribute, the content should not be truncated.
There should be no whitespace at the beginning or end.
See also
descriptionHtml
.
- descriptionHtml: Optional[str]
HTML description.
See
description
for extraction details.The format is not the raw HTML from the source webpage. See the HTML normalization specification for details.
- features: Optional[List[str]]
List of features.
They are usually listed as bullet points in product webpages.
See also
additionalProperties
.
- gtin: Optional[List[Gtin]]
List of standardized GTIN product identifiers associated with the product, which are unique for the product across different sellers.
See also:
mpn
,productId
,sku
.
- images: Optional[List[Image]]
All product images.
The main image (see
mainImage
) should be first in the list.Images only displayed as part of the product description are excluded.
- metadata: Optional[ProductMetadata]
Data extraction process metadata.
- mpn: Optional[str]
Manufacturer part number (MPN).
A product should have the same MPN across different e-commerce websites.
See also:
gtin
,productId
,sku
.
- price: Optional[str]
Price at which the product is being offered.
It is a string with the price amount, with a full stop as decimal separator, and no thousands separator or currency (see
currency
andcurrencyRaw
), e.g."10500.99"
.If
regularPrice
is notNone
,price
should always be lower thanregularPrice
.
- productId: Optional[str]
Product identifier, unique within an e-commerce website.
It may come in the form of an SKU or any other identifier, a hash, or even a URL.
See also:
gtin
,mpn
,sku
.
- regularPrice: Optional[str]
Price at which the product was being offered in the past, and which is presented as a reference next to the current price.
It may be labeled as the original price, the list price, or the maximum retail price for which the product is sold.
See
price
for format details.If
regularPrice
is notNone
, it should always be higher thanprice
.
- size: Optional[str]
Size or dimensions.
Pertinent to products such as garments, shoes, accessories, etc.
It is extracted as displayed (e.g.
"XL"
).See also
color
,style
.
- sku: Optional[str]
Stock keeping unit (SKU) identifier, i.e. a merchant-specific product identifier.
See also:
gtin
,mpn
,productId
.
- style: Optional[str]
Style.
Pertinent to products such as garments, shoes, accessories, etc.
It is extracted as displayed (e.g.
"polka dots"
).See also
color
,size
.
- variants: Optional[List[ProductVariant]]
List of variants.
When slightly different versions of a product are displayed on the same product page, allowing you to choose a specific product version from a selection, each of those product versions are considered a product variant.
Product variants usually differ in
color
orsize
.The following items are not considered product variants:
Different products within the same bundle of products.
Product add-ons, e.g. premium upgrades of a base product.
Only variant-specific data is extracted as product variant details. For example, if variant-specific versions of the product description do not exist in the source webpage, the description attributes of the product variant are not filled with the base product description.
Extracted product variants may not include those that are not visible in the source webpage.
Product variant details may not include those that require multiple additional requests (e.g. 1 or more requests per variant).
- class zyte_common_items.ProductVariant(**kwargs)
Product
variant.See
Product.variants
,ProductVariantExtractor
,ProductVariantSelectorExtractor
.- classmethod from_list(items: Optional[List[Dict]], *, trail: Optional[str] = None) List
Read items from a list.
- additionalProperties: Optional[List[AdditionalProperty]]
List of name-value pais of data about a specific, otherwise unmapped feature.
Additional properties usually appear in product pages in the form of a specification table or a free-form specification list.
Additional properties that require 1 or more extra requests may not be extracted.
See also
features
.
- availability: Optional[str]
Availability status.
The value is expected to be one of:
"InStock"
,"OutOfStock"
.
- currency: Optional[str]
Price currency ISO 4217 alphabetic code (e.g.
"USD"
).See also
currencyRaw
.
- currencyRaw: Optional[str]
Price currency as it appears on the webpage (no post-processing), e.g.
"$"
.See also
currency
.
- gtin: Optional[List[Gtin]]
List of standardized GTIN product identifiers associated with the product, which are unique for the product across different sellers.
See also:
mpn
,productId
,sku
.
- images: Optional[List[Image]]
All product images.
The main image (see
mainImage
) should be first in the list.Images only displayed as part of the product description are excluded.
- mpn: Optional[str]
Manufacturer part number (MPN).
A product should have the same MPN across different e-commerce websites.
See also:
gtin
,productId
,sku
.
- price: Optional[str]
Price at which the product is being offered.
It is a string with the price amount, with a full stop as decimal separator, and no thousands separator or currency (see
currency
andcurrencyRaw
), e.g."10500.99"
.If
regularPrice
is notNone
,price
should always be lower thanregularPrice
.
- productId: Optional[str]
Product identifier, unique within an e-commerce website.
It may come in the form of an SKU or any other identifier, a hash, or even a URL.
See also:
gtin
,mpn
,sku
.
- regularPrice: Optional[str]
Price at which the product was being offered in the past, and which is presented as a reference next to the current price.
It may be labeled as the original price, the list price, or the maximum retail price for which the product is sold.
See
price
for format details.If
regularPrice
is notNone
, it should always be higher thanprice
.
- size: Optional[str]
Size or dimensions.
Pertinent to products such as garments, shoes, accessories, etc.
It is extracted as displayed (e.g.
"XL"
).See also
color
,style
.
- sku: Optional[str]
Stock keeping unit (SKU) identifier, i.e. a merchant-specific product identifier.
See also:
gtin
,mpn
,productId
.
- class zyte_common_items.ProductMetadata(**kwargs)
Metadata class for
zyte_common_items.Product.metadata
.- dateDownloaded: Optional[str]
Date and time when the product data was downloaded, in UTC timezone and the following format:
YYYY-MM-DDThh:mm:ssZ
.
- probability: Optional[float]
The probability (0 for 0%, 1 for 100%) that the resource features the expected data type.
For example, if the extraction of a product from a given URL is requested, and that URL points to the webpage of a product with complete certainty, the value should be 1. If with complete certainty the webpage features a job listing instead of a product, the value should be 0. When there is no complete certainty, the value could be anything in between (e.g. 0.96).
Product list
- class zyte_common_items.ProductList(**kwargs)
Product list from a product listing page of an e-commerce webpage.
It represents, for example, a single page from a category.
url
is the only required attribute.- classmethod from_list(items: Optional[List[Dict]], *, trail: Optional[str] = None) List
Read items from a list.
- breadcrumbs: Optional[List[Breadcrumb]]
Webpage breadcrumb trail.
- categoryName: Optional[str]
Name of the product listing as it appears on the webpage (no post-processing).
For example, if the webpage is one of the pages of the Robots category,
categoryName
is'Robots'
.
- metadata: Optional[ProductListMetadata]
Data extraction process metadata.
- pageNumber: Optional[int]
Current page number, if displayed explicitly on the list page.
Numeration starts with 1.
- products: Optional[List[ProductFromList]]
List of products.
It only includes product information found in the product listing page itself. Product information that requires visiting each product URL is not meant to be covered.
The order of the products reflects their position on the rendered page. Product order is top-to-bottom, and left-to-right or right-to-left depending on the webpage locale.
- class zyte_common_items.ProductFromList(**kwargs)
Product from a product list from a product listing page of an e-commerce webpage.
See
ProductList
,ProductFromListExtractor
,ProductFromListSelectorExtractor
.- classmethod from_list(items: Optional[List[Dict]], *, trail: Optional[str] = None) List
Read items from a list.
- currency: Optional[str]
Price currency ISO 4217 alphabetic code (e.g.
"USD"
).See also
currencyRaw
.
- currencyRaw: Optional[str]
Price currency as it appears on the webpage (no post-processing), e.g.
"$"
.See also
currency
.
- metadata: Optional[ProbabilityMetadata]
Data extraction process metadata.
- price: Optional[str]
Price at which the product is being offered.
It is a string with the price amount, with a full stop as decimal separator, and no thousands separator or currency (see
currency
andcurrencyRaw
), e.g."10500.99"
.If
regularPrice
is notNone
,price
should always be lower thanregularPrice
.
- productId: Optional[str]
Product identifier, unique within an e-commerce website.
It may come in the form of an SKU or any other identifier, a hash, or even a URL.
- regularPrice: Optional[str]
Price at which the product was being offered in the past, and which is presented as a reference next to the current price.
It may be labeled as the original price, the list price, or the maximum retail price for which the product is sold.
See
price
for format details.If
regularPrice
is notNone
, it should always be higher thanprice
.
- class zyte_common_items.ProductListMetadata(**kwargs)
Metadata class for
zyte_common_items.ProductList.metadata
.
Article
- class zyte_common_items.Article(**kwargs)
Article, typically seen on online news websites, blogs, or announcement sections.
url
is the only required attribute.- classmethod from_list(items: Optional[List[Dict]], *, trail: Optional[str] = None) List
Read items from a list.
- articleBody: Optional[str]
Clean text of the article, including sub-headings, with newline separators.
Format:
trimmed (no whitespace at the beginning or the end of the body string),
line breaks included,
no length limit,
no normalization of Unicode characters.
- articleBodyHtml: Optional[str]
Simplified and standardized HTML of the article, including sub-headings, image captions and embedded content (videos, tweets, etc.).
Format: HTML string normalized in a consistent way.
- breadcrumbs: Optional[List[Breadcrumb]]
Webpage breadcrumb trail.
- dateModified: Optional[str]
Date when the article was most recently modified.
Format: ISO 8601 format: “YYYY-MM-DDThh:mm:ssZ” or “YYYY-MM-DDThh:mm:ss±zz:zz”.
With timezone, if available.
- dateModifiedRaw: Optional[str]
Same date as
dateModified
, but :before parsing/normalization, i.e. as it appears on the website.
- datePublished: Optional[str]
Publication date of the article.
Format: ISO 8601 format: “YYYY-MM-DDThh:mm:ssZ” or “YYYY-MM-DDThh:mm:ss±zz:zz”.
With timezone, if available.
If the actual publication date is not found, the value of
dateModified
is used instead.
- datePublishedRaw: Optional[str]
Same date as
datePublished
, but :before parsing/normalization, i.e. as it appears on the website.
- description: Optional[str]
A short summary of the article.
It can be either human-provided (if available), or auto-generated.
- inLanguage: Optional[str]
Language of the article, as an ISO 639-1 language code.
Sometimes the article language is not the same as the web page overall language.
- metadata: Optional[ArticleMetadata]
Data extraction process metadata.
- class zyte_common_items.ArticleMetadata(**kwargs)
Metadata class for
zyte_common_items.Article.metadata
.- dateDownloaded: Optional[str]
Date and time when the product data was downloaded, in UTC timezone and the following format:
YYYY-MM-DDThh:mm:ssZ
.
- probability: Optional[float]
The probability (0 for 0%, 1 for 100%) that the resource features the expected data type.
For example, if the extraction of a product from a given URL is requested, and that URL points to the webpage of a product with complete certainty, the value should be 1. If with complete certainty the webpage features a job listing instead of a product, the value should be 0. When there is no complete certainty, the value could be anything in between (e.g. 0.96).
Article list
- class zyte_common_items.ArticleList(**kwargs)
Article list from an article listing page.
The
url
attribute is the only required attribute, all other fields are optional.- classmethod from_list(items: Optional[List[Dict]], *, trail: Optional[str] = None) List
Read items from a list.
- articles: Optional[List[ArticleFromList]]
List of article details found on the page.
The order of the articles reflects their position on the page.
- breadcrumbs: Optional[List[Breadcrumb]]
Webpage breadcrumb trail.
- metadata: Optional[ArticleListMetadata]
Data extraction process metadata.
- class zyte_common_items.ArticleFromList(**kwargs)
Article from an article list from an article listing page.
See
ArticleList
.- classmethod from_list(items: Optional[List[Dict]], *, trail: Optional[str] = None) List
Read items from a list.
- articleBody: Optional[str]
Clean text of the article, including sub-headings, with newline separators.
Format:
trimmed (no whitespace at the beginning or the end of the body string),
line breaks included,
no length limit,
no normalization of Unicode characters.
- datePublished: Optional[str]
Publication date of the article.
Format: ISO 8601 format: “YYYY-MM-DDThh:mm:ssZ” or “YYYY-MM-DDThh:mm:ss±zz:zz”.
With timezone, if available.
If the actual publication date is not found, the date of the last modification is used instead.
- datePublishedRaw: Optional[str]
Same date as
datePublished
, but :before parsing/normalization, i.e. as it appears on the website.
- inLanguage: Optional[str]
Language of the article, as an ISO 639-1 language code.
Sometimes the article language is not the same as the web page overall language.
- metadata: Optional[ProbabilityMetadata]
Data extraction process metadata.
- class zyte_common_items.ArticleListMetadata(**kwargs)
Metadata class for
zyte_common_items.ArticleList.metadata
.
Business place
- class zyte_common_items.BusinessPlace(**kwargs)
Business place, with properties typically seen on maps or business listings.
url
is the only required attribute.- classmethod from_list(items: Optional[List[Dict]], *, trail: Optional[str] = None) List
Read items from a list.
- actions: Optional[List[NamedLink]]
List of actions that can be performed directly from the URLs on the place page, including URLs.
- additionalProperties: Optional[List[AdditionalProperty]]
List of name-value pais of any unmapped additional properties specific to the place.
- aggregateRating: Optional[AggregateRating]
The overall rating, based on a collection of reviews or ratings.
- containedInPlace: Optional[ParentPlace]
If the place is located inside another place, these are the details of the parent place.
- metadata: Optional[BusinessPlaceMetadata]
Data extraction process metadata.
- openingHours: Optional[List[OpeningHoursItem]]
Ordered specification of opening hours, including data for opening and closing time for each day of the week.
- priceRange: Optional[str]
How is the price range of the place viewed by its customers (from z to zzzz).
- reservationAction: Optional[NamedLink]
The details of the reservation action, e.g. table reservation in case of restaurants or room reservation in case of hotels.
- starRating: Optional[StarRating]
Official star rating of the place.
- timezone: Optional[str]
Which timezone is the place situated in.
Standard: Name compliant with IANA tz database (tzdata).
- class zyte_common_items.BusinessPlaceMetadata(**kwargs)
Metadata class for
zyte_common_items.BusinessPlace.metadata
.- dateDownloaded: Optional[str]
Date and time when the product data was downloaded, in UTC timezone and the following format:
YYYY-MM-DDThh:mm:ssZ
.
- probability: Optional[float]
The probability (0 for 0%, 1 for 100%) that the resource features the expected data type.
For example, if the extraction of a product from a given URL is requested, and that URL points to the webpage of a product with complete certainty, the value should be 1. If with complete certainty the webpage features a job listing instead of a product, the value should be 0. When there is no complete certainty, the value could be anything in between (e.g. 0.96).
Real estate
- class zyte_common_items.RealEstate(**kwargs)
Real state offer, typically seen on real estate offer aggregator websites.
url
is the only required attribute.- classmethod from_list(items: Optional[List[Dict]], *, trail: Optional[str] = None) List
Read items from a list.
- additionalProperties: Optional[List[AdditionalProperty]]
A name-value pair field holding information pertaining to specific features. Usually in a form of a specification table or freeform specification list.
- area: Optional[RealEstateArea]
Real estate area details.
- breadcrumbs: Optional[List[Breadcrumb]]
Webpage breadcrumb trail.
- currencyRaw: Optional[str]
Currency associated with the price, as appears on the page (no post-processing).
- datePublished: Optional[str]
Publication date of the real estate offer.
Format: ISO 8601 format: “YYYY-MM-DDThh:mm:ssZ”
With timezone, if available.
- datePublishedRaw: Optional[str]
Same date as datePublished, but before parsing/normalization, i.e. as it appears on the website.
- description: Optional[str]
The description of the real estate.
Format:
trimmed (no whitespace at the beginning or the end of the description string),
line breaks included,
no length limit,
no normalization of Unicode characters,
no concatenation of description from different parts of the page.
- metadata: Optional[RealEstateMetadata]
Contains metadata about the data extraction process.
- numberOfRooms: Optional[int]
The number of rooms (excluding bathrooms and closets) of the real estate.
- realEstateId: Optional[str]
The identifier of the real estate, usually assigned by the seller and unique within a website, similar to product SKU.
- class zyte_common_items.RealEstateMetadata(**kwargs)
Metadata class for
zyte_common_items.RealEstate.metadata
.- dateDownloaded: Optional[str]
Date and time when the product data was downloaded, in UTC timezone and the following format:
YYYY-MM-DDThh:mm:ssZ
.
- probability: Optional[float]
The probability (0 for 0%, 1 for 100%) that the resource features the expected data type.
For example, if the extraction of a product from a given URL is requested, and that URL points to the webpage of a product with complete certainty, the value should be 1. If with complete certainty the webpage features a job listing instead of a product, the value should be 0. When there is no complete certainty, the value could be anything in between (e.g. 0.96).
Job posting
- class zyte_common_items.JobPosting(**kwargs)
A job posting, typically seen on job posting websites or websites of companies that are hiring.
url
is the only required attribute.- classmethod from_list(items: Optional[List[Dict]], *, trail: Optional[str] = None) List
Read items from a list.
- baseSalary: Optional[BaseSalary]
The base salary of the job or of an employee in the proposed role.
- dateModified: Optional[str]
The date when the job posting was most recently modified.
Format: ISO 8601 format: “YYYY-MM-DDThh:mm:ssZ”
With timezone, if available.
- dateModifiedRaw: Optional[str]
Same date as dateModified, but before parsing/normalization, i.e. as it appears on the website.
- datePublished: Optional[str]
Publication date of the job posting.
Format: ISO 8601 format: “YYYY-MM-DDThh:mm:ssZ”
With timezone, if available.
- datePublishedRaw: Optional[str]
Same date as datePublished, but before parsing/normalization, i.e. as it appears on the website.
- description: Optional[str]
A description of the job posting including sub-headings, with newline separators.
Format:
trimmed (no whitespace at the beginning or the end of the description string),
line breaks included,
no length limit,
no normalization of Unicode characters.
- descriptionHtml: Optional[str]
Simplified HTML of the description, including sub-headings, image captions and embedded content.
- employmentType: Optional[str]
Type of employment (e.g. full-time, part-time, contract, temporary, seasonal, internship).
- hiringOrganization: Optional[HiringOrganization]
Information about the organization offering the job position.
- jobLocation: Optional[JobLocation]
A (typically single) geographic location associated with the job position.
- jobStartDate: Optional[str]
Job start date
Format: ISO 8601 format: “YYYY-MM-DDThh:mm:ssZ”
With timezone, if available.
- jobStartDateRaw: Optional[str]
Same date as jobStartDate, but before parsing/normalization, i.e. as it appears on the website.
- metadata: Optional[JobPostingMetadata]
Contains metadata about the data extraction process.
- class zyte_common_items.JobPostingMetadata(**kwargs)
Metadata class for
zyte_common_items.JobPosting.metadata
.- dateDownloaded: Optional[str]
Date and time when the product data was downloaded, in UTC timezone and the following format:
YYYY-MM-DDThh:mm:ssZ
.
- probability: Optional[float]
The probability (0 for 0%, 1 for 100%) that the resource features the expected data type.
For example, if the extraction of a product from a given URL is requested, and that URL points to the webpage of a product with complete certainty, the value should be 1. If with complete certainty the webpage features a job listing instead of a product, the value should be 0. When there is no complete certainty, the value could be anything in between (e.g. 0.96).
Search Request templates
- class zyte_common_items.SearchRequestTemplate(**kwargs)
Request template to build a search
Request
.- classmethod from_list(items: Optional[List[Dict]], *, trail: Optional[str] = None) List
Read items from a list.
- body: Optional[str]
Jinja template for
Request.body
.It must be a plain
str
, notbytes
or a Base64-encodedstr
. Base64-encoding is done byrequest()
after rendering this value as a Jinja template.Defining a non-UTF-8 body is not supported.
- headers: Optional[List[Header]]
List of
Header
, forRequest.headers
, where everyname
andvalue
is a Jinja template.When a header name template renders into an empty string (after stripping spacing), that header is removed from the resulting list of headers.
- metadata: Optional[SearchRequestTemplateMetadata]
Data extraction process metadata.
- class zyte_common_items.SearchRequestTemplateMetadata(**kwargs)
Metadata class for
zyte_common_items.SearchRequestTemplate.metadata
.- dateDownloaded: Optional[str]
Date and time when the product data was downloaded, in UTC timezone and the following format:
YYYY-MM-DDThh:mm:ssZ
.
- probability: Optional[float]
The probability (0 for 0%, 1 for 100%) that the resource features the expected data type.
For example, if the extraction of a product from a given URL is requested, and that URL points to the webpage of a product with complete certainty, the value should be 1. If with complete certainty the webpage features a job listing instead of a product, the value should be 0. When there is no complete certainty, the value could be anything in between (e.g. 0.96).
Custom items
Subclass Item
to create your own item classes.
- class zyte_common_items.base.ProbabilityMixin(**kwargs)
Provides
get_probability()
to make it easier to access the probability of an item or item component that is nested under its metadata attribute.
Social media post
Represents a single social media post.
Read an item from a dictionary.
Read items from a list.
Returns the item probability if available, otherwise
None
.Details of the author of the post.
No easily identifiable information can be contained in here, such as usernames.
The timestamp at which the post was created.
Format: Timezone: UTC. ISO 8601 format: “YYYY-MM-DDThh:mm:ssZ”
The list of hashtags contained in the post.
The list of URLs of media files (images, videos, etc.) linked from the post.
Contains metadata about the data extraction process.
The identifier of the post.
Details of reactions to the post.
The text content of the post.
The URL of the final response, after any redirects.
Metadata class for
zyte_common_items.SocialMediaPost.metadata
.Date and time when the product data was downloaded, in UTC timezone and the following format:
YYYY-MM-DDThh:mm:ssZ
.The probability (0 for 0%, 1 for 100%) that the resource features the expected data type.
For example, if the extraction of a product from a given URL is requested, and that URL points to the webpage of a product with complete certainty, the value should be 1. If with complete certainty the webpage features a job listing instead of a product, the value should be 0. When there is no complete certainty, the value could be anything in between (e.g. 0.96).
The search text used to find the item.
Contains paths to fields with the description of issues found with their values.