Web scraper and data processing job for retail websites

Experienced web scraper and data processor to develop daily scraping service off 500+ retail product websites and one single affiliate network. Please only apply if you have referenceable web scraping and automation processing experience at this scale. Please explain the relevance of your experience within your application. Objective: To scrape products from 500+ retailer UK websites, all of which have 1,000’s of products (including colour/size/style variants). The data to be scraped is retail product data on the retailer's product detail page, including all available images (at full resolution). Example: [login to view URL] To scrape commission details for the same retailers (advertisers) from a single affiliate network. These products need to be scraped daily and then: • Stripped down to a subset of products based on a simple, single identification criteria (details provided to successful application). E.g. 5,000 products from a retail website might be reduced to 200 to be saved. (it might be that this reduction can be done before the data is scraped) • Mapped to a provided predefined category taxonomy – e.g. John Lewis might have Mens / Clothing / Tops / Polo Shirts and this will be mapped to Mens / Clothing / Shirts / Polo Shirts • Appended with the appropriate tracking URL elements for a single affiliate network which will be confirmed to successful applicant (e.g. Affiliate Window, Commission Junction) • Enriched with Shopify product tags based on data within the products title, description, colour, brand etc • Enriched products with data from the a single affiliate network which will be confirmed to successful applicant (commission % etc.) • Consolidated in to a single file/repository • Enriched with a calculation of the product price / affiliate commissions % (e.g. £30 RRP / 5% Commission = £1.50) • Uploaded to a Shopify website The successful applicant will setup the required services and database/repositories on an Amazon AWS machine.