Find Information from Websites

Need to compile a database of 3500 Real Estate agents in the UK on the (publically available) ARLA database ([login to view URL]) This is split into 3 parts. 1) Putting together a UK post code list (about 1-3 hours of cut and pasting in Excel from Wikipedia) 2) Doing Searches and using a web scraper to extract data 3) removing dupes and working with Exceptions .. like the crowded London W1A postcode. Instructions... FIRST PART - GET ALL UK POST CODES 1) Put together a database of all UK postcodes. This may exist somewhere if it doesn’t then you can get it at Wikipedia 2) Go to [login to view URL] - this has the 120 Post Code Regions. 3) Each of these has Post Code regions has 10-40 sub regions. 4) There will be about (120 x 20 avg) 2000-3000 post codes to search. 5) Putting together the list of UK post codes will take 2-5 hours (assuming inteligent cut and pasting and Excel) SECOND PART - WEB SCRAPE ARLA SITE 6) For each of the post codes do a search here - [login to view URL] 7) Each search will return up to 20 results. Each of these results need to be "opened" and scraped 8) Each page needs to have the data entered in an XLS (see the template) 9) The address needs to be parsed into Add1,2,3 and Town. The post code must be in a separate field. 10) The "Services" and "Employees" fields have multiple entires and need to be seprated in one field with semi colons 11) The "Bcode" is a postion of the URL. The last 7 digits after the "- b" and before the .asx of the URL. THIRD PART - REMOVE DUPLICATES 11) The scrapping will return 20 x 3000 .. Up to 60000 results. Many of these will be duplicates (i.e. overlapping postcodes) 12) The duplicates need to be removed (do this by looking at the Bcode) Exceptions Certain postiodes wont return any results e.g. W1A and For those try adding a postscipt to them after a space e.g. W1A 1AA. For the postscripts use 1AA, 2AA, 3AA, 4AA, 5AA, 6AA etc etc. If they still don’t return anything then do 1BB, 2BB etc etc