web scrapping and running it on the cloud

Lezárva Kiadva: Jan 18, 2014 Kiszállításkor fizetve
Lezárva Kiszállításkor fizetve

Only experienced freelancers who are ready to start it asap and devote time to it should apply.

This project is about scraping data from a public website. The dataset is large and so, the code will have to employ a sampling strategy (as suggested below).

Part 1:

The portal has data on customers for each firm. We have to first sample the firms (I will provide the list) and second, we will have to sample the customers. Customers data is spread out over multiple pages. Say, there are 1000 pages with 10 customers data on each page. Sampling will be on page numbers (say, 5% of random sample from pages 1,2...1000) and all the customers' data on sampled page numbers will have to be scrapped.

I have some python code who are almost good to run after some fixing. But they are for all-out scrapping. For the sake of time and resources, I would go for sampling.

However, one can also do it in Java or with other web languages.

Part 2: The job will take many days and I would suggest using a cheap cloud service for it (unless you have captive resources). One will also set it in a way that I keep getting the data continuously through cloud (Dropbox, or anything cheaper). I have run python code for this site on AWS and running upto 10 termins on one micro instance is feasible (possibly more). I won't suggest hitting the website at a higher rate.

This is not an one time job. Since the job will take multiple days (depending up on server capacity and the resources employed), it may go for 30 days. You must be willing to be ready to devote time to monitor it on regular basis.

Only experienced freelancers who are ready to start it asap and devote time to it should apply.

Felhőrendszerek Webes adatgyűjtés

Projektazonosító: #5333591

A projektről

33 ajánlat Távolról teljesíthető projekt Utoljára aktív: Feb 24, 2014

33 szabadúszó tett átlagosan 752$ összegű árajánlatot erre a munkára

alizza

Hello I have own cloud server that i use to scrape websites.I can use my own cloud server for your project.I am interesting on your project. I assure you 100% accurate and high quality results. Waiting for your positi Továbbiak

$515 USD 10 napon belül
(28 vélemény)
6.0
FINGERRPRINT

Hi, we have wide range of experience in similar type of jobs. 100% Quality and satisfaction guaranteed with us. Could you please see our client feedback in review section for hassle free service, Hope you would cons Továbbiak

$4123 USD 30 napon belül
(20 vélemény)
5.7
ShiaFirst

Hi there, I have automated many web scraping manuals tasks before and clients have posted positive feedbacks for me, I can show you proof. I will develop an automated software or script for this scraping and then run i Továbbiak

$400 USD 30 napon belül
(16 vélemény)
5.7
uumarkhalid31

hi, i am expert in web scraping and interested in this project, let me do this work with perfection, accuracy and according to your requirements thanks

$526 USD 10 napon belül
(35 vélemény)
5.3
proauthor

Dear employer, I've extensive experience in web scraping and I can proceed with your project. Please send in more details. I can give you a java application (we can schedule it on AWS). Hope that works for you. I'm Továbbiak

$750 USD 30 napon belül
(26 vélemény)
5.1
mhmhz

Hi I can scrape the site using my own application in c#, the application will be multi-threads. Also i have my own vps which i can test on too. So i can monitor the collected data and sent you the results at f Továbbiak

$2105 USD 20 napon belül
(14 vélemény)
5.1
Toperfection

Dear barnwal,Greetings! We are one of the most trusted freelancer on Freelancer site,we have a team of 18 multiskilled research operators those have a excellent working skills in Research/Data collection/Product ad Továbbiak

$263 USD 3 napon belül
(20 vélemény)
5.1
MagedGazzar

I've an experience in this field for about 5 years and can do this project for you from scratch with java if you're interested please contact me.

$250 USD 10 napon belül
(2 vélemény)
4.7
PSYCH0CODE

Hey! I use Python as my main tool for everything. I can provide you fast deploy and support. Always work until the end. If you have any question please contact me.

$350 USD 30 napon belül
(3 vélemény)
3.7
Abhishek92Kumar

Hello Sir I am interested in your project and I can start immediately I am a professional web scraper and I have done many web scraping projects before, I can show you a sample before awarding the project Looking f Továbbiak

$250 USD 4 napon belül
(13 vélemény)
3.6
phpdudes

HI barnwal, My bid is only a very rough estimate based on the description. In order to properly make an estimate i will need a bit more info. 1) What's the website ? What info do you need actually (as in . all o Továbbiak

$1450 USD 30 napon belül
(3 vélemény)
3.6
mtavartkiladze

Hello, I have done many this kind of projects and I'm sure I can help you please contact me so we can talk about it. Thanks

$526 USD 3 napon belül
(2 vélemény)
2.8
helmot

Hello Dear. You know that everything is possible with Python and BeautifulSoup. Please take a look at my profile to see the past web scraping projects. I have a server and If you want I can run the programs on my o Továbbiak

$368 USD 10 napon belül
(2 vélemény)
2.1
MasterExcel

I am Data Entry, MS Word and MS Excel Expert. i am very much professional in this work i am pretty sure that you cant find a best person for this job like me so i am ready to work on your project with low rate and high Továbbiak

$250 USD 10 napon belül
(1 Visszajelzés)
1.7
kamal20

sir, I'm experienced freelancer working from 2 years. have a nice web scrapping company with 20 men. every one is dedicate and perfect. I can start my job from right now and should complete devote time only. willing Továbbiak

$250 USD 20 napon belül
(2 vélemény)
2.0
Superteam786

Hi, i can do this job, Expert in web scraping, data mining, email mining, interested in your project, i have done many of projects like this , i have experience, and samples, i can show you, i can give you 100% accurat Továbbiak

$277 USD 10 napon belül
(7 vélemény)
1.6
mmagr99

------------------------------------------------------Expert here Ready to start with more than 9 years working experience...........................................Please check my past reviews for more satisfaction-- Továbbiak

$721 USD 20 napon belül
(1 Visszajelzés)
0.0
OP3NSOURC3

I understand you project requirements and i am ready to start on this right away................................

$3157 USD 30 napon belül
(0 vélemény)
0.0
balamuralip

Well sir i am having good experience with web scraping and is Better to Consider Rackspace Cloud Files for Storage in terms of cheaper price with out sacrifice of quality and as you asked you prefer Digital Ocean Cheap Továbbiak

$388 USD 3 napon belül
(0 vélemény)
2.4