I am looking for a Node.js or Python developer for a webcrawling or scraping project

Befejezett Kiadva: 2 évvel ezelőtt Kiszállításkor fizetve
Befejezett

I am looking for a Node.js or Python developer for a webcrawling or scraping project. I don't know how much work it will be, so we will do the work step by step to see how it goes. Therefore I need to know:

1- your hourly rate

2- your experience with Python or Node.js or anything else that can do the job on a webserver.

3- your experience with building a system as described below

4- how much time you have available to work on the project (hours available per day)

If needed, I will invite other developers to build this system as well, depending on how it goes. I have a running Ubuntu webserver with Python already installed. Node.js is not installed yet, so if we would change to that, then that needs to be installed first.

The system that we will be building:

I want to build a system that crawls urls (sometimes single urls and sometimes any url in the source of a page), which then returns the status codes to a database. The urls that should be crawled, will probably also come from the database or a sitemap.

This system should run with cron jobs (or anything else which is time based). It should check the urls based on a variable frequency, from every second to once a day. It should alert when the stored value is different than the existing value in the database, as an example: when the http status code changes from 200 to 503. Later on, the system should also have the option to check DNS records, or pieces of content in the source code of a page.

Next to crawling a page, I also want a script to check for javascript errors in a webpage. This should be similar to the Inspector as you find in many browsers. Any javascript error(s) should then be stored in the database and send an alert when the value in the database is different than the existing value.

The alert system will send alerts via email / SMS / WhatsApp and maybe others.

I have been checking for code to crawl urls for status codes, the fastest code I have now is in Python. I am not sure if something like Node.js could be faster, so if you are an expert on this, then please advise me what to do and why. If you have a better idea, I am open for that.

To finalize the system, we will be adding code to check any changes 2 or 3 extra times before sending an alert. Otherwise the system could be sending hundreds of alerts when a server with multiple websites goes down for 10 minutes...

The first steps to build it will be:

1- to setup a single crawler (I already have code in Python for a list of urls) which will inspect urls and display the status codes on a webpage

2- to connect a database, so it checks the list in the database and then updates the results to the database again (if different than existing value)

3- setup the cron job, so that it runs automatically

After these steps, we will evaluate the result and take it from there.

Python Node.js JavaScript Linux PHP

Projektazonosító: #32636768

A projektről

13 ajánlat Távolról teljesíthető projekt Utoljára aktív: 2 évvel ezelőtt

Odaítélve:

lowkeyusername

Hi there, 1- I prefer to work on a fixed payment but if an hourly rate is a must, then it really just depends on how much it would take to do this (probably not more than a day or two). 2- I've built some algorithms in Továbbiak

€4 EUR / óra
(0 értékelés)
0.0

13 szabadúszó tett átlagosan 16€/órás árajánlatot erre a munkára

liveexperts123

Hi there,I'm biddin on your project "I am looking for a Node.js or Python developer for a webcrawling or scraping project" I have read your project description and i'm an expert in Python and machine learning therefore Továbbiak

€43 EUR / óra
(33 vélemény)
6.5
smsaurabhv

Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON Továbbiak

€6 EUR / óra
(104 vélemény)
6.0
josanta

Hello there! I understand that you need to scrape data from a website. I have skills and tricks to make us successful in collecting every target data from the target website using Python. Can we talk please so that I c Továbbiak

€10 EUR / óra
(32 vélemény)
4.7
veljkodimcic111

Hello client I am a professional Python developer with 7+ years of experience in Python such as web scraping, bot, Flask , Django and Machine Learning such as NLP, Deep Learning, Artificial Intelligence , Tensor Overfl Továbbiak

€6 EUR / óra
(3 vélemény)
2.6
Prabakaran141

I will be able to provide an extended range of skills that you will find extremely useful. I have expert knowledge in all the major software programs used in the IT industry. What positive attribute I bring to the tab Továbbiak

€4 EUR / óra
(1 Visszajelzés)
0.0