repeated get requests using Node.js (or python) language to specific website, without getting blocked

€8-30 EUR

Törölve

Kiadva ekkor:

majdnem 5 évvel ezelőtt

€8-30 EUR

Teljesítéskor fizetve

I need to make a get request every 1-2 second (an interval between 1000ms and 2000ms, the shortest possible) to a page of this specific website: [login to view URL] (e.g. '[login to view URL]') I already wrote down some code, but as of now I've encountered two problems: a) the scraper gets recognized after x requests (or even at the first request) and gets blocked (bot protections returns an encrypted document) B) I get IP banned (403) after x requests I found out that the headers are really important and could be a key to bypass the protection. SO, what you have to do would be just provide me the snippet to achieve this and prove that the script doesn't get blocked after running for hours.

Node.js

Python

Web Scraping

Projektazonosító: 19992985

A projektről

4 ajánlat

Távolról teljesíthető projekt

Aktiválva: 5 évvel ezelőtt

Szeretne pénzt keresni?

E-mail cím

A Freelancer oldalán történő árajánlatadás előnyei

Határozzon meg költségvetést és időkeretet

Kapja meg fizetését a munkáért

Vázolja ajánlatát

Ingyen regisztrálhat és adhat árajánlatot munkákra

4 szabadúszó adott átlagosan €21 EUR összegű árajánlatot erre a munkára

@UndeadBug

Hi, So, to prevent getting banned or to make script run longer, there are a few options: 1) We use proxies along with requests so that each request is essentially different than others. 2) We use random user agents because a website always recognizes the user agent that you use to request and if same user agent keeps repeating requests in short durations, it blocks that. 3) Above 2 options should help a lot but in the worst case, we will have to introduce some more delay between requests. Some sites have rules where they allow 'x' no of requests per minute but if you pass that, they'll block you. Like, google search blocks you if you make somewhere around more than 10 requests per minute. Also, there's an option to use selenium which has less chances of being blocked but that totally depends on the website. That's all I have to say about what you need. Please contact me if you wanna implement these things. I cannot guarantee anything because all of it depends on the website itself.

€20 EUR 1 napon belül

5,0

(23 értékelés)

5,7

@bilalkamoon

Hello sir, I am an expert Python developer specialized in Web Scraping. You are right about the headers, they are the most important thing when scraping, and I am an expert in mimicing real requests headers, so I can make sure that the website would not recongnise that the visitor of the page is a bot

€18 EUR 1 napon belül

5,0

(11 értékelés)

4,0

@vivi1990

Good day! I'm a licensed full stack programming developer and designer. I have many experiences in laravel, wordpress, CI, python as backend. And I had several experiences in angular.js, react.js, node.js, Vue.js, material ,bootstrap as frontend. I have many experiences in c#,c++,c,java programming. I'm interested in your project, please feel free to check my clients reviews, my profile and if you are interested too, we can discuss more details. Thank you very much.

€23 EUR 1 napon belül

5,0

(1 értékelés)

2,6

@kshnkvn

In order not to be banned, you need to check the requester and after receiving 403 change the proxy. I have a lot of experience in scraping and I can show you how to do this on Python with well-documented code.

€24 EUR 1 napon belül