Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for a proxy or pool of IPs #321

Closed
aharris02 opened this issue Sep 26, 2020 · 4 comments · Fixed by #352
Closed

Add support for a proxy or pool of IPs #321

aharris02 opened this issue Sep 26, 2020 · 4 comments · Fixed by #352
Assignees

Comments

@aharris02
Copy link

aharris02 commented Sep 26, 2020

Description

Left image is a server on AWS. Right is powershell on my local computer. As far as I can tell, the AWS server's requests are blocked because the AWS region IP blocks are easily discoverable.
image

Possible solution

I'm sure there are other solutions, but adding support for a rotating block of proxied IP addresses would help reduce blocks. This can be done in the app directly by changing the IP address each time the browser session is started and stopped. If we want to change the IP address for each query, that would need to be implemented as a separate proxy server.

https://scrapoxy.io/
https://zenscrape.com/how-to-build-a-simple-proxy-rotator-in-node-js/

@jef
Copy link
Owner

jef commented Sep 26, 2020

Related to #113 (comment)

@matjam
Copy link

matjam commented Sep 26, 2020

clearly the next step is a worker mode where you can have workers running on multiple machines and they report back to the central node that then posts the actual status. So if you have 30 people all running the checker, you have good coverage, and if one person gets blocked, the others will still be ok.

@fabienvauchelles

This comment was marked as spam.

@Mr-Tech-13
Copy link

I could play around with that today. Thanks for the update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants