GSOCKS Web Scraper API supports three integration methods, each offering unique benefits:
Realtime.
Realtime.
This method is synchronous. This means that upon sending your job submission request, you will have to keep the connection open until we successfully finish your job or return an error. This integration method is great if you would like to send JSON payloads with scraping and parsing job descriptions, including some advanced scraping parameters, to our API.
Push-Pull (supports batch queries).
Push-Pull (supports batch queries).
This integration method is asynchronous. This means that upon submitting your job, we will quickly return you a JSON with your job info, including the URLs for downloading the result/checking the job status. Once we are done processing your job, we will POST a notification about job completion to your server, at which point you can go ahead and download the results. This integration method also enables uploading results straight to your (AWS S3 or Google Cloud Storage).
Proxy Endpoint.
Proxy Endpoint.
This method is also synchronous (like Realtime), but instead of using our service via a RESTful interface, you can use our endpoint like a proxy. Use Proxy Endpoint if you've used proxies before and would just like to get unblocked content from us.
✅ Push-Pull is the recommended method for reliably handling large-scale data scraping tasks.
Connection Time-To-Live (TTL).
All API connections have a TTL of 150 seconds. In rare cases, connections may timeout before receiving a response. This can happen due to high system load or when submitting highly complex scraping jobs.