Logo New Black

Documentation

Get started scraping the web in as little as 2 minutes

API Documentation

Detailed API documentation explaining all the APIs key functionality

Scraping & Integration Tutorials

Guided examples on how to integrate and use ScrapeNetwork

Frequent Questions

Quickly find answers for the most
common issues

Getting Started

Advanced API Functionality

Getting Started

Utilizing ScrapeNetwork is simple. All you need to do is send the desired URL for scraping to the API, along with your API key. The API will then return the HTML response from the specified URL, making the web scraping process a breeze.

API Key & Authentication

ScrapeNetwork employs API keys for request authentication. To access the API, simply sign up for an account and incorporate your unique API key into each request, ensuring a seamless and secure experience.

If you haven’t signed up for an account yet then

Making Requests

You can send GET requests to ScrapeNetwork via our API end point:
 
https://app.scrapenetwork.com/api?

No matter the method employed to utilize the service, we strongly suggest implementing a 60-second timeout within your application to attain the highest possible success rates, particularly when dealing with difficult-to-scrape domains.

Requests to the API Endpoint

ScrapeNetwork exposes a single API endpoint for you to send GET requests. Simply send a GET request to https://api.scrapenetwork.com/api? with two query string parameters and the API will return the HTML response for that URL:

  • api_key which contains your API key, and
  • request_url which contains the url you would like to scrape

Ensure that your requests to the API endpoint are structured in the following manner:

				
					curl "https://app.scrapenetwork.com/api?api_key=APIKEY&request_url=http://httpbin.org/ip"

				
			

To access additional API functionality when sending a request to the ScrapeNetwork endpoint, you can include the corresponding query parameters at the end of the URL.

For example, if you want to enable request as a mobile device, then add device=mobile to the request:

				
					curl "https://app.scrapenetwork.com/api?api_key=APIKEY&request_url=http://httpbin.org/ip&device=mobile"

				
			

You can use multiple parameters by separating them with the “&” symbol.

				
					curl "https://app.scrapenetwork.com/api?api_key=APIKEY&request_url=http://httpbin.org/ip&device=mobile&country_code=us"

				
			

API Status Codes

After each API request, ScrapeNetwork returns a specific status code, indicating the success, failure, or occurrence of any other error. If a request fails, ScrapeNetwork will make additional attempts for up to 60 seconds to obtain a successful response from the target URL. If all retries are unsuccessful, a 500 error response will be sent, signifying a failed request.

Note: To ensure your request doesn’t time out before the API has an opportunity to complete all retries, it’s important to set your timeout to a duration of 60 seconds.

When a request remains unsuccessful after 60 seconds of retry attempts, you won’t incur any charges for that failed request. Charges only apply to successful requests, specifically those returning 200 and 404 status codes.

Occasionally, errors may arise, so it’s important to handle them on your end. You can configure your code to retry the request immediately, and in most instances, it will succeed. If a request consistently fails, verify that it is configured correctly. Alternatively, if you repeatedly receive ban messages from an anti-bot system, reach out to our support team by creating a ticket, and we will attempt to circumvent the anti-bot for you.

In the event that you receive a successful 200 status code response from the API but encounter a CAPTCHA within the response, kindly reach out to our support team. They will add it to our CAPTCHA detection database, and in the future, the API will recognize it as a ban and automatically retry the request.

Here are the potential status codes you may encounter:

Status CodeDetails
200 Successful response
404Page requested does not exist.
410Page requested is no longer accessible.
500Despite retrying for 60 seconds, the API failed to obtain a successful response.
429You are submitting requests too quickly, surpassing your allowed concurrency limit.
403You have used up all your API credits.

Once Logged In

You’ll see you all the information you need: Dashboard, View Invoices, My Account & Logs. 

Dashboard
In this section, you’ll find your API key, have the option to regenerate it, access an overview of your usage statistics, and view your current plan.

View Invoices
On the “View Invoices” tab, you’ll find a list of your paid invoices.

My Account 
In this section, you can view your existing plan, switch plans, or cancel your subscription. Additionally, this is the ideal location for setting up or updating your credit card information.

Advanced API Functionality

Customize API Functionality

ScrapeNetwork enables you to customize the API’s functionality by adding additional parameters to your requests. The API will accept the following parameters:

ParameterDescription
device_type Set your requests to use mobile or desktop user agents by setting device_type=desktop or device_type=mobile.
This parameter does not increase the cost of the API request.

Device Type

If your use case requires you to exclusively use either desktop or mobile user agents in the headers it sends to the website then you can use the device_type parameter.

  • Set device_type=desktop to have the API set a desktop (e.g. iOS, Windows, or Linux) user agent. 
    Note:  This behavior is the default and will have the same effect if the parameter is not set.
     
  • Set device_type=mobile to have the API set a mobile (e.g. iPhone or Android) user agent.
				
					curl "https://app.scrapenetwork.com/api?api_key=APIKEY&request_url=http://httpbin.org/ip&device_type=mobile"