Documentation
Get started scraping the web in as little as 2 minutes
Scraping & Integration Tutorials
Guided examples on how to integrate and use ScrapeNetwork
Don’t have an API key? Sign up here and get 5,000 free API credits
Getting Started
Advanced API Functionality
Getting Started
Utilizing ScrapeNetwork is simple. All you need to do is send the desired URL for scraping to the API, along with your API key. The API will then return the HTML response from the specified URL, making the web scraping process a breeze.
API Key & Authentication
ScrapeNetwork employs API keys for request authentication. To access the API, simply sign up for an account and incorporate your unique API key into each request, ensuring a seamless and secure experience.
If you haven’t signed up for an account yet then
- Get started with our free trial offering 5,000 complimentary API credits – sign up here!
Making Requests
You can send GET requests to ScrapeNetwork via our API end point:
https://app.scrapenetwork.com/api?
No matter the method employed to utilize the service, we strongly suggest implementing a 60-second timeout within your application to attain the highest possible success rates, particularly when dealing with difficult-to-scrape domains.
Requests to the API Endpoint
ScrapeNetwork exposes a single API endpoint for you to send GET requests. Simply send a GET request to https://api.scrapenetwork.com/api?
with two query string parameters and the API will return the HTML response for that URL:
api_key
which contains your API key, andrequest_url
which contains the url you would like to scrape
Ensure that your requests to the API endpoint are structured in the following manner:
curl "https://app.scrapenetwork.com/api?api_key=APIKEY&request_url=http://httpbin.org/ip"
To access additional API functionality when sending a request to the ScrapeNetwork endpoint, you can include the corresponding query parameters at the end of the URL.
For example, if you want to enable request as a mobile device, then add device=mobile
to the request:
curl "https://app.scrapenetwork.com/api?api_key=APIKEY&request_url=http://httpbin.org/ip&device=mobile"
You can use multiple parameters by separating them with the “&” symbol.
curl "https://app.scrapenetwork.com/api?api_key=APIKEY&request_url=http://httpbin.org/ip&device=mobile&country_code=us"
API Status Codes
After each API request, ScrapeNetwork returns a specific status code, indicating the success, failure, or occurrence of any other error. If a request fails, ScrapeNetwork will make additional attempts for up to 60 seconds to obtain a successful response from the target URL. If all retries are unsuccessful, a 500
error response will be sent, signifying a failed request.
Note: To ensure your request doesn’t time out before the API has an opportunity to complete all retries, it’s important to set your timeout to a duration of 60 seconds.
When a request remains unsuccessful after 60 seconds of retry attempts, you won’t incur any charges for that failed request. Charges only apply to successful requests, specifically those returning 200
and 404
status codes.
Occasionally, errors may arise, so it’s important to handle them on your end. You can configure your code to retry the request immediately, and in most instances, it will succeed. If a request consistently fails, verify that it is configured correctly. Alternatively, if you repeatedly receive ban messages from an anti-bot system, reach out to our support team by creating a ticket, and we will attempt to circumvent the anti-bot for you.
In the event that you receive a successful 200 status code response from the API but encounter a CAPTCHA within the response, kindly reach out to our support team. They will add it to our CAPTCHA detection database, and in the future, the API will recognize it as a ban and automatically retry the request.
Here are the potential status codes you may encounter:
Status Code | Details |
---|---|
200 | Successful response |
404 | Page requested does not exist. |
410 | Page requested is no longer accessible. |
500 | Despite retrying for 60 seconds, the API failed to obtain a successful response. |
429 | You are submitting requests too quickly, surpassing your allowed concurrency limit. |
403 | You have used up all your API credits. |
Once Logged In
You’ll see you all the information you need: Dashboard, View Invoices, My Account & Logs.
Dashboard
In this section, you’ll find your API key, have the option to regenerate it, access an overview of your usage statistics, and view your current plan.
View Invoices
On the “View Invoices” tab, you’ll find a list of your paid invoices.
My Account
In this section, you can view your existing plan, switch plans, or cancel your subscription. Additionally, this is the ideal location for setting up or updating your credit card information.
Advanced API Functionality
Customize API Functionality
ScrapeNetwork enables you to customize the API’s functionality by adding additional parameters to your requests. The API will accept the following parameters:
Parameter | Description |
---|---|
device_type | Set your requests to use mobile or desktop user agents by setting device_type=desktop or device_type=mobile. This parameter does not increase the cost of the API request. |
Device Type
If your use case requires you to exclusively use either desktop or mobile user agents in the headers it sends to the website then you can use the device_type
parameter.
- Set
device_type=desktop
to have the API set a desktop (e.g. iOS, Windows, or Linux) user agent.
Note: This behavior is the default and will have the same effect if the parameter is not set. - Set
device_type=mobile
to have the API set a mobile (e.g. iPhone or Android) user agent.
curl "https://app.scrapenetwork.com/api?api_key=APIKEY&request_url=http://httpbin.org/ip&device_type=mobile"
Getting Started
Utilizing ScrapeNetwork is simple. All you need to do is send the desired URL for scraping to the API, along with your API key. The API will then return the HTML response from the specified URL, making the web scraping process a breeze.
API Key & Authentication
ScrapeNetwork employs API keys for request authentication. To access the API, simply sign up for an account and incorporate your unique API key into each request, ensuring a seamless and secure experience.
If you haven’t signed up for an account yet then
- Get started with our free trial offering 5,000 complimentary API credits – sign up here!
Making Requests
You can send GET requests to ScrapeNetwork via our API end point:
https://app.scrapenetwork.com/api?
No matter the method employed to utilize the service, we strongly suggest implementing a 60-second timeout within your application to attain the highest possible success rates, particularly when dealing with difficult-to-scrape domains.
Requests to the API Endpoint
ScrapeNetwork exposes a single API endpoint for you to send GET requests. Simply send a GET request to http://api.scrapenetwork.com/api?
with two query string parameters and the API will return the HTML response for that URL:
api_key
which contains your API key, andurl
which contains the url you would like to scrape
You should format your requests to the API endpoint as follows:
"http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip"
Simple Code
import requests
payload = {'api_key': 'APIKEY', 'url': 'https://httpbin.org/ip'}
r = requests.get('http://api.scrapenetwork.com/api', params=payload)
print r.text
# Scrapy users can simply replace the urls in their start_urls and parse function
# ...other scrapy setup code
start_urls = ['http://api.scrapenetwork.com/api?api_key=APIKEY&url=' + url]
def parse(self, response):
# ...your parsing logic here
yield scrapy.Request('http://api.scrapenetwork.com/api?api_key=APIKEY&url=' + url, self.parse)
To enable other API functionality when sending a request to the API endpoint simply add the appropriate query parameters to the end of the ScrapeNetwork URL.
For example, if you want to enable Javascript rendering with a request, then add render=true
to the request:
"http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip&render=true"
Simple Code
import requests
payload = {'api_key': 'APIKEY', 'url':'https://httpbin.org/ip', 'render': 'true'}
r = requests.get('http://api.scrapenetwork.com/api', params=payload)
print r.text
# Scrapy users can simply replace the urls in their start_urls and parse function
# ...other scrapy setup code
start_urls = ['http://api.scrapenetwork.com?api_key=APIKEY&url=' + url +'&render=true']
def parse(self, response):
# ...your parsing logic here
yield scrapy.Request('http://api.scrapenetwork.com/?api_key=APIKEY&url=' + url +'&render=true', self.parse)
To use two or more parameters, simply separate them with the “&” sign.
"http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip&render=true&country_code=us"
Simple Code
import requests
payload = {'api_key': 'APIKEY', 'url':'https://httpbin.org/ip', 'render': 'true', 'country_code': 'us'}
r = requests.get('http://api.scrapenetwork.com/api', params=payload)
print r.text
# Scrapy users can simply replace the urls in their start_urls and parse function
# ...other scrapy setup code
start_urls = ['http://api.scrapenetwork.com/api?api_key=APIKEY&url=' + url +'&render=true' + ‘&country_code=true’]
def parse(self, response):
# ...your parsing logic here
yield scrapy.Request('http://api.scrapenetwork.com/api?api_key=APIKEY&url=' + url +'&render=true' + ‘&country_code=true’, self.parse)
API Status Codes
After each API request, ScrapeNetwork returns a specific status code, indicating the success, failure, or occurrence of any other error. If a request fails, ScrapeNetwork will make additional attempts for up to 60 seconds to obtain a successful response from the target URL. If all retries are unsuccessful, a 500
error response will be sent, signifying a failed request.
Note: To ensure your request doesn’t time out before the API has an opportunity to complete all retries, it’s important to set your timeout to a duration of 60 seconds.
When a request remains unsuccessful after 60 seconds of retry attempts, you won’t incur any charges for that failed request. Charges only apply to successful requests, specifically those returning 200
and 404
status codes.
Occasionally, errors may arise, so it’s important to handle them on your end. You can configure your code to retry the request immediately, and in most instances, it will succeed. If a request consistently fails, verify that it is configured correctly. Alternatively, if you repeatedly receive ban messages from an anti-bot system, reach out to our support team by creating a ticket, and we will attempt to circumvent the anti-bot for you.
In the event that you receive a successful 200 status code response from the API but encounter a CAPTCHA within the response, kindly reach out to our support team. They will add it to our CAPTCHA detection database, and in the future, the API will recognize it as a ban and automatically retry the request.
Here are the potential status codes you may encounter:
Status Code | Details |
---|---|
200 | Successful response |
404 | Page requested does not exist. |
410 | Page requested is no longer available. |
500 | After retrying for 60 seconds, the API was unable to receive a successful response. |
429 | You are sending requests too fast, and exceeding your concurrency limit. |
403 | You have used up all your API credits. |
Once Logged In
You’ll see you all the information you need: Dashboard, View Invoices, My Account & Logs.
Dashboard
In this section, you’ll find your API key, have the option to regenerate it, access an overview of your usage statistics, and view your current plan.
View Invoices
On the “View Invoices” tab, you’ll find a list of your paid invoices.
My Account
In this section, you can view your existing plan, switch plans, or cancel your subscription. Additionally, this is the ideal location for setting up or updating your credit card information.
Advanced API Functionality
Customize API Functionality
ScrapeNetwork enables you to customize the API’s functionality by adding additional parameters to your requests. The API will accept the following parameters:
Parameter | Description |
---|---|
device_type | Set your requests to use mobile or desktop user agents by setting device_type=desktop or device_type=mobile. This parameter does not increase the cost of the API request. |
Device Type
If your use case requires you to exclusively use either desktop or mobile user agents in the headers it sends to the website then you can use the device_type
parameter.
- Set
device_type=desktop
to have the API set a desktop (e.g. iOS, Windows, or Linux) user agent. Note: This is the default behavior. Not setting the parameter will have the same effect. - Set
device_type=mobile
to have the API set a mobile (e.g. iPhone or Android) user agent.
Note: The device type you set will be overridden if you use keep_headers=true
and send your own user agent in the requests header.
import requests
payload = {'api_key': 'APIKEY', 'url':'https://httpbin.org/ip', 'device_type': 'mobile'}
r = requests.get('http://api.scrapenetwork.com/api', params=payload)
print r.text
# Scrapy users can simply replace the urls in their start_urls and parse function
# ...other scrapy setup code
start_urls = ['http://api.scrapenetwork.com/api?api_key=APIKEY&url=' + url + 'device_type=mobile']
def parse(self, response):
# ...your parsing logic here
yield scrapy.Request('http://api.scrapenetwork.com/api?api_key=APIKEY&url=' + url + 'device_type=mobile', self.parse)
Getting Started
Utilizing ScrapeNetwork is simple. All you need to do is send the desired URL for scraping to the API, along with your API key. The API will then return the HTML response from the specified URL, making the web scraping process a breeze.
API Key & Authentication
ScrapeNetwork employs API keys for request authentication. To access the API, simply sign up for an account and incorporate your unique API key into each request, ensuring a seamless and secure experience.
If you haven’t signed up for an account yet then
- Get started with our free trial offering 5,000 complimentary API credits – sign up here!
Making Requests
You can send GET requests to ScrapeNetwork via our API end point:
https://app.scrapenetwork.com/api?
No matter the method employed to utilize the service, we strongly suggest implementing a 60-second timeout within your application to attain the highest possible success rates, particularly when dealing with difficult-to-scrape domains.
You should format your requests to the API endpoint as follows:
Requests to the API Endpoint
ScrapeNetwork exposes a single API endpoint for you to send GET requests. Simply send a GET request to https://api.scrapenetwork.com/api?
with two query string parameters and the API will return the HTML response for that URL:
api_key
which contains your API key, andrequest_url
which contains the url you would like to scrape
Ensure that your requests to the API endpoint are structured in the following manner:
"http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip"
Simple Code
const request = require('request-promise');
request(`http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip`)
.then(response => {
console.log(response)
})
.catch(error => {
console.log(error)
})
To use two or more parameters, simply separate them with the “&” sign.
"http://api.scrapenetwork.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true&country_code=us"
Simple Code
const request = require('request-promise');
request(`http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip&render=true&country_code=us`)
.then(response => {
console.log(response)
})
.catch(error => {
console.log(error)
})
API Status Codes
After each API request, ScrapeNetwork returns a specific status code, indicating the success, failure, or occurrence of any other error. If a request fails, ScrapeNetwork will make additional attempts for up to 60 seconds to obtain a successful response from the target URL. If all retries are unsuccessful, a 500
error response will be sent, signifying a failed request.
Note: To ensure your request doesn’t time out before the API has an opportunity to complete all retries, it’s important to set your timeout to a duration of 60 seconds.
When a request remains unsuccessful after 60 seconds of retry attempts, you won’t incur any charges for that failed request. Charges only apply to successful requests, specifically those returning 200
and 404
status codes.
Occasionally, errors may arise, so it’s important to handle them on your end. You can configure your code to retry the request immediately, and in most instances, it will succeed. If a request consistently fails, verify that it is configured correctly. Alternatively, if you repeatedly receive ban messages from an anti-bot system, reach out to our support team by creating a ticket, and we will attempt to circumvent the anti-bot for you.
In the event that you receive a successful 200 status code response from the API but encounter a CAPTCHA within the response, kindly reach out to our support team. They will add it to our CAPTCHA detection database, and in the future, the API will recognize it as a ban and automatically retry the request.
Here are the potential status codes you may encounter:
Status Code | Details |
---|---|
200 | Successful response |
404 | Page requested does not exist. |
410 | Page requested is no longer accessible. |
500 | Despite retrying for 60 seconds, the API failed to obtain a successful response. |
429 | You are submitting requests too quickly, surpassing your allowed concurrency limit. |
403 | You have used up all your API credits. |
Once Logged In
You’ll see you all the information you need: Dashboard, View Invoices, My Account & Logs.
Dashboard
In this section, you’ll find your API key, have the option to regenerate it, access an overview of your usage statistics, and view your current plan.
View Invoices
On the “View Invoices” tab, you’ll find a list of your paid invoices.
My Account
In this section, you can view your existing plan, switch plans, or cancel your subscription. Additionally, this is the ideal location for setting up or updating your credit card information.
Advanced API Functionality
Customize API Functionality
ScrapeNetwork enables you to customize the API’s functionality by adding additional parameters to your requests. The API will accept the following parameters:
Parameter | Description |
---|---|
device_type | Set your requests to use mobile or desktop user agents by setting device_type=desktop or device_type=mobile. This parameter does not increase the cost of the API request. |
Device Type
If your use case requires you to exclusively use either desktop or mobile user agents in the headers it sends to the website then you can use the device_type
parameter.
- Set
device_type=desktop
to have the API set a desktop (e.g. iOS, Windows, or Linux) user agent.
Note: This behavior is the default and will have the same effect if the parameter is not set. - Set
device_type=mobile
to have the API set a mobile (e.g. iPhone or Android) user agent.
const request = require('request-promise');
request('http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip&device_type=mobile')
.then(response => {
console.log(response)
})
.catch(error => {
console.log(error)
})
Getting Started
Using ScrapeNetwork is easy. Just send the URL you would like to scrape to the API along with your API key and the API will return the HTML response from the URL you want to scrape.Utilizing ScrapeNetwork is simple. All you need to do is send the desired URL for scraping to the API, along with your API key. The API will then return the HTML response from the specified URL, making the web scraping process a breeze.
API Key & Authentication
ScrapeNetwork employs API keys for request authentication. To access the API, simply sign up for an account and incorporate your unique API key into each request, ensuring a seamless and secure experience.
If you haven’t signed up for an account yet then
- Get started with our free trial offering 5,000 complimentary API credits – sign up here!
Making Requests
You can send GET requests to ScrapeNetwork via our API end point:
https://app.scrapenetwork.com/api?
No matter the method employed to utilize the service, we strongly suggest implementing a 60-second timeout within your application to attain the highest possible success rates, particularly when dealing with difficult-to-scrape domains.
Requests to the API Endpoint
ScrapeNetwork exposes a single API endpoint for you to send GET requests. Simply send a GET request to https://api.scrapenetwork.com/api?
with two query string parameters and the API will return the HTML response for that URL:
api_key
which contains your API key, andrequest_url
which contains the url you would like to scrape
Ensure that your requests to the API endpoint are structured in the following manner:
"http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip"
Simple Code
To use two or more parameters, simply separate them with the “&” sign.
"http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip&render=true&country_code=us"
Simple Code
API Status Codes
After each API request, ScrapeNetwork returns a specific status code, indicating the success, failure, or occurrence of any other error. If a request fails, ScrapeNetwork will make additional attempts for up to 60 seconds to obtain a successful response from the target URL. If all retries are unsuccessful, a 500
error response will be sent, signifying a failed request.
Note: To ensure your request doesn’t time out before the API has an opportunity to complete all retries, it’s important to set your timeout to a duration of 60 seconds.
When a request remains unsuccessful after 60 seconds of retry attempts, you won’t incur any charges for that failed request. Charges only apply to successful requests, specifically those returning 200
and 404
status codes.
Occasionally, errors may arise, so it’s important to handle them on your end. You can configure your code to retry the request immediately, and in most instances, it will succeed. If a request consistently fails, verify that it is configured correctly. Alternatively, if you repeatedly receive ban messages from an anti-bot system, reach out to our support team by creating a ticket, and we will attempt to circumvent the anti-bot for you.
In the event that you receive a successful 200 status code response from the API but encounter a CAPTCHA within the response, kindly reach out to our support team. They will add it to our CAPTCHA detection database, and in the future, the API will recognize it as a ban and automatically retry the request.
Here are the potential status codes you may encounter:
Status Code | Details |
---|---|
200 | Successful response |
404 | Page requested does not exist. |
410 | Page requested is no longer accessible. |
500 | Despite retrying for 60 seconds, the API failed to obtain a successful response. |
429 | You are submitting requests too quickly, surpassing your allowed concurrency limit. |
403 | You have used up all your API credits. |
Once Logged In
You’ll see you all the information you need: Dashboard, View Invoices, My Account & Logs.
Dashboard
In this section, you’ll find your API key, have the option to regenerate it, access an overview of your usage statistics, and view your current plan.
View Invoices
On the “View Invoices” tab, you’ll find a list of your paid invoices.
My Account
In this section, you can view your existing plan, switch plans, or cancel your subscription. Additionally, this is the ideal location for setting up or updating your credit card information.
Advanced API Functionality
Customize API Functionality
ScrapeNetwork enables you to customize the API’s functionality by adding additional parameters to your requests. The API will accept the following parameters:
Parameter | Description |
---|---|
device_type | Set your requests to use mobile or desktop user agents by setting device_type=desktop or device_type=mobile. This parameter does not increase the cost of the API request. |
Device Type
If your use case requires you to exclusively use either desktop or mobile user agents in the headers it sends to the website then you can use the device_type
parameter.
- Set
device_type=desktop
to have the API set a desktop (e.g. iOS, Windows, or Linux) user agent.
Note: This behavior is the default and will have the same effect if the parameter is not set. - Set
device_type=mobile
to have the API set a mobile (e.g. iPhone or Android) user agent.
<?php
$url = "http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip&device_type=mobile";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
print_r($response);
Getting Started
Utilizing ScrapeNetwork is simple. All you need to do is send the desired URL for scraping to the API, along with your API key. The API will then return the HTML response from the specified URL, making the web scraping process a breeze.
API Key & Authentication
ScrapeNetwork employs API keys for request authentication. To access the API, simply sign up for an account and incorporate your unique API key into each request, ensuring a seamless and secure experience.
If you haven’t signed up for an account yet then
- Get started with our free trial offering 5,000 complimentary API credits – sign up here!
Making Requests
You can send GET requests to ScrapeNetwork via our API end point:
https://app.scrapenetwork.com/api?
No matter the method employed to utilize the service, we strongly suggest implementing a 60-second timeout within your application to attain the highest possible success rates, particularly when dealing with difficult-to-scrape domains.
Requests to the API Endpoint
ScrapeNetwork exposes a single API endpoint for you to send GET requests. Simply send a GET request to http://api.scrapenetwork.com/api?
with two query string parameters and the API will return the HTML response for that URL:
api_key
which contains your API key, andurl
which contains the url you would like to scrape
You should format your requests to the API endpoint as follows:
"http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip"
Simple Code
require 'net/http'
require 'json'
params = {
:api_key => "APIKEY",
:url => "http://httpbin.org/ip"
}
uri = URI('http://api.scrapenetwork.com/api')
uri.query = URI.encode_www_form(params)
website_content = Net::HTTP.get(uri)
print(website_content)
To enable other API functionality when sending a request to the API endpoint simply add the appropriate query parameters to the end of the ScrapeNetwork URL.
For example, if you want to enable Javascript rendering with a request, then add render=true
to the request:
"http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip&render=true"
Simple Code
require 'net/http'
require 'json'
params = {
:api_key => "APIKEY",
:url => "http://httpbin.org/ip",
:render => true
}
uri = URI('http://api.scrapenetwork.com/api')
uri.query = URI.encode_www_form(params)
website_content = Net::HTTP.get(uri)
print(website_content)
To use two or more parameters, simply separate them with the “&” sign.
"http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip&render=true&country_code=us"
Simple Code
require 'net/http'
require 'json'
params = {
:api_key => "APIKEY",
:url => "http://httpbin.org/ip",
:render => true,
:country_code => “us”
}
uri = URI('http://api.scrapenetwork.com/api')
uri.query = URI.encode_www_form(params)
website_content = Net::HTTP.get(uri)
print(website_content)
API Status Codes
After each API request, ScrapeNetwork returns a specific status code, indicating the success, failure, or occurrence of any other error. If a request fails, ScrapeNetwork will make additional attempts for up to 60 seconds to obtain a successful response from the target URL. If all retries are unsuccessful, a 500
error response will be sent, signifying a failed request.
Note: To ensure your request doesn’t time out before the API has an opportunity to complete all retries, it’s important to set your timeout to a duration of 60 seconds.
When a request remains unsuccessful after 60 seconds of retry attempts, you won’t incur any charges for that failed request. Charges only apply to successful requests, specifically those returning 200
and 404
status codes.
Occasionally, errors may arise, so it’s important to handle them on your end. You can configure your code to retry the request immediately, and in most instances, it will succeed. If a request consistently fails, verify that it is configured correctly. Alternatively, if you repeatedly receive ban messages from an anti-bot system, reach out to our support team by creating a ticket, and we will attempt to circumvent the anti-bot for you.
In the event that you receive a successful 200 status code response from the API but encounter a CAPTCHA within the response, kindly reach out to our support team. They will add it to our CAPTCHA detection database, and in the future, the API will recognize it as a ban and automatically retry the request.
Here are the potential status codes you may encounter:
Status Code | Details |
---|---|
200 | Successful response |
404 | Page requested does not exist. |
410 | Page requested is no longer available. |
500 | After retrying for 60 seconds, the API was unable to receive a successful response. |
429 | You are sending requests too fast, and exceeding your concurrency limit. |
403 | You have used up all your API credits. |
Once Logged In
You’ll see you all the information you need: Dashboard, View Invoices, My Account & Logs.
Dashboard
In this section, you’ll find your API key, have the option to regenerate it, access an overview of your usage statistics, and view your current plan.
View Invoices
On the “View Invoices” tab, you’ll find a list of your paid invoices.
My Account
In this section, you can view your existing plan, switch plans, or cancel your subscription. Additionally, this is the ideal location for setting up or updating your credit card information.
Customize API Functionality
ScrapeNetwork enables you to customize the API’s functionality by adding additional parameters to your requests. The API will accept the following parameters:
Parameter | Description |
---|---|
device_type | Set your requests to use mobile or desktop user agents by setting device_type=desktop or device_type=mobile. This parameter does not increase the cost of the API request. |
Device Type
If your use case requires you to exclusively use either desktop or mobile user agents in the headers it sends to the website then you can use the device_type
parameter.
- Set
device_type=desktop
to have the API set a desktop (e.g. iOS, Windows, or Linux) user agent. Note: This is the default behavior. Not setting the parameter will have the same effect. - Set
device_type=mobile
to have the API set a mobile (e.g. iPhone or Android) user agent.
Note: The device type you set will be overridden if you use keep_headers=true
and send your own user agent in the requests header.
require 'net/http'
require 'json'
params = {
:api_key => "APIKEY",
:url => "http://httpbin.org/ip",
:device_type => “mobile”
}
uri = URI('http://api.scrapenetwork.com/api')
uri.query = URI.encode_www_form(params)
website_content = Net::HTTP.get(uri)
print(website_content)
Getting Started
Utilizing ScrapeNetwork is simple. All you need to do is send the desired URL for scraping to the API, along with your API key. The API will then return the HTML response from the specified URL, making the web scraping process a breeze.
API Key & Authentication
ScrapeNetwork employs API keys for request authentication. To access the API, simply sign up for an account and incorporate your unique API key into each request, ensuring a seamless and secure experience.
If you haven’t signed up for an account yet then
- Get started with our free trial offering 5,000 complimentary API credits – sign up here!
Making Requests
You can send GET requests to ScrapeNetwork via our API end point:
https://app.scrapenetwork.com/api?
No matter the method employed to utilize the service, we strongly suggest implementing a 60-second timeout within your application to attain the highest possible success rates, particularly when dealing with difficult-to-scrape domains.
Requests to the API Endpoint
ScrapeNetwork exposes a single API endpoint for you to send GET requests. Simply send a GET request to http://api.scrapenetwork.com/api?
with two query string parameters and the API will return the HTML response for that URL:
api_key
which contains your API key, andurl
which contains the url you would like to scrape
You should format your requests to the API endpoint as follows:
"http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip"
Simple Code
try {
String apiKey = "APIKEY";
String url = "http://api.scrapenetwork.com/api?api_key=" + apiKey + "&url=http://httpbin.org/ip";
URL urlForGetRequest = new URL(url);
String readLine = null;
HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
conection.setRequestMethod("GET");
int responseCode = conection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
StringBuffer response = new StringBuffer();
while ((readLine = in.readLine()) != null) {
response.append(readLine);
}
in.close();
System.out.println(response.toString());
} else {
throw new Exception("Error in API Call");
}
} catch (Exception ex) {
ex.printStackTrace();
}
To enable other API functionality when sending a request to the API endpoint simply add the appropriate query parameters to the end of the ScrapeNetwork URL.
For example, if you want to enable Javascript rendering with a request, then add render=true
to the request:
"http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip&render=true"
Simple Code
try {
String apiKey = "APIKEY";
String url = "http://api.scrapenetwork.com/api?api_key=" + apiKey + "&url=http://httpbin.org/ip&render=true";
URL urlForGetRequest = new URL(url);
String readLine = null;
HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
conection.setRequestMethod("GET");
int responseCode = conection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
StringBuffer response = new StringBuffer();
while ((readLine = in.readLine()) != null) {
response.append(readLine);
}
in.close();
System.out.println(response.toString());
} else {
throw new Exception("Error in API Call");
}
} catch (Exception ex) {
ex.printStackTrace();
}
To use two or more parameters, simply separate them with the “&” sign.
"http://api.scrapenetwork.com/api?api_key=APIKEY&url=http://httpbin.org/ip&render=true&country_code=us"
Simple Code
try {
String apiKey = "APIKEY";
String url = "http://api.scrapenetwork.com?api_key=" + apiKey + "&url=http://httpbin.org/ip&render=true&country_code=us";
URL urlForGetRequest = new URL(url);
String readLine = null;
HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
conection.setRequestMethod("GET");
int responseCode = conection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
StringBuffer response = new StringBuffer();
while ((readLine = in.readLine()) != null) {
response.append(readLine);
}
in.close();
System.out.println(response.toString());
} else {
throw new Exception("Error in API Call");
}
} catch (Exception ex) {
ex.printStackTrace();
}
API Status Codes
After each API request, ScrapeNetwork returns a specific status code, indicating the success, failure, or occurrence of any other error. If a request fails, ScrapeNetwork will make additional attempts for up to 60 seconds to obtain a successful response from the target URL. If all retries are unsuccessful, a 500
error response will be sent, signifying a failed request.
Note: To ensure your request doesn’t time out before the API has an opportunity to complete all retries, it’s important to set your timeout to a duration of 60 seconds.
When a request remains unsuccessful after 60 seconds of retry attempts, you won’t incur any charges for that failed request. Charges only apply to successful requests, specifically those returning 200
and 404
status codes.
Occasionally, errors may arise, so it’s important to handle them on your end. You can configure your code to retry the request immediately, and in most instances, it will succeed. If a request consistently fails, verify that it is configured correctly. Alternatively, if you repeatedly receive ban messages from an anti-bot system, reach out to our support team by creating a ticket, and we will attempt to circumvent the anti-bot for you.
In the event that you receive a successful 200 status code response from the API but encounter a CAPTCHA within the response, kindly reach out to our support team. They will add it to our CAPTCHA detection database, and in the future, the API will recognize it as a ban and automatically retry the request.
Here are the potential status codes you may encounter:
Status Code | Details |
---|---|
200 | Successful response |
404 | Page requested does not exist. |
410 | Page requested is no longer available. |
500 | After retrying for 60 seconds, the API was unable to receive a successful response. |
429 | You are sending requests too fast, and exceeding your concurrency limit. |
403 | You have used up all your API credits. |
Once Logged In
You’ll see you all the information you need: Dashboard, View Invoices, My Account & Logs.
Dashboard
In this section, you’ll find your API key, have the option to regenerate it, access an overview of your usage statistics, and view your current plan.
View Invoices
On the “View Invoices” tab, you’ll find a list of your paid invoices.
My Account
In this section, you can view your existing plan, switch plans, or cancel your subscription. Additionally, this is the ideal location for setting up or updating your credit card information.
Advanced API Functionality
Customize API Functionality
ScrapeNetwork enables you to customize the API’s functionality by adding additional parameters to your requests. The API will accept the following parameters:
Parameter | Description |
---|---|
device_type | Set your requests to use mobile or desktop user agents by setting device_type=desktop or device_type=mobile. This parameter does not increase the cost of the API request. |
Device Type
If your use case requires you to exclusively use either desktop or mobile user agents in the headers it sends to the website then you can use the device_type
parameter.
- Set
device_type=desktop
to have the API set a desktop (e.g. iOS, Windows, or Linux) user agent. Note: This is the default behavior. Not setting the parameter will have the same effect. - Set
device_type=mobile
to have the API set a mobile (e.g. iPhone or Android) user agent.
Note: The device type you set will be overridden if you use keep_headers=true
and send your own user agent in the requests header.