Rampage Blogs

Introduction:

Client URL, commonly known as cURL, is a command-line tool for sending and receiving data over the Internet using URLs. Compatible with all the providers, this guide will cover what cURL is, what it’s commonly used for, and some of our most frequently asked questions.

Formerly referred to as httpget or urlget, cURL saw its initial release in 1996. cURL is a versatile program that is supported out-of-the-box on almost every modern operating system and supports over 20 popular protocols. cURL also allows integration with proxies, setting cookies, and adding authentication.

Popular API tools like Postman give a user interface that allows you to make different requests to URLs for receiving and processing requests if the command line isn't your forte. Alternatively, using the cURL command directly in your terminal gives the same functionality.

Here’s an example of the most basic use of cURL:

curl https://www.rampageproxies.com

Here, we’re making a HTTP "GET" request.

Options

cURL allows for options to be used as part of the syntax. Options give a range of customisations to your requests and come in two choices: short or long. Short options are single-dashed characters, whereas long options are double-dashed with their name. For example:

Name	Option	Description
Proxy	-x OR --proxy	Used to specify a proxy
Verbose	-v OR --verbose	Provides detailed information about the request and response
Output	-o OR --output	Saves the output to a file
User	-u OR --user	Provides a username and password for authentication
Header	-h OR --header	Used to add a custom header to the request
Insecure	-k OR --insecure	Ignores SSL certificate errors
Information	-i OR --include	Include the response headers in the output

A full list of options can be found by using the command below or looking at the curl manpage

curl -h all

cURL is case sensitive, so make sure you check for the correct case when using different options. Something useful to know is that the command-line parser in curl always parses the entire line, meaning you can put the options anywhere you like; they can also appear after the URL:

curl https://www.rampageproxies.com -v -i

Installing cURL

If you’re using a modern version of Windows, cURL will already be preinstalled and supported, the same way modern macOS does. Linux-based systems may require some installation.

If you're unsure whether or not you have i installeed, check by trying to get the manual:

curl –manual

To install on a Debian-based Linux system, run the following command:

sudo apt install curl

For other non-Debian-based systems, follow these instructions.

If you’re using an older version of Windows, you might need to install cURL. You can do this quickly by downloading and unzipping.

What will you need to start using proxies?

Once you’re set-up, you’ll need to choose which proxy provider to use in cURL. The good news is that any of our proxy providers will work with cURL. This is the same for residential, mobile, ISP, or DC proxies. All the providers Rampage sells can be easily accessed through our dashboard.

You’ll then need to get the following from the proxy you’ve generated:

• Protocol (cURL uses HTTP by default)
• Proxy username and password
• Port (This will vary per provider)

All proxies generated through the Rampage dashboard have the format IP:PORT:USERNAME:PASSWORD.

Finally, all you need to do is tell cURL to use the proxy you choose.

How to use cURL proxies

A proxy acts as a middleman between your computer and the internet. It intercepts your requests and serves them on your behalf. This can be particularly useful when you’re trying to avoid IP-based bans/blocks.

Let's start with a basic example of a cURL command with a proxy. We used a static Oxylabs proxy for our example:

curl -x “{PROXY USERNAME}:{PROXY PASSWORD}@{PROXY IP ADDRESS}:{PROXY PORT}” ipinfo.io

Note: remember you can use either “-x” or “--proxy”; both do the same thing.

In the example above, we’ve used the “-x” option to specify the proxy. We’ve re-formatted the proxy to USERNAME:PASSWORD@IP:PORT. Because we’ve also not specified the protocol, cURL will default to HTTP.

Alternatively, you can choose to specify the username and passwords separately. This is less popular in most cases, as it's usually easier to include the whole string in one. Taking into account the correct case, adding the option of "-U":

curl –proxy -U {PROXY USERNAME}:{PROXY PASSWORD} -x {PROXY IP ADDRESS}{PROXY PORT} ipinfo.io

If you were looking to use a different proxy protocol, such as SOCKS5, you’d need to define that as part of the syntax:

curl -x "socks5://{PROXY USERNAME}:{PROXY PASSWORD}@{PROXY ADDRESS}:{PROXY PORT}" ipinfo.io

Note: Getting the authentication wrong as part of the proxy (such as wrong credentials) will result in an error 407 response code. If you see this, we’d recommend checking if the username and password are correct.

Other proxy protocols, such as socks4://, socks4a://, socks5://, or socks5h://, can be used; however, this varies depending on the SOCKS version the proxy you’re using supports.

Testing a proxy with cURL

Referring to our previous examples, we can use cURL to test single proxies. Testing against a site such as ipinfo allows you to see various information about the proxy you’re testing:

curl -x "{PROXY USERNAME}:{PROXY PASSWORD}@{PROXY ADDRESS}:{PROXY PORT}" https://ipinfo.io

This will then provide you with a JSON response of useful information about the proxy:

This is particularly useful for checking information like proxy location, the IP address or just checking if the proxy is online.

Note: occasionally, the location may display incorrectly. If the location doesn’t match, the databases behind these external services may be out of date and show the wrong location.

What can cURL do?

cURL can be used in many ways; here’s a non-exhaustive list of popular applications:

Checking the status of a web server:

You can use cURL to check the status code of a website in the response headers.

Web scraping:

Scrape data from web pages directly from the command line, the perfect place to apply proxies from one of our 10 providers.

Uploading/download a file to a server:

cURL can be used to upload or download files from servers easily.

FAQ’s

We’ve provided some extra information on cURL, answering some of our frequently asked questions:

How do I get more information from my cURL command?

To get more information, use the option “-v” or “--verbose,” which will provide a much more detailed response on the communication between client and server, including information such as the SSL/TLS handshake details. This is particularly useful in debugging connectivity issues, content debugging, or inspecting headers.

How do I ignore SSL certificate errors?

Using the options “-k” or “--insecure” will allow cURL to ignore any SSL certificate. It’s worth noting that this is at your own risk, as it could compromise the security of your connection, potentially exposing you to MITM attacks.

Do I need to use proxies when using cURL?

Although proxies are not a requirement, they are essential in ensuring you’re able to continue block-free in use cases such as web scraping. The type of proxies you use will depend on the use case. At Rampage, we sell all types of proxies. We’d recommend checking out the residential proxies. Alternatively, we have ISP and DC proxies available. Note that proxies aren't the only thing that can be blocked while web scraping; various anti-bot solutions also exist that cURL/proxies won't be able to help you with.

Conclusion

cURL is versatile, easily accessible, and free. We’ve covered the basics and more in this guide. Beyond this, we’d highly recommend checking out the documentation or manpage. cURL itself is widely documented, providing easy access to all its information.