Rampage Blogs

GETting to Know cURL

We covered the importance and usefulness of cURL in a previous guide. If you’re unfamiliar with it, click here to learn more- this is an excellent place to start.

If you’re already familiar with cURL but want to expand your knowledge, here’s the next best place to start.

What is a GET Request?

A GET request is how you pull data from a specific webpage. For example, when you type a URL into your browser and hit enter, your browser sends a GET request to the server at that address, asking to retrieve the HTML code for that page.

Although other HTTP request methods can also prompt a server to send HTML back, each has a different purpose. POST is specifically meant to send data from the client to the server. When you log in, pressing the “login” button triggers an HTTP POST request, sending your username and password to the server to be authenticated. Unlike GET, which simply retrieves information, POST is about sending data for processing. This makes POST great for submitting forms, updating profiles, or uploading files, where you’re actively providing information to the server instead of just asking for it.

GET is simple. It doesn’t send any data; it just requests information. This could be to retrieve a web page, an image, a video, or various other resources. The most simple GET request we can do is to our website (or any others):

curl https://www.rampageproxies.com

Performing this cURL in our command line will return the entire HTML of the website. Pre-warning: your terminal will be flooded with HTML code.

Sending GET Requests

GET makes one of nine standard HTTP protocols. In our previous post, we discussed using proxies and how they automatically default to the HTTP protocol unless told otherwise. This is similar to using cURL without proxies, if no specific HTTP method is given, it defaults to the GET method. This means that simply providing a URL without any other options will perform just perform a GET request.

For this example, we’ll use a simple request/response demonstration service httpbin. There’s a variety of useful options that go alongside cURL, making it a versatile and powerful tool.

Headers

HTTP headers let the client and server share extra info during an HTTP request. Doing this lets us see the HTTP response headers, server, date, content type, and length.

To include the headers, pass the -i or --include option:

curl -i http://www.httpbin.org/headers

This provides an output with three main parts. The HTTP status line, the response headers, and the response body:

HTTP/1.1 200 OK
Date: Mon, 04 Nov 2024 08:52:50 GMT
Content-Type: application/json
Content-Length: 172
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.7.1",
    "X-Amzn-Trace-Id": "Root=1-67288b62-03b7f2cc5357f88c09d2faa0"
  }
}

In addition to this, you can make a GET request without including the body; this is done by using the --head option:

curl --head http://www.httpbin.org/headers

Finally, it's also posible to use a custom user agent in the request by using "User-Agent" as part of the argument like so:

curl -H "User-Agent: CustomUserAgent" http://www.httpbin.org/headers

Parameters

GET request parameters let you send extra data to the server directly within the URL. This is useful for filtering results, passing user input, or making dynamic queries.

To send parameters with a GET request, use the -G option along with -d:

curl -G -d "test1=value" -d "test2=value2" http://httpbin.org/get

In this example, both “test1” and “test2” would be the parameters, with the “value” being what you’d want to pass on with them. Parameters can also be included in the URL. This is sometimes quicker and easier to do:

curl http://www.httpbin.org/get?test1=value1&test2=value2

JSON’s

Providing the site supports the format, cURL can request data in a JSON format. Servers that support multiple content types may check the header and return data in the requested format if available. If the server cannot fulfill the request in the specified JSON format, it could respond in a different format or send an error message.
To request a JSON, use the following -H option followed by “"Accept: application/json":

curl -H "Accept: application/json" http://www.httpbin.org

If successful, you’ll be served with a JSON of the site in the terminal. You might then find it useful to use the -o or --output to save it to a file elsewhere for easy viewing:

curl -H "Accept: application/json" http://www.httpbin.org -o output.json

Following Re-directs

By default, cURL will not automatically follow re-directs. Websites may set up automatic re-directions from one page to another. Without specifying, cURL gets lost here, where you’ll need to say explicitly.

This is done using the -Lor location option as so:

curl -L 'www.http://httpbin.org/redirect-to?url=http://httpbin.org/get'

It's also helpful to know that cURL can limit the number of redirects it follows using the --max-redirs option:

curl -L --max-redirs 5 http://www.httpbin.org/redirect/3

This command follows up to 5 redirects.

Here’s a really useful table, including all the options we’ve discussed above:

Option	Long Version	Short Version	Description
Include	--include	-i	Shows the HTTP response headers along with the response body.
Head	--head	-I	Sends a HEAD request, only fetching the headers without the body.
Data	--data	-d	Sends data as part of a POST request by default. With -G, adds data to the URL as params.
Get	--get	-G	Ensures the request is a GET, appending data from -d to the URL.
Header	--header	-H	Adds custom headers to your request (e.g., Accept: application/json).
Output	--output	-o	Saves the response to a file instead of displaying it in the terminal.
Location	--location	-L	Follows redirects automatically so you don’t have to chase them down manually.

Conclusion

If you weren’t a cURL master before, you might be on your way to it now. We’ve covered the basics of GET and the useful options alongside it inside cURL. If this is the first post you’ve visited, we recommend further reading our cURL guides, such as using cURL with proxies.