How to Send a cURL Get Request

A cURL GET request retrieves data from a specified resource, usually a URL. GET requests are one of the most widely used HTTP methods, as they are used whenever you access something on the internet.

While you usually do that through a browser, the cURL get command lets you bypass all the GUI drawing and collect only the HTML data.

Getting Started With cURL

cURL comes pre-installed in many modern operating systems such as Windows 10 and MacOS. Some Linux distributions may have cURL available. However, you’ll often need to install it manually:

sudo apt-get install curl

If your Windows operating system is older than 10, installing cURL manually will be required . Once that is complete, open up the Terminal (or Command Prompt in Windows) and check if cURL is working properly:

curl --help

Sending a cURL GET Request

The cURL GET method is the default action whenever you type in “curl” and any URL as such:

curl https://iproyal.com

Alternatively, you can specify the GET method (by using “-G” or “–get”), which will always follow the structure of:

curl --get [url]

Additionally, you can add cURL query parameters. These are added by including a “-d”. You can use as many of these options as you need; the only restriction is that they are added after specifying that it’s a cURL GET query.

curl --get -d "parameter=value" -d "parameter2=value2" -d "parameter=value3" https://iproyal.com

Note that all data that are being sent with the cURL get request is added in quotation marks.

You can also send a cURL GET with parameters to several URLs at once. Actions will be performed on each URL in sequence.

curl --get -d "parameter=value" -d "parameter2=value2" -d "parameter=value3" https://iproyal.com https://iproyal.com/proxies-by-location/north-america/united-states/

Finally, if a website has redirects, you have to send a cURL GET request with parameters to follow them. Otherwise, it’ll only attempt to reach the defined URL(s). Use the “-L” parameter to do follow redirects:

curl -G -L https://iproyal.com

Adding or Retrieving HTTP Headers With a cURL GET Request

An important part of a cURL GET command is HTTP headers. Some websites may block requests with default cURL GET request parameters. Setting HTTP headers, especially the user agent, may help evade the block:

curl --get -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" https://iproyal.com/

User agents are set with “-A” and a string that defines the user agent. Using popular browser user agents is a great way to reduce the likelihood of getting blocked.

You can also retrieve headers from a website, along with the rest of the cURL GET body by using the “-i” (short for “include”). Alternatively, you can use “–head” to ignore the cURL GET request body and retrieve the headers only:

curl --get -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" --head https://iproyal.com/

Retrieving Data in Specific Formats

You can create a cURL with parameters so that you receive data in a format that’s preferable. JSON formats are frequently used as without the cURL params, the data received is a jumbled mess:

curl -G -H "Accept: application/json" https://iproyal.com

You’ll be setting a parameter option with “-H” and entering “Accept: application/json”. Note that it will only work if the server can respond in such a way. Many websites may not respond with a JSON, however, it’s quite common for APIs.

Authentication With cURL Request Parameters

Finally, to access some websites, you’ll need to provide credentials. There are two main authentication options—usernames and passwords or access tokens. The latter are included in the header as in the previous example:

curl -G -H "API-Access-Token: YOUR_TOKEN" https://iproyal.com

For usernames and passwords, the cURL query parameter “–user” is used:

curl -G --user "USERNAME:PASSWORD" https://iproyal.com

You can also include your username in a cookie, which may provide different content (especially when you’re dealing with login or post-login pages):

curl -G --user "USERNAME:PASSWORD" --cookie "username=Tom" https://iproyal.com

Conclusion

cURL is very useful for retrieving data from websites with the GET request method. You can also use other programs to automate the process. Here’s a basic cURL GET example, which you can modify with the parameters we’ve outlined above:

curl --get -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" https://iproyal.com/

Sending a browser user agent should often be included, as you might get blocked otherwise.

Finally, we’ve only touched upon all of the available cURL GET params. You can find them all by typing in:

curl --help all

If you want to learn more about cURL, we recommend our cURL POST request blog post.