Linux cURL

From Ever changing code
Jump to navigation Jump to search

Using curl

The example below use curl to send requests to which is simple HTTP Request & Response Service, the guide follow introduction-by-example principle.

Basics of HTTP

  • haproxy configuration Fantastic first section on how http works, header format and assembly when passing proxy servers

Common curls

Single website pinning

Debugging problems with a site that's already live, it's helpful to bypass CDN and proxy layers when requesting content from the site by sending those web requests directly to a specific IP address without using the site's public DNS records. This practice of "pinning" a web request directly to a server can be accomplished by changing your /etc/hosts file, which will cause requests for a specified domain name (e.g: " ") to be routed from your local machine to a specified IP address (e.g: until the changes you've made to /etc/hosts are reverted.

# get ip of your Loadbalancer
dig +short # choose one

# dns name pinning
curl -v --resolve ""
curl -v --resolve ""

# multiple pinnings :80 & :443 to use localhost version of the website
curl -v --resolve "" \
        --resolve ""

#       --resolve [DOMAIN]:[PORT]:[IP] # the option format

Resolve DNS without using local cache

sudo systemd-resolve --flush-caches # if you wish to flush dns; curl -I --resolve "$DOMAIN:80:$(dig +short $DOMAIN @ | head -1)" http://$DOMAIN
systemd-resolve --statistics | grep 'Current Cache Size' # should still be -> Current Cache Size: 0

# --dns-servers - this solution still uses local cache
curl --dns-servers _MY_DNS_SERVER_IP_ https://_MY_DOMAINS_
curl --dns-servers

Note: libcurl and c-ares backend

curl: (4) A requested feature, protocol or option was not found built-in in this libcurl due to a build-time decision.

From curl man page:
This option requires that libcurl was built with a resolver backend that supports this operation.
The c-ares backend is the only such one.  (Added in 7.33.0)

Use curl in shell scripts

When using Curl in shell scripts, always pass -fSSL, which:

  • -f treats non-2xx/3xx responses as errors
  • -sS disables the progress meter
  • -L handles HTTP redirects

Occasionally, if you consider some non-2xx/3xx responses to be non-fatal, then omit -f then use logic to treat these responses as success.

curl -fsSL

$ curl -sS -w '%{http_code}\n' -o /dev/null 
  # -w '%{http_code}' write the HTTP response code to standard output
  # -o <filename> write the HTTP response body to the given file, not to standard output


Probe to monitor an endpoint

curl --connect-timeout 5 --retry 12 --retry-delay 1 --retry-max-time 60

GET request

Fire up a terminal and execute the following command:


This command will fetch the HTML for the HTTPBin homepage. Notice that we've not mentioned the protocol or the HTTP method. They've defaulted to HTTP and GET, respectively.

HEAD request

Sometimes you may want to see just the headers that the server would return when it's issued a GET request. In such a case, we'd use the HEAD method. The head request is issued with -I (capital i) or --head.

curl -I    
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 01 Dec 2016 18:50:56 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 12150
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Connection: keep-alive

Verbose output

Sometimes, you may want to debug a problem because the curl command hasn't returned what it was supposed to or just hasn't worked at all. In such cases, you can use the verbose mode using -v or --verbose to get more output from curl. Let's modify the previous HTTP request to become verbose.

curl -Iv

Now there's a lot more output in the terminal. Let's see what it means.

Lines prefixed with an asterisk (*) show additional work curl has done. For example:

* Rebuilt URL to:
*   Trying
* Connected to ( port 80 (#0)

The first line shows that curl added a slash at the end of the URL.

The second line shows that curl has resolved the URL to an IP address.

Finally, the third line shows that curl has connected to the URL on port 80.

Lines prefixed with a greater-than (>) sign show the data curl has sent to the server. For example:

> HEAD / HTTP/1.1
> Host:
> User-Agent: curl/7.47.0
> Accept: */*

Lines prefixed with a less-than (<) sign show the data curl has received from the server. For example:

< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: nginx
Server: nginx

Multiple requests

You can also send requests to multiple URLs by separating them by space. Here's how:


The above example will send two GET requests, both to the same URL, one after the other. You should see two little teapots printed in your terminal.

POST request

You can send a POST request by using the -d (or --data) flag. It͛s up to you to properly URL encode the data or specify it as JSON. We'll see the examples for both of those. Here's how you'd send URL encoded data as a part of the POST body

curl -d 'name=john+doe&course=linux'

The response sent by HTTPBin will simply echo what you͛ve sent.


curl -d '{"name":"John Doe","course":"Linux"}'

Notice that in both the examples, the data is enclosed in single quotes. Using double quotes is okay but it may cause problems if the body also contains double quotes like JSON. Using single quotes is the safer approach.

Setting headers, authentication, authorization

You can set headers by using the -H flag. In our previous example, we've sent JSON without setting the Content-Type header to application/json. Here's the previous example modified to set the header:

curl -H 'Content-Type: application/json' -d '{"name":"John Doe","course":"Linux"}'

You set the header as a key-value pair consisting of the header name followed by a colon and the value.

Posting form data

You can send form data by using the -F flag. Use this flag for every form field that you want to send. Here's how you'd send form data:

curl -F "name=John Doe" -F "course=linux"

Notice that we've written the name as John Doe without a plus sign in between. The response received will have the Content-Type header set to multipart/form-data.

Note SSL connection is not using -H hostname from headers but from DNS name; use below

curl -H 'Host:'

Tokens, bearer tokens and Basic auth

curl -kH 'Authorization-Token: aaaa!bbbb!ccc' -v

Uploading files

Using curl, you can upload files both as a part of a form field or to a ReST API endpoint. We'll first create a file and then upload it both as a part of a form and to a ReST endpoint.

echo "hello, world" >> file.txt     
curl -F "file=@file.txt"

Since we're using the -F flag, the file is uploaded as a part of the form. You specify the file to upload by an @ followed by the path to the file. Since our file is in the current directory, we just specify the file name.

Uploading to a ReST endpoint is similar. We just use the -d flag instead.

curl -d @file.txt

Notice now that there are no quotes around file.txt. This is because we want the contents of the file to be the body of the POST request and not string “@file.txt” itself.

PUT request

ReST APIs use the PUT request to update an existing resource that the server manages. Making a PUT request with curl is similar to making a POST request. Here’s how you’d make a PUT request:

curl -X PUT -d '{"name":"Jane Doe"}'

DELETE request

Similarly, ReST APIs use DELETE request to delete an existing resource. Here's how to make a delete request:

curl -X DELETE

Basic authentication

Curl includes support for basic authentication. With basic authentication, you send your username and password as a part of the URL. There's a two ways in which you can do this in curl. Here's how:

curl -u john:abc123

In the first one you specify the username and password using the -u (short for --user) flag and curl appends this to the URL you provide. In the second case, you do it manually. Basic authentication does not encrypt your credentials in any way and is thus insecure unless used with HTTPS.

Setting cookies

You can set cookies using the -b (short for --cookie) flag. You specify the key and the value for the cookie with the flag. Here͛s an example showing how to set two cookies:

curl --cookie "name=john;course=linux beginner"

Notice that cookies are separated by a semicolon and that you do not need to encode spaces as a plus sign.This brings us to the end of the guide on using curl. The next part of the guide will introduce you to HTTP headers.

HTTP headers

The HTTP protocol allows a client to send a request to a server. With every HTTP request and response, there are associated headers that provide some more information. Let's make a HEAD request and see what we get:

curl -Iv     
*   Trying
* Connected to ( port 80 (#0)
> HEAD /get HTTP/1.1
> Host:> User-Agent: curl/7.49.1
> Accept: */*
< HTTP/1.1 200 OKHTTP/1.1 200 OK
< Server: nginxServer: nginx
< Date: Wed, 14 Dec 2016 16:23:11 GMTDate: Wed, 14 Dec 2016 16:23:11 GMT
< Content-Type: application/json
Content-Type: application/json
< Content-Length: 186
Content-Length: 186
< Connection: keep-alive
Connection: keep-alive
< Access-Control-Allow-Origin: *
Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
Access-Control-Allow-Credentials: true

In the output that's printed, the following are headers: Host, User-Agent, Accept, Server, Content-Type, Content-Length, Connection, Access-Control-Allow-Origin, and Access-Control-Allow-Credentials.

Headers can be classified as request and response headers that contain information about the request and response, respectively, or as general headers that contain information about both request and response. There's also another class of headers known as entity headers that contain more information about an entity, such as a file (i.e. it's length and MIME-type). Here's some of the most common headers and what they mean.

Authorization header

The Authorization header is used to provide authentication information such as bearer tokens.The server then uses this information to find out if the request should be processed further or not, depending on the validity of the authentication information provided. Example:

Authorization: Bearer 3beca038a248ff027d0445342fe285

Cache-Control header

The Cache-Control header decides how long applications, such as the browser, can keep a local copy of the data. This helps improve the efficiency since it helps avoid a round-trip to the server. Cache-Control has various directives that control various aspects of caching, such as the length of time data may be cached or if it shouldn't be cached at all. Example:

Cache-Control: max-age=100

Expires header

The Expires header is also used for caching and specifies the date and time after which a particular cached resource is considered stale. When both Expires and Cache-Control header are set, the Cache-Control header takes higher priority. Example:

Expires: Wed, 21 Oct 2015 07:28:00 GMT

Connection header

The Connection header specifies whether a connection should last for the duration of the request or if it should stay open, allowing it to be reused for subsequent requests. The default value is "close" which creates a connection that only lasts for the duration of the request. Setting the header to "keep-alive" allows it to be persistent. Example:

Connection: keep-alive

Accept header

The Accept header tells the server what kind of content the client is expecting. The value of this header is the MIME type of the content. Example:

Accept: application/json

Cookie header

The cookie header contains all the cookies and their values. Cookies are used to identify the users, maintain sessions, and so on. Example:

Cookie: name=John;course=Linux

Content-Length header

The Content-Length header specifies the length of the content in bytes. Example:

Content-Length: 111020

Content-Type header

The Content-Type header tells the client the MIME-type of the content that it has received. Example:

Content-Type: application/json

There are a lot more headers than that can be covered in a simple guide, which brings us to the end of the guide on curl and HTTP headers.