Linux cURL

Using curl

The best way to understand curl is to use it. This guide is an introduction-by-example. We'll use curl to send requests to httpbin.org which is an HTTP client testing service. Let's get started.

GET request

Fire up a terminal and execute the following command:

curl www.httpbin.org

This command will fetch the HTML for the HTTPBin homepage. Notice that we've not mentioned the protocol or the HTTP method. They've defaulted to HTTP and GET, respectively.

HEAD request

Sometimes you may want to see just the headers that the server would return when it's issued a GET request. In such a case, we'd use the HEAD method. The head request is issued with -I (capital i) or --head.

curl -I www.httpbin.org    
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 01 Dec 2016 18:50:56 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 12150
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Connection: keep-alive

Verbose output

Sometimes, you may want to debug a problem because the curl command hasn't returned what it was supposed to or just hasn't worked at all. In such cases, you can use the verbose mode using -v or --verbose to get more output from curl. Let's modify the previous HTTP request to become verbose.

curl -Iv www.httpbin.org

Now there's a lot more output in the terminal. Let's see what it means.

Lines prefixed with an asterisk (*) show additional work curl has done. For example:

* Rebuilt URL to: www.httpbin.org/
*   Trying 23.22.14.18...
* Connected to www.httpbin.org (23.22.14.18) port 80 (#0)

The first line shows that curl added a slash at the end of the URL.

The second line shows that curl has resolved the URL to an IP address.

Finally, the third line shows that curl has connected to the URL on port 80.

Lines prefixed with a greater-than (>) sign show the data curl has sent to the server. For example:

> HEAD / HTTP/1.1
> Host: www.httpbin.org
> User-Agent: curl/7.47.0
> Accept: */*

Lines prefixed with a less-than (<) sign show the data curl has received from the server. For example:

< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: nginx
Server: nginx

Multiple requests

You can also send requests to multiple URLs by separating them by space. Here's how:

curl httpbin.org/status/418 httpbin.org/status/418

The above example will send two GET requests, both to the same URL, one after the other. You should see two little teapots printed in your terminal.

POST request

You can send a POST request by using the -d (or --data) flag. It͛s up to you to properly URL encode the data or specify it as JSON. We'll see the examples for both of those. Here's how you'd send URL encoded data as a part of the POST body

curl -d 'name=john+doe&course=linux' httpbin.org/post

The response sent by HTTPBin will simply echo what you͛ve sent.

To send JSON:

curl -d '{"name":"John Doe","course":"Linux"}' httpbin.org/post

Notice that in both the examples, the data is enclosed in single quotes. Using double quotes is okay but it may cause problems if the body also contains double quotes like JSON. Using single quotes is the safer approach.

Setting headers

You can set headers by using the -H flag. In our previous example, we've sent JSON without setting the Content-Type header to application/json. Here's the previous example modified to set the header:

curl -H 'Content-Type: application/json' -d '{"name":"John Doe","course":"Linux"}' httpbin.org/post

You set the header as a key-value pair consisting of the header name followed by a colon and the value. Posting form data

You can send form data by using the -F flag. Use this flag for every form field that you want to send. Here's how you'd send form data:

curl -F "name=John Doe" -F "course=linux"  httpbin.org/post

Notice that we've written the name as John Doe without a plus sign in between. The response received will have the Content-Type header set to multipart/form-data.

Uploading files

Using curl, you can upload files both as a part of a form field or to a ReST API endpoint. We'll first create a file and then upload it both as a part of a form and to a ReST endpoint.

echo "hello, world" >> file.txt     
curl -F "file=@file.txt" httpbin.org/post

Since we're using the -F flag, the file is uploaded as a part of the form. You specify the file to upload by an @ followed by the path to the file. Since our file is in the current directory, we just specify the file name.

Uploading to a ReST endpoint is similar. We just use the -d flag instead.

curl -d @file.txt httpbin.org/post

Notice now that there are no quotes around file.txt. This is because we want the contents of the file to be the body of the POST request and not string “@file.txt” itself.

PUT request

ReST APIs use the PUT request to update an existing resource that the server manages. Making a PUT request with curl is similar to making a POST request. Here’s how you’d make a PUT request:

curl -X PUT -d '{"name":"Jane Doe"}' httpbin.org/put

DELETE request

Similarly, ReST APIs use DELETE request to delete an existing resource. Here's how to make a delete request:

curl -X DELETE httpbin.org/delete

Basic authentication

Curl includes support for basic authentication. With basic authentication, you send your username and password as a part of the URL. There's a two ways in which you can do this in curl. Here's how:

curl -u john:abc123 httpbin.org/basic-auth/john/abc123
curl john:abc123@httpbin.org/basic-auth/john/abc123

In the first one you specify the username and password using the -u (short for --user) flag and curl appends this to the URL you provide. In the second case, you do it manually. Basic authentication does not encrypt your credentials in any way and is thus insecure unless used with HTTPS.

Setting cookies

You can set cookies using the -b (short for --cookie) flag. You specify the key and the value for the cookie with the flag. Here͛s an example showing how to set two cookies:

curl --cookie "name=john;course=linux beginner" httpbin.org/cookies

Notice that cookies are separated by a semicolon and that you do not need to encode spaces as a plus sign.This brings us to the end of the guide on using curl. The next part of the guide will introduce you to HTTP headers.

HTTP headers

The HTTP protocol allows a client to send a request to a server. With every HTTP request and response, there are associated headers that provide some more information. Let's make a HEAD request and see what we get:

curl -Iv httpbin.org/get     
*   Trying 23.22.14.18...
* Connected to httpbin.org (23.22.14.18) port 80 (#0)
> HEAD /get HTTP/1.1
> Host: httpbin.org> User-Agent: curl/7.49.1
> Accept: */*
> 
< HTTP/1.1 200 OKHTTP/1.1 200 OK
< Server: nginxServer: nginx
< Date: Wed, 14 Dec 2016 16:23:11 GMTDate: Wed, 14 Dec 2016 16:23:11 GMT
< Content-Type: application/json
Content-Type: application/json
< Content-Length: 186
Content-Length: 186
< Connection: keep-alive
Connection: keep-alive
< Access-Control-Allow-Origin: *
Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
Access-Control-Allow-Credentials: true

In the output that's printed, the following are headers: Host, User-Agent, Accept, Server, Content-Type, Content-Length, Connection, Access-Control-Allow-Origin, and Access-Control-Allow-Credentials.

Headers can be classified as request and response headers that contain information about the request and response, respectively, or as general headers that contain information about both request and response. There's also another class of headers known as entity headers that contain more information about an entity, such as a file (i.e. it's length and MIME-type). Here's some of the most common headers and what they mean.

Authorization header

The Authorization header is used to provide authentication information such as bearer tokens.The server then uses this information to find out if the request should be processed further or not, depending on the validity of the authentication information provided. Example:

Authorization: Bearer 3beca038a248ff027d0445342fe285

Cache-Control header

The Cache-Control header decides how long applications, such as the browser, can keep a local copy of the data. This helps improve the efficiency since it helps avoid a round-trip to the server. Cache-Control has various directives that control various aspects of caching, such as the length of time data may be cached or if it shouldn't be cached at all. Example:

Cache-Control: max-age=100

Expires header

The Expires header is also used for caching and specifies the date and time after which a particular cached resource is considered stale. When both Expires and Cache-Control header are set, the Cache-Control header takes higher priority. Example:

Expires: Wed, 21 Oct 2015 07:28:00 GMT

Connection header

The Connection header specifies whether a connection should last for the duration of the request or if it should stay open, allowing it to be reused for subsequent requests. The default value is "close" which creates a connection that only lasts for the duration of the request. Setting the header to "keep-alive" allows it to be persistent. Example:

Connection: keep-alive

Accept header

The Accept header tells the server what kind of content the client is expecting. The value of this header is the MIME type of the content. Example:

Accept: application/json

Cookie header

The cookie header contains all the cookies and their values. Cookies are used to identify the users, maintain sessions, and so on. Example:

Cookie: name=John;course=Linux

Content-Length header

The Content-Length header specifies the length of the content in bytes. Example:

Content-Length: 111020

Content-Type header

The Content-Type header tells the client the MIME-type of the content that it has received. Example:

Content-Type: application/json

There are a lot more headers than that can be covered in a simple guide, which brings us to the end of the guide on curl and HTTP headers.