Linux cURL
Using curl
The example below use curl to send requests to httpbin.org which is simple HTTP Request & Response Service, the guide follow introduction-by-example principle.
Basics of HTTP
- haproxy configuration Fantastic first section on how http works, header format and assembly when passing proxy servers
Common curls
Single website pinning
Debugging problems with a site that's already live, it's helpful to bypass CDN and proxy layers when requesting content from the site by sending those web requests directly to a specific IP address without using the site's public DNS records. This practice of "pinning" a web request directly to a server can be accomplished by changing your /etc/hosts file, which will cause requests for a specified domain name (e.g: "www.example.com ") to be routed from your local machine to a specified IP address (e.g: 127.0.0.1) until the changes you've made to /etc/hosts are reverted.
# get ip of your Loadbalancer
dig +short k8s.example.com # choose one
# dns name pinning
curl -v --resolve "project1.private.com:80:10.10.10.10" http://project1.private.com/docs/index.html
curl -v --resolve "project1.private.com:443:10.10.10.10" https://project1.private.com/docs/index.html
# multiple pinnings :80 & :443 to use localhost version of the website
curl -v --resolve "project1.private.com:80:127.0.0.1" http://project1.private.com \
--resolve "project1.private.com:443:127.0.0.1" https://project1.private.com
# --resolve [DOMAIN]:[PORT]:[IP] # the option format
Resolve DNS without using local cache
sudo systemd-resolve --flush-caches # if you wish to flush dns DOMAIN=tvp.info; curl -I --resolve "$DOMAIN:80:$(dig +short $DOMAIN @8.8.8.8 | head -1)" http://$DOMAIN systemd-resolve --statistics | grep 'Current Cache Size' # should still be -> Current Cache Size: 0 # --dns-servers - this solution still uses local cache curl --dns-servers _MY_DNS_SERVER_IP_ https://_MY_DOMAINS_ curl --dns-servers 8.8.8.8 https://tvp.info
Note: libcurl and c-ares backend
curl: (4) A requested feature, protocol or option was not found built-in in this libcurl due to a build-time decision.
From curl man page: This option requires that libcurl was built with a resolver backend that supports this operation. The c-ares backend is the only such one. (Added in 7.33.0)
Use curl in shell scripts
When using Curl in shell scripts, always pass -fSSL, which:
-ftreats non-2xx/3xx responses as errors-sSdisables the progress meter-Lhandles HTTP redirects
Occasionally, if you consider some non-2xx/3xx responses to be non-fatal, then omit -f then use logic to treat these responses as success.
curl -fsSL http://httpbin.org
$ curl -sS -w '%{http_code}\n' -o /dev/null http://httpbin.org
200
# -w '%{http_code}' write the HTTP response code to standard output
# -o <filename> write the HTTP response body to the given file, not to standard output
Resources:
Probe to monitor an endpoint
curl --connect-timeout 5 --retry 12 --retry-delay 1 --retry-max-time 60 service-1.acme.com:9000/sessions/new
GET request
Fire up a terminal and execute the following command:
curl www.httpbin.org
This command will fetch the HTML for the HTTPBin homepage. Notice that we've not mentioned the protocol or the HTTP method. They've defaulted to HTTP and GET, respectively.
HEAD request
Sometimes you may want to see just the headers that the server would return when it's issued a GET request. In such a case, we'd use the HEAD method. The head request is issued with -I (capital i) or --head.
curl -I www.httpbin.org HTTP/1.1 200 OK Server: nginx Date: Thu, 01 Dec 2016 18:50:56 GMT Content-Type: text/html; charset=utf-8 Content-Length: 12150 Access-Control-Allow-Origin: * Access-Control-Allow-Credentials: true Connection: keep-alive
Verbose output
Sometimes, you may want to debug a problem because the curl command hasn't returned what it was supposed to or just hasn't worked at all. In such cases, you can use the verbose mode using -v or --verbose to get more output from curl. Let's modify the previous HTTP request to become verbose.
curl -Iv www.httpbin.org
Now there's a lot more output in the terminal. Let's see what it means.
Lines prefixed with an asterisk (*) show additional work curl has done. For example:
* Rebuilt URL to: www.httpbin.org/ * Trying 23.22.14.18... * Connected to www.httpbin.org (23.22.14.18) port 80 (#0)
The first line shows that curl added a slash at the end of the URL.
The second line shows that curl has resolved the URL to an IP address.
Finally, the third line shows that curl has connected to the URL on port 80.
Lines prefixed with a greater-than (>) sign show the data curl has sent to the server. For example:
> HEAD / HTTP/1.1 > Host: www.httpbin.org > User-Agent: curl/7.47.0 > Accept: */*
Lines prefixed with a less-than (<) sign show the data curl has received from the server. For example:
< HTTP/1.1 200 OK HTTP/1.1 200 OK < Server: nginx Server: nginx
Multiple requests
You can also send requests to multiple URLs by separating them by space. Here's how:
curl httpbin.org/status/418 httpbin.org/status/418
The above example will send two GET requests, both to the same URL, one after the other. You should see two little teapots printed in your terminal.
POST request
You can send a POST request by using the -d (or --data) flag. It͛s up to you to properly URL encode the data or specify it as JSON. We'll see the examples for both of those. Here's how you'd send URL encoded data as a part of the POST body
curl -d 'name=john+doe&course=linux' httpbin.org/post
The response sent by HTTPBin will simply echo what you͛ve sent.
POST JSON
curl -d '{"name":"John Doe","course":"Linux"}' httpbin.org/post
Notice that in both the examples, the data is enclosed in single quotes. Using double quotes is okay but it may cause problems if the body also contains double quotes like JSON. Using single quotes is the safer approach.
Setting headers, authentication, authorization
You can set headers by using the -H flag. In our previous example, we've sent JSON without setting the Content-Type header to application/json. Here's the previous example modified to set the header:
curl -H 'Content-Type: application/json' -d '{"name":"John Doe","course":"Linux"}' httpbin.org/post
You set the header as a key-value pair consisting of the header name followed by a colon and the value.
- Posting form data
You can send form data by using the -F flag. Use this flag for every form field that you want to send. Here's how you'd send form data:
curl -F "name=John Doe" -F "course=linux" httpbin.org/post
Notice that we've written the name as John Doe without a plus sign in between. The response received will have the Content-Type header set to multipart/form-data.
Note SSL connection is not using -H hostname from headers but from DNS name; use below
curl -H 'Host: project1.private.com' http://127.0.0.1/something
Tokens, bearer tokens and Basic auth
curl -kH 'Authorization-Token: aaaa!bbbb!ccc' httpbin.org/api/HealthCheck -v
Uploading files
Using curl, you can upload files both as a part of a form field or to a ReST API endpoint. We'll first create a file and then upload it both as a part of a form and to a ReST endpoint.
echo "hello, world" >> file.txt curl -F "file=@file.txt" httpbin.org/post
Since we're using the -F flag, the file is uploaded as a part of the form. You specify the file to upload by an @ followed by the path to the file. Since our file is in the current directory, we just specify the file name.
Uploading to a ReST endpoint is similar. We just use the -d flag instead.
curl -d @file.txt httpbin.org/post
Notice now that there are no quotes around file.txt. This is because we want the contents of the file to be the body of the POST request and not string “@file.txt” itself.
PUT request
ReST APIs use the PUT request to update an existing resource that the server manages. Making a PUT request with curl is similar to making a POST request. Here’s how you’d make a PUT request:
curl -X PUT -d '{"name":"Jane Doe"}' httpbin.org/put
DELETE request
Similarly, ReST APIs use DELETE request to delete an existing resource. Here's how to make a delete request:
curl -X DELETE httpbin.org/delete
Basic authentication
Curl includes support for basic authentication. With basic authentication, you send your username and password as a part of the URL. There's a two ways in which you can do this in curl. Here's how:
curl -u john:abc123 httpbin.org/basic-auth/john/abc123 curl john:abc123@httpbin.org/basic-auth/john/abc123
In the first one you specify the username and password using the -u (short for --user) flag and curl appends this to the URL you provide. In the second case, you do it manually. Basic authentication does not encrypt your credentials in any way and is thus insecure unless used with HTTPS.
Setting cookies
You can set cookies using the -b (short for --cookie) flag. You specify the key and the value for the cookie with the flag. Here͛s an example showing how to set two cookies:
curl --cookie "name=john;course=linux beginner" httpbin.org/cookies
Notice that cookies are separated by a semicolon and that you do not need to encode spaces as a plus sign.This brings us to the end of the guide on using curl. The next part of the guide will introduce you to HTTP headers.
HTTP headers
The HTTP protocol allows a client to send a request to a server. With every HTTP request and response, there are associated headers that provide some more information. Let's make a HEAD request and see what we get:
curl -Iv httpbin.org/get * Trying 23.22.14.18... * Connected to httpbin.org (23.22.14.18) port 80 (#0) > HEAD /get HTTP/1.1 > Host: httpbin.org> User-Agent: curl/7.49.1 > Accept: */* > < HTTP/1.1 200 OKHTTP/1.1 200 OK < Server: nginxServer: nginx < Date: Wed, 14 Dec 2016 16:23:11 GMTDate: Wed, 14 Dec 2016 16:23:11 GMT < Content-Type: application/json Content-Type: application/json < Content-Length: 186 Content-Length: 186 < Connection: keep-alive Connection: keep-alive < Access-Control-Allow-Origin: * Access-Control-Allow-Origin: * < Access-Control-Allow-Credentials: true Access-Control-Allow-Credentials: true
In the output that's printed, the following are headers: Host, User-Agent, Accept, Server, Content-Type, Content-Length, Connection, Access-Control-Allow-Origin, and Access-Control-Allow-Credentials.
Headers can be classified as request and response headers that contain information about the request and response, respectively, or as general headers that contain information about both request and response. There's also another class of headers known as entity headers that contain more information about an entity, such as a file (i.e. it's length and MIME-type). Here's some of the most common headers and what they mean.
Authorization header
The Authorization header is used to provide authentication information such as bearer tokens.The server then uses this information to find out if the request should be processed further or not, depending on the validity of the authentication information provided. Example:
Authorization: Bearer 3beca038a248ff027d0445342fe285
Cache-Control header
The Cache-Control header decides how long applications, such as the browser, can keep a local copy of the data. This helps improve the efficiency since it helps avoid a round-trip to the server. Cache-Control has various directives that control various aspects of caching, such as the length of time data may be cached or if it shouldn't be cached at all. Example:
Cache-Control: max-age=100
Expires header
The Expires header is also used for caching and specifies the date and time after which a particular cached resource is considered stale. When both Expires and Cache-Control header are set, the Cache-Control header takes higher priority. Example:
Expires: Wed, 21 Oct 2015 07:28:00 GMT
Connection header
The Connection header specifies whether a connection should last for the duration of the request or if it should stay open, allowing it to be reused for subsequent requests. The default value is "close" which creates a connection that only lasts for the duration of the request. Setting the header to "keep-alive" allows it to be persistent. Example:
Connection: keep-alive
Accept header
The Accept header tells the server what kind of content the client is expecting. The value of this header is the MIME type of the content. Example:
Accept: application/json
Cookie header
The cookie header contains all the cookies and their values. Cookies are used to identify the users, maintain sessions, and so on. Example:
Cookie: name=John;course=Linux
Content-Length header
The Content-Length header specifies the length of the content in bytes. Example:
Content-Length: 111020
Content-Type header
The Content-Type header tells the client the MIME-type of the content that it has received. Example:
Content-Type: application/json
There are a lot more headers than that can be covered in a simple guide, which brings us to the end of the guide on curl and HTTP headers.