uses of curl linux command with examples

Advertisements

According to various definitions in the internet, curl is a tool to transfer data from or to a server. Curl supports the following protocols. DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP. I was working with content delivery networks(cdn) and came across this command to get the headers related to the caching of the objects. Later when I checked the use cases, I found it very interesting. So I'm sharing the points here.
Starting with the cdn related commands.

if we use -v option with curl with the url of the object (in this case cdn we are using in example is amazon cloufront), it will give all the request headers and response headers.

randeep@Randeep:~$ curl -v http://abcdefghi.cloudfront.net/images/CLogo_bollywood_192.png
* Hostname was NOT found in DNS cache
*   Trying 54.230.190.xxx...
* Connected to abcdefghi.cloudfront.net (54.230.190.xxx) port 80 (#0)
> GET /images/CLogo_bollywood_192.png HTTP/1.1
> User-Agent: curl/7.35.0
> Host: abcdefghi.cloudfront.net
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: image/png
< Content-Length: 8046
< Date: Mon, 16 Feb 2015 02:04:03 GMT
< Last-Modified: Wed, 25 Jun 2014 12:10:59 GMT
< ETag: "530dd4308aaedc70a3aae9b0889bd"
< Accept-Ranges: bytes
* Server AmazonS3 is not blacklisted
< Server: AmazonS3
< Age: 203184
< X-Cache: Hit from cloudfront
< Via: 1.1 29b3cfc63bec85046291502a4.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: EGjPC9oHhVYc6gdczCygXeAGJtPGIuYrklTMaNlko98RQ==
< Connection: keep-alive
<
Followed by PNG data.

In case if you want to see only the headers in the response, you can use -I option with curl

randeep@Randeep:~$ curl -I http://abcdefghi.cloudfront.net/images/CLogo_bollywood_192.png
HTTP/1.1 200 OK
Content-Type: image/png
Content-Length: 8046
Date: Mon, 16 Feb 2015 02:04:03 GMT
Last-Modified: Wed, 25 Jun 2014 12:10:59 GMT
ETag: "530dd4308aaedc7063aae9b0889bd"
Accept-Ranges: bytes
Server: AmazonS3
Age: 203184
X-Cache: Hit from cloudfront
Via: 1.1 29b3cfc63bec85091502a4.cloudfront.net (CloudFront)
X-Amz-Cf-Id: EGjPC9oHhV6gdczCygX7qEPGIuYrklTMaNlko98RQ==
Connection: keep-alive

Checking whether the content can be cached or not:
In this example content can be cached.
randeep@Randeep:~/aws$ curl -H "Cache-control: only-if-cached" -D - -o /dev/null "http://54.161.158.165:8080/

videos/abc.mp4"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/1.1 200 OK
Date: Sun, 12 Apr 2015 12:18:41 GMT
Server: Apache/2.2.15 (CentOS)
Last-Modified: Wed, 30 Apr 2014 04:02:06 GMT
ETag: "3b32-1df0d06-4f83a9c9a6b80"
Accept-Ranges: bytes
Content-Length: 31395078
Cache-Control: max-age=604800, public
Connection: close
Content-Type: video/mp4

 14 29.9M   14 4568k    0     0   371k      0  0:01:22  0:00:12  0:01:10  255k
 15 29.9M   15 4787k    0     0   334k      0  0:01:31  0:00:14  0:01:17  128k^C

In this example, content is not cached and cant be.
randeep@Randeep:~/aws$ curl -H "Cache-control: only-if-cached" -D - -o /dev/null "http://54.161.158.165/videos/abc.mp4"  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/1.1 504 Not CachedDate: Sun, 12 Apr 2015 12:21:37 GMT
Connection: keep-alive
Cache-Control: no-store
Content-Type: text/html
Content-Language: en
Content-Length: 340100   340  100   340    0     0  46008      0 --:--:-- --:--:-- --:--:-- 48571
randeep@Randeep:~/aws$

sending multiple headers in curl: 
curl -H "Cache-control: only-if-cached" -H "Range: bytes=0-1024" -D - -o /dev/null "http://tata.cdn.bitgravity.com/abc.mp4"

Other interesting uses of curl:

Fetching/Downloading files:
To download a file

Syntax:
$ curl -O URL
Example:
$ curl -O http://www.apache.org/dyn/closer.cgi/trafficserver/trafficserver-5.2.0.tar.bz2
The above example will download the archive file named trafficserver-5.2.0.tar.bz2 in the current directory.

To download multiple files

Syntax:
$ curl -O URL1 -O URL2
Example:
$ curl -O http://www.apache.org/dyn/closer.cgi/trafficserver/trafficserver-5.2.0.tar.bz2 -O http://apache.bytenet.in/tomcat/tomcat-8/v8.0.18/bin/apache-tomcat-8.0.18.tar.gz
The above command will download both trafficserver-5.2.0.tar.bz2, apache-tomcat-8.0.18.tar.gz

To download a file with restricted network speed

Syntax:
$ curl --limit-rate 1000B -O http://download.thinkbroadband.com/1GB.zip
Example:
randeep@Randeep:~$ curl --limit-rate 1000B -O http://download.thinkbroadband.com/1GB.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 1024M    0 45455    0     0   1005      0  12d 08h  0:00:45  12d 08h  1229
The above command will download the file 1GB.zip but within the speed limit of 1000Bytes per second.

FTP:
FTP is file transfer protocol.
Downloading a file from ftp:
$ curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/xss.php

Uploading a file to ftp
$ curl -u ftpuser:ftppass -T myfile.txt ftp://ftp.testserver.com

sending mail using curl:
Using curl, we can also send mails. All we need is an smtp server and user credentials.

curl --url "smtps://smtp.example.com:465" --ssl-reqd   --mail-from "me@example.com" --mail-rcpt "you@example.com"   --upload-file content.txt --user "me@example.com:mypassword" --insecure

 That's it. Read more about curl and comment if you find something very useful.

Various usage of find command can be found here.