Whenever we use the internet, whether for social media, browsing, or downloading or uploading files, we are fundamentally transferring data from or to servers. Often, this is done by the browser using the HTTP or HTTPS protocol.
Maybe you may also have wondered, whether is it only accessible by the browser? The answer is no. As part of the client-server architecture, both can communicate by relying on requests and responses through the terminal.
What is curl?
Curl is a tool for transferring data from or to servers using various protocols. Although often used with HTTP or HTTPS protocols, curl supports many other protocols such as IMAP, IMAPS, POP3, POP3S, FTP, FTPS, SMB, SMBS and many more. This tool also has some useful features related to use authentication, SSL connection, proxy support, file transfer resume and so on.
Why use curl?
If you are a programmer, consuming resources via REST API or creating your own is a very common thing to do. But sometimes, we need tools to check the REST API endpoints, and Postman is one of the most common to use.
As an alternative, curl can be greatly useful. Employing the terminal, this tool has the capability to make requests with various HTTP methods such as GET, POST, PUT, and so on. Apart from that, curl also allows users to automate the task, meaning that it might be very powerful as the interactions can be carried out without the need for user intervention.
Differences between wget and curl
If you are quite familiar with Linux and the terminal, one of the tools that might come to your mind to download a file via the terminal is wget. In essence, it might get the job done and occasionally you may prefer this tool too. Wget also supports recursive download unlike curl, which is the main benefit for using wget over curl.
However, in terms of versatility, curl leaves the wget way behind. By versatility, it means that the curl command has the plethora of supported protocols that we have mentioned. This is because curl includes libcurl, whereas wget does not. For this reason, wget command only supports HTTP, HTTPS, and FTP
Essential curl command list
While there’s actually a lot you can do with curl, this tutorial will only focus on the HTTP and HTTPS protocols.
Simple commands
curl https://example.com
You can use this command to retrieve all of the content from the current webpage, including the HTML code contained within. By default, received output will be displayed in stdout.
curl -o dummy.html https://example.com
This command allows the user to save the associated page into a file. Here, we
give the file name “dummy.html” and we set it as an output with -o
. This
file will display the same page if we open it with a browser, but it will only
run locally. However, it doesn’t always work as it should for any reasons such
as missing dependencies.
curl -o ubuntu.iso https://repo.usk.ac.id/iso/ubuntu/22.04.1/ubuntu-22.04.1-desktop-amd64.iso
The same command can also be used to download a file. This command will download the Ubuntu 22.04.1 ISO file from the official Ubuntu site, https://ubuntu.com and we save it with ubuntu.iso as its name.
curl -i https://example.com
The -i
or --include
flag means we are requesting a response to the server
in the form of both the header and body response. By default, the server will
only return the request in the form of a body response. You can also request a
header without a body by adding -I
flag instead of -i
.
Curl for REST API
By default, curl will make a request with a GET method just like some of the examples above. Though, there are several other methods you can specify according to your needs, such as POST, PUT, DELETE.
To demonstrate, I will use DummyJSON made by Muhammad Ovi (Owais). You can simply use either the website or clone his Github repository to follow along. For this example, I will use the website https://dummyjson.com to show you how it works. You can also get more information from the documentation by visiting https://dummyjson.com/docs .
GET Requests
curl https://dummyjson.com/users
Reference: https://dummyjson.com/docs/auth
This command will return the data in JSON format instead of HTML. While doing this in terminal, you may find it quite difficult to read as it is not so well structured. To fix this and make it more readable you can use piping with the jq
command. Simply add | jq
at the end of the above command, so the command will look like this, **curl https://dummyjson.com/users | jq**
. While doing this, you can save the username and password combinations from a single account so we can use it in the next POST request.
POST Requests
curl -d "username=kdulyt&password=5t6q4KC7O" https://dummyjson.com/auth/login
Reference:https://dummyjson.com/docs/auth
It simulates the form that is commonly used to do authentication on a website.
-d
flag specifies the data we want to send to the server. You can obtain the
data from the previous command. You can also try to input the incorrect data
or omit this data altogether, so then you receive the "Authentication Problem"
response.
If you also include the -i
flag, the header will show up, telling you that
the server returns a different status code. To identify whether your request
is successful or fail, check the status code. Successful requests will result
in a response with status code 200 and failed requests with status code 403.
If you look closely at the response body, the server provides information related to the token that we are going to use for the next POST request. Save this token value so we can use it later.
curl -X GET https://dummyjson.com/auth/RESOURCE -H "Accept: application/json" -H "Authorization: Bearer {token}”
**Success response: **
not found!
Reference:https://dummyjson.com/docs
The token we got earlier can be used to access the resource we want. The -X
flag is used to specify the HTTP method we want. -H
will set the value we
provide to the header for our request.
We want to get a response in JSON format, so we must inform the server with
"Accept: application/json"
. Next, we need to provide a token to access the
resources we need which is usually referred to as JWT (JSON Web
Token).
Another thing to note is that authorization and authentication are usually done within the header. In the previous example, both username and password are inserted within the body.
Some API Resources may require a username or password to be placed in the
header. If so, you can use the --basic
flag as a means of authentication. In
the command below, we also use -u which means user.
curl -u name:password --basic https://example.com
curl -X POST -H "Content-Type: application/json" -d '{"title":"BMW Pencil"}' https://dummyjson.com/products/add
Success response:
{
"id":101,
"title":"BMW Pencil"
}
Reference:https://dummyjson.com/docs/products
Now, you are adding the product in the form of JSON to the server using a POST request. We also need to define the content-type we are going to send in the request header as we have the payload involved.
PUT Requests
curl -X PUT -H "Content-Type: application/json" -d '{"title":"iPhone Galaxy +1"}' https://dummyjson.com/products/1
Success response:
{
"id": 1,
"title": "iPhone Galaxy +1", //Update only applied to title
"price": 549,
"stock": 94,
"ratings": 4.69,
"images": [
"https://i.dummyjson.com/data/products/1/1.jpg",
"https://i.dummyjson.com/data/products/1/2.jpg",
"https://i.dummyjson.com/data/products/1/3.jpg",
"https://i.dummyjson.com/data/products/1/4.jpg",
"https://i.dummyjson.com/data/products/1/thumbnail.jpg"
],
"thumbnail": "https://i.dummyjson.com/data/products/1/thumbnail.jpg",
"description": "An apple mobile which is nothing like apple",
"brand": "Apple",
"category": "smartphones"
}
Reference:https://dummyjson.com/docs/products
The command is basically the same as the previous one. The only difference lies in the PUT method, where the method will be used to update existing data. We also need to specify the data that we are going to change or update.
DELETE Request
curl -X DELETE ‘https://dummyjson.com/products/1’
Success response:
{
"id": 1,
"title": "iPhone 9",
"description": "An apple mobile which is nothing like apple",
"price": 549,
"discount Percentage": 12.96,
"ratings": 4.69,
"stock": 94,
"brand": "Apple",
"category": "smartphones",
"thumbnail": "https://i.dummyjson.com/data/products/1/thumbnail.jpg",
"images": [
"https://i.dummyjson.com/data/products/1/1.jpg",
"https://i.dummyjson.com/data/products/1/2.jpg",
"https://i.dummyjson.com/data/products/1/3.jpg",
"https://i.dummyjson.com/data/products/1/4.jpg",
"https://i.dummyjson.com/data/products/1/thumbnail.jpg"
],
"isDeleted": true,
"deletedOn": "2023-02-07T15:53:51.461Z"
}
Reference:https://dummyjson.com/docs/products** **
This command is used to delete the existing records. To do so, we use a DELETE request.
Summary
As explained earlier, curl still has many functions depending on what protocol is used. The examples above are a summary of the most frequently used commands to perform HTTP requests & responses. Besides being useful in software and application development, curl will make it very easy for us to perform pentesting on websites that depend on REST API, especially modern websites that use a lot of microservices architecture.