Curl Tutorials for REST API

Whenever we use the internet, whether for social media, browsing, or downloading or uploading files, we are fundamentally transferring data from or to servers. Often, this is done by the browser using the HTTP or HTTPS protocol.

Maybe you may also have wondered, whether is it only accessible by the browser? The answer is no. As part of the client-server architecture, both can communicate by relying on requests and responses through the terminal.

What is curl?

Curl is a tool for transferring data from or to servers using various protocols. Although often used with HTTP or HTTPS protocols, curl supports many other protocols such as IMAP, IMAPS, POP3, POP3S, FTP, FTPS, SMB, SMBS and many more. This tool also has some useful features related to use authentication, SSL connection, proxy support, file transfer resume and so on.

Why use curl?

If you are a programmer, consuming resources via REST API or creating your own is a very common thing to do. But sometimes, we need tools to check the REST API endpoints, and Postman is one of the most common to use.

As an alternative, curl can be greatly useful. Employing the terminal, this tool has the capability to make requests with various HTTP methods such as GET, POST, PUT, and so on. Apart from that, curl also allows users to automate the task, meaning that it might be very powerful as the interactions can be carried out without the need for user intervention.

Differences between wget and curl

If you are quite familiar with Linux and the terminal, one of the tools that might come to your mind to download a file via the terminal is wget. In essence, it might get the job done and occasionally you may prefer this tool too. Wget also supports recursive download unlike curl, which is the main benefit for using wget over curl.

However, in terms of versatility, curl leaves the wget way behind. By versatility, it means that the curl command has the plethora of supported protocols that we have mentioned. This is because curl includes libcurl, whereas wget does not. For this reason, wget command only supports HTTP, HTTPS, and FTP

Essential curl command list

While there’s actually a lot you can do with curl, this tutorial will only focus on the HTTP and HTTPS protocols.

Simple commands

curl https://example.com

You can use this command to retrieve all of the content from the current webpage, including the HTML code contained within. By default, received output will be displayed in stdout.

curl -o dummy.html https://example.com

This command allows the user to save the associated page into a file. Here, we give the file name “dummy.html” and we set it as an output with -o. This file will display the same page if we open it with a browser, but it will only run locally. However, it doesn’t always work as it should for any reasons such as missing dependencies.

curl -o ubuntu.iso https://repo.usk.ac.id/iso/ubuntu/22.04.1/ubuntu-22.04.1-desktop-amd64.iso

The same command can also be used to download a file. This command will download the Ubuntu 22.04.1 ISO file from the official Ubuntu site, https://ubuntu.com and we save it with ubuntu.iso as its name.

curl -i https://example.com

The -i or --include flag means we are requesting a response to the server in the form of both the header and body response. By default, the server will only return the request in the form of a body response. You can also request a header without a body by adding -I flag instead of -i.

Curl for REST API

By default, curl will make a request with a GET method just like some of the examples above. Though, there are several other methods you can specify according to your needs, such as POST, PUT, DELETE.

To demonstrate, I will use DummyJSON made by Muhammad Ovi (Owais). You can simply use either the website or clone his Github repository to follow along. For this example, I will use the website https://dummyjson.com to show you how it works. You can also get more information from the documentation by visiting https://dummyjson.com/docs .

GET Requests

curl https://dummyjson.com/users

Reference: https://dummyjson.com/docs/auth

This command will return the data in JSON format instead of HTML. While doing this in terminal, you may find it quite difficult to read as it is not so well structured. To fix this and make it more readable you can use piping with the jq command. Simply add | jq at the end of the above command, so the command will look like this, **curl https://dummyjson.com/users | jq**. While doing this, you can save the username and password combinations from a single account so we can use it in the next POST request.

POST Requests

curl -d "username=kdulyt&password=5t6q4KC7O" https://dummyjson.com/auth/login

Reference:https://dummyjson.com/docs/auth

It simulates the form that is commonly used to do authentication on a website. -d flag specifies the data we want to send to the server. You can obtain the data from the previous command. You can also try to input the incorrect data or omit this data altogether, so then you receive the "Authentication Problem" response.

If you also include the -i flag, the header will show up, telling you that the server returns a different status code. To identify whether your request is successful or fail, check the status code. Successful requests will result in a response with status code 200 and failed requests with status code 403.

If you look closely at the response body, the server provides information related to the token that we are going to use for the next POST request. Save this token value so we can use it later.

curl -X GET https://dummyjson.com/auth/RESOURCE -H "Accept: application/json" -H "Authorization: Bearer {token}”

**Success response: **

not found!

Reference:https://dummyjson.com/docs

The token we got earlier can be used to access the resource we want. The -X flag is used to specify the HTTP method we want. -H will set the value we provide to the header for our request.

We want to get a response in JSON format, so we must inform the server with "Accept: application/json". Next, we need to provide a token to access the resources we need which is usually referred to as JWT (JSON Web Token).

Another thing to note is that authorization and authentication are usually done within the header. In the previous example, both username and password are inserted within the body.

Some API Resources may require a username or password to be placed in the header. If so, you can use the --basic flag as a means of authentication. In the command below, we also use -u which means user.

curl -u name:password --basic https://example.com





curl -X POST -H "Content-Type: application/json" -d '{"title":"BMW Pencil"}' https://dummyjson.com/products/add

Success response:

{
  "id":101,
  "title":"BMW Pencil"
}

Reference:https://dummyjson.com/docs/products

Now, you are adding the product in the form of JSON to the server using a POST request. We also need to define the content-type we are going to send in the request header as we have the payload involved.

PUT Requests

curl -X PUT -H "Content-Type: application/json" -d '{"title":"iPhone Galaxy +1"}' https://dummyjson.com/products/1

Success response:

{
  "id": 1,
  "title": "iPhone Galaxy +1", //Update only applied to title
  "price": 549,
  "stock": 94,
  "ratings": 4.69,
  "images": [
    "https://i.dummyjson.com/data/products/1/1.jpg",
    "https://i.dummyjson.com/data/products/1/2.jpg",
    "https://i.dummyjson.com/data/products/1/3.jpg",
    "https://i.dummyjson.com/data/products/1/4.jpg",
    "https://i.dummyjson.com/data/products/1/thumbnail.jpg"
  ],
  "thumbnail": "https://i.dummyjson.com/data/products/1/thumbnail.jpg",
  "description": "An apple mobile which is nothing like apple",
  "brand": "Apple",
  "category": "smartphones"
}

Reference:https://dummyjson.com/docs/products

The command is basically the same as the previous one. The only difference lies in the PUT method, where the method will be used to update existing data. We also need to specify the data that we are going to change or update.

DELETE Request

curl -X DELETE ‘https://dummyjson.com/products/1’

Success response:

{
  "id": 1,
  "title": "iPhone 9",
  "description": "An apple mobile which is nothing like apple",
  "price": 549,
  "discount Percentage": 12.96,
  "ratings": 4.69,
  "stock": 94,
  "brand": "Apple",
  "category": "smartphones",
  "thumbnail": "https://i.dummyjson.com/data/products/1/thumbnail.jpg",
  "images": [
    "https://i.dummyjson.com/data/products/1/1.jpg",
    "https://i.dummyjson.com/data/products/1/2.jpg",
    "https://i.dummyjson.com/data/products/1/3.jpg",
    "https://i.dummyjson.com/data/products/1/4.jpg",
    "https://i.dummyjson.com/data/products/1/thumbnail.jpg"
  ],
  "isDeleted": true,
  "deletedOn": "2023-02-07T15:53:51.461Z"
}

Reference:https://dummyjson.com/docs/products** **

This command is used to delete the existing records. To do so, we use a DELETE request.

Summary

As explained earlier, curl still has many functions depending on what protocol is used. The examples above are a summary of the most frequently used commands to perform HTTP requests & responses. Besides being useful in software and application development, curl will make it very easy for us to perform pentesting on websites that depend on REST API, especially modern websites that use a lot of microservices architecture.