Simple cURL requests

Whenever it comes to parsing some web content, cURL is a sweet tool for that. It is as simple as

curl -X GET https://christianrolle.medium.com

which in this case returns some HTML.

However there is more to it. You will see when messing around with cURL the HTTPbin is a sweet public API.

cURL request header

The request header can be set with option -H

curl -H "Accept: application/json" "https://httpbin.org/anything?search=Test"

And since that API is able to respond to a JSON request its response looks like

{
  "args": {
    "search": "Test"
  }, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.81.0", 
    "X-Amzn-Trace-Id": "Root=1-63daca6a-3c9f6f944b65a5877c455039"
  }, 
  "json": null, 
  "method": "GET", 
  "origin": "142.250.181.238", 
  "url": "https://httpbin.org/anything?search=Test"
}

With a custom user agent using option -A

curl -A "Special agent Version 0.0.7" "https://httpbin.org/anything"

And claiming a referer with option -e

curl -e "http://google.com" "https://httpbin.org/anything"

cURL response header

Diving into the response header with option -I

curl -I "https://httpbin.org/anything"

provides some insights

HTTP/2 200 
date: Wed, 01 Feb 2023 20:58:58 GMT
content-type: application/json
content-length: 344
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

cURL request parameters

Sending additional request parameters requires option -d

curl -X POST -d "search=Test" -d "size=10" "https://httpbin.org/anything"

Or execute a multipart upload with option -F

curl -X POST -F image=@test.png "https://httpbin.org/anything"

cURL cookie handling

Downloading a cookie and saving it locally with the cookie-jar option -c

curl -c cookie.txt https://medium.com

And it seems to be alive for a long time

cat cookie.txt

#HttpOnly_.medium.com TRUE / FALSE 1706820414  __cfruid 8a6d93effb0f5c6bf6a5702ec13c1c98058e7aa2-1675282510

We could manipulate the cookie and set it with option -b

curl -b cookie.txt "https://medium.com"

Resources

• cURL man page
• HTTPbin