http(s) debugging cheat sheet

Martin Luther is attributed with the utterance "You have to watch how people talk". What works for the famous bible translation applies to APIs as well. Despite Open Standards and a standard body, APIs do their own thing, not necessarily documented as they are.

While it is reasonable easy to watch them inside a browser using Developer Tools (also here) it gets tricky when you want to watch an application like macOS Calendar, Thunderbird, Slack, Bodo (for Jira).

This is your cheat-sheet.

Setup your HTTP(s) Forensics

• Install the application you want to investigate (Yeah, a no brainer)
• Install a HTTP debugger, one of them (I use Charles Proxy)
• Proxyman (Mac)
• Charles Proxy (Mac, Win, Linux)
• Fiddler (Win)
• Configure your HTTP debugger to be able to analyse https traffic to your choosen calendar server
• Install PostMan and curl
• Have a place where you save your investigation results in Markdown, /docs in a GitHub repo is a good place
• Configure your HTTP debugger to intercept the http traffic on your target domain. This works different for each OS and debugger, so read the manual!
• SSL Proxying for Charles Proxy
• Decrypt HTTPS for Fiddler
• SSL Proxying with Proxyman
• Fetch a cup of your favorite beverage, we are good to go

Investigate

• Switch on your HTTP debugger (this line is sponsored by Captain Obvious)
• Execute one command in your custom app. Typically something like "login" or "add account" or "File Open" (that's my favorite in Office apps, fires 4 or more HTTP request when done against a http endpoint that understands webDAV)
• Look at the raw results. Each debugger has a nice gui that separates headers, cookies and body payload, but you want to look at raw data:
• Your request starts with METHOD /route HTTP_Version, e.g. POST /login HTTP/1.1 Everything until the first empty line is header, followed eventually with a body. Knowing methods helps to set expectations. See also RFC 7231, RFC 5789, RFC 2518, RFC 4918, RFC 3744 and RFC 4791
• Your response starts with a status line HTTP/1.1 StatusCode StatusMessage e.g. HTTP/1.1 200 OK Again: everything until the first empty line is header, followed by the optional response body
• It can get trickier when you app is already using HTTP/2.0 or later since it allows streaming or non-text payloads like gRPC
• Document that in markdown, bracket the http with three backticks, so it renders as source
• Repeat for other commands in your app

Re-enactment

What fun is detective work without verifying results. This is where you turn to Postman or if you have it - curl.

You want to use parameters for your hostname and the user specific parts (username, passwords) and you need to have a clear idea what are variable return values. Good candidtes to watch out for are cookies or header values. You need to extract these values for chaining to the next request. With a little practise you should be able to make Postman behave like the original app

Parting words

• Check the terms of service of the application you are investigating. While public end-point are, well, public, you might have agreed in your T&C not to touch them
• This isn't an instuction for hacking, you still need to be you - with your credentials. Nevertheless you might stumble over "security by obscurity" or other annoyances
• Any app that uses http instead of https needs to die a horrible death
• Reading the API spec is potentially faster

As usual YMMV