Martin Luther is attributed with the utterance "You have to watch how people talk". What works for the famous bible translation applies to APIs as well. Despite Open Standards and a standard body, APIs do their own thing, not necessarily documented as they are.

While it is reasonable easy to watch them inside a browser using Developer Tools (also here) it gets tricky when you want to watch an application like macOS Calendar, Thunderbird, Slack, Bodo (for Jira).

This is your cheat-sheet.

Setup your HTTP(s) Forensics

• Install the application you want to investigate (Yeah, a no brainer)
• Install a HTTP debugger, one of them (I use Charles Proxy)
• Proxyman (Mac)
• Charles Proxy (Mac, Win, Linux)
• Fiddler (Win)
• Configure your HTTP debugger to be able to analyse https traffic to your choosen calendar server
• Install PostMan and curl
• Have a place where you save your investigation results in Markdown, /docs in a GitHub repo is a good place
• Configure your HTTP debugger to intercept the http traffic on your target domain. This works different for each OS and debugger, so read the manual!
• SSL Proxying for Charles Proxy
• Decrypt HTTPS for Fiddler
• SSL Proxying with Proxyman
• Fetch a cup of your favorite beverage, we are good to go

Investigate

• Switch on your HTTP debugger (this line is sponsored by Captain Obvious)
• Execute one command in your custom app. Typically something like "login" or "add account" or "File Open" (that's my favorite in Office apps, fires 4 or more HTTP request when done against a http endpoint that understands webDAV)
• Look at the raw results. Each debugger has a nice gui that separates headers, cookies and body payload, but you want to look at raw data:
• Your request starts with METHOD /route HTTP_Version, e.g. POST /login HTTP/1.1 Everything until the first empty line is header, followed eventually with a body. Knowing methods helps to set expectations. See also RFC 7231, RFC 5789, RFC 2518, RFC 4918, RFC 3744 and RFC 4791
• Your response starts with a status line HTTP/1.1 StatusCode StatusMessage e.g. HTTP/1.1 200 OK Again: everything until the first empty line is header, followed by the optional response body
• It can get trickier when you app is already using HTTP/2.0 or later since it allows streaming or non-text payloads like gRPC
• Document that in markdown, bracket the http with three backticks, so it renders as source
• Repeat for other commands in your app

Re-enactment

What fun is detective work without verifying results. This is where you turn to Postman or if you have it - curl.

You want to use parameters for your hostname and the user specific parts (username, passwords) and you need to have a clear idea what are variable return values. Good candidtes to watch out for are cookies or header values. You need to extract these values for chaining to the next request. With a little practise you should be able to make Postman behave like the original app

Parting words

• Check the terms of service of the application you are investigating. While public end-point are, well, public, you might have agreed in your T&C not to touch them
• This isn't an instuction for hacking, you still need to be you - with your credentials. Nevertheless you might stumble over "security by obscurity" or other annoyances
• Any app that uses http instead of https needs to die a horrible death
• Reading the API spec is potentially faster

As usual YMMV

When dealing with large amounts of data, using Streams allows processing happen the moment data arrives, not just when data is complete. Streaming is core to reactive programming. This blog entry describes an approach where the vert.x EventBus sits between requester and resource

The scenario

Image created using WebSequenceDiagramms

title Vert.x EventBus

participant Requester
participant EventBus
participant DataSource

Requester->EventBus: request DataSet
EventBus->DataSource: forward request
DataSource->EventBus: reply with data
EventBus->Requester: forward reply

A requester (e.g. the handler of a HTTP Listener) sends a request via the EventBus using a request-response pattern with EventBus.request(...). Simple and easy. The problem with this: one request has one response. That doesn't work for streaming data.

Taking a page from the military

The standard pattern for military commands is:

1. Utter the command
2. Accnowledge the command
3. Execute the command (Conquer 14 countries, might take time. For Germans: Liberate 14 countries)
4. Report completion of command

Applying to the EventBus

Following the pattern above the first request/response only will establish the Intent (btw. Intent Based Leadership is a smoking hot topic). Item 2 and 3 will be handled by a publish and subscribe pattern.

So our scenario now looks like this:

Image created using WebSequenceDiagramms

title Vert.x EventBus Streaming

participant Requester
participant EventBus
participant DataSource

Requester->EventBus: start listening\non temp address
note over Requester, DataSource: Start request/response
Requester->EventBus: request Stream\notify on temp address
EventBus->DataSource: forward request
DataSource->EventBus: reply withaccnowledgement
EventBus->Requester: forward response
note over Requester, DataSource: End of request/response
note over Requester, DataSource: Start publish/subscribe
DataSource->EventBus: publish first data
EventBus->Requester: forward response
DataSource->EventBus: publish more data
EventBus->Requester: forward response
DataSource->EventBus: publish last data
EventBus->Requester: forward response
Requester->EventBus: end listening\non temp address
note over Requester, DataSource: End of publish/subscribe

To implement this, I'm taking advantage of EventBus' DeliveryOptions that allow me to set header values. I define a header StreamListenerAddress that my data source will use for publishing data:

// Error handling omitted
public void initiateStreamResponse(final String dataAddress, final JsonObject requestMessage, Promise<Void> didItWork) {
 final String streamListenerAddress = "tempListenerAddresses." + UUID.randomUUID().toString();
 final EventBus eventBus = this.getVertx().eventBus();
 final MessageConsumer<JsonObject> dataReceiver = eventBus.consumer(streamListenerAddress);
 dataReceiver.handler(handler -> {
  final boolean isFirst = Boolean.parseBoolean(headers.get("first"));
  final boolean isComplete = Boolean.parseBoolean(headers.get("complete"));
  /*
       Here goes the code feeding into the requester's logic e.g. a chunked HTTP response
       or a websocket publish or a gRPC push. isFirst and isComplete can be true at the
       same time when there is only a single response
  */
     ditItWork.complete();  
 });
 DeliveryOptions deliveryOptions = new DeliveryOptions();
 deliveryOptions.addHeader("StreamListenerAddress",streamListenerAddress);
 eventBus.request(dataAddress, requestMessage, deliveryOptions, ar -> {
   if (ar.succeeded()) {
    final Message<Object> resultMessage = ar.result();
    final boolean success = Boolean.parseBoolean(resultMessage.headers().get(Constants.HEADER_SUCCESS));
    if (!success) {
     consumer.unregister();
     didItWork.fail(new Error("Request for Data unsuccessfull"));
    }

   } else {
    consumer.unregister();
    didItWork.fail(ar.cause());
   }
  });
}

What next?

• In Part 2 I will describe the data source part of this approach
• In Part 2 I will wrap that in observable and observer

I'm using this pattern in the Keep API, YMMV