The totally inofficial guide to Verse on Premises

Now that CNGD8ML is upon us, it is story time. Read about the why, who, what and what to watch out for.

To successfully deploy Verse, make sure to carefully read and implement the installation instructions. The availability of Verse makes Domino the most versatile eMail platform around, offering you the choice of: Notes Client, Outlook, POP2, IMAP4, iNotes, Verse, iOS, Android. Anywhay, here we go:

The back story

Verse on premises was a long (out)standing promise to the IBM customer base. Not everybody is ready to embrace the cloud, but interested in the new way to work. In SmartCloud Notes, the backend for Verse in the Cloud, all search is powered by Apache SOLR. If Verse got delivered as is, that would have required substantial hardware and skill investments for the on-premises customers.

So I made a bet with Michael Alexander, whom I worked with on TPTSNBN, that we could use standard Domino capabilities, not requiring Solr. Based on prototypes with vert.x and Java8 we gained confidence and got the go ahead to build the search component as OSGi plug-in (in Java6). So the search part (not the UI or other functionality) is on me.

The team(s)

There were two distinct teams working on the delivery of Verse on Premises (VoP): The core Verse team, that owns UI, functionality and features for both cloud and on premises and the search plugin team responsible to replace the Solr capabilities with native Domino calls.
The former is rather large, distributed between the US, Ireland and China. The later was led by the distinguished engineer David Byrd and just a few core coding members: David, Michael, Christopher, Raj and myself.
We were supported by a team of testers in Belarus and the Philippines. The test teams wrote hundreds of JUnit and Postman tests, just for the search API.

The Orangebox

Each project needs a good code name. The original Verse code name was Sequoia, which is reflected in the name of the plugins for core and UI functionality.

The search component, not being part of RealVerse™, needed a different name. In an initial high level diagram, outlining the architecture for management, the search component was drawn as an orange box. Since we "just" had to code "the orange box". The name stuck and led to our code name "Project OrangeBox" (PoB).

You can find Orangebox and POB in multiple places (including notes.ini variables and https calls the browser makes). So now you know where it is coming from.

The challenges

We set out to build POB with the following constraints:

• No changes to the Verse UI code
  The UI makes heavy use of JAGR for aggregation and sends SOLR syntax queries over HTTPS. Short of the message bodies almost everything the Verse client requests is a SOLR search query, including opening the inbox.
  For your enlightenment: Open the developer tools in Chrome and watch what is transmitted over the wire.
  The UI code is completely identical to the code running in the cloud, no splitting of code streams was a design mandate - and no changes from the running production code
• Java8 would not be ready in time
  So no Vert.x library, no RxJava (it uses threads heavily and that kills the Domino Java6 JVM performance), no lambda functions. Initially I used RxJava to process resultsets into the required JSON. However that didn't survive our load tests, mainly attributed to the JVM struggling with rapid thread transitions
• Only public APIs
  Any of the calls made from the search plugin to Domino use only publicly documented Java APIs. Since we frequently load tested the application, I learned a big deal on behaviour under heavy load. One example: My initial approach was to use a list of item names I needed and then use document.getFirstItem in a loop to retrieve them. Turns out, under heavy load it is faster to use document.getItems and loop through those, picking the values I needed
• Light on template changes
  In VoP 1.0 the only changes to the mail template are a few additional views. So you can keep your modified R9 template (eventually) for now - unless you messed with the search order in views and folders.
  However the Verse Client requires data for its list column, that is not in any current view (column). As a result we have to read each and every document that's displayed in the list. Nirmala our Chief of Performance made us work hard to get this fast.
  One of the nice side effects of this effort: Jim Cooper, C expert in the core Domino team, provided code updates (also included in the upcoming FP8), that dramatically improve read performance for any document - a big bonus for all Domino based applications

To solve the challenges I devised an architecture where any incoming query would run through code determining the execution strategy. If determined feasible, VoP would just read a view or folder (eventually with a category and a date limit).
When that wouldn't fit we hand it off to a fulltext query. Raj designed and coded the Query parser that, for the use cases of VoP, translates the SOLR query syntax into Notes FTSearch syntax. Quite a feast in code, kudos to Raj!

Operating it

Make sure, you read the instructions carefully. There are a number of preconditions to be met. Missing any of these and VoP will not work for that users. My suggestion to auto-fix some of them didn't make it into VoP 1.0

• Your server needs to have https active (and please: no self-signed certificates)
• There are a few extra design elements needed. Instead of providing a complete new template, you can continue to use your existing template and just add the new views. Only catch: if you have messes with the sort order, you need to reverse that changes. Want to be on the save side? Use the stock 9.0.1 mail template as starting point!
• The mail files need to have fulltext index active (typically a given for iNotes shops, mostly off for Notes shops)
• The special: The databases need to have folder references enabled. Read the instruction (16g) on that one carefully
• The mail server needs designer or manager access to the mail files
• In a cluster: the clustered mail databases must have the same file path (relative to the data directory) and name on all servers

Watch the log

The users will not get error indicators beyond: "Talk to your admin", but the log files (the one in workspace/logs) will capture details what is missing. It will also log any other errors that might arise. VoP was subjected to thousands of automation test as well as human testing. Nevertheless, it is software, you know where this is going. The PMR system is your friend (and ours too)!

Under the hood

Watching the HTTPS traffic is quite enlightening. I usually use the Chrome tools for it. Go and watch!

To be super clear: Nothing of the data going forth (HTTPS Form submissions) and back (JSON) is documented or guaranteed to stay and can change in any future version (Very much like iNotes, see my articles about that). So don't count on it.

For adventurous souls: the search plugin is using Apache WINK which gives some information away at its base URL /pob/api/search. Go and check it out. Some of the URLs you will find there:

• /pob/api/search/inbox
  Returns an extended inbox in JSON format
• /pob/api/search/suggestimportant
  Lists the people Verse believe belong into
• /pob/api/search/needsaction
  Incoming mail that has been flagged as "Needs Action"
• /pob/api/search/waitingform
  outgoing mail that has been flagged as "Waiting for"
• If you happen to be a member of the LocalDomainAdmins or OrangeBoxAdmins group, there are a few more, which are secret and undocumented - and can't be subject to a PMR:
  • /pob/api/search/info
    Shows the version number and some cache information
  • /pob/api/search/monitor
    Real time statistics on search performance (also to be seen in the server stats!)
  • /pob/api/search/parser?query=something
    Takes a SOLR syntax query and shows its Domino FTIndex equivalent. Keep in mind: it isn't an universal translator, but tailored to Verse use cases (you don't want to learn SOLR syntax anyway)
  • /pob/api/search/design
    Various options like show/update design information or extract the current "Verse additional design elements" Template:
    • ?createtemplate=1 extracts the template with the additional Verse design elements. Fresh and crispy
    • ?showviews=1 renders the view design of the additional Verse views for inspection (to check if they are right)
    • ?replicaids=current submits a background job creating all Verse preconditions (FTIndex, Views, FolderRef) in the current database
    • ?replicaids=id,id,id same as above, but for the databases with the given replica id
    • ?serverwide=1 samee as above, but for all mail files (pulled from names.nsf) on that server. Do this only after a backup. You have been warned! Actually don't do this. It's not supported
    Use on your own risk
  As mentioned: Use at your own peril! (and not on a production system!)

As usual YMMV