wissel.net

Usability - Productivity - Business - The web - Singapore & Twins

From Blogsphere to a Static Site (Part 4) - Comment backend


The blog needed a comment function. While there are social options around (Facebook, Disqus etc), I decided I want to roll my own. Partly because I want tighter control and partly, well, because I could. My comment backend would:

  • Provide a REST API to create comments in a JSON structure. The comment body will be Markdown. Reading would provide comments in ready to use HTML (I hear howling from the API crowd). No delete or update functionality
  • Cleanup content considered harmful (code injection) and optional sport Captcha
  • Store all content in a NoSQL database, in my case CouchDB (or Cloudant with its 20G free plan)
  • Cache all queries for comment in an online cache to limit calls to the database
  • Initially run on Domino, later on liberty or the raw JVM
  • Initially also update Domino using a web service - so during transition no comments would get lost

In its initial incarnation the Comment servlet is a OSGi plugin that listens to the /comments URL implemented as Wink servlet. So the class of interest is the one defining the service. We have one method for post, one for get and a helper function

/**
 * Wink implementation of Comment service
 */
@Workspace(workspaceTitle = "Blog Comments", collectionTitle = "Create or display comments")
@Path(value = "/comments")
@Produces(MediaType.TEXT_HTML)
@Consumes(MediaType.APPLICATION_JSON)
public class CommentService extends CommentResponse {

    private final Logger logger = Logger.getLogger(this.getClass().getName());

    @POST
    public Response createComment(@Context HttpServletRequest request) {
        final Monitor mon = MonitorFactory.start("CommentService#createComment");
        String result = "Sorry I can't process your comment at this time";
        ResponseBuilder builder = Response.ok();

        try {
            InputStream in = request.getInputStream();
            BlogComment comment = BlogComment.load(in);
            in.close();

            if (comment != null) {
                this.captureSubmissionDetails(request, comment);
                result = CommentManager.INSTANCE.saveComment(comment, true);
            } else {
                builder.status(Status.NOT_ACCEPTABLE);
            }
            builder.entity(result).type(MediaType.TEXT_HTML_TYPE);

        } catch (Exception e) {
            String errorMessage = e.getMessage();
            builder.entity((((errorMessage == null) || errorMessage.equals("")) ? "Undefined error" : errorMessage)).type(
                    MediaType.TEXT_HTML_TYPE);
            Utils.logError(this.logger, e);
        }

        mon.stop();
        return builder.build();
    }

    @GET
    public Response getComments(@QueryParam("parentid") final String parentid) {
        Response response = null;
        final Monitor mon = MonitorFactory.start("CommentService#getComments");
        final ResponseBuilder builder = Response.ok();
        final Collection<BlogComment> bc = CommentManager.INSTANCE.loadComments(parentid);

        if ((bc == null) || bc.isEmpty()) {
            builder.status(Status.NO_CONTENT);
        } else {
            response = this.renderOutput(bc, "comment.mustache");
        }
        mon.stop();
        return (response == null) ? builder.build() : response;
    }

    private void captureSubmissionDetails(HttpServletRequest request, BlogComment comment) {
        final Monitor mon = MonitorFactory.start("CommentService#captureSubmissionDetails");
        try {
            @SuppressWarnings("rawtypes")
            Enumeration hn = request.getHeaderNames();
            if (hn != null) {
                while (hn.hasMoreElements()) {
                    String key = hn.nextElement().toString();
                    comment.addParameter(key, request.getHeader(key));
                }
            }
            @SuppressWarnings("rawtypes")
            Enumeration pn = request.getParameterNames();
            if (pn != null) {
                while (pn.hasMoreElements()) {
                    String key = pn.nextElement().toString();
                    String[] values = request.getParameterValues(key);
                    comment.addParameters(key, values);

                    if (key.equals("referer")) {
                        comment.setReferer(values[0]);
                    } else if (key.equals("user-agent")) {
                        comment.setUserAgent(values[0]);
                    }

                }
            }
            @SuppressWarnings("rawtypes")
            Enumeration an = request.getAttributeNames();
            if (an != null) {
                while (an.hasMoreElements()) {
                    try {
                        String key = an.nextElement().toString();
                        comment.addAttribute(key, String.valueOf(request.getAttribute(key)));
                    } catch (Exception e) {
                        // No action here
                    }
                }
            }
            comment.addParameter("REMOTE_HOST", request.getRemoteHost());
            comment.addParameter("REMOTE_ADDR", request.getRemoteAddr());
            comment.addParameter("REMOTE_USER", request.getRemoteUser());

            // Needed for Captcha
            comment.setRemoteAddress(request.getRemoteAddr());
        } catch (Exception e) {
            Utils.logError(this.logger, e);
            // But no further action here!
        }
        mon.stop();
    }
}

Read more

Posted by on 2017-05-04 07:32 | Comments (0) | categories:

The Decline and Fall of IBM


I wrote this quite a while ago, never finished the article until now. Enjoy.

Yeah right!

With this words Robert X Cringley a.k.a Mark Stephens celebrates himself when he asserts something clever. His book The Decline and Fall of IBM created quite some stir and was a hot topic of discussion especially among IBMers and alumni.
So I got myself a copy and had a look. Clearly he has an axe to grind with IBM and everybody is invited. Many came. Half of the book consists of mostly grieving comments ranging from 2007 until 2014. With all this contributions, the content remains light on substance. After all it costs you less than a late.

Some of the stated observations are spot on, like "IBM is a sales organisation", others while looking like observations, but rather are opinions (e.g. "IBM lost its way") and statements that made me feel: "why didn't the IBM board ask him to run IBM? He seems to know so much better!", exactly like an arbitrary spectator of the world cup in a pub can tell you what a team did wrong.

I'm not saying, that all is well in IBM, that would be a fools view. An organisation with a size and workforce exceeding several countries does live in challenging times. An economic system, that values growth over everything is problematic (In biology an organism that grows indefinitely is called cancer) at least and transiting to a global workforce on this scale is unparalleled. I share his view that the dance around the golden calf known as shareholder value might be praying to a false good.
Here are some comments to Cringley's statements, that crossed my mind (paraphrasing):

  • Cringley: "In IBM management is royalty, the sales force the nobility and technical people are the peasants. No peasant can dream to become royalty".
    Yeah right! IBM fellows, IBM Distinguished Engineers or members of the IBM Academy of Technology will disagree. The irony here: in the "good old IBM (of Watson)" there was no career path for technical experts. In the IBM of today there is
  • I worked a lot with our engineers in India and China. They are hard working, ready to learn and, by now, quite experienced. Asserting that they are less capable that their American counterparts, seems quite arrogant to me. Yes, they were unexperienced a decade ago, but that's a long time in IT years. Also: there are capable and incapable engineers everywhere. Pinning it on a specific country or region is (insert your own statement of backwards here). A real issue however are IBM's processes, that had been designed to cater to get less experienced people on board. They need an overhaul.
  • Cringley: "IBM should not sell the Intel servers to Lenovo".
    Yeah right! Cringley portrays it as the complete exit out of a server growth market. However IBM still has Intel based technology in their PureSystems and acquired skills and know how through the Softlayer acquisition how to build the special segment of Intel based machines that run in cloud size data centres. So instead of exiting a segment, it looks to me like eliminating duplicate product lines
  • Cringley: "IBM should port AIX to Intel".
    Yeah right! AIX runs on one class of IBM machines (System P), while Linux runs on everything from Softlayer Bluemix to Mainframe. Linux outperforms AIX on System P in quite some workloads. So what makes a better investment? Port AIX or infuse the security know how of AIX into Linux? (the file system options are there already)
  • Cringley: "IBM should ditch the Power architecture and switch to Intel"
    Yeah right! IBM has vast know how in building processors that will get lost when stopping to develop them. Asset utilisation isn't ditching assets, but making them more competitive. Intel and ARM aren't the only shops who can design processors, just have a look at OpenPOWER
  • Cringley: Hadoop will make mainframes obsolete.
    Yeah right! The technology is around for a decade, Google, its inventor, already moved on. Asserting a single technology will kill the I/O beasts known as Big Iron, looks like an inflight magazine statement. Besides the fact, that IBM offers a robust Hadoop implementation for a while already, which you can run on zLinux, if you choose so.
  • The book contains quite some areas, like current staff morale, the process culture or the layers of management that highlight pain. However where he is wrong: IBM isn't blind to the issues and there are forces inside working for the betterment (even if there is a case of a Knowing-Doing-Gap)
Of course, YMMY, so read it for yourself

Posted by on 2017-05-01 12:35 | Comments (0) | categories: IBM

From Blogsphere to a Static Site (Part 3) - Generating pages


The rendering engine I choose is mustache which describes itself as "logic-less templating engine". My main criteria was the availability on multiple platforms including Java and JavaScript (I might port the rendering part to NodeJS at some time in the future).
The only logic mustache supports is conditional rendering based on the presence or absence of an element. When an element is present and is an array (or a collection in Java) the body of the template gets repeated for each element in the array. A scalar value hence is treated as an array with one value only.

Mustache is simple to use. All you need is a data bean (in Java, a JSON structure for JavaScript) and a text file containing placeholders with the property names. E.g. <h1>{{title}}</h1> will render a headline with the title property of you data object. In Java that would be either a public variable or a call to getTitle according to the bean specification. The blog rendering code therefore is quite simple:

private void renderOneEntry(BlogEntry be, Mustache mustache) throws IOException {

        String location = this.config.destinationDirectory + be.getNewURL();
        String outDirs = location.substring(0, location.lastIndexOf("/"));
        File dirs = new File(outDirs);
        if (!dirs.exists()) {
            dirs.mkdirs();
        }
        // Set the current context
        for (LinkItem cat : be.getCategory()) {
            String c = cat.name;
            this.allCategories.get(c.toLowerCase()).active = true;
        }
        this.allDateCategories.get(be.getDateYear()).active = true;

        if (be.getSeries() != null) {
            String series = be.getSeries();
            if (this.allSeries.containsKey(series)) {
                this.allSeries.get(series).get(be.getNewURL()).active = true;
            }
        }

        // Prepare to write out
        ByteArrayOutputStream out = new ByteArrayOutputStream(102400);
        Writer pw = new PrintWriter(out);

        // This is where the magic happens
        mustache.execute(pw, be);
        pw.flush();
        pw.close();
        this.saveIfChanged(out.toByteArray(), location);

        // Cleanup
        for (LinkItem cat : be.getCategory()) {
            String c = cat.name;
            this.allCategories.get(c.toLowerCase()).active = false;
        }
        this.allDateCategories.get(be.getDateYear()).active = false;

        if (be.getSeries() != null) {
            String series = be.getSeries();
            if (this.allSeries.containsKey(series)) {
                this.allSeries.get(series).get(be.getNewURL()).active = false;
            }
        }
    }

The actual rendering is just the line mustache.execute(pw, be); The code around it prepares and resets the collections that might render on a page like categories, series or month and year. Also of interest is this.saveIfChanged(out.toByteArray(), location); which only saves results back to disk if it actually has changed. Don't be mistaken: any change in layout will lead to a newly rendered page, so this is quite important to save as needed and not more (you don't want to have tons of identical files that only differ in their time stamp)


Read more

Posted by on 2017-05-01 11:23 | Comments (0) | categories: Blog

From Blogsphere to a Static Site (Part 2) - Cleaning up the HTML


Blogsphere allows to create RichText and plain HTML entries. To export them I need to grab the HTML, either the manual entered or the RichText generated on, clean it up (especially for my manual entered HTML) and then replace image sources and internal links using the new URL syntax. To make this happen I created 2 functions that saved images and attachments and created a lookup list, so the HTML cleanup has a mapping table to work with

private void saveImage(Document doc) {
        String sourceDirectory = this.config.sourceDirectory + this.config.imageDirectory;
        try {
            String subject = doc.getItemValueString("ImageName");
            Date created = doc.getCreated().toJavaDate();
            @SuppressWarnings("rawtypes")
            Vector attNames = this.s.evaluate("@AttachmentNames", doc);
            String description = doc.getItemValueString("ImageName");
            String oldURL = this.config.oldImageLocation + doc.getItemValueString("ImageUNID") + "/$File/";
            SimpleDateFormat sdf = new SimpleDateFormat("yyyy");
            String year = sdf.format(created);
            FileEntry fe = this.imgEntries.add(subject, oldURL, description, created);

            for (Object attObj : attNames) {
                try {
                    String attName = attObj.toString();
                    String newURL = this.config.webBlogLocation + this.config.imageDirectory + year + "/" + attName;
                    fe.add(attName, newURL, description, created);
                    String outDir = sourceDirectory + year + "/";
                    this.ensureDirectory(outDir);
                    EmbeddedObject att = doc.getAttachment(attName);
                    att.extractFile(outDir + attName);
                    Utils.shred(att);
                } catch (NotesException e) {
                    e.printStackTrace();
                } catch (Exception e2) {
                    e2.printStackTrace();
                }
            }

        } catch (NotesException e) {
            e.printStackTrace();
        }

    }

    private void saveImageFromURL(String href, String targetName) {

        String fetchFromWhere = "https://" + this.config.bloghost + href;
        try {
            byte[] curImg = Request.Get(fetchFromWhere).execute().returnContent().asBytes();
            this.saveIfChanged(curImg, targetName);
        } catch (ClientProtocolException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

    }

With images saved the HTML cleanup can proceed. As mentioned before I'm using JSoup to process crappy HTML. It allows for easy extraction of elements and attributes, so processing of links an images is just a few lines


Read more

Posted by on 2017-04-17 09:29 | Comments (0) | categories: Blog

From Blogsphere to a Static Site (Part 1) - Objects


The migration tooling revolves around data, so getting the data classes right is important. The data classes need to be designed to allow it to be populated either from the Blogsphere NSF or from a collection of JSON files (so the blog generation can continue when the NSF is gone). For the blog we need 3 objects:
  • BlogEntry: The main data containing a blog entry including its meta data
  • BlogComment: An entry with a comment for a Blog in a 1:n relation
  • FileEntry: information about downloadable files (needed for export)

There will be auxiliary data classes like Config, RenderInstructions, Blogindex. Their content is derived from the data stored in the main object or, in case of Config, from disk.

Data classes in the Blog

Read more

Posted by on 2017-04-15 04:06 | Comments (0) | categories: Blog

From Blogsphere to a Static Site (Part 0) - Requirements


Readers of this blog might have noticed, that blog layout and blog URLs have changed (a while ago). This blog now serves static HTML pages using a nginx web server (get used to nginx, it's coming in Connections Pink too). I will document the steps and code I used to get there. Step 0 is: define the requirements and evaluate the resulting constraints:
  • Export of all Blogsphere content to HTML, including the conversion of entries written in RichText
  • No export of configuration or layout required
  • New site structure that shows articles in year and month folders
  • Modular templating system with includes for repeated pieces (e.g. header, footer, sidebar)
  • Summary pages for year, month and categories
  • Summary page for article series
  • Existing comments to become part of the html page
  • Repeatability: export must be able to repeat, but not overwrite a page that hasn't actually changed
  • Storage of exported pages in a file structure as JSON files
  • Rendering of static site from Notes or from JSON directory
  • Redirection file, so old links get a proper redirection to the new URL
  • Have a comment database for new comments
  • No pagination for any of the summary pages (I might change my mind on that one)
  • Cleanup messy HTML automatically, fix syntax and URLs to posts and images
  • Optimized HTML, CSS and JS for speedy delivery
I had a look at Jekyll, which is the templating engine GitHub is using. I would have allowed me to just commit a new file and Github would render for me. Unfortunately Jekyll fell short of the article series and category overview pages.

Read more

Posted by on 2017-04-12 11:00 | Comments (0) | categories: Blog

Project Deep Purple - IBM Notes Native for iOS


We all heard the announcements around Project Pink headed by Jason R. Gary, the future of IBM Connections. Attending the conference we could admire him all clad out in pink.

However there is something more going on (and I'm not talking about his "don style" haircut). Besides the pink suit, Jason has been spotted in a deep purple jacket on several occasions, like his FossASIA talk. Digging deeper it seems IBM collaboration projects are now colour coded. Purple is chosen quite deliberately: The color purple is a rare occurring color in nature and as a result is often seen as having sacred meaning. Purple combines the calm stability of blue and the fierce energy of red. The color purple is often associated with royalty, nobility, luxury, power, and ambition. Purple also represents meanings of wealth, extravagance, creativity, wisdom, dignity, grandeur, devotion, peace, pride, mystery, independence, and magic.

You might have guessed it: it is the next generation of IBM Notes! Not one of the fix feature packs, but an entire new generation. Under the influence Jason admitted: "We took the Notes Client source code and compiled it with XCode for iOS. Guess what: it worked. Cocoa needs some work, but it isn't rocket science"

There you have it! After banning Notes clients from your desktop, instead of relying on the IBM Client Application Access, you just can launch IBM Notes Native for iOS™ on your iPad and continue working. Any improvements and updates are automatically rolled out using the Apple app shop. Availability will be April 1st, 2018.

We live in interesting times!


Posted by on 2017-04-01 10:23 | Comments (1) | categories: IBM Notes

Goodbye IBM, hello Salesforce!


The Ministry of Manpower in Singapore is running a campaign "A new career at 55". Intrigued by it, I decided to give it a shot.

I will be joining Salesforce in Singapore as Cloud Solution Architect this Monday.

My 11 year tenure in IBM thus came to its end. With the new co-location policy sweeping though IBM, I realised, that staying in Singapore will not get me any closer to Notes than the December delivery of Verse on premises. Moving with my offspring in JC wasn't an option.

Working with the "Yellow bubble" always was fun and I intend to continue to participate there. Over the years the community propelled me to one of the top XPages experts on Stackoverflow, adopted my word creation XAgents and always made me feel welcome.

I had the opportunity to contribute code back to the community via OpenNTF on github. Check them out:

  • DominoDAV
    A webDAV implementation for Domino attachments. It allows you to fully round-trip edit office documents in a browser. It is extensible, so you could make views look like spreadsheets etc.
  • Swiftfile Java for Notes
    We had to pick a different name (AFSfNC) to add to the confusion. The project is a Java plugin implementation of Swiftfile, the little tool that would predict what folder you would file a message to. In todays lingo one would call it: Cognitive tag prediction (in Notes Folders and Tags could be used interchangeable)
  • Out of Office
    a Rest API that allows to check the OOO status of a given user
  • DominoRED
    Linking Domino and NodeRED. Very much work in progress

So let the adventure "From sensei to n00b" begin. See you on the other side!


Posted by on 2017-04-01 01:02 | Comments (e) | categories: IBM Salesforce

@Formula on JSON


When you look at "modern" programming styles, you will find novel concepts like isomorphic (runs on client or server), Idempotency (same call, same result), Immutable (functions never mess with the parameters or global state) or map operations (working on a set of data without looping).

I put "modern" deliberately in quotes, since these ideas have been around since Lisp (or for the younger of you: since you sorted all blocks by colour and size in kindergarden). In the Lotus world we got our share of this with the venerable @Formula language (the functions, not the commands), IBM Notes inherited from Lotus 1-2-3. While it has served us well, so far it has been confined to the realm of the NSF.

Not any more! Thanks to Connections Pink and the ever ingenious Maureen Leland, @Formula will come to a JSON structure near you soon. As far as I understood the plan: each @Function will serve as an endpoint to a (serverless) microservice that executes on values provided, returning a new value object that can be chained to the next call stream style. I'm very excited about this new development. Watch out for news about Connections Livegrid™.

Time for Maureen to undust her blog!


Posted by on 2017-04-01 12:19 | Comments (1) | categories: IBM Notes

Agile Outsourcing


The problem

Outsourcing is a "special" animal. Typically the idea is to save cost by letting a service provider execute work. The saving cost happens because the service provider is supposed to be able to perform this actions at scale. Increasingly outsourcing deals are motivated by a skill squeeze. So instead of maintaining in-house expertise, rely on the vendors to keep the light on.
This is where the trouble starts. Negotiations on outsourcing contracts revolves around price (drive it down) and the SLA (add as many 9 behind the comma as possible). The single outcome of such contracts is extreme risk aversion. For illustration here is the impact of SLA levels :
SLA Total annual Downtime
98% 7 days, 6h, 12min
99% 3 days, 15h, 36min
99.9% 8h, 45min, 36sec
99.99% 52min, 34sec
99.999% 5min, 16sec
99.9999% 32 sec
The fixation on SLA has a clinical term: OCD. Any change is considered as dangerous as someone holding a knife at your throat and asked you to dance.
Looking at some of the figures (I can't share) I would claim that short of highly parallel (and expensive) transaction system anything above 99.9% is wishful thinking. That doesn't deter negotiators to aim for a "look how many 9th I got" trophy. (While the Buddha reminds us: one cause of suffering is to close your eyes to reality). Expensive SLA violation clauses let outsourcers freeze all system, since any change (read: patches, upgrades, enhancements) is rightly identified as grave risk (to the profits).
So all sorts of processes and checks get implemented to vet any change request and in practise avoid them.
This usually leads to a lot of bureaucracy and glacial progress. As a result discontent, especially on the use of non-transactional system grows: Stuff like outdated eMail clients, lack of mobile support etc. etc.
The relation between oursourcer and oursourcee grows, inevitably, challenging over time. Does it have to be that way?

Some fresh thinking

Just move to cloud might not be the answer (or everybody would be there, it's such a nice place). So what could be done? Here are some thoughts:
  • Kiss goodby the wholesale SLA agreement. Classify systems based on business impact. A booking system for an airline surly deserves three nines (I doubt that four would make sense), while a website can live with one nine (as long as it distributed over the year)
  • Take a page from the PaaS offerings: each element of the environment has a measurement and a price. So the outsourcing provider can offer ala card services instead of freezing the environment. A catalogue entry could be "Running a current and patched DB/2", another entry could be "Run a legacy IIS, version xx"
  • Customer and provider would agree on an annual catalogue value, based on the starting environment and any known plan at the time
  • The catalogue would allow to decommission unneeded system and replace them with successors without much hassle (out with PHP, in with node.js)
  • Automate, Automate, Automate - An outsourcer without DevOps (Puppet, Chef and tight monitoring) didn't get the 2017 message
  • Transparency: Running systems over processes, Customer satisfaction over unrealistic SLA, Automation over documentation (I hear the howling), Repeatable procedures over locked down environments
What do you think?

Posted by on 2017-02-08 07:00 | Comments (3) | categories: Software