When solving problems for clients, I like to use worked examples to help me understand how well a particular technology might benefit the client.

It’s for this reason that I created and Open Sourced the PivotApp. This app is very useful for Javascript and NodeJS technology evaluation, because it does some interesting and non-trivial things (as opposed to the relatively useless, ubiquitous, and canonical ToDo App).

We’re Live!

You can have a live look at the PivotApp here. You can get a feel for the kinds of thing that the PivotApp does. I encourage you to give it a quick try before reading on. (If for some reason the link isn’t working, it probably means I’m doing some maintenance on it.)

Exercise your Data

It’s one thing to create a simple data storage or data retrieval app. It’s another thing to pivot it, bin it, date-order it, aggregate it, ‘facet-ize’ it, and adjust it so that it can be further manipulated in-memory and on the browser. The PivotApp does all these things, giving us some confidence that the underlying JavaScript techniques (including Functional Programming and Metadata DSLs) work.

Animate to See Outliers

The PivotApp leverages the D3 data-driven documents platform to create cool animated graphs. But this isn’t just animation for the sake of being cool; animation is great for seeing how outliers affect different aspects of the same data: a benefit sometimes called Object Constancy.

I Never Metadata DSL I Didn’t Like

The PivotApp has a module that encapsulates its business rules in a single JSON metadata object, using a mini-spec called a Domain-Specific Language, or DSL. This is really nice, for a couple of reasons:

1. I want those business rules where I can see them! I don’t want them spread out all over the module, much less the entire app. Plus, I hate it when business rules are implemented by code. I’ve gotten a lot of mileage out of the concept of “data your app”–where possible, use data to make your app do something, not code.
2. I want to be confident that I can apply a rule whenever I want. I can do this because my DSL is a general-purpose mini-language, and it doesn’t care which values I want to plot on a graph axis, for example.

MongoDB Is Nice Because It’s Fast

PivotApp uses MongoDB as its persistent store. It does this because it’s very useful to have a little experimental platform that gives us some performance data for comparison purposes.

For example, on the web, there are some interesting discussions about the relative performance aspects between the Hadoop ecosystem and MongoDB. By putting MongoDB in PivotApp, we can figure out if (say) Hadoop MapReduce is better/stronger/faster than MongoDB Aggregation Pipeline.

It’s all about verification and validation–looking before you leap! After all, both MongoDB and Hadoop (not to mention SQL-based stores) aren’t that easy to learn and deploy. In the best case, we do our own mini-analysis prior to committing lots of development resources or clustering hardware.

NodeJS Is Viable

A lot of shops have gone away from .Net and Java mid-tiers and moved to NodeJS. The PivotApp gives us some confidence that this strategy could be a winner.

There’s just less code that you have to write. Plus, the same JavaScript code can run on the server as in the browser, a capability called Universal JavaScript.

This is really nice, too. So-called “full stack” programmers are perfectly capable of shifting gears between C#, Java, and JavaScript. But that doesn’t mean they should!

Let’s say you’ve committed your shop to Functional Programming. The techniques for doing this in Java look very, very different from the equivalent JavaScript techniques. Yes, a sufficiently bright programmer can do both. But why not just lower that conceptual burden?

I can tell you that when I use NodeJS, it’s hard for me to even tell if I’m writing server code or browser code. As Forrest Gump once said: “That’s good! One less thing.”

React/Redux and Functional Programming

PivotApp is built on React/Redux and adheres strictly to Functional Programming standards. By doing this, we can gather some further datapoints that tell us whether using pure functions is a good idea, or just a fad.

Previous versions of the PivotApp ran on the JQuery and AngularJS platforms. Having done all of them, I can say without reservation that the ReactJS incarnation of PivotApp is easier to read and maintain. Mostly this is because of further lowering of conceptual burden: using the least JavaScript syntax is a good thing.

But even better: using FP lets us write more complete unit tests. Since everything is a function, it’s far easier for our test harness to mock the outside world (because there’s almost nothing in it ). Smaller unit tests, easier to write. That translates to faster and cheaper deliveries with less rework.

Experiments Are Critical

I’ll wrap this up with this claim: Any sufficiently complex problem needs some modeling and experimentation. The modeling should be “quick and dirty”–a tiny percentage of the overall cost of the software–but it should lower the project’s ignorance by giving us insight into where the hard problems really are.

This isn’t any different from other engineering disciplines, really. What’s strange is that there are so many software projects that just assume total knowledge up front.

Agile methodologies don’t solve this problem–they only give us rapid failure feedback (which isn’t bad, but it’s solving a different problem ).

You have to play in the problem space awhile before you can spot its boundaries and sharp corners!