When solving problems for clients, I like to use worked examples to help me understand how well a particular technology might benefit the client.
You can have a live look at the PivotApp here. You can get a feel for the kinds of thing that the PivotApp does. I encourage you to give it a quick try before reading on. (If for some reason the link isn’t working, it probably means I’m doing some maintenance on it.)
Exercise your Data
Animate to See Outliers
The PivotApp leverages the D3 data-driven documents platform to create cool animated graphs. But this isn’t just animation for the sake of being cool; animation is great for seeing how outliers affect different aspects of the same data: a benefit sometimes called Object Constancy.
I Never Metadata DSL I Didn’t Like
The PivotApp has a module that encapsulates its business rules in a single JSON metadata object, using a mini-spec called a Domain-Specific Language, or DSL. This is really nice, for a couple of reasons:
- I want those business rules where I can see them! I don’t want them spread out all over the module, much less the entire app. Plus, I hate it when business rules are implemented by code. I’ve gotten a lot of mileage out of the concept of “data your app”–where possible, use data to make your app do something, not code.
- I want to be confident that I can apply a rule whenever I want. I can do this because my DSL is a general-purpose mini-language, and it doesn’t care which values I want to plot on a graph axis, for example.
MongoDB Is Nice Because It’s Fast
PivotApp uses MongoDB as its persistent store. It does this because it’s very useful to have a little experimental platform that gives us some performance data for comparison purposes.
For example, on the web, there are some interesting discussions about the relative performance aspects between the Hadoop ecosystem and MongoDB. By putting MongoDB in PivotApp, we can figure out if (say) Hadoop MapReduce is better/stronger/faster than MongoDB Aggregation Pipeline.
It’s all about verification and validation–looking before you leap! After all, both MongoDB and Hadoop (not to mention SQL-based stores) aren’t that easy to learn and deploy. In the best case, we do our own mini-analysis prior to committing lots of development resources or clustering hardware.
NodeJS Is Viable
A lot of shops have gone away from .Net and Java mid-tiers and moved to NodeJS. The PivotApp gives us some confidence that this strategy could be a winner.
I can tell you that when I use NodeJS, it’s hard for me to even tell if I’m writing server code or browser code. As Forrest Gump once said: “That’s good! One less thing.”
React/Redux and Functional Programming
PivotApp is built on React/Redux and adheres strictly to Functional Programming standards. By doing this, we can gather some further datapoints that tell us whether using pure functions is a good idea, or just a fad.
But even better: using FP lets us write more complete unit tests. Since everything is a function, it’s far easier for our test harness to mock the outside world (because there’s almost nothing in it ). Smaller unit tests, easier to write. That translates to faster and cheaper deliveries with less rework.
Experiments Are Critical
I’ll wrap this up with this claim: Any sufficiently complex problem needs some modeling and experimentation. The modeling should be “quick and dirty”–a tiny percentage of the overall cost of the software–but it should lower the project’s ignorance by giving us insight into where the hard problems really are.
This isn’t any different from other engineering disciplines, really. What’s strange is that there are so many software projects that just assume total knowledge up front.
Agile methodologies don’t solve this problem–they only give us rapid failure feedback (which isn’t bad, but it’s solving a different problem ).
You have to play in the problem space awhile before you can spot its boundaries and sharp corners!
This is something very cool which I have been envisioning to do for a quite while. This provides a rapid approach for data analysis and cleanup before ingestion. Would it be possible to integrate JSON forms into your architecture for describing & wrapping form rendering metadata based on the granularity with in a JSON collection that is directly related to the files being consumed? I want a model to be driven out of files metadata. For example, a file may have a long key such as Country, State, County, Gender, names & phone number. In this example, it might be possible to correlate country as a drop down whose selection drive another drop down to state, so on so forth and the details presented in a grid. This is one kind of metadata template but one should let the user choose the mapping based on the granularity disposition with in the collection. Also, I am not quite certain if DSL layer for business rules could be driven to attempt cross collection validation based on the metadata of relationships between collections. I hope I am not thinking too weird, some of it I did for a client and left to do other parts of the frame for a later date.
Yes, I think that JSON Forms might be a better metadata DSL than the one I came up with. It’s usually better to ride a popular Open Source wave, since then the forms DSL should give us leverage across multiple problem domains. And as you point out, my DSL doesn’t handle compound keys or key dependencies.
I haven’t tried to attack the cross-collection validation problem–however, I would look at leveraging joi browser to see how much mileage I could get out of it.