The constrained visions of analytics.
Let’s suspend reality and take the following scenario to start us off. The starship Enterprise has dropped out of warp in an unmapped part of the galaxy. Captain Kirk, the CEO of the ship, has one thing on his mind. “I need to get the nearest green skinned lady at all costs?” He is goal oriented and the means by which it will be achieved is of minor importance as long as he achieves his mission for the benefit of the federation. He believes that by following through with his plan he is doing good by interacting with new alien worlds and only he has the ability to do this in such a way. Such an unconstrained view of processes. Spock, his next in command, is however on the away missions too. He sees this happening, again, and endeavours to get Kirk to follow the prime directive so that any new interactions will follow the approved processes. He knows through experience that he shouldn’t allow Kirk to charge forward but instead follow the processes and rules of which they have agreed on before. He is not so much worried about what they do as to how they do it for he knows Kirk is a limited, flawed human and hasn’t thought about the wider consequences. The constrained view. Here we have in a microcosm the opposing visions of two modes of thinking. The upside to Kirk’s vision is that they are constantly interacting with new worlds, building relationships and gaining new experiences. The downsides are that by not following the approved processes he is putting himself and his crew in danger and inevitably the expendable extra will take the fall.
Back on terra firma however it is difficult to work in an analytics job strictly within the confines of a purely constrained or unconstrained view without becoming the victim of our own ideological bent. Kirk had it easy. He was the fat emperor handing out commands with no thoughts as to the consequences. For us it is very much different. To be 100% constrained in our business all we would think about is process, following guidelines and painting by numbers. To be 100% unconstrained the results, which would be all that matters would come at the cost of quality, traceability and consensus. We could of course strive every time for the beautifully exotic alien but at what cost? What employment tribunal would stand up for us?
In the world in which we work balance is everything. The constrained view is the trap holding back the greyhound, not allowing the passion and raw energy blast its way towards the goal. Conversely when the trap is opened we cannot take the fully goal oriented, base instinct ‘must catch the rabbit at all costs’ approach either. This would result in blindly slobbering towards the rabbit, climbing over the fallen to be the first to the end, surely resulting in very low-quality, unaccountable work. The balance between the two is the ideal approach but the balance isn’t fixed. It is the pendulum constantly to-ing and fro-ing between visions. All the way to one side we are too constrained, all the way to the other side we are too unconstrained. Controlling this or more importantly finding its rhythm, and knowing when to blast towards the end and when to take the methodical approach, is where we find consistent results blossom.
In any aspect of work where there is a need for out-of-the-box thinking (as in our analytical lives) a certain amount of single-minded confidence in your own vision is necessary. This allows us to break the mould and envision things beyond the scope of what came before. However, when the adrenaline rush wears off and getting to that vision is necessary, predetermined and analytical processes must be followed, notes must be taken so as to make traceable the thought process and documents must be created to allow others to follow it again if necessary. Any number of CRISP-DM or KDD methodologies vies for dominance here.
In dealing in other company’s data the customer initially has a dominantly unconstrained vision. The message that usually comes from the client is, “Show us our business as we have never seen it before.” The process by which we do this is largely unimportant to them to begin with as they just want results and they want them by whatever means. So we take on board their data to show them what the possible is. For us, on receiving their data for a proof of concept, the results are the first thing that comes to mind and thus an unconstrained vision is also the first point of call.
‘This is what I must show. The customer must see proof of value, ease of use and versatility. The customer must be wowed and be shown that within a short period of a few days It was possible to analyse their data to such a level as to show them aspects of their business they had never seen or even thought about before’.
This is the Kirk in us dealing only with the results. After this initial end-goal inner brainstorming the pendulum swings back and the sober reality of actually doing it within the confines of our skill both intellectually and technologically kick in.
‘What process do we follow to achieve this and how do I go about doing it?’
In the next few days to few weeks the pendulum continues to swing back and forth. We follow methodically the analytical process to a certain point, then step back, take stock of where we are and envision what the result of the next phase should be. Then, back into the process and analysis to achieve this goal. This process of back and forth between visions continues until we finally have the level of desired results we set out initially to achieve. We then present to the customer the results of what has been found within their data and usually they are pleased enough to then go back and ask how we did it. The customer swings back from their initial unconstrained view of only needing results to a more constrained vision of wanting to know how these results were achieved.
Truly in the business world no one should or could live entirely within the confines of a particular vision. Even the most egomaniacal CEO must know, deep down, that results are born through both inspiration and process and if he were to have analysts without the balance needed then his business would surely fail….at least the next audit anyway. One could say, if one was so inclined, that the unconstrained envisioning is part of the process and hence is inherently constrained but this is a Pandora’s Box of second guessing better left to the philosopher. We could try arguing out the finer points of this but I’m afraid this would leave our predominantly scientific brains weeping in the corner searching for the logical door.
So, to conclude, for a truly analytical approach to our work it is necessary that a balance is maintained. To see a process or piece of work through from beginning to end both a constrained and unconstrained vision of things should be employed. The work process needs the ‘to and fro’ between both visions and, if we relied on either, we would find that the end-result would not be as consistent, reliable or reproducible as it should and this would be a huge problem for the ongoing and consistent analysis that is needed. The real issue is how and when to take each approach. Both methods by themselves will get a result but as we have seen already the starship Enterprise leeches expendable extras every week due to the single-minded approach of Kirk so a mix of them will give us a result that is both quality and more importantly, for lower-echelon workers, traceable, reproducible and safe.