You mean joining the Children of the Magenta Line?

Across many fields, and particularly in technology, data is often synonymous with quantifiable data. If it fits in a spreadsheet, it’s data. If it’s indexible on Google, it’s data. If it’s a Tweet that raises (or lowers) my Klout score, it’s data.

You eloquently point out the irony of data-related discourse in Silicon Valley: the very same people who wax poetic about cybernetics, Hadoop, and “… for America” inhabit an industry and culture that thrives upon nuanced personal relationships, off-the-record conversations, and subtle indicators of status and style that find only marginal correlation with measurable, web-based indicators.

It would seem that there is a place for both measurable and unmeasurable data. Perhaps the trend of this decade will be the degree to which they begin to interoperate — say, as Twitter conversations, FourSquare check-ins, and various implementations of Augmented Reality.

Great article 🙂

It strikes me that there is an interesting balance between the process of incremental improvements in what we know and the attempt to jump to a different perspective to get a new insight.

The big data systems at present appear to me to be rather better at the former than the latter – i.e. they use past and present personal and peer performance to make proposals and projections (way too many ps)

Where they are poorer appears to be in bringing in the jumps in perspective, the dislocations and lateral thinking insights. Perhaps they overly encourage confirmation bias.

So the question I would have asked had I been at Foo Camp would have been “Are there ways that big data can help people make those leaps and encounter some of the ‘unknown unknowns'”

My hunch is that the answer might come down to whether you believe in free will or rigid determinism – i.e. the “is there anything more to the world than axioms and logical inference”….

Anyway… thanks for a really interesting article.

Peter