Some dirt on dirt data

Sometimes I think back to last Sept when I got a chance to see data in a different role. That is, as a central player in solving civilization-scale challenges. So much has been done, yet there is so much more to do. Just as one example there is soil data. More and more data is being gathered on soil moisture, weather and crop conditions, but new storage techniques, analytical methods and search algorithms are required, as a U of Wis. (Go, Bucky Badger!) researcher said at the conference. - Jack Vaughan




Related
Data-wrangling-a-key-to-meeting-civilization-scale-challenges - SearchDataManagement.com, Sept 2014 

Momentous Tweets in the week of our Lord Jan 25 2015

HBR - We've written a lot of great stuff on digital business models http://s.hbr.org/1tGWJrk

The Onion @TheOnion • 21h21 hours ago Top Story In Sports: Marshawn Lynch Delivers Eloquent 45-Minute Address On Privacy In The Modern Age http://onion.com/1zH2lVj

Mark Madsen @markmadsen • 21h21 hours ago RT @kdnuggets: Can Microsoft make R easy? http://flip.it/gIOeg #rstats http://flip.it/A1eV1

#Hortonworks #Data #Governance Initiative targets #Hadoop data management, better data policies http:// http://bit.ly/1CVCFDF

Francois Petitjean @LeDataMiner · Jan 28

.@kdnuggets #Code for learning the #Structure of #GraphicalModels for #BigData released - http://goo.gl/0BhnHh 

Biggest misnaming of place in history? Greenland! NASA | Greenland's Ice Layers Mapped in 3D: http://youtu.be/u0VbPE0TOtQ

Curt Monash shows you where the #bigdata innovation is. http://www.dbms2.com/2015/01/19/where-the-innovation-

The Internet of Things just got a watchdog: FTC issues official report. #Thingnado #IoT http://bit.ly/15PtUOY 

"There be data here"... lots of #BigData #OpenData ... http://bit.ly/1sWtO26  #abdsc via @DataScienceCtrl

Fast Clustering of #BigData Sets: http://bit.ly/1BBm8ml  #abdsc #DataScience via @DataScienceCtrl

Accenture's Vince Dell'Anno discusses the data supply chain http://bit.ly/1CinPYJ 

Big Data from Space

Big Data from Space - "The question of how to ensure space-based knowledge is used for the common good has become pressing with the dawning of a new space age, in which satellites have become affordable for private interests," writes NOAA head Kathy Sullivan on Davos Blog. Food for thought. - Jack Vaughan



https://agenda.weforum.org/2015/01/how-big-data-from-space-helps-life-on-earth/
https://www.google.com/webhp?hl=en&tab=ww#hl=en&tbs=qdr:y&q=kathy+sullivan+NOAA+space+data+opendata+%22climate+corporation%22

Between the buttons

Sometime ago I read a review of a book - "The Information: a history, a theory, a flood" by James Gleick. The review was by Sam Anderson and appeared in NYT Sunday Magazine June 26, 2011. That whole year was a blur, lately I am discovering, and I find it hard to believe I totally missed this thing. Cause it sees like a semi-mystical tome about technology, which is one of my suits.

Gleick it seems discussed the fact that every era thinks its is one of information overload. I get it. Folks immemorial feel something essential is being overthrown during the natural course of communications progress. Certainly the telegraph upset the cozy world of the semaphore, and the telephone unwound the telegraph of things. And on.

Glieck as expounded upon by Anderson picks up on some points that bear some noodling. Let's start with what they said that George Boole said that I did not even know about: "The symbols zero and one in the system of logic are nothing and the universe."

Anderson views the Web as an almanac - as a vast interlocking set of databases that seeks to comprise ALL PREVIOUS TEXT. But in the dance of All or Nothing at All you need two to tango. All text runs against communications. Thus Anderson saddles up upon a basic idea of communications to describe the problem incumbent with today's brand of "too much information."

He interestingly describes the leap from harnessing of electricity to telegraph as a leap based on interruptions of circuit flow - breaks in continuity, coded to hold meaning. And as he describes it you can see Morse's experience possibly affecting Boole's thinking.  (We could add Weiner and Shannon.)

- "We need to remember the value of nothing. 

- "We need to organize our internal absences to create meaning." 

- "It’s like breathing: you can’t inhale all day. We need to learn to make peace with the information we don’t know, to embrace the zeroes, to relearn the pleasures of hunger, need, interruption, restraint." 

Sam Anderson finds you have to leave the space, you cant fill up the glass, there are not just ones, there are zeros. You could put Monk, Lacy, Ellington, Basie, Morton, lot of jazz in this discussion.

-"We need to remember the value of nothing. We need to organize our internal absences to create meaning. "

Related

http://www.nytimes.com/2011/06/26/magazine/an-accidental-experimental-masterpiece.html

https://www.goodreads.com/book/show/8701960-the-information

Randomly
-Fortunately for Western Union, the telegram became the money transfer.

-When I look at SI's web site, I see the Web less as a new threat but a resurgence of the original threat to newspapers and magazine that was TV.

-Derived from this story : Here's a good quote from Benedict Anderson, scholar: "reading a newspaper is like reading a novel whose author has abandoned any thought of a coherent plot." The story we are looking at [ http://www.nytimes.com/2011/06/26/magazine/an-accidental-experimental-masterpiece.html ] juxtaposes thoughts on Glieck's book with what an old style almanac was, and how the Internet is now the almanac and the title of the story describes this:  An accidental experimental masterpiece - and I guess that's what I think news is like. 

Data tweets I have known during the Week of January 18, 2015

Top news this week...Microsoft to acquire Revolution Analytics... the company is the chief independent purveyor of R, which is fitting in nicely with Excel and some other skills in the world of data science. Very interesting 'early acquisition' in the Satya Nadella reign http://t.co/TarHUV15DY ... U.Tex students to use Watson to deliver #IBMWatson @JackVaughanatTT reports: http://bit.ly/1CxHlgr ... HDP 2.1: Apache Falcon for Data Governance in Hadoop http://t.co/WwUpeVXvRF ... Facebook opening up Deep Learning software - looking to vie with Google. http://bit.ly/1um9oLc ... Google's data supremacy: should we be worried? http://wp.me/p2WnJJ-ff ... New Report Evaluates Technological Alternatives to Bulk Data Collection http://t.co/EgliCMTqgE

The best laid plans of the mice and the man

EPIAC XIV, though undedicated, was already at work in deciding
how many refrigerators,
how many lamps,
how many turbine generators,
how many hub caps,
how many dinner plates,
how many doorknobs,
how many rubber heels,
how many television sets,
how many pinochle decks – how many everything America and her customers could have and how much they would cost. – Kurt Vonnegut, Player Piano, 1951

Kurt Vonnegut's depiction of a computer somewhat resembling Eckert's and Mauchly's ENIAC derived from his experiences working for General Electric in Schenectady and his readings of Norbert Wiener, whose "The Human Use of Human Beings" was influential for a time. The "EPIAC" state-controller described in 'Player Piano' is also a riff on statistical logistics planning that emerged out of World War II efforts.

Wiener's cybernetics work and its applicability to state planning also captured the imagination of forward looking Latin American leaders in the 1970s. e. But this is real world story, with realworld blood as told in Eden Medina's "Cybernetic Revolutionaries" [MIT Press, 2013], which concerns the ill fate of Chilean socialist leader Salvador Allende.. Her story was in turn artfully retold in part in a recent New Yorker story, "The Planning Machin," where Evgeny Morozov weaves the story of Allende and cybernetics into a discussion of such recent phenomena as big data, open data, the Internet of Things, Nest and Uber.

The piece finds similarities between the burgeoning slaught of interconnected devices and data driven business models, as others have found between long moribund AI and upstart machine learning and cognitive computing applications. History, here in the form of the Chilean cybernetic experience, provides a guide as to what can go wrong.

TBC

http://mitpress.mit.edu/books/cybernetic-revolutionaries - MIT Press
http://www.newyorker.com/magazine/2014/10/13/planning-machine - New Yorker, Oct 2014

Talking Data: Protecting online data privacy was the big 2014 trend

The last Talking Data Podcast of 2014 is a bit of a walk down Twitter Lane with data privacy issues in 2014. Ed Burns and I discuss a Pew poll that looks at the attitude of Americans toward online privacy. They aren't comfortable with their data sharing, but they sure do like that social infrastructure services at gratis. Uber and its loping missteps on the way to killing the hackney as he is now known, and what that means for data professionals. There is more, including some discussion of HortonWorks IPO. On Christmas eve my version was truncated. Might wait until the Christmas smoke clears to sort through that. - Jack Vaughan

Talking Data: Protecting online data privacy was the big2014 trend -MP#

That was the year that was big data ala Hadoop, NoSQL

The year 2014 saw progress in big data architecture development and deployment, as users gained more experience with NoSQL alternatives to relational databases, and Hadoop 2 gained traction for operational analytics uses beyond the distributed processing framework's original batch processing role. Those trends were detailed in a variety of SearchDataManagment pieces. Big data in review: 2014

The roots of machine learning

Neural networks and artificial intelligence have been on my mind over many years, while I spent most days studying middleware, something different. At the heart of neurals were backward propagations and feedback loops – all somewhat related to cybernetics, which flowered in the 1940s into the early 1970s. One of the first implements of cybernetics was the thermostat.

In early 2013 I started working at SearchDataManagement, writing about Big Data. At the end of  this year I have devoted some time to book learning about machine learning. A lot happened while Rip Van Vaughan was catching some z's.  So something told me to go back to one of my old blogs and see where I left off with feedback. If you pick through it you will find Honeywell and the thermostat and automatic pilot, etc. My research told me the first auto pilot arose from a combo of the thermostat (Honeywell) and advanced gyroscopes (Sperry).

I spent hours looking at the thermostat, its mercury, its coil. It had an alchemical effect. I remember wondering if the thermostat could be a surveillance bug. Now we have Nest, which uses the thermostat as a starting point for collecting data for machine learning process. - Jack Vaughan

[It is funny how the old arguments about the autopilot appeared as memes in this year of machine learning. This link, which includes Tom Wolfe's mirthful take on the autopilot in The Right Stuff… is here rather as a place-marker for background: The Secret Museum of Cybernetics - JackVaughan's Radio Weblog, March 2004(also reposted on Moon Traveller with a slew of feedback errata). Probably valuable to cite Nicholas Carr's The Glass Cage, published this year, which takes as its premise, society's growing inabilities, many brought on by automation. Several serious airplane crashes where pilots' skills seemed overly lulled by automation form a showcase in The Glass Cage, ]

From Wolfe’s The Right Stuff:

“Engineers were ... devising systems for guiding rockets into space, through the use of computers built into the engines and connected to accelerometers for monitoring the temperature, pressure, oxygen supply, and other vital conditions of the Mercury capsule and for triggering safety procedures automatically -- meaning they were creating  with computers, systems in which machines could communicate with one another, make decisions, take action, all with tremendous speed and accuracy .. Oh, genius engineers! “