Big Data from Space

Big Data from Space - "The question of how to ensure space-based knowledge is used for the common good has become pressing with the dawning of a new space age, in which satellites have become affordable for private interests," writes NOAA head Kathy Sullivan on Davos Blog. Food for thought. - Jack Vaughan



https://agenda.weforum.org/2015/01/how-big-data-from-space-helps-life-on-earth/
https://www.google.com/webhp?hl=en&tab=ww#hl=en&tbs=qdr:y&q=kathy+sullivan+NOAA+space+data+opendata+%22climate+corporation%22

Between the buttons

Sometime ago I read a review of a book - "The Information: a history, a theory, a flood" by James Gleick. The review was by Sam Anderson and appeared in NYT Sunday Magazine June 26, 2011. That whole year was a blur, lately I am discovering, and I find it hard to believe I totally missed this thing. Cause it sees like a semi-mystical tome about technology, which is one of my suits.

Gleick it seems discussed the fact that every era thinks its is one of information overload. I get it. Folks immemorial feel something essential is being overthrown during the natural course of communications progress. Certainly the telegraph upset the cozy world of the semaphore, and the telephone unwound the telegraph of things. And on.

Glieck as expounded upon by Anderson picks up on some points that bear some noodling. Let's start with what they said that George Boole said that I did not even know about: "The symbols zero and one in the system of logic are nothing and the universe."

Anderson views the Web as an almanac - as a vast interlocking set of databases that seeks to comprise ALL PREVIOUS TEXT. But in the dance of All or Nothing at All you need two to tango. All text runs against communications. Thus Anderson saddles up upon a basic idea of communications to describe the problem incumbent with today's brand of "too much information."

He interestingly describes the leap from harnessing of electricity to telegraph as a leap based on interruptions of circuit flow - breaks in continuity, coded to hold meaning. And as he describes it you can see Morse's experience possibly affecting Boole's thinking.  (We could add Weiner and Shannon.)

- "We need to remember the value of nothing. 

- "We need to organize our internal absences to create meaning." 

- "It’s like breathing: you can’t inhale all day. We need to learn to make peace with the information we don’t know, to embrace the zeroes, to relearn the pleasures of hunger, need, interruption, restraint." 

Sam Anderson finds you have to leave the space, you cant fill up the glass, there are not just ones, there are zeros. You could put Monk, Lacy, Ellington, Basie, Morton, lot of jazz in this discussion.

-"We need to remember the value of nothing. We need to organize our internal absences to create meaning. "

Related

http://www.nytimes.com/2011/06/26/magazine/an-accidental-experimental-masterpiece.html

https://www.goodreads.com/book/show/8701960-the-information

Randomly
-Fortunately for Western Union, the telegram became the money transfer.

-When I look at SI's web site, I see the Web less as a new threat but a resurgence of the original threat to newspapers and magazine that was TV.

-Derived from this story : Here's a good quote from Benedict Anderson, scholar: "reading a newspaper is like reading a novel whose author has abandoned any thought of a coherent plot." The story we are looking at [ http://www.nytimes.com/2011/06/26/magazine/an-accidental-experimental-masterpiece.html ] juxtaposes thoughts on Glieck's book with what an old style almanac was, and how the Internet is now the almanac and the title of the story describes this:  An accidental experimental masterpiece - and I guess that's what I think news is like. 

Data tweets I have known during the Week of January 18, 2015

Top news this week...Microsoft to acquire Revolution Analytics... the company is the chief independent purveyor of R, which is fitting in nicely with Excel and some other skills in the world of data science. Very interesting 'early acquisition' in the Satya Nadella reign http://t.co/TarHUV15DY ... U.Tex students to use Watson to deliver #IBMWatson @JackVaughanatTT reports: http://bit.ly/1CxHlgr ... HDP 2.1: Apache Falcon for Data Governance in Hadoop http://t.co/WwUpeVXvRF ... Facebook opening up Deep Learning software - looking to vie with Google. http://bit.ly/1um9oLc ... Google's data supremacy: should we be worried? http://wp.me/p2WnJJ-ff ... New Report Evaluates Technological Alternatives to Bulk Data Collection http://t.co/EgliCMTqgE

The best laid plans of the mice and the man

EPIAC XIV, though undedicated, was already at work in deciding
how many refrigerators,
how many lamps,
how many turbine generators,
how many hub caps,
how many dinner plates,
how many doorknobs,
how many rubber heels,
how many television sets,
how many pinochle decks – how many everything America and her customers could have and how much they would cost. – Kurt Vonnegut, Player Piano, 1951

Kurt Vonnegut's depiction of a computer somewhat resembling Eckert's and Mauchly's ENIAC derived from his experiences working for General Electric in Schenectady and his readings of Norbert Wiener, whose "The Human Use of Human Beings" was influential for a time. The "EPIAC" state-controller described in 'Player Piano' is also a riff on statistical logistics planning that emerged out of World War II efforts.

Wiener's cybernetics work and its applicability to state planning also captured the imagination of forward looking Latin American leaders in the 1970s. e. But this is real world story, with realworld blood as told in Eden Medina's "Cybernetic Revolutionaries" [MIT Press, 2013], which concerns the ill fate of Chilean socialist leader Salvador Allende.. Her story was in turn artfully retold in part in a recent New Yorker story, "The Planning Machin," where Evgeny Morozov weaves the story of Allende and cybernetics into a discussion of such recent phenomena as big data, open data, the Internet of Things, Nest and Uber.

The piece finds similarities between the burgeoning slaught of interconnected devices and data driven business models, as others have found between long moribund AI and upstart machine learning and cognitive computing applications. History, here in the form of the Chilean cybernetic experience, provides a guide as to what can go wrong.

TBC

http://mitpress.mit.edu/books/cybernetic-revolutionaries - MIT Press
http://www.newyorker.com/magazine/2014/10/13/planning-machine - New Yorker, Oct 2014

Talking Data: Protecting online data privacy was the big 2014 trend

The last Talking Data Podcast of 2014 is a bit of a walk down Twitter Lane with data privacy issues in 2014. Ed Burns and I discuss a Pew poll that looks at the attitude of Americans toward online privacy. They aren't comfortable with their data sharing, but they sure do like that social infrastructure services at gratis. Uber and its loping missteps on the way to killing the hackney as he is now known, and what that means for data professionals. There is more, including some discussion of HortonWorks IPO. On Christmas eve my version was truncated. Might wait until the Christmas smoke clears to sort through that. - Jack Vaughan

Talking Data: Protecting online data privacy was the big2014 trend -MP#

That was the year that was big data ala Hadoop, NoSQL

The year 2014 saw progress in big data architecture development and deployment, as users gained more experience with NoSQL alternatives to relational databases, and Hadoop 2 gained traction for operational analytics uses beyond the distributed processing framework's original batch processing role. Those trends were detailed in a variety of SearchDataManagment pieces. Big data in review: 2014

The roots of machine learning

Neural networks and artificial intelligence have been on my mind over many years, while I spent most days studying middleware, something different. At the heart of neurals were backward propagations and feedback loops – all somewhat related to cybernetics, which flowered in the 1940s into the early 1970s. One of the first implements of cybernetics was the thermostat.

In early 2013 I started working at SearchDataManagement, writing about Big Data. At the end of  this year I have devoted some time to book learning about machine learning. A lot happened while Rip Van Vaughan was catching some z's.  So something told me to go back to one of my old blogs and see where I left off with feedback. If you pick through it you will find Honeywell and the thermostat and automatic pilot, etc. My research told me the first auto pilot arose from a combo of the thermostat (Honeywell) and advanced gyroscopes (Sperry).

I spent hours looking at the thermostat, its mercury, its coil. It had an alchemical effect. I remember wondering if the thermostat could be a surveillance bug. Now we have Nest, which uses the thermostat as a starting point for collecting data for machine learning process. - Jack Vaughan

[It is funny how the old arguments about the autopilot appeared as memes in this year of machine learning. This link, which includes Tom Wolfe's mirthful take on the autopilot in The Right Stuff… is here rather as a place-marker for background: The Secret Museum of Cybernetics - JackVaughan's Radio Weblog, March 2004(also reposted on Moon Traveller with a slew of feedback errata). Probably valuable to cite Nicholas Carr's The Glass Cage, published this year, which takes as its premise, society's growing inabilities, many brought on by automation. Several serious airplane crashes where pilots' skills seemed overly lulled by automation form a showcase in The Glass Cage, ]

From Wolfe’s The Right Stuff:

“Engineers were ... devising systems for guiding rockets into space, through the use of computers built into the engines and connected to accelerometers for monitoring the temperature, pressure, oxygen supply, and other vital conditions of the Mercury capsule and for triggering safety procedures automatically -- meaning they were creating  with computers, systems in which machines could communicate with one another, make decisions, take action, all with tremendous speed and accuracy .. Oh, genius engineers! “

AI re-emergence : Study to Examine Effects of Artificial Intelligence

Able New York Times technology writer John Markoff (he has been far away the star of my RJ-11 blog) had two of three (count em, three) AI articles in the Dec 16 Times. One discusses Paul Allen's AI2 institute work; the other discusses a study being launched at Stanford with the goal to look at how technology reshapes roles of humans. Dr. Eric Horvitz of MS Research will lead a committee with Russ Altman, a Stanford professor of bioengineering and computer science. The committee will include Barbara J. Grosz, a Harvard University computer scientist; Yoav Shoham, a professor of computer science at Stanford; Tom Mitchell, the chairman of the machine learning department at Carnegie Mellon University; Alan Mackworth, a professor of computer science at the University of British Columbia; Deirdre K. Mulligan, a lawyer and a professor in the School of Information at the University of California, Berkeley. The last, Mulligan, is the only one who immediately with some cursory Googling appears to be ready to accept that there are some potential downsides to AI re-emergence. It looks like Horvitz has an initial thesis formed ahead of the committee work. That is that, based on a TED presentation ("Making friends with AI") , while he understand some people's issues with AI, that the methods of AI will come to support people's decisions in a nurturing way. The theme would be borne out further if we look at the conclusion of an earlier Horvitz'z organized study on AI's ramifications (that advances were largely positive and progress relatively graceful). Let's hope the filters the grop implement tone down the rose-colored learning machine that enforces academics' best hopes. – Jack Vaughan

Facebook imbroglio

Recently I wrote a story for SearchDataManagement that largely centered on one of this year's big data imbroglios. That is the Facebook-Cornell Emotional Contagion study. This was the topic at Friday night symposium (Oct 8) capping the first day of the Conference on Digital Experimentation at MIT.   You could sum up the germ of imp of the story as: it is okay to personalize web pages with financial purpose, and to fine tune your methods thereof, but could that overstep an ethical boundary?

On the MIT panelist but not covered in my brief story was Jonathan Zitrain of Harvard University Law. For me, he put the contagion study into context- contrasting and comparing it to the Stanford Prison experiment and the Tuskeegee prisoner studies, which came under scrutiny and helped lead the way to some standards of decorum for psychological and medical experiments.  "There out to be a baseline protection," he said. There is a fiduciary responsibility, a need for a custodial and trusting relationship with subjects, that at least are an objective in science studies of humans.

Now, this responsibility, forwarded by Zitrain and others, remains unstated and vague. Clearly, the Internet powers that be are ready to move on to other topics, and let the Facebook experiment fade into the recesses, as even Snowden's NSA revelations have. I think a professional organization is needed – that sounds old school, I know and I don’t care. As with civil engineering, there is no need for a new generation to figure out what is overstepping – for waiting until the bridge collapses. – Jack Vaughan

Related

http://cyber.law.harvard.edu/people/jzittrain
http://searchdatamanagement.techtarget.com/opinion/Facebook-experiment-points-to-data-ethics-hurdles-in-digital-research
http://codecon.net/