Monday, July 4, 2016

Simplexity kilt the cat


Simplexity by Jeffrey Kluger (subtitled Why Simple Things Become Complex and how complex things can be made simple). The book by the then (2008) Time reporter describes a ''newly emerging Science"(Maybe si, maybe no) meant to provide a cross disciplinary view on systems ranging from ant colonies to stock market. The "SFI" San Jose Instituite is the sort of anchor source (let by Prof Gell-Mann). Along the way (p31) he speaks w Brandeis economist Blake Lebaron who has been studying simulations of stock market trading. He has seen repeating patterns: 1-Traders wobble about; 2-one finds a useful stratagem; 3-the others mimic it; 4-diminishing returns set in. THe repeating patterns I think may be relevant to the general flow of the tech bubble that many of us live within. Whether it is ASPs or Deep Learning or Hadoop - a scatter occurs in the discover stage, then, a coalescence around a mean ensues, until a new scatter occurs in a new discovery stage. Think Whac-A-Mole.

  https://www.amazon.com/Simplexity-Simple-Things-Become-Complex/dp/B002YNS18EComplex/dp/B002YNS18E

Simplexity kilt the cat

Simplexity by Jeffrey Kluger (subtitled Why Simple Things Become Complex and how complex things can be made simple). The book by the then (2008) Time reporter describes a ''newly emerging Science"(Maybe si, maybe no) meant to provide a cross disciplinary view on systems ranging from ant colonies to stock market. The "SFI" San Jose Instituite is the sort of anchor source (let by Prof Gell-Mann). Along the way (p31) he speaks w Brandeis economist Blake Lebaron who has been studying simulations of stock market trading. He has seen repeating patterns: 1-Traders wobble about; 2-one finds a useful stratagem; 3-the others mimic it; 4-diminishing returns set in. THe repeating patterns I think may be relevant to the general flow of the tech bubble that many of us live within. Whether it is ASPs or Deep Learning or Hadoop - a scatter occurs in the discover stage, then, a coalescence around a mean ensues, until a new scatter occurs in a new discovery stage. Think Whac-A-Mole.


https://www.amazon.com/Simplexity-Simple-Things-Become-https://www.amazon.com/Simplexity-Simple-Things-Become-Complex/dp/B002YNS18EComplex/dp/B002YNS18E

Thursday, June 30, 2016

The nature of field data gathering has changed

The nature of field data gathering has changed, as mobile devices and notepad computers find wider circulation. Surveys that once went through arduous roll-up processes are now gathered and digitized quickly. Now, a new stage of innovation is underway, as back-end systems enable users to employ field data for near-real-time decision making. An example in the geographic information system (GIS) space is ESRI's Survey123 for ArcGIS, which was formally introduced at ESRI's annual user conference, held this week in San Diego. To read the rest of the story.


See also Be there when the GIS plays Hadoop

Tuesday, May 10, 2016

Less Moore's Law in Store

Quantum computers wait in wings as
Moore's Law slows to a crawl.
Source: IBM
Fair to say our sister blog turned into "The Saturday Evening Review of John Markhoff" a long time ago. Well, at Amazing Techno Futures, the news feeds are good - and we could do worse than to track John Markhoff, who has been covering high tech at NYTImes for lo these many years. And I will not turn into a pumpkin if I hijack my own hijack of John.

For your consideration: His May 5 article on Moore's Law. He rightly points out this at inception was more an observation than a law, but Intel's Gordon Moore's 1965 eureka that the number of components that could be etched onto the surface of a silicon wafer was doubling at regular intervals stood the test of what today passes for time.

The news hook is a decision by the Semiconductor Industry Assn's to discontinue its Technology Roadmap for Semiconductors, based I take it on the closing of the Moore's Law era. IEEE will take up where this leaves off, with a forecasting roadmap [system] that tracks a wider swath of technology. Markhoff suggests that Intel hasn't entirely accepted the end of this line.

Possible parts of that swath, according to Markhoff, are quantum computing and  graphene.  The heat of the chips has been the major culprit blocking Moore's Law further run. Cost may be the next bugaboo. So far, parallelism has been the answer.

Suffice it to say, for some people at least, Moore's Law has chugged on like a beautiful slow train of time. With the Law in effect people at Apple, Sun, Oracle, etc. could count on things being better tomorrow than they were today in terms of features and functionality. So the new future, being less predictable, is a bit more foreboding.

I had my uh-ha moment on something like this in about 1983 when I was working on my master's thesis on Local Area Networks. This may not completely be a story about Moore's Law.. But I think it has a point.

Intel was working at the time to place the better part of the Ethernet protocol onto an Ethernet controller (in total maybe it was a 5-chip set). This would replace at least a couple of PC boards worth of circuitry that were the only way at the time to make an Ethernet node.


I was fortunate enough to get a Mostek product engineer on the phone to talk about the effect the chip would have on the market - in those days it was pretty much required that there were alternative sources for important chips, in this case Mostek. The fella described to me the volume that was anticipated over 5 or so years, and the pricing of the chip over that time. I transcribed his data points to a graph paper, and, as the volume went up, the price went down. Very magical moment. - Jack Vaughan

Sunday, April 3, 2016

Are dark pools coming to big data?

In Feb Barclays and Credit Suisse settled with the SEC which uncovered their nefarious high frequency manipulations in their dark trading pools. What’s with that? To go figure a good place is Flash Boys. But it is not an easy read.

Flash Boys delves into the netherworld of Wall St trading in the 2000s – where the devil is in the latency, and professional ethics is shit out of luck. Writer Michael Lewis paints a picture of an obsessively complex world of finance that attracts the underside of human aspiration. That echoes The Big Short, his earlier piece and a quite successful film in 2015.

But here the technological complexity that serves the finance engine rather gets the better of the story - ultimately Flash Boys pales a bit in comparison to The Big Short, as a result. We have a worthy hero at the center of the tale in Brad Katsuyama of Royal Bank of Canada, but the story can be stumbly as it tries to convey his efforts to uncover the culprits in the dark pools of high frequency trading – that would be the people that have the wherewithal to eavesdrop on the market,  spoof your intention, and buy stock in mass quantities at prices slightly lower than what you will pay to buy it from them. Brad could be the Cisco Kid that heads them off at the pass – if the road to get there weren’t so durn rocky.

 I’d suggest that many of the wonders of big data today resemble the wonders of stock market technology that front runs it. Publish and subscribe middleware and fantastically tuned algorithms are common to both phenomena.  Network latency can be the boogie man in both cases. Yes, while nearly no one was looking, online big data made a high frequency trading market out of advertising. The complexity is such that few can truthfully claim to understand it. And that lack of understanding is an opening for a con, as it was in The Big Short and the Flash Boys. When you believe in things that you dont understand then you suffer. - Jack Vaughan

Sunday, March 20, 2016

On the eve of the White House Water Summit

From On the Waterfront
References to The Manhattan Project ( for example, "We need a new Manhattan Project" to address fill in the blank)  are overdone. But we need something on the order of something to deal with water. California knows what it is like to live with this life blood threatened – Israel too. It is good cast attention on it – and that might happen to some extent this week as The White House Water Summit takes place.

One of the issues that must be addressed is data about water. It is not as good as data on oil, or stocks, but it should be. In the New York Times op-ed column Charles Fishman writes about water and data, and how weak efforts are to categorize, detail and report on water use.  

Imagine if NOAA only reported on weather every fifth day. That is analogous to the water reports of the U.S. government, according to Fishman, who says, where water is concerned, we spend five years rolling up a report on a single year. The biggest problem, says Fishman, is water's invisibility, here and globally.

He focused on the fact that water census is done only every five years - that gives us only 20% view of the total water experience. He points to Flint, Toledo, the Colorado basin as recent water crises and notes that adequately monitoring the water doesn't assure results, but that inadequately monitoring the water is criminal what with so much monitoring of Wall Street, Twitter Tweets or auto traffic. Any call for more monitoring of course is up against today's version of the 1800's Know-Nothing movement.

Fishman tells us that good information does three things: 1- it creates demand for more information; 2- it changes people's behavior; and, 3- it ignites innovation.

But what is next? My little look-see into this area uncovered an overabundance of data formats for representing data. It seems a first step for water data improvements might come with the application of  modern big data technology to the problem of multiple formats.


Sunday, March 13, 2016

Four tales of data


Scenario - Four young students on Spring Break go to Honduras in search of the roots of big data. Each comes back with a story. Together they tell of a struggle entwined. Look at data anew through the eyes of four groovy undergrads. Not yet rated.


Oceans of data - Hundreds of meters below the surface of the ocean, Laura Robinson probes the steep slopes of massive undersea mountains. She's on the hunt for thousand-year-old corals that she can test in a nuclear reactor to discover how the ocean changes over time. Big data is her co-pilot

https://www.ted.com/talks/laura_robinson_the_secrets_i_find_on_the_mysterious_ocean_floor

Lord, bless your data - Thomas Bayes writer F.T. Flam [I am not making this up] says set out to calculate the probability of God's existence. This was back in the 19th Century in jolly old England. The math was difficult and really beyond the ken of calculation of the time - until the recent profusion of clustered computer power came around the corner in the early 2000s.

https://en.wikipedia.org/wiki/Thomas_Bayes

Autopilot and the roots of cybernation - Elmer Sperry's son Elmer’s son, Lawrence Burst Sperry, nicknamed Gyro, was best known for inventing the autopilot utilizing the concepts developed by his father for the gyroscope. Alas he was lost off the Channel when only 31.

https://www.pinterest.com/pin/568649890431580951/

I was just trying to help people. The story by Veronique Greenwood tells us she wrote of her experience in a letter to the New England Journal of Medicine, and was subsequently warned by her bosses not to do that kind of query again. Assumedly HIPPA privacy concerns are involved – so get out the anonymizer and gun it, right?!

http://itsthedatatalking.blogspot.com/2014/10/calling-dr-data-dr-null-dr-data-for.html

-----

Data does baseball


Brought to you by MBA@Syracuse: Tools of Baseball Analytics

Data, Humans on the Side




Good things - The recent PBS show The Human Side of Data has much to recommend it. As someone who labors in the big data vineyard as a reporter and commentator, I appreciate its succinct high level view on one of the defining memes of now. I had the chance to speak with my colleague Ed Burns on the topic for the Talking Data Podcast, and thought I’d add to the discussion here.

There were some beautiful animated pictures of data crossing the world – be it air traffic in a normal day or one tweet spreading. A theme was that humans had to create narratives around the data (per Jack Dorsey), and to follow the trail from the data point to the actual real-world event (Jer Thorpe). What makes a culture collectively change its view of data? - one participant asks. What is the context of the data? – several query.




Cause for pause things - And that takes us to an underlying issue with the show.. which is that there is this unspoken progressive notion that we are getting better – that a Ivy League youngster who studied statistics and grew up with the Web for pablum, soda, and bread can do better than the dimwits that went before. It could be true. But correlation is not causation.To phrase a coin. -Jack Vaughan