Notes for a future article on Mahout

Mahout is changing. Its changed over the years from recommender to core engine for the math part.. What users do is put a surface on it, and tweak the algorithms. Contrast that to using a product like Datameer. It may relatively be a black box in terms of how it does what it does. But there is some assurity that it is a path that is tested and you should have less of an adventure in implementing it.  Just as you may enlist vendor field engineers in order to do the implementation, you may get to a place in your Mahout build, and opt to bring in a consultant.
Let's look http://www.slideshare.net/chrishalton/build-vs-buy-strategy at build v buy basics.

Algorithms, audits and CDOs

Storified AI of the week

Simplexity kilt the cat

Simplexity by Jeffrey Kluger (subtitled Why Simple Things Become Complex and how complex things can be made simple). The book by the then (2008) Time reporter describes a ''newly emerging Science"(Maybe si, maybe no) meant to provide a cross disciplinary view on systems ranging from ant colonies to stock market. The "SFI" San Jose Instituite is the sort of anchor source (let by Prof Gell-Mann). Along the way (p31) he speaks w Brandeis economist Blake Lebaron who has been studying simulations of stock market trading. He has seen repeating patterns: 1-Traders wobble about; 2-one finds a useful stratagem; 3-the others mimic it; 4-diminishing returns set in. THe repeating patterns I think may be relevant to the general flow of the tech bubble that many of us live within. Whether it is ASPs or Deep Learning or Hadoop - a scatter occurs in the discover stage, then, a coalescence around a mean ensues, until a new scatter occurs in a new discovery stage. Think Whac-A-Mole.

  https://www.amazon.com/Simplexity-Simple-Things-Become-Complex/dp/B002YNS18EComplex/dp/B002YNS18E

Simplexity kilt the cat

Simplexity by Jeffrey Kluger (subtitled Why Simple Things Become Complex and how complex things can be made simple). The book by the then (2008) Time reporter describes a ''newly emerging Science"(Maybe si, maybe no) meant to provide a cross disciplinary view on systems ranging from ant colonies to stock market. The "SFI" San Jose Instituite is the sort of anchor source (let by Prof Gell-Mann). Along the way (p31) he speaks w Brandeis economist Blake Lebaron who has been studying simulations of stock market trading. He has seen repeating patterns: 1-Traders wobble about; 2-one finds a useful stratagem; 3-the others mimic it; 4-diminishing returns set in. THe repeating patterns I think may be relevant to the general flow of the tech bubble that many of us live within. Whether it is ASPs or Deep Learning or Hadoop - a scatter occurs in the discover stage, then, a coalescence around a mean ensues, until a new scatter occurs in a new discovery stage. Think Whac-A-Mole.


https://www.amazon.com/Simplexity-Simple-Things-Become-https://www.amazon.com/Simplexity-Simple-Things-Become-Complex/dp/B002YNS18EComplex/dp/B002YNS18E

The nature of field data gathering has changed

The nature of field data gathering has changed, as mobile devices and notepad computers find wider circulation. Surveys that once went through arduous roll-up processes are now gathered and digitized quickly. Now, a new stage of innovation is underway, as back-end systems enable users to employ field data for near-real-time decision making. An example in the geographic information system (GIS) space is ESRI's Survey123 for ArcGIS, which was formally introduced at ESRI's annual user conference, held this week in San Diego. To read the rest of the story.


See also Be there when the GIS plays Hadoop

Less Moore's Law in Store

Quantum computers wait in wings as
Moore's Law slows to a crawl.
Source: IBM

Fair to say our sister blog turned into "The Saturday Evening Review of John Markhoff" a long time ago. Well, at Amazing Techno Futures, the news feeds are good - and we could do worse than to track John Markhoff, who has been covering high tech at NYTImes for lo these many years. And I will not turn into a pumpkin if I hijack my own hijack of John.

For your consideration: His May 5 article on Moore's Law. He rightly points out this at inception was more an observation than a law, but Intel's Gordon Moore's 1965 eureka that the number of components that could be etched onto the surface of a silicon wafer was doubling at regular intervals stood the test of what today passes for time.

The news hook is a decision by the Semiconductor Industry Assn's to discontinue its Technology Roadmap for Semiconductors, based I take it on the closing of the Moore's Law era. IEEE will take up where this leaves off, with a forecasting roadmap [system] that tracks a wider swath of technology. Markhoff suggests that Intel hasn't entirely accepted the end of this line.

Possible parts of that swath, according to Markhoff, are quantum computing and  graphene.  The heat of the chips has been the major culprit blocking Moore's Law further run. Cost may be the next bugaboo. So far, parallelism has been the answer.

Suffice it to say, for some people at least, Moore's Law has chugged on like a beautiful slow train of time. With the Law in effect people at Apple, Sun, Oracle, etc. could count on things being better tomorrow than they were today in terms of features and functionality. So the new future, being less predictable, is a bit more foreboding.

I had my uh-ha moment on something like this in about 1983 when I was working on my master's thesis on Local Area Networks. This may not completely be a story about Moore's Law.. But I think it has a point.

Intel was working at the time to place the better part of the Ethernet protocol onto an Ethernet controller (in total maybe it was a 5-chip set). This would replace at least a couple of PC boards worth of circuitry that were the only way at the time to make an Ethernet node.

I was fortunate enough to get a Mostek product engineer on the phone to talk about the effect the chip would have on the market - in those days it was pretty much required that there were alternative sources for important chips, in this case Mostek. The fella described to me the volume that was anticipated over 5 or so years, and the pricing of the chip over that time. I transcribed his data points to a graph paper, and, as the volume went up, the price went down. Very magical moment. - Jack Vaughan

http://www.nytimes.com/2016/05/05/technology/moores-law-running-out-of-room-tech-looks-for-a-successor.html

Are dark pools coming to big data?

In Feb Barclays and Credit Suisse settled with the SEC which uncovered their nefarious high frequency manipulations in their dark trading pools. What’s with that? To go figure a good place is Flash Boys. But it is not an easy read.

Flash Boys delves into the netherworld of Wall St trading in the 2000s – where the devil is in the latency, and professional ethics is shit out of luck. Writer Michael Lewis paints a picture of an obsessively complex world of finance that attracts the underside of human aspiration. That echoes The Big Short, his earlier piece and a quite successful film in 2015.

But here the technological complexity that serves the finance engine rather gets the better of the story - ultimately Flash Boys pales a bit in comparison to The Big Short, as a result. We have a worthy hero at the center of the tale in Brad Katsuyama of Royal Bank of Canada, but the story can be stumbly as it tries to convey his efforts to uncover the culprits in the dark pools of high frequency trading – that would be the people that have the wherewithal to eavesdrop on the market,  spoof your intention, and buy stock in mass quantities at prices slightly lower than what you will pay to buy it from them. Brad could be the Cisco Kid that heads them off at the pass – if the road to get there weren’t so durn rocky.

 I’d suggest that many of the wonders of big data today resemble the wonders of stock market technology that front runs it. Publish and subscribe middleware and fantastically tuned algorithms are common to both phenomena.  Network latency can be the boogie man in both cases. Yes, while nearly no one was looking, online big data made a high frequency trading market out of advertising. The complexity is such that few can truthfully claim to understand it. And that lack of understanding is an opening for a con, as it was in The Big Short and the Flash Boys. When you believe in things that you dont understand then you suffer. - Jack Vaughan

You and the night and the machine intelligence