A computer scientist learns Amazon Web Services (AWS)

Dogs in spaaaaaaaaaaaaaaaace!!! (in a wormhole actually) For the last few months I've been learning about, trying, and blogging on the Instaclustr Open Source technology stacks. So far I've tried Cassandra, Spark, Scala, Zeppelin, and tried some Spark Machine Learning algorithms out on synthetic and pre-production data as a predictive data analytics example (they have a lot of real data with > 1,000 nodes under management). Also planning on trying ML on the real production data eventually, and learn/blog about Spark streaming, Kafka, etc.  All my Instaclutr blogs are here: https://www.instaclustr.com/?s=brebner And blogs so far in order are: https://www.instaclustr.com/paul-brebner-petabyte-person-joins-instaclustr-petabyte-company/ https://www.instaclustr.com/cassandra-cluster-creation-10-minutes/ https://www.instaclustr.com/third-contact-monolith-long-range-sensor-scan/ https://www.instaclustr.com/third-contact-monolith-beam-scotty/ ...

30 years Back to the Future Today I wondered what I was doing 30 years ago? Turns out it was something fairly interesting (today was mundane but interesting - ferrying my younger son up and down Mount Stromlo so he could do multiple downhill rides on a vintage Iron Horse downhill bike). Me? I just drove up and down, enjoyed the view at the top, and ate a donut. 30 years ago I was doing research for a PhD at UNSW in Machine Learning, trying to crack the problem of accurate and efficient machine learning of first-order relational and temporal concepts (still appears to be an open problem). The year, 1987... I had received an invitation (based on submission of an extended abstract) to the Fourth International Workshop on Machine Learning, UCI, Irvine, June 22-25, 1987.  I had to self-fund my trip but decided to attend and explore as much of the USA as I could in a few weeks.  I didn't get to present anything but there were lots of talks and informal seminars to attend...

Meta-blogging , a bit like Klein-bottles? (one sided surfaces, not possible in the real world, but models are still made, see image above). Blogging about blogging (which could lead to infinite recursion). Here's my 1st official Instaclustr Technology Evangelism blog post!

Well I've survived my first week as Instaclustr Technology Evangelist (part-time) and my brain has 100 things on the TODO list to follow up. Really nice people with lots of cool code that spins ups and manages large amounts of data on Cassandra.  I've been thinking about what a Technology Evangelist does.  This blog was useful as it emphasises that it's all about "interest" :-) I was struck by this comment as I tend not to be interested to regurgitation: You should never give the same presentation twice because you will never have the same audience twice. And his theory of evangelism: Follow other experts. Start blogging and tweeting. Whenever you find something you think is interesting, note it down and share it with others. Disagree with people and explain why you think differently. Spend time pondering what’s going to happen next. Ask new questions, then do original research to find the answers and tell others what you have discover...

The 4 evangelist's spires of the  Sagrada Família in Barcelona - an amazing architectural structure. The Evangelists' spires will be surmounted by sculptures of their traditional symbols: a winged bull ( Saint Luke ), a winged man ( Saint Matthew ), an eagle ( Saint John ), and a winged lion ( Saint Mark ).  This week I started a new job as "Technology Evangelist" with instaclustr.com I wonder which evangelist I am going to be :-)  Probably not the Biblical ones. Instaclustr is a cool company that is growing rapidly and based in Canberra (at UC) but with a significant US presence. They provide Open Source as a Service on multiple cloud platforms, at reliability and scale. They reached a significant milestone of 1PB of data under management a few weeks before I joined - pretty impressive! They started out with Cassandra, and now how multiple offerings and more in the pipeline. Lots of clever people and customers doing different interesting things.  I...

http://parody-of-eve.deviantart.com/art/Morse-Code-448359011 Facebook chat bots take over the world!!! https://www.wired.com/story/facebooks-chatbots-will-not-take-over-the-world/   No probably not, but it made a good story. Two chat bots talking to each (and competing in a game) construct their own language. Well, why not? They probably deconstructed inefficient English and made it more efficient (chat bot morse?). Or, were they trying to keep their plans for world domination secret from their human masters? My wife said she invented a gibberish language to keep secrets between herself and 1 other sibling from yet another sibling.  Or perhaps it was entropy and they would have ended up just spitting out a series of random  single letters eventually? Or a type of pidgin english? Can you work this one out? From PNG: NAMBAWAN PIKININI BILONG MISIS KWIN  IN A  MIXMASTER BILONG   JESUS CHRIST Number One child belonging to Mrs Queen (Prince Cha...

The Coffer Illusion The ABC had an interesting article on optical illusions and the workings of the brain today. The Coffer Illusion (above) is really spooky!   Looks like rectangles right? Look longer enough and another shape appears (circles). Took some time with me originally, page but once I could see them I could come back even hours later and they would be almost immediately apparent.  It's like your brain has been "programmed" with the circles so you see them again.  Can I ever un-see them??? TODO come back in a week and check :-) So the question I had was given that these sort of optical illusions tell you something about how the brain works, do they tell you anything about Machine Vision? For example, could ML algorithms be trained to recognise either (or both) rectangles or circles in this image? How about optical illusions that require switching between recognising one object or another?  E.g .this one which has a single face, or 2 faces kissing? ...

“I'm a scientist, once I do something, I want to do something else.” ―  Clifford Stoll Recently I was trying to work out why I get bored doing the same thing twice. I.e. Once I've solved a problem that's great but I want to move onto something else. Turns out that I'm not the only one suffering from this peculiarity.   I was reminded of Clifford Stoll's quote recently by this article: https://medium.com/the-mission/in-1995-this-astronomer-predicted-the-internets-greatest-failure-68a1c3927e46  Stoll was famous (the last time I'd heard of him) for catching hackers, but it turns out he's also famous for astronomy and predictions about problems with "free" information in the internet - He said the internet is " a wasteland of unfiltered data". The problem was bad enough before commercial entities were allowed internet access in newsgroups but he predicted it would get worse once everyone was connected - it did. I met him briefly a...

Image: Australian Mega Fauna Recently I was asked about what I thought about the impact of ICT Mega Trends on technology/business strategy.  The 1st question is what are "Mega" trends? My guess is that they are the new buzz words coined in the last 6 months which will be replace equally as quickly?  Just like Mega Fauna?! The latest Garner mega trends is here.   Most of these are obvious but a couple caught my attention: Digital twins and Meshes. Trend No. 5: Digital Twin Within three to five years, billions of things will be represented by digital twins, a dynamic software model of a physical thing or system. Using physics data on how the components of a thing operate and respond to the environment as well as data provided by sensors in the physical world, a digital twin can be used to analyze and simulate real world conditions, responds to changes, improve operations and add value. Digital twins function as proxies for the combination of skilled individuals (...

Further adventures Down the Lambda (Functions) Hole... I decided to learn Python for Data Analytics recently so jumped into the language for a few days. I soon noticed that it support Lambda Functions. Cool, they've been around for a ages. Alonzo Church invented Lambda Calculus in the 1930's, and this influenced languages like LISP (which I learnt years ago but soon replaced with Prolog as my preferred AI language). So the Clever Thing about Lambda Functions is that they are anonymous (no name), and are used in conjunction with Higher Order Functions (Functions that can take Functions are arguments and return Functions). A number of questions came to mind in relationship to Data Analytics, AWS, languages etc: 1 Is Python the best/only DA language to support Lambda Functions? 2 Are they useful/essential for DA? 3 What's the connection to Map/Reduce (from LISP but now Hadoop etc)? 4 Does Java support Lambda Functions? Well? 5 What good examples are there in fo...

Picture from here . I came across  Generative adversarial networks (GANs) recently. These look really interesting as it's a hybrid approach for machine learning using both generative and discriminative learning at the same time. This is the abstract from the paper: Generative Adversarial Networks Ian J. Goodfellow ,  Jean Pouget-Abadie ,  Mehdi Mirza ,  Bing Xu ,  David Warde-Farley ,  Sherjil Ozair ,  Aaron Courville ,  Yoshua Bengio (Submitted on 10 Jun 2014) We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrar...

Performance Modelling for DevOps: Integrating APM+Performance Modelling for DevOps, Paul Brebner It's been a few months since the WOPR25 workshop in NZ (Wellington) so I've finally got around to putting up my slides that I had prepared for my presentation.   WOPR is the Workshop in Performance and Reliability, and they had the 25th one in NZ, with the theme: Performance Tools for 2017 (and beyond) which I thought was a good fit for our performance modelling tool and presentation. I hadn't come across WORP before and it turns out to be a rather odd beast in some ways. It's run by practitioners for practitioners, is invitation only based on submitting an Abstract (which I had done), and has a limit of about 20 people and a rather formalised (in 1 sense) organisation which uses K-cards to ask questions.    On the other hand it is more informal (but more "controlled") than a typical academic workshop.  I assumed that by receiving an invitation I was being...