Wednesday, August 29, 2018

What I could not undiscover about Unikernels.

Unikernels http://unikernel.org/ have been around for a while but remain a relatively unknown technology. The core idea of a Unikernel is that it is a specialised operating system that throws away pretty much everything except network and disk drivers and application code, making for a tiny custom operating system that is usually single purpose, and typically claim to be more secure and efficient than traditional operating systems because they have disposed of all the unneeded code and in doing so, also reduced the Unikernels "attack surface".

Let's be clear about one thing - my view of Unikernels is oriented only toward my personal use case for them - that's the context of this post... and there are use cases for Unikernels that do not match my use cases.... I'm just saying so this isn't seen as a criticism of Unikernels, it's just the telling of my journey on that path.

I discovered Unikernels several years ago and took a particular interest in the Rump Kernel http://rumpkernel.org/ which is essentially NetBSD stripped down to little more than the network drivers, the disk drivers and your application code.

The minimal nature of Unikernels is immensely appealing.... tiny operating systems occupying only a handful of megabytes, more secure than traditional operating systems.

I thought I'd arrived early at a technology that was going to explode with popularity because Unikernels made so much damn sense...... security is a big deal and Unikernels were exceptionally secure because it's just an application, there's really nothing you can log in to on a Unikernel.  Contrast this with Linux where most machines are ready and waiting for users to log in, and hackers, once they get access can cause untold damage.

Being first to a new technology that explodes with growth is potentially a big opportunity, so I bet big on Unikernels.  There was work to be done with Unikernels because they were very complex to build and configure, and when I got on board they were not running at all on the cloud platforms.

I invested years building http://www.bootrino.com - software to make it easy to load and run Unikernels on all the major cloud platforms.  That was the gap I saw - making the complex Unikernel technology easy to run in the cloud.  I got my software built and it was fun and I learned a huge amount along the way, but Unikernels didn't explode with popularity, and although the project is still up and running, effectively it is abandoned because it meets no driving market need.

Why didn't Unikernels catch on?

Mainly because of Docker and serverless computing.  Serverless computing is a good enough solution for improving security, and Docker's lightweight virtualisation were two technologies that filled the market space that I thought Unikernels were going to occupy.  I'd had some experience with Docker and found it to be complex and unwieldy and it seemed likely people would gravitate to the better Unikernel technology once they discovered it.  When serverless hit the scene it was clear this wasn't going away any time soon, and it was also clear that serverless had taken some of the heat out of the application security issue - want a more secure application? Just run it as serverless.  You can't log in to a Unikernel, but you can't log in to a serverless function either. In the time it took me to build bootrino, Docker took over the world.

Unikernels had also been attacked as being "undebuggable" and "unfit for production".  I thought these claims were of no real value and put forward by people who had invested alot in backing the competing container technology.  Nevertheless the criticism seemed to get alot of traction with detractors of the Unikernel concept but it was never a criticism I cared much about - being "undebuggable" was never something that I encountered as a problem.

The flaws with the Unikernel concepts.

While I was building bootrino I discovered some things about Unikernels that were hard to undiscover.

One thing I learned was that sure - Unikernels are extremely minimal, and have no user login system, making them extremely secure, BUT Linux can be built that way too.  Embedded Linux systems such as Yocto Linux https://www.yoctoproject.org allow you to build tiny operating systems that consist only of a kernel plus the application code, occupying only a handful of megabytes.  Hey- that looks alot like a Unikernel!  Purists might object and say they are worlds apart because a Unikernel has a flat address space or some other distinction but I don't care about that stuff.... from a practical perspective - for the factors that I cared about - a stripped back Yocto Linux system is largely the same as a Unikernel.

The next thing I discovered that was hard to ignore was the symmetric multiprocessing story - Unikernels don't have such a story.  They are inherently single core.  That's a big problem in todays multicore world, and whilst there are ways around it by running many parallel single core Unikernels, it became clear to me that in fact the Linux kernel is a really good thing - and lot's of good stuff like handling SMP.  And as a result of my research I discovered also that the Linux kernel can be only a handful of megabytes anyway.  Taking full advantage of multi processors is not something that should be the responsibility of an application - it's a specialised and very tricky thing to get right, and its probably best done by some other bit of software - what should we call that?  Let's call it a kernel.  So what's with the Unikernels obsession with getting rid of the kernel?  There's not alot to be gained from having no kernel, and there is alot to be lost.  Hmmm.

Along the way I discovered you could do all sorts of tricky stuff like building Linux operating systems to run purely from RAM - such as Alpine Linux and TinyCore Linux do.  Again started to become even more blurry to me what the compelling advantage was of a Unikernel over a Run-From-RAM Linux operating system.

And finally the thing that I just couldn't ignore was that Linux let you configure it into very small and secure configurations with full SMP support AND you could run ANY existing Linux languages or applications on those tiny systems.  Unikernels generally seemed to need special builds and toolchains and configuration systems and recompilations to get them to run applications.  This was a huge advantage for Linux in a tiny configuration versus Unikernels.

I spent alot of time getting tiny operating systems to boot on the major cloud computing platforms. One of the times that most influenced my thinking about Unikernels was when I was trying to get MirageOS https://mirage.io - one of the leading Unikernel implementations - working on Amazon, Google and Digital Ocean. MirageOS is interesting in that you can compile the same MirageOS program into either a Unikernel, or into a Linux binary. I got to the point that I had the same MirageOS compiled application running on Amazon, Google and Digital Ocean under Linux, and I also had it running as a Unikernel successfully on Google Compute Engine - it did not work as a Unikernel on Amazon and Digital Ocean.  This was a key moment.  I sat there thinking OK - the very same MirageOS code works well on all three cloud platforms, and works fine Google as a Unikernel, but not the other two.  And I thought "Unikernel or Linux - it's exactly the same practical outcome".  The Linux version was running with nothing more than the MirageOS application binary, a Linux kernel of a handful of megabytes plus a few extra libraries - incredibly minimal.  That moment was a turning point, and it became very hard to see Unikernels as having substantial benefit over Linux at that point. In fact Linux had made it easier and more practical to implement MirageOS due to its better driver support. The only perceptible difference was the MirageOS/Linux implementation was 10 - 15 megabytes larger, which I didn't care about. I wondered then if all this Unikernel effort, and all the associated configuration, programming and loss of SMP, was about trying to save the space taken by 15 megabytes of Linux kernel? Cause the fact is, you can run your system as nothing more than a Linux kernel, plus your application and library code and absolutely nothing more, if you can be bothered configuring it that way.

No doubt unikernel purists will point out that there is a big difference between the definition of a unikernel versus a minimally configured Linux system such as Yocto Linux or Alpine or TinyCore.  Or perhaps they might point out that Unikernels can be only a handful of kilobytes and can boot in milliseconds.  I respect those things but my personal interests never needed those things. I was interested in practical outcomes for my personal use cases and less interested in purist definitions.  

I had come to see that it was possible to build very very small Linux systems, compatible with all Linux software, with all the security advantages of Unikernels, but with the benefit of symmetric multiprocessing and all the Linux driver support and community support.  

In practical terms I'd discovered, and could not undiscover, that for my personal interests and my personal use cases - anything that Unikernels can do, Linux can do better.  What's the real difference between embedded Linux systems and Unikernels?  Not much.  And besides, none of it really mattered anyway because the market wasn't beating a door down for this technology in the way it was flooding toward Docker and serverless. To put it a little brutally but factually - (almost) no one cares about Unikernels because there's no compelling market need.  Technologies that the market really needs get ripped out of your hands and used - to butcher the words of Marc Andressen. Unikernels do not have product/market fit.

And speaking of zero product/market fit - I'm proud of http://www.bootrino.com - the system I built for booting Run-From-RAM operating systems in the cloud, and I have immense respect for the incredible intellect and capability of every person in the Unikernel community - all infinitely smarter and better programmers than me - I was after all just a user of the technology, not one of the heroic individuals building the Unikernel technology - but in the end I just couldn't undiscover that Linux makes a better Unikernel-like OS than Unikernels do. 

The core point of this blog post being that Unikernels have a place in the world, but maybe not for running general purpose applications. Unikernels shine where millisecond boot times count and where you want to squeeze out every single possible instruction and run everything in ring 0.


If you are wanting to run general purpose applications with Unikernel-like minimalism and in Unikernel-like isolation then perhaps consider something like Yocto Linux or Alpine Linux running from RAM or TinyCore Linux in Run-From-RAM mode, and configure out the user login system

I had bet on the horse that wasn't the winner, I'd invested years of work into it, which I don't regret and had alot of fun with.  Off to start the next thing.......

Andrew Stuart
andrew.stuart@supercoders.com.au


Monday, March 28, 2016

Follow up to my Babel 6 rant

I wrote a blog post yesterday ranting about Babel 6. It was posted to Hacker News and viewed 10,000 times within one day.

It's the next day and my head is cooler so I want to say a few more things.

The first thing to acknowledge is my great appreciation for the hard work that open source developers do. They choose to contribute their time and considerable professional expertise writing software for free.  That's no small deal, so I say thank-you to the Babel 6 developers.

Babel 6 is amazing.  Being able to compile ES2015 and have it run on any browser is a true game changer.  ES5 is a real pain and I am eternally grateful to those people who enable me to write ES2015 instead of ES5.

The reason people bother to complain about something is that they care about that thing.  If they did not care then they would simply be indifferent. So the origin of my rant is that I am clearly deeply involved in the usage of Babel 6.

The thing I value more than anything is my own time.  I have nowhere near enough time.  It is my most scarce resource. So when software causes me to lose time in very large chunks, I get extremely frustrated.  Writing a rant like that is the outcome of months of losing vast chunks of time to Babel 6 configuration.  It's not a spur of the moment hotheaded outburst.

And although I delivered the message to the Babel 6 folks in an overly pointed manner, sometimes there is value in people receiving not only a message but also the level of emotional intensity.  The rant level of intensity is far more likely to draw the attention of the message recipient than a quietly spoken, factually driven list of issues posted to github.

An effective rant, whilst certainly emotional, should not be a personal attack on people.  Hopefully this one was not.

And for the recipient of a rant, it's an opportunity to hear from your software users who are deeply engaged .  As a software developer it can be very hard to grasp how a user experiences your system, so when you get a rant hopefully it is not just an attack but contains valuable insights that can help the developer to understand the user experience .

So thanks to the Babel 6 folks for an excellent product.  I'm not at all backing away from the content of my rant but I do want to color it further with my appreciation for your work, and indeed for the work of all open source developers.

Sunday, March 27, 2016

Babel 6 - useless by default - a lesson in how NOT to design software.

One of the hardest things about learning modern JavaScript programming is wrapping your head around the build systems - all the non-programming stuff you have to get going to run your code.

And for mortal humans it is *really hard* to grasp what the heck this complex ecosystem of build tools does and how they fit together, how to configure them and how to drive them.  REALLY hard.  Beginner JavaScript programmers who wish to do modern development face a massive learning curve just to get started. If you think it's easy then have climbed the learning curve and done your learning already and forgotten how hard it was.

So the easier the build tools are to install and use, the better.  Ideally, for common use cases, the end user will need to do precisely nothing apart from install the build tool to get basic and even advanced level usage out of it.

And Babel is one of the core parts of that build system.  It compiles ES2015 code to ES5 that can be run on any browser.

Out of the box, Babel doesn't do anything.  Literally useless by default.  Want to do something?  You MUST configure it.  And of course that means you have to start learning how to drive and configure Babel.

The first step in learning a new system is the hardest and most confusing and most time consuming.  To configure one tiny thing a vast array of questions come up.  Am I really meant to do this?  Should I be changing this setting?  Why do I need to do this? What does it do?  Am I breaking something?  Are there more things to configure? Am I working with the latest information or has the system changed since the documentation I am reading?  Why have I found three sets of conflicting instructions to do this configuration? And the biggest question of all - why do the instructions on this web page not match what I see on my screen? And then of course, "I've made the changes as instructed, why doesn't it work, what's wrong?". All these questions come from a hazy fog of ignorance of what the software is, how it holds together, what it's purpose is and why it works the way it does. You groan inwardly and sigh - fuck, how deep is this "simple" rabbit hole going to be? EVERYONE goes through this phase of learning a new technology.

If a build tool requires the user to make even one single configuration setting then they have imposed a huge learning cost on that user.  Possibly many hours of time and possibly (even probably) failure in trying to reach their goal.  For many users the cognitive load will be too high.  And the user may not even succeed in doing configuration that the developers consider to be basic and trivial.

Conversely, if a build tool requires a user to configure *absolutely nothing*, then they have saved the end user hours and driven them to a "pit of success".

Too many JavaScript build tools think that developers care about the build tool.  They think that developers want to understand the build tools, and want to learn and configure and program those build tools.  I'm not here to program build tools.  I am here to program application code.  I want my JavaScript build tools to anticipate likely use cases, come pre-configured to do as much as possible out of the box.

Experienced Babel 6 users will dismiss my assertion that configuration is hard. "It's easy!" they'll say.  You change one file: .babelrc, npm install your presets, install your plugins and modify your update your webpack config and you're good to go!  It's 60 seconds work so you are speaking garbage. "

Experienced developers forget how incredibly hard and time consuming and confusing it is for unexperienced users to do those 60 seconds worth of tasks.

It should only be necessary to configure JavaScript build tools for optimisation of final production code.  Forcing users to configure everything up front is premature optimization, wastes users valuable time, imposes a cognitive load on the user and makes the entire JavaScript development process massively more complex and hard and for some users trying to get started, will kill their enthusiasm and will kill the entire process for them

So when Babel 6 decided that it would do nothing out of the box, it did the opposite of conventional wisdom in software usability, which is to include sane defaults that anticipate the most common use cases and provide them, ideally with zero requirement for setup and configuration.

Babel should come pre configured with the kitchen sink of JavaScript development - async await, decorators etc etc.  They should throw everything in there that a developer might want to use.  The expert use case for Babel is to strip it down to the minimum.  It should not be the basic use case to build it up to usable. I shouldn't need to even think about how to configure Babel until I'm so far into my development process that I have already scaled the other learning curves that were essential such as learning ES2015 and browser programming.

Babel wants to exist, be seen and heard and talked about.  But in fact it is plumbing that you should really never have to give a thought to unless you're walking a very unusual road.  Babel's demands for you to configure it is like your taps in the kitchen needing you to get under the house with the wrench and hook up the hot water yourself, install a hot water tank, install solar panels and set the temperature in order to use it.  Not stuff you should need to do.  Babel should generally be neither seen or heard.  The Internet should not be plastered with blog posts instructing you on how to configure Babel, some right, some wrong, some out of date, some conflicting.  The developers should have configured it once, properly, so you don't need to think about it.

And finally, the cost to the Babel project itself is support issues and questions and problems from people who haven't managed to get things to configure, or didn't even know they were meant to configure - only that something isn't working.  The Babel project is shooting itself in the foot and making work for itself. The Babel developers should always be focused on how to reduce and remove the need for configuration, to find effective ways to preconfigure, and to find ways to allow people to use Babel with ever less knowledge that it even exists.

OK I'm headed back to see how to configure Babel to use async and await.  I really wish it was configured out of the box. What a waste of time.

P.S. and by the way I do know about Babel presets. That's too much configuration too.  The right amount of configuration is none.

Saturday, March 5, 2016

Get ready for our first civil war against robots - here's what it will look like.

Clearly we are already in a world in which robots are an integral part of war.  Nations use robots to fight against each other and against terrorists.

So far, however, we have not had a civil war against our own robots - against the robots who serve us. But that civil war against our servant robots is coming.  And sooner rather than later.

The first civil war against our own robots will be the battle against driverless trucks, which are, of course, robots.

In France, taxi drivers protested against uber.  It got violent.  The taxi drivers face losing their income to this disruptive force.

But the French taxis drivers protest has three constraints that prevent it turning into an all out war:


  • The first constraint is that it's not easy to identify an Uber car - they look like any other car, so it's hard to target them.  
  • The second constraint is that even if Uber cars could be easily identified, there's a natural reluctance to direct violence at them because Uber drivers are people and there are significant consequences for directing violence at another person.
  • The third constraint is that whilst there are alot of taxi drivers, there's not a huge number of them.  Wikipedia says that in the United States in 2012 there were 233,900 taxi drivers.


The war against robot trucks will have no such constraints.


  • Driverless trucks are big and hard to hide, and they have no driver, which makes them easy to spot and thus easy targets. They are unprotected.
  • There are no people in driverless trucks, so if done carefully, they can be attacked with minimal risk of hurting humans. The consequences of violence against robots are not as harsh as the consequences of violence against humans (yet).
  • According to the American Trucking Association there are over 3 million truck drivers in the U.S.A.  That's alot of potential "combatants" against the robots.

Driverless trucks will mean that alot of people lose everything, and when people have nothing to lose they become more willing to respond in an extreme manner.

There will be a war against driverless trucks.  It's gonna be a big one.  It will certainly be explosive and it's coming to the streets near you.






Wednesday, February 3, 2016

A full stack developer is only half as focused as a half stack developer

Stop obsessing about employing full stack developers.

JavaScript front end development is more than enough for one person to specialise in.

It is to your advantage to employ people who want to focus only on the front end.

I am seeing employers who insist on people being full stack and as a result miss out on getting great developers who are focused entirely on the web browser front end.


Friday, December 11, 2015

Atlassian's success reminds what a crappy place Australia is for tech business.

Right now Australia is brimming with excitement at the IPO of Atlassian - Australia's greatest software company success.

Success has many fathers and no doubt the government will want somehow to be one of Atlassian's dads.


Whilst Australia should be immensely proud and admiring of Atlassian's achievements, we should in fact hang our heads in shame.


Australia is a crappy country for technology business.

Successive Liberal governments have been at best ignorant of technology and at worst openly hostile towards it.  The Howard's government's Senator Richardson had the reputation for being a luddite whose primary interest during the birth of the world wide web was to protect the interests of established media.

The Howard Government invested nothing of any significance in Australia's technology future.


Australia had a brief foray into being a world leader in technology when Kevin Rudd announced the National Broadband Network which would roll fibre optic cable to the vast majority of Australian homes.  Steve Wozniak liked it so much that it was reported that he was applying for citizenship on the strength of it.


Labor soon lost government to the Liberals led by Tony Abbott who ripped the NBN plan to shreds and with current Prime Minister Malcolm Turnbull instead gave us a half assed patchwork excuse for a broadband network, cobbled together from whatever bits of cable TV coax and telephone copper network were already lying around. 

In an age where the state of the art broadband to the home in the U.S.A. is 10 gigabits, Australia is currently building (it claims) a National Broadband Network at great expense that promises a blistering 12 to 100 megabits per second at the top end. And there's no sign of the NBN in most parts of Australia anyway because it's clear the government was really only keeping the project alive under duress.

The Abbott Government was deeply ignorant of technology.  Tony Abbott smirking his catchphrase "I'm no Bill Gates, but..." whenever any question of technology came up. No doubt about that.


Speaking of our fibre-to-the-home smashing current Prime Minister Malcolm Turnbull, he has just released the latest government innovation initiative and of course it doesn't do what is needed to genuinely foster innovation.  Instead, the government remains focused on giving money to risk averse venture capitalists and pointless incubators that are little more than desks and chairs. 


In 2015 innovation is as likely to come from one, two or three people sitting in the lounge room coding away.  This latest innovation strategy does nothing to support the thousands of programmers who just need some help getting the bills paid whilst they sit in the lounge room and code for 18 months to get that first version up and running.  

Instead it's the time honored formula of driving people to spend their time not coding, but writing business plans for the government funded venture capitalists to reject until the business is successful anyway (sorry "has traction").

I don't know alot about Atlassian but it looks to me like they did it with no help from the Australian Government nor from the venture capitalists who purport to support innovative businesses but really are focused on getting their next fund cashed up by the government. Atlassian's enormous success serves to illustrate that Australia has come up with only one Atlassian-sized success story. 


That's what we have to be ashamed of - we have created only one Atlassian.  Where are the rest?


Australia is a country so hostile to technology and so poorly supported by Government and bumbling venture capitalists that I can recommend strongly that if you are a young entrepreneur and have the means to leave, then do so as fast as you can.  


Get out of Australia and get to Silicon Valley.  Atlassian is an outlier - the exception.  Don't take Atlassian as a sign that you can do it from Sydney or Melbourne - you're not Steve Jobs, you're not Mark Zuckerberg and you're probably not the next Atlassian.


You'll have a much better chance if you get out to where things really happen. Atlassian has succeeded despite being an Australian company, not because it is an Australian company. 


Being an Australian business confers no particular advantage in the tech world. Get thee to Silicon Valley young person.


andrew.stuart@supercoders.com.au

Wednesday, September 2, 2015

What "I don't have time" really means.

When someone explains why they didn't do something, one of the most common reasons is "I don't have time."

And it's a good reason.  Everyone understands.  Everyone is busy.  No-one has time.

Didn't get that work done?  Didn't have time.

Can't manage to see friends?  No time.

Never manage to see the grandkids? So hard to find the time.

Can't get to the gym?  No time for that either.


No one holds you to account when the reason is "I don't have time."  YOU don't hold you to account when you say to yourself "Why aren't I doing that important thing? Well I don't have time of course." Everyone understands.


But the truth is that "I don't have time." is just an EXCUSE.  It's a get-out-of-jail-free card. It's a socially acceptable way of concealing the less palatable truth of why something didn't happen.


When someone says "It's hard to find the time to do that", what they are REALLY saying is "That's not high enough priority to me.  Everything else in my life is a higher priority."


So next time you hear someone say "I don't have time", convert those words to "All the other things in my life are higher priority than that activity and that's why I didn't do it.".


There's the truth - everyone has time, it's just a question of whether they are willing to make it a high enough priority to actually do.


And next time you find yourself saying "I just don't have time for that.", trying saying instead "Everything else is a higher priority, sorry.".  Even if you only say it to yourself you'll get closer to the truth about why you do and don't do things.