I recorded my DJ set last night – dubby, gritty downtempo – and have just finished uploading it for your enjoyment:

March 2016 at the Dungeon

Notes about getting better power management out of Linux when running on a Thinkpad X300, like the one I have been so frustrated with recently that I gave it the hostname “aggravator”. I get maybe 90 minutes out of its battery, currently. I was thinking about replacing its DVD player with a second battery, but I’m reluctant to throw even more money at this thing when its reliability has been so underwhelming thus far.

KreativeKorp Relay Fonts has a glorious array of retrocomputing typefaces pulled from sources I haven’t thought about in years.

The junk removal people have come and gone. With them went all the random objects that have been sitting in the backyard at Rock Star Parking, unclaimed and unloved, since last summer’s Floodland truck unloading. In addition, they departed with an assortment of mysterious objects that had taken up residence under the porch for no apparent reason. Why, exactly, did we have two portable charcoal barbecues? Where did that knit bag of rubber stamps come from? Who put that big black plastic bag next to my motorcycle, and what was inside it? (None of us were brave enough to open it and find out.) And what kind of person brings a goddamn swamp cooler to a party and then forgets to take it home again afterward?

Also gone: the massive slab of seating furniture which would have been the first component of the now-cancelled couch-fortress project, the coffee table I’ve disliked since the day my ex-wife surprised me with it, and some bags of clothing which did not belong to any of this house’s past or present occupants. Alas: the cans of paint, piles of lead-acid batteries, and terrifyingly rusty propane cylinder were problems they were unwilling to solve. I guess it’s time to break out the box of thermite.

Resources for Amateur Compiler Writers

After sixteen years of membership, I just closed my Metafilter account. End of an era: it was the last piece of the old Web, the one I was excited about and emotionally invested in, the one that felt like a community – or a community of communities, really. No more; Metafilter’s culture has been growing steadily crankier over the years, less fun and more nitpicky, and has finally sunk below the “never read the comments” threshold.

The web is over; it’s basically just television, now, with a few exceptions. Reddit is still fun, though it’s more of a playground than a community. Blogs are over, message boards all seem to have gotten swallowed up by Facebook, and Facebook itself doesn’t have very much going on anymore.

I’m not sure what comes next, but I’m looking forward to finding out about it.

I didn’t get any photos, but the verdict was clear after Saturday night’s impromptu gathering: you totally can make delicious grilled-cheese sandwiches with a flamethrower. Use a soft cheese and keep the slices thin (so it’ll melt quickly), don’t butter the outside of the bread (it catches fire), and smash the sandwich between a couple of cast iron skillets (or use a panini press) if you’d rather not char the outside. 15-20 seconds of heat will do it, but you can go longer if you want the bread to get super crunchy.

Also, don’t assume that those propane cylinders you haven’t touched since last summer are still full.

A nice solid side-table lamp made of oak and epoxy, with step-by-step photos.

Colorization using optimization:

In this paper we present a simple colorization method that requires neither precise image segmentation, nor accurate region tracking. Our method is based on a simple premise: neighboring pixels in space-time that have similar intensities should have similar colors. We formalize this premise using a quadratic cost function and obtain an optimization problem that can be solved efficiently using standard techniques. In our approach an artist only needs to annotate the image with a few color scribbles, and the indicated colors are automatically propagated in both space and time to produce a fully colorized image or sequence. We demonstrate that high quality colorizations of stills and movie clips may be obtained from a relatively modest amount of user input.

O-expressions are a new syntax for Lisp-like languages, with an implementation called Liso which is based on Racket.

I’ve been playing with similar ideas recently, from a slightly more functional-programming perspective. Everyone agrees, for the most part, that delimiter pairs create structure, and that commas have higher precedence than semicolons. The idea that any consecutive sequence of non-delimiter punctuation characters constitutes an “operator” token is useful, and I like the use of INDENT/DEDENT as another form of bracketed delimiters. Is it possible, given the additional constraint of “no reserved identifiers”, to construct a language sufficiently general that a language like Python, Javascript, or (my own) Radian could be constructed on top of it as a macro library?

Aluminum extrusion channels for LED strip lighting.

I’ve just discovered a language called Sisal, developed in the early ’80s, which was based on the same ideas that inspired Radian. It was quickly forgotten, as most languages are, but a group of students resurrected it in 2010 and built a fine-grained parallelism backend for it, based on pthreads. Which was, again, the whole point of Radian. It looks like they pulled it off about a year before I got Radian doing the same tricks, but the project seems to have lapsed back into obscurity.

bookmark: useful notes on writing makefiles.

A straightforward and well-presented article about the hopelessness of the currently-dominant computing model with some sensible suggestions for a better way.

I had a startling thought on the bus this morning which led to a pretty exciting idea. What if I combined the work I’ve been doing on lightweight unikernels with all the work I did on Radian, turning Radian into a tool for building lightweight services?

I still think the Radian language is a good one; it’s clean and neat, and its semantics are constrained in just such a way that it’s easy to interact with asynchronous processes, and thus to split work up among multiple workers. As an experienced developer, I could do a lot of productive work with that language – but I think it’d be just as easy for a newbie to pick up as would python or javascript.

As a new language, though, it suffers from the unavoidable defect of being new: there is no language community, there are no libraries, there are no example programs, there are no tutorials, there is barely any documentation. Not if I made it my full-time job and worked the kind of hours I put in when I was doing Object Basic could I make up for that lack myself.

As a tool, it also has significant limitations. The constraints of its immutable dataflow semantics make it very difficult to interact with code written in other, less-constrained languages. That makes it very difficult to interact with the operating system and all of its libraries. Its parallel execution model and its dataflow semantics make it very difficult to debug using a conventional debugger: the traditional mode of stepping through line by line and watching the variables change is basically impossible, because there is nothing in the compiled code which cleanly matches up with the original source lines or variable names.

Finally, and worst of all, I was never able to find a problem for which Radian was a uniquely powerful solution. I looked into datamining, and then into numerics, but in each case I got bogged down in the immense amount of work necessary to get the core libraries up to the point that someone not already dedicated to the language would care to use it.

So, the new idea.

Unikernel-based services are still quite new, and *lightweight* unikernel services are still quite a cutting-edge activity. I’ve been having a great time poking around in the gritty guts of the x86 architecture but that’s because it reminds me of coding in the wild west of the late ’80s and early ’90s. Quite a barrier to entry!

What if I turned radian into a dedicated tool for building standalone microservices? The output would not be an executable program, but a bootable disk image you could start up inside a VM.

This would neatly turn all of radian’s disadvantages into strengths – or inevitable constraints of the target environment, at least. Can’t interact with system libraries? Well, isolation is the whole point of a lightweight unikernel service. Can’t step through with a debugger? Who cares, it’s probably running in the cloud anyway, so you debug it via log files – which is pretty much exactly the way I used to debug Radian programs anyway. Can’t link your favorite library into Radian? Doesn’t matter, just boot it up in a separate microservice, then make them sling packets at each other!

Radian’s strengths, meanwhile, would remain strengths. It’s all about slinging data between parallel asynchronous processes, and what else are microservices? Lightweight tasks are built right into the language, as is a synchronization mechanism which would fit perfectly on top of a socket-based communication mechanism.

Furthermore, there’s basically nothing else out there that does this. The only comparable system is MirageOS, which basically wraps OCaml and a bunch of unix libraries into a bootable VM. And that’s great, but it’s OCaml, and it’s a *much* heavier solution than I’d be presenting.

Radian, under this vision, would become shell scripting for the cloud: letting anyone with the tiniest trace of programming knowledge go from idea to running microservice in a matter of seconds. I am undoubtedly feeling unreasonably excited about this idea right now, but if I could accomplish this, it seems like it might be the most valuable thing I ever did with my career.

So. What would I have to do in order to bring this idea into reality? The radian compiler wouldn’t need any changes; all the work would be on the runtime side. I would certainly have to write some better documentation, though. I’d then replace the fleet kernel’s C runtime with the Radian runtime and rework the radian allocator and threads accordingly. I’d also have to write a little bit of code for emitting a disk image, and for invoking qemu on it.

As far as the radian standard library goes, I’d scrap the fancy but unfinished bignums package and bang out something simple and reliable based on good ol’ int64 and float64. The major piece of new work I’d have to do would be the development of a socket interface. Since I’d be going for an all-in-one runtime library, though, I could scrap the clunky ffi workarounds that used to make this sort of thing so laborious, and just build it right in! Call straight into the kernel, why not.

At that point… we’ve got sockets, tools for working with data going in and out of sockets, tools for managing complex asynchronous processes involving multiple sockets, and a tool that turns source code into a running VM in a snap of the fingers. What’s next? Time to play.

This would be an insanely formidable task if I were thinking about starting it from scratch, but I’ve already done all the hard parts! What I’ve just described sounds more like… assembling pieces.

Wow. Could I actually make all those years of effort pay off? I think I might try it.

The Website Obesity Crisis: starts off pointing out just how absurdly large websites have become, then continues – in remarkably straightforward, plain-spoken terms – to highlight everything I hate about modern web design. Such a satisfying read.

two projects:
– router sets up first VM, exposes libvirtd proxy, launches boot process; implements firewall policies between fleet VMs
– shell implementation (bash fork? just posix sh? something new?) uses DNS-SD to locate libvirtd proxy, implements process management in terms of VMs

then, port/compile the programs I want to use as unikernel VM images.

I’ve made significant progress toward the completion of a new, non-POSIX-oriented unikernel library in the form of a baremetal C99 runtime, but even as I’m cruising down the home stretch I think I will put the project aside. Sure, my unikernel library would be tighter and lighter than the rest, if I finished it, but it really doesn’t add all that much, and working on it means I haven’t been making any progress on the stuff that is truly unique to my vision.

IncludeOS is marketed as a cloud-service tool, but when I look at what they’re actually building, it looks very much like what I’m doing. Their project is C++, mine is C; their architecture is a bit more full-featured, where mine is deliberately minimal… but it’s still basically the same idea, and I generally agree with all of their architectural choices. So… why not join forces? I can always dust off my fleet-kernel later, if there’s some problem it would solve better; one of the great things about this VM/unikernel architecture is that you can have as many kernel architectures as you like running simultaneously.

The part I really need to build, in order to get my vision for fleet off the ground, is the router/shell system that will make unikernel-based programs useful as everyday workstation tools. I’ve grown increasingly antsy as development of my unikernel library has dragged on, preventing me from getting to the shell; it’s time to refocus and get on with it.

Maybe I can contribute some of my stuff to IncludeOS along the way, too.

Useful information about electric bicycle commuting in Seattle, including links to relevant RCW statutes and recommendations for equipment.

I miss the web. I didn’t expect it to be over so quickly. What’s next, I wonder? Should we try to build another one, or is the concept fundamentally flawed?

The proper genre for Thanksgiving is science-fiction:

It has come to my attention that people are woefully uninformed about certain episodes in the Thanksgiving narrative. For example, almost no one mentions the part where Squanto threatens to release a bioweapon buried under Plymouth Rock that will bring about the apocalypse.

Mr. S, an ordinary American, is minding his own business outside his East Coast home when he is suddenly abducted by short large-headed creatures like none he has ever seen before. They bring him to their ship and voyage across unimaginable distances to an alien world both grander and more horrible than he could imagine. The aliens have godlike technologies, but their society is dystopian and hivelike. Enslaved at first, then displayed as a curiosity, he finally wins his freedom through pluck and intelligence. Despite the luxuries he enjoys in his new life, he longs for his homeworld.

ne, the nice editor:

ne is a free (GPL’d) text editor based on the POSIX standard that
runs (we hope) on almost any UN*X machine. ne is easy to use for
the beginner, but powerful and fully configurable for the wizard, and most
sparing in its resource usage. If you have the resources and the patience to
use emacs or the right mental twist to use vi then
probably ne is not for you. However, if you need an editor that:

• compiles without effort everywhere (or almost everywhere), is packaged for
  all Linux distributions, and ported to other operating systems (such as Mac OS X);
• is fast, small, powerful and simple to use;
• has standard keystrokes (e.g., copy is CTRL-C);
• uses little bandwidth, so it is ideal for email, editing through phone line (or
  slow GSM/GPRS/UMTS) connections;
• has a very compact internal text
  representation, so you can easily load and modify very large
  files…

… then you should try ne.

From IEEE Spectrum, Bosch’s Giant Robot Can Punch Weeds To Death:

At IROS last month, researchers from a Bosch startup called Deepfield Robotics presented a paper on “Vision-Based High-Speed Manipulation for Robotic Ultra-Precise Weed Control,” which has like four distinct exciting-sounding phrases in it. We wanted to write about it immediately, but Deepfield asked us to hold off a bit until their fancy new website went live, which it now has. This means that we can show you video of their enormous agricultural robot that can autonomously detect and physically obliterate individual weeds in a tenth of a second.

Oh yeah, this project to marry unikernels with the Qubes system is pretty much exactly what I have been going for with Fleet.

A few solid hours hacking on fleet last night got me to a basic C “hello world”. The C library has everything from string.h and the simple character I/O functions from stdio.h, and the kernel has just enough of a driver interface to make simple file operations work. I’m using the legacy serial ports for now, since the drivers are trivial; stdin maps to COM1, stdout is COM2, and stderr writes to the host console via port E9.

This is the part where it starts to get really interesting. If you handwave past the part where most of the C library isn’t there yet, it’s now possible to compile a couple of ordinary C programs, link them against the fleet library, run them in VMs, and pipe data between them in grand old Unix shell style. It’s all very normal- except that these processes are just as comprehensively isolated from each other as if they were running on separate physical machines.

This fleet project is a lot of fun, combining a shiny new idea with an excuse to take a crack at a lot of classic problems.

The next layer after the startup code should be something to do with drivers and the low-level kernel apparatus, but it all felt a bit vague, so I decided to start with the C standard library interface and work my way down, letting that drive the rest of the kernel architecture.

There are dozens of free C library implementations available, but I have not been able to find one that will work for my project. I don’t want POSIX support, don’t need hardware portability, and won’t have a Unix-style system call interface underneath. And while I’m building this in the style of an embedded firmware project, it’s actually designed to run on a purely virtual machine, so I don’t need or want a lot of code dealing with legacy PC hardware.

Oh, well, I’m writing my own C library. Of course I’ll fill it in with a lot of existing components, but this architecture is apparently weird enough that the framework is up to me.

I did write the string library myself, though, because I thought it would be fun. There sure is a lot of weirdness in there – it’s been 23 years since I learned C, and I can’t say I had ever noticed the existence of strspn, strxfrm, or strcoll – but now I’ve written ’em and built test suites too.

I factored the lowest-level portion of the fleet code out as a standalone library which I’ve named ‘startc’ and posted on github. I also announced it on reddit.com/r/programming and on hackernews. Of course it feels a wee bit nerve-racking to post something experimental like this for the world to examine, but it’s a good exercise as it forces me to get all the loose ends tied up and to really think carefully about the interfaces between modules. So far the reception has been generally positive, which is nice. I have no idea whether anyone will actually use the library, but perhaps someone will get through the early stages of a similar project more quickly by looking at its source code, and that would make me feel good.

According to The Death Clock, I have about a billion seconds left.

That… seems reasonable.