The Internet of Things is upon us, and being able to efficiently communicate with those things is going to become more and more difficult as more things get connected. I’ve been working as a consultant for Nexia Home Intelligence for the last year, and they were kind enough to let me blog a bit about what it is we’re doing to help them handle the large number of connected devices involved in running a hosted home automation solution.

TL;DR – Elixir/Erlang are Awesome

After trying out several Ruby-based solutions to the issues we faced handling the crush of 50k internet-connected devices, including EventMachine and Celluloid, running on MRI and jRuby, we spiked a solution on Elixir, which runs on the Erlang virtual machine. Leveraging the Erlang VM and out-of-the-box OTP frameworks, along with a few more advanced tricks, we’ve built a system that can easily handle our target workload.

Connections, Connections, Connections

There are lots of “things” in the world, and with IPv6 slowly gaining traction, many more of them will have an IP address (or at least internet connectivity of some kind). In many cases, the communication with these devices may be mostly one-way, where the device simply reports information back to a central server somewhere. In those cases, a simple HTTP-based API may well be able to handle the communications needs of your devices, and normal scaling suggestions for web servers will probably serve you just fine. However, there are certain classes of devices (locks, cameras, and thermostats, for example), that require bidirectional, near-real-time communication, and it is this kind of device that we’re dealing with.

In the Beginning…

Our original implementation leveraged Ruby’s EventMachine, which is an evented I/O system for Ruby (similar to Node.js, if that helps). Devices would connect directly to the Ruby TCP/IP server via SSL, and communicate with our back-end Rails-based system via Resque. Outbound data from the Rails app was sent to RabbitMQ, with a header identifying the specific device to which the message was to be directed. The device servers all subscribed to the RabbitMQ exchange, and whichever one held the connection to the device would send out the results. This system held up well under stead-state load in the 3-6000 connection range. However, there were some issues…

The Stampede

There are several kinds of problems you’ll run into when you try to build a system that can handle 10s to 100s of thousands of simultaneous TCP/IP connections. The first one is “how do I handle all of these things connecting in rapid succession.” This was, in fact, the original impetus for investigating splitting the TCP/IP handling side of things from the “Business Logic” side. And, because all problems in Computer Science can be solved with an additional level of indirection, we decided to add an additional layer in front of the business logic to simply terminate SSL connections and route raw data to a pool of workers, which would handle the business logic and communication with the back-end. Given we are familiar with EventMachine, it seemed to be a no-brainer to simply split the original implementation in two, with some additional logic to handle things like message ordering now that data flowing to/from the device was transiting via a second hop over RabbitMQ instead of the more direct path from before. So that’s what we did.

Houston, we have a problem…

We quickly found that the new system was unable to keep up with the load of even 5,000 devices, much less our goal of something closer to 50,000 devices on a single machine. Things were not looking good. Using rubyprof, we found (and fixed) several performance-related issues with Ruby’s amqp-gem, which made some significant performance improvements. However, at this point we were CPU-bound with no obvious single bottleneck left to go after to further improve performance. It appeared that the single-threaded nature of EventMachine, along with some additional amqp-gem related performance issues (even when load-balancing against many instances on a multi-core machine), were going to sink this implementation.

A Side-Track Through Celluloid.io

Given our core application was a Ruby on Rails application, we really wanted to stay on Ruby for this solution, so we spent some time spiking a solution on Celluloid.io to see if its actor-based, multi-threaded model (and using jRuby as our underlying ruby implementation) would help resolve our issues. Running on jRuby, and using the march-hare gem, which is a lightweight wrapper around the java-based AMQP client, we hoped to be able to hit our targets. And, while I must admit that we abandoned this effort without a significant amount of profiling time, it only got us to about 15k connections before it fell over, mostly due to huge memory consumption (on the order of 30 GB). My understanding (learned much later) is that this may indicate that we did something with Celluloid that we shouldn’t have, but we didn’t have time to continue down this route to try to fix the memory leak at this time.

Elixir to the Rescue

Finally, we felt it was necessary to try something a bit more drastic. Having been introduced to Elixir a few months ago, and understanding that the Erlang runtime was designed in large part for this kind of problem, we approached the team at Nexia with the suggestion that we take a few weeks and spike a solution using Elixir. We chose Elixir over Erlang because its syntax was much closer to the Ruby that the developers of the existing Rails application, which should make transitioning this work to their core team easier. So, we started to learn more about Elixir, Erlang, and the OTP framework that promised to help us build a scalable, robust system that could, conceivably, provide 99.9999999% uptime (ok, maybe we’re not that good, but Erlang can be).

A quick note – Elixir is a great language on top of the amazing Erlang runtime, and the OTP framework and libraries really provide most of the features we’re leveraging for this application, so you can mostly replace Elixir with Erlang in this post, and I’ll try to call out Elixir-specific stuff if there is any.

Welcoming the Herd With Open Arms…

or, how do you accept 1000 connections per second on a single machine, with one OS process. Most of the example TCP/IP (or SSL, in our case) server examples you find on the internet generally do something like: 1. Create a listen socket 2. block on that socket until a client connects (accept) 3. hand off the newly-accepted socket to some other process 4. Goto 1

In Elixir, this would look something like this tail-recursive call that implements our loop above for an SSL-based connection:

defp do_listen(listen_socket) do
 {:ok, socket} = :ssl.transport_accept(listen_socket)
 :ok = :ssl.ssl_accept(socket)
 endpoint = TcpSupervisor.start_endpoint(socket)
 :ssl.controlling_process(socket, endpoint)
 :gen_server.cast endpoint, {:start}
 do_listen(listen_socket)
end

Notice that you’ve now single-threaded your application’s ability to accept new connections, which will eventually cause the operating system to simply refuse new connections on your listening port if you can’t keep up with accepting them in a timely manner. There are some things you can tweak to get more time (especially, increasing the listen backlog for your service to allow more pending connections), but eventually you’re going to have to do something about that single listener. In Elixir, the answer is to spin up multiple “acceptor” processes, each of which blocks on the same listen port (yes, you can do this!). When a new connection arrives, it will awake the next available waiting process and that process will handle that connection. This pattern has allowed us to get to 1000 connections/second on a single server quite easily (and we haven’t really found out what the upper limit was). The code is obviously a bit more complex. First, we have a supervisor that owns the listen socket, and its children will be the acceptor processes:

defmodule Owsla.TcpListenerSupervisor do
 use Supervisor.Behaviour
 def start_link(port, acceptor_count, backlog) do
 :supervisor.start_link({ :local, :listener_sup}, __MODULE__, [port, acceptor_count, backlog])
 end
 def init([port, acceptor_count, backlog]) do
 :ssl.start()
 {:ok, listen_socket} = create_listen_socket(port, backlog)
 spawn(fn -> 
 Enum.each(1..acceptor_count,
 fn (_) -> start_listener() end
 )
 end)
 tree = [ worker(Owsla.TcpAcceptor, [listen_socket], restart: :permanent) ]
 supervise(tree, strategy: :simple_one_for_one)
 end
 def create_listen_socket(port, backlog) do
 tcp_options = [
 :binary,
 {:packet, :line}, 
 {:reuseaddr, true}, 
 {:active, false},
 {:backlog, backlog}
 ]
 :gen_tcp.listen(port, tcp_options)
 end
 def start_listener() do
 :supervisor.start_child(:listener_sup, [])
 end 
end

Next, we have the acceptors themselves:

defmodule Owsla.TcpAcceptor do
 use GenServer.Behaviour
 @ssl_options [{:certfile, "deviceserver.crt"}, {:keyfile, "deviceserver.key"},
 {:ciphers, [{:dhe_rsa,:aes_256_cbc,:sha256},
 {:dhe_dss,:aes_256_cbc,:sha256},
 {:rsa,:aes_256_cbc,:sha256},
 {:dhe_rsa,:aes_128_cbc,:sha256},
 {:dhe_dss,:aes_128_cbc,:sha256},
 {:rsa,:aes_128_cbc,:sha256},
 {:dhe_rsa,:aes_256_cbc,:sha},
 {:dhe_dss,:aes_256_cbc,:sha},
 {:rsa,:aes_256_cbc,:sha},
 {:dhe_rsa,:'3des_ede_cbc',:sha},
 {:dhe_dss,:'3des_ede_cbc',:sha},
 {:rsa,:'3des_ede_cbc',:sha},
 {:dhe_rsa,:aes_128_cbc,:sha},
 {:dhe_dss,:aes_128_cbc,:sha},
 {:rsa,:aes_128_cbc,:sha},
 {:rsa,:rc4_128,:sha},
 {:rsa,:rc4_128,:md5},
 {:dhe_rsa,:des_cbc,:sha},
 {:rsa,:des_cbc,:sha} 
 ]}]
 def start_link(listen_socket) do
 :gen_server.start_link(__MODULE__, listen_socket, [])
 end
 def init(listen_socket) do
 :gen_server.cast self, {:listen}
 {:ok, listen_socket }
 end
 def handle_cast( {:listen}, listen_socket) do
 do_listen(listen_socket)
 end
 defp do_listen(listen_socket) do
 case :gen_tcp.accept(listen_socket) do
 {:ok, socket} ->
 case :ssl.ssl_accept(socket, @ssl_options) do
 {:ok, ssl_socket} ->
 endpoint = Owsla.TcpSupervisor.start_endpoint(ssl_socket)
 :ssl.controlling_process(ssl_socket, endpoint)
 :gen_server.cast endpoint, {:start}
 do_listen(listen_socket)
 {:error, :closed} ->
 do_listen(listen_socket)
 end
 {:error, :closed} -> do_listen(listen_socket)
 {:error, _} -> { :stop, :error, [] }
 end
 end
end

The following lines are the happy path of accepting an SSL connection:

{:ok, ssl_socket} ->
 endpoint = Owsla.TcpSupervisor.start_endpoint(ssl_socket)
 :ssl.controlling_process(ssl_socket, endpoint)
 :gen_server.cast endpoint, {:start}
 do_listen(listen_socket)

which:

However, now we have 1000 of these running at a time, with a TCP/IP backlog of 2000 connections, and we have no issue handling 1000 connections/second. Note that we also haven’t tweaked those numbers at all – this was our first guess, and it “just worked” so we left it alone (but configurable). It’s possible these are non-optimal, YMMV, IANAL, etc. From there, the individual endpoint processes forward messages across RabbitMQ to our back-end systems. There were some additional challenges there, which I’ll talk about in another post, but we're now handling 50k concurrent connections pretty easily on a single machine.

Chalk one up for the enemies of patent trolls: The Supreme Court on Monday threw out a request for trial from alleged patent troll Soverain Software.

The case, called Soverain Software LLC. v. Newegg Inc., is one of three that are expected to go before the Supreme Court this year. While the Court will likely hear the remaining cases, which deal with finer points of patent law, its dismissal of Soverain speaks to the potential frivolousness of its claims.

Soverain acquired the rights to numerous pieces of code tied to the online shopping cart, developed in the 1990s. In recent years, Soverain has gone on a litigious tear, suing more than two dozen companies including Amazon, Nordstrom, Macy's and Newegg, an online retailer, which all use shopping carts for internet sales.

Soverain had some success suing on the state level, where a Texas jury awarded the Chicago-based company $2.5 million in damages against Newegg. However, Soverain lost on appeal last year in U.S. District Court for the Eastern District of Texas, which ruled the shopping cart patents owned by Soverain were too general.

Patent trolls typically acquire rights to fallow or soon-to-expire patents with no intention of using the patent. Often patent trolls set up shell companies whose only assets are the patents, which means they have no real revenues or assets. Their sole purpose is to harass small businesses, which usually settle rather than pay for extended and costly litigation.

Patent law was originally written to protect the patent holder, making it easier for the patent holder to prevail in court. For the patent infringer to win, rather, the defendant must prove exceptional circumstances--namely that the patentee acted in bad faith and made baseless claims. This is hard to do. While the patent holder can be awarded "treble damages," or three times the damage claimed, the most the infringer can ever collect is attorney fees. 

The remaining cases before the Supreme Court will deal with these finer points.

Congress is examining legislation that would fight patent trolls and their frivolous lawsuits by making them liable for court costs, should they lose their cases.

Small businesses mounted 3,400 legal defenses in 2011 for patent cases, a 32 percent increase over the prior year, according to a research paper from 2012 by Boston University law professors James Bessen and Michael J. Meurer.  That cost to small companies was about $11 billion in 2011, also a 32 percent increase over the prior year.

The total median awards to trolls is now nearly twice as high as those to legitimate patent holders, whose median reward fell about 30 percent to $4 billion, according to a 2013 report by PriceWaterhouseCoopers.

IMAGE: iStock

Last updated: Jan 13, 2014

The 34 year old C code is the C code for ed and sed from Unix Version 7. I’ve been getting it to compile on a modern POSIXish system.

Some changes had to be made. But not very many.

The union hack

sed uses a union to save space in a struct. Specifically, the reptr union has two sub structs that differ only in that one of them has a char *re1 field where the other has a union reptr *lb1. In the old days it was possible to access members of structs inside unions without having to name the intermediate struct. For example the code in the sed implementation uses rep->ad1 instead of rep->reptr1.ad1. That’s no longer possible (I’m pretty sure this shortcut was already out of fashion by the time K&R was published in 1978, but I don’t have a copy to hand).

I first changed the union to a struct that had a union inside it only for the two members that differed:

struct reptr {
 char *ad1;
 char *ad2;
 union {
 char *re1;
 struct reptr *lb1;
 } u;
 char *rhs;
 FILE *fcode;
 char command;
 char gfl;
 char pfl;
 char inar;
 char negfl;
} ptrspace[PTRSIZE], *rep;

The meant changing a few “union reptr” to “struct reptr”, but most of the member accesses would be unchanged. ->re1 had to be changed to ->u.re1, but that’s a simple search and replace.

It wasn’t until I was explaining this ghastly use of union to Paul a day later that I realised the union is a completely unnecessary space-saving micro-optimisation. We can just have a plain struct where only one of the two fields re1 and lb1 were ever used. That’s much nicer, and so is the code.

The rise of headers

In K&R C if the function you were calling returned int then you didn’t need to declare it before calling it. Many functions that in modern times return void, used to return int (which is to say, they didn’t declare what they returned, so it defaulted to int, and if the function used plain return; then that was okay as long as the caller didn’t use the return value). exit() is such a function. sed calls it without declaring it first, and that generates a warning:

sed0.c:48:3: warning: incompatible implicit declaration of built-in function ‘exit’ [enabled by default]

I could declare exit() explicitly, but it seemed simpler to just add #include <stdlib.h>. And it is.

ed declares some of the library functions it uses explicitly. Like malloc():

char *malloc();

That’s a problem because the declaration of malloc() has changed. Now it’s void *malloc(size_t). This is a fatal violation of the C standard that the compiler is not obliged to warn me about, but thankfully it does.

The modern fix is again to add header files. Amusingly, version 7 Unix didn’t even have a header file that declared malloc().

When it comes to mktemp() (which is also declared explicitly rather than via a header file), ed has a problem:

tfname = mktemp("/tmp/eXXXXX");

2 problems in fact. One is that modern mktemp() expects its argument to have 6 “X”s, not 5. But the more serious problem is that the storage pointed to by the argument will be modified, and the string literal that is passed is not guaranteed to be placed in a modifiable data region. I’m surprised this worked in Version 7 Unix. These days not only is it not guaranteed to work, it doesn’t actually work. SEGFAULT because mktemp() tries to write into a read-only page.

And the 3rd problem is of course that mktemp() is a problematic interface so the better mkstemp() interface made it into the POSIX standard and mktemp() did not.

Which brings me to…

Unfashionable interfaces

ed uses gtty() and stty() to get and set the terminal flags (to switch off ECHO while the encryption key is read from the terminal). Not only is gtty() unfashionable in modern Unixes (I replaced it with tcgettattr()), it was unfashionable in Version 7 Unix. It’s not documented in the man pages; instead, tty(4) documents the use of the TIOCGETP command to ioctl().

ed is already using a legacy interface in 1979.

This entry was posted on 2013-09-01 at 15:50:23 and is filed under programming.

Tags: ed, sed, version 7 unix

MOSCOW, January 14. /ITAR-TASS/. Russian search engine Yandex was granted access to public posts made by Facebook users in CIS states and Turkey (countries where Yandex officially operates). According to an announcement made by Yandex, the sides have signed an agreement.

Previously the search engine only indexed personal and community pages on Facebook. As of now, Yandex has access to the whole database and can display posts and comments as search results. The only search engine which was granted similar access is Microsoft’s Bing.

Currently Facebook posts can be found using Yandex blog search; later they will be added to general web search. Comments are also not indexed yet, only posts themselves. Yandex representatives highlight that private posts are not indexed.

Facebook data will also be used by Yandex to improve search quality. The search engine will be able to add newly posted articles and videos, popular on Facebook, to relevant searches. Facebook popularity of content will also determine ranking of search results.

Yandex is the largest internet company operating in East Europe, valued at $13.9 after its Nasdaq IPO in 2011. Its key service is the search engine, which drivers the company’s revenue through ad sales. In Q3, 2013, the company’s revenue amounted to 10.2 billion rubles, 4.4 billion after EBITDA adjustment with a net profit of 5 billion rubles. Yandex shareholders are founder and general director Arkadiy Volozh, Baring Vostok Capital Partners fund and others.

Facebook is the world’s largest social network, valued at $137.2 billion /Nasdaq/. Monthly audience is estimated at 1.19 billion users. Q3 2013 earnings are in the range of $2 billion, operational profit - $736 million, net profit - $425 million.

Tony Fadell, in an interview today with Om Malik, on why Nest sold to Google:

Consider Fadell’s record with the iPod at Apple. Clearly, he knows how to do hardware: design, utility, software integration. But he also knows scale — creating devices that sell tens of millions of units. And he knows iteration: not just building and shipping an iPod, but building and shipping new iPods year after year, each better, in some way, than the last.

One of Alan Kay’s numerous oft-cited quotations is, “People who are really serious about software should partner with an OEM in Asia.” No, wait, that’s not what he said. What he said is, “People who are really serious about software should make their own hardware.” That’s never been true of Google, putting aside Motorola (which they seemingly acquired more for its patent portfolio than for its phone hardware acumen) and the niche Google Search Appliance.

In a sense, Google has always followed Kay’s adage. The software that Google was most serious about — web search, Gmail, and so forth — ran in the cloud, and with the company’s legendary data centers, they effectively built their own hardware.

Google now has a division with a remarkable consumer hardware track record. Nest and Fadell now have the financial resources to work faster. Money doesn’t solve scaling problems, but the actual solutions to scaling problems always cost money. Google’s Nest acquisition has very little to do with selling thermostats and smoke detectors in particular. Instead, it’s about Google having the ability to do consumer hardware right, in general.

360 view of my DIY PiPad Raspberry Pi tablet.

It seems that every day a manufacturer comes out with a new tablet computer. Thinner, lighter, faster, but it seems that they all look about the same and accomplish roughly the same things. When I set out to build my Raspberry Pi tablet I wanted something different. I wanted an all-in-one system that was usable, portable, and Linux based. Additionally, it had to look good. Since I wanted to use it on flights the device couldn’t freak out the TSA or the old lady sitting next to me.

My custom PiPad tablet.

Early in 2013 I started accumulating parts. The Raspberry Pi runs off of 5V so I knew it could be powered from a cell phone charger. Most touchscreens I could find were 12V though, making the electrical work more complex. After a bit of searching I finally found what I was looking for: a touchscreen monitor with a 5V HDMI to LVDS converter from a site called Chalk-Elec.com. I plugged the screen in as soon as I received it and to my delight it worked perfectly with the Pi, even the capacitive touchscreen. Now I knew my dream of a Raspberry Pi tablet was possible.

According to Parkinson’s Law, “Work expands so as to fill the time available for its completion.” Thus was the case for my Pi tablet. Two weeks before Maker Faire Bay Area I was helping a guy in the Shed Tech Support queue that needed some help with his Maker Faire project. Helping him got my creative juices flowing and I decided I wanted a Maker Faire project too. Crazy – right? I had all the parts and now I had an ambitious deadline that couldn’t allow for expansion. Fortunately I had started some preliminary design work so I *kind of* knew what I was going for. I happened to have access to a CNC machine, some 1/2″ Baltic birch plywood, and a relatively large sheet of scrap carbon fiber laying around to form the basis of the frame.

Cut files were created in Vectric's Aspire. Components were scanned and traced for fitment.

The files were cut on my CNC machine.

Due to time constraints and cut path layer confusion, I used a hobby knife to cut the frame.

Cutting access ports in the frame with a hobby knife.

Cutting through layers of plywood.

A bit from the top of the frame has to be cut out too.

Cutting in the rest of the components.

Frame with all cuts complete.

Looking good on this side.

This side looks good too.

I later found out this was far too much foam tape.

Measuring for the SD card slot.

My DIY drill level looks ghetto but it worked great.

Clearing out the SD card slot with a coping saw.

Smoothing out the SD slot with a strip of sand paper.

Almost ready for the top frame.

It took a bit of sanding to get the LCD to fit correctly.

Finally got it sitting flush into the frame.

Everything fits! Time to stain.

All hardware was removed for finishing.

The carbon fiber backing was glued into place.

Final component installation. Notice the missing Ethernet and USB port?

The completed setup running RaspBMC (the touchscreen driver doesn't work in RaspBMC.)

‹›

View All

(Full parts listing and design files can be found here.)

After several 4am nights I completed the Raspberry Pi tablet (aka, PiPad) the day before my flight. The build wasn’t without its issues (I had to remove one USB port and the Ethernet jack due to clearance problems.), but everything worked and I was happy with the results. But what about the TSA?

My PiPad caught the attention of the flight attendant, but only because she liked the movie I was watching.

This image was taken while on my flight to San Francisco. It didn’t raise an eyebrow going through security. On the plane though, a flight attendant kept walking by looking closely at the home-built gadget I had on my lap. At one point I could feel her looking over my shoulder and was sure she was going to say something. She nudged me (I thought it was over at this point.) and said, “I love that movie – you’re coming up to the best part!” It turns out that she’d been catching glimpses of “Talladega Nights” that I had playing using RaspBMC. I’ve taken the PiPad on most flights since and no one has said a word.

Eben Upton signed the carbon fiber back of my PiPad!

I’d e-mailed Raspberry Pi Foundation founder Eben Upton a few times for work, but didn’t have the chance to meet him at Maker Faire Bay Area. I finally managed to catch up with him at Maker Faire New York though. Eben is probably the most humble, down to earth person I’ve ever met. I really can’t say enough about him. After a long chat I showed him the PiPad. He gave it several compliments and after a few minutes of playing with it, he graciously signed the back at my request. His signature looks amazing on the carbon fiber!

Overall I’m very happy with my Raspberry Pi tablet. It does what I want it to do and has been a great way to demonstrate the capabilities of the Raspberry Pi at Maker Faires. (Perhaps you noticed it?) The 10,000mAh battery provides a usable six hours of run-time and the device gets constant compliments from makers. (Including Bunnie Huang!) I do wish I would have used a battery that provides power while plugged in. Other changes I’d make would be mostly software related. It’s difficult to double-click on icons reliably and the N-Trig touch driver isn’t supported by RaspBMC, (but can be compiled into the kernel if I could ever get it figured out). I’ve also considered adding a camera and an IR sensor… maybe if I build another one.

A few weeks into the making of Her, Spike Jonze’s new flick about romance in the age of artificial intelligence, the director had something of a breakthrough. After poring over the work of Ray Kurzweil and other futurists trying to figure out how, exactly, his artificially intelligent female lead should operate, Jonze arrived at a critical insight: Her, he realized, isn’t a movie about technology. It’s a movie about people. With that, the film took shape. Sure, it takes place in the future, but what it’s really concerned with are human relationships, as fragile and complicated as they’ve been from the start.

Of course on another level Her is very much a movie about technology. One of the two main characters is, after all, a consciousness built entirely from code. That aspect posed a unique challenge for Jonze and his production team: They had to think like designers. Assuming the technology for AI was there, how would it operate? What would the relationship with its “user” be like? How do you dumb down an omniscient interlocutor for the human on the other end of the earpiece?

When AI is cheap, what does all the other technology look like?

For production designer KK Barrett, the man responsible for styling the world in which the story takes place, Her represented another sort of design challenge. Barrett’s previously brought films like Lost in Translation, Marie Antoinette, and Where the Wild Things Are to life, but the problem here was a new one, requiring more than a little crystal ball-gazing. The big question: In a world where you can buy AI off the shelf, what does all the other technology look like?

Technology Shouldn’t Feel Like Technology

One of the first things you notice about the “slight future” of Her, as Jonze has described it, is that there isn’t all that much technology at all. The main character Theo Twombly, a writer for the bespoke love letter service BeautifulHandwrittenLetters.com, still sits at a desktop computer when he’s at work, but otherwise he rarely has his face in a screen. Instead, he and his fellow future denizens are usually just talking, either to each other or to their operating systems via a discrete earpiece, itself more like a fancy earplug anything resembling today’s cyborgian Bluetooth headsets.

In this “slight future” world, things are low-tech everywhere you look. The skyscrapers in this futuristic Los Angeles haven’t turned into towering video billboards a la Blade Runner; they’re just buildings. Instead of a flat screen TV, Theo’s living room just has nice furniture.

This is, no doubt, partly an aesthetic concern; a world mediated through screens doesn’t make for very rewarding mise en scene. But as Barrett explains it, there’s a logic to this technological sparseness. “We decided that the movie wasn’t about technology, or if it was, that the technology should be invisible,” he says. “And not invisible like a piece of glass.” Technology hasn’t disappeared, in other words. It’s dissolved into everyday life.

Here’s another way of putting it. It’s not just that Her, the movie, is focused on people. It also shows us a future where technology is more people-centric. The world Her shows us is one where the technology has receded, or one where we’ve let it recede. It’s a world where the pendulum has swung back the other direction, where a new generation of designers and consumers have accepted that technology isn’t an end in itself–that it’s the real world we’re supposed to be connecting to. (Of course, that’s the ideal; as we see in the film, in reality, making meaningful connections is as difficult as ever.)

Theo Twombly still sits at a desktop computer when he’s at work, but otherwise he rarely has his face in a screen.

Jonze had help in finding the contours of this slight future, including conversations with designers from New York-based studio Sagmeister & Walsh and an early meeting with Elizabeth Diller and Ricardo Scofidio, principals at architecture firm DS+R. As the film’s production designer, Barrett was responsible for making it a reality.

Throughout that process, he drew inspiration from one of his favorite books, a visual compendium of futuristic predictions from various points in history. Basically, the book reminded Barrett what not to do. “It shows a lot of things and it makes you laugh instantly, because you say, ‘those things never came to pass!’” he explains. “But often times, it’s just because they over-thought it. The future is much simpler than you think.”

That’s easy to say in retrospect, looking at images of Rube Goldbergian kitchens and scenes of commute by jet pack. But Jonze and Barrett had the difficult task of extrapolating that simplification forward from today’s technological moment.

Theo’s home gives us one concise example. You could call it a “smart house,” but there’s little outward evidence of it. What makes it intelligent isn’t the whizbang technology but rather simple, understated utility. Lights, for example, turn off and on as Theo moves from room to room. There’s no app for controlling them from the couch; no control panel on the wall. It’s all automatic. Why? “It’s just a smart and efficient way to live in a house,” says Barrett.

Today’s smartphones were another object of Barrett’s scrutiny. “They’re advanced, but in some ways they’re not advanced whatsoever,” he says. “They need too much attention. You don’t really want to be stuck engaging them. You want to be free.” In Barrett’s estimation, the smartphones just around the corner aren’t much better. “Everyone says we’re supposed to have a curved piece of flexible glass. Why do we need that? Let’s make it more substantial. Let’s make it something that feels nice in the hand.”

Theo’s phone in the film is just that–a handsome hinged device that looks more like an art deco cigarette case than an iPhone. He uses it far less frequently than we use our smartphones today; it’s functional, but it’s not ubiquitous. As an object, it’s more like a nice wallet or watch. In terms of industrial design, it’s an artifact from a future where gadgets don’t need to scream their sophistication–a future where technology has progressed to the point that it doesn’t need to look like technology.

All of these things contribute to a compelling, cohesive vision of the future–one that’s dramatically different from what we usually see in these types of movies. You could say that Her is, in fact, a counterpoint to that prevailing vision of the future–the anti-Minority Report. Imagining its world wasn’t about heaping new technology on society as we know it today. It was looking at those places where technology could fade into the background, integrate more seamlessly. It was about envisioning a future, perhaps, that looked more like the past. “In a way,” says Barrett, “my job was to undesign the design.”

The Holy Grail: A Discrete User Interface

The greatest act of undesigning in Her, technologically speaking, comes with the interface used throughout the film. Theo doesn’t touch his computer–in fact, while he has a desktop display at home and at work, neither have a keyboard. Instead, he talks to it. “We decided we didn’t want to have physical contact,” Barrett says. “We wanted it to be natural. Hence the elimination of software keyboards as we know them.”

Again, voice control had benefits simply on the level of moviemaking. A conversation between Theo and Sam, his artificially intelligent OS, is obviously easier for the audience to follow than anything involving taps, gestures, swipes or screens. But the voice-based UI was also a perfect fit for a film trying to explore what a less intrusive, less demanding variety of technology might look like.

Indeed, if you’re trying to imagine a future where we’ve managed to liberate ourselves from screens, systems based around talking are hard to avoid. As Barrett puts it, the computers we see in Her“don’t ask us to sit down and pay attention” like the ones we have today. He compares it to the fundamental way music beats out movies in so many situations. Music is something you can listen to anywhere. It’s complementary. It lets you operate in 360 degrees. Movies require you to be locked into one place, looking in one direction. As we see in the film, no matter what Theo’s up to in real life, all it takes to bring his OS into the fold is to pop in his ear plug.

Looking at it that way, you can see the audio-based interface in Her as a novel form of augmented reality computing. Instead of overlaying our vision with a feed, as we’ve typically seen it, Theo gets a one piped into his ear. At the same time, the other ear is left free to take in the world around him.

Barrett sees this sort of arrangement as an elegant end point to the trajectory we’re already on. Think about what happens today when we’re bored at the dinner table. We check our phones. At the same time, we realize that’s a bit rude, and as Barrett sees it, that’s one of the great promises of the smartwatch: discretion.

“They’re a little more invisible. A little sneakier,” he says. Still, they’re screens that require eyeballs. Instead, Barrett says, “imagine if you had an ear plug in and you were getting your feed from everywhere.” Your attention would still be divided, but not nearly as flagrantly.

Of course, a truly capable voice-based UI comes with other benefits. Conversational interfaces make everything easier to use. When every different type of device runs an OS that can understand natural language, it means that every menu, every tool, every function is accessible simply by requesting it.

That, too, is a trend that’s very much alive right now. Consider how today’s mobile operating systems, like iOS and ChromeOS, hide the messy business of file systems out of sight. Theo, with his voice-based valet as intermediary, is burdened with even less under-the-hood stuff than we are today. As Barrett puts it: “We didn’t want him fiddling with things and fussing with things.” In other words, Theo lives in a future where everything, not just his iPad, “just works.”

Theo lives in a future where everything, not just his iPad, “just works.”

AI: the ultimate UX challenge

The central piece of invisible design in Her, however, is that of Sam, the artificially intelligent operating system and Theo’s eventual romantic partner. Their relationship is so natural that it’s easy to forget she’s a piece of software. But Jonze and company didn’t just write a girlfriend character, label it AI, and call it a day. Indeed, much of the film’s dramatic tension ultimately hinges not just on the ways artificial intelligence can be like us but the ways it cannot.

Much of Sam’s unique flavor of AI was written into the script by Jonze himself. But her inclusion led to all sorts of conversations among the production team about the nature of such a technology. “Anytime you’re dealing with trying to interact with a human, you have to think of humans as operating systems. Very advanced operating systems. Your highest goal is to try to emulate them,” Barrett says. Superficially, that might mean considering things like voice pattern and sensitivity and changing them based on the setting or situation.

Even more quesitons swirled when they considered how an artificially intelligent OS should behave. Are they a good listener? Are they intuitive? Do they adjust to your taste and line of questioning? Do they allow time for you to think? As Barrett puts it, “you don’t want a machine that’s always telling you the answer. You want one that approaches you like, ‘let’s solve this together.’”

In essence, it means that AI has to be programmed to dumb itself down. “I think it’s very important for OSes in the future to have a good bedside manner.” Barrett says. “As politicians have learned, you can’t talk at someone all the time. You have to act like you’re listening.”

As we see in the film, though, the greatest asset of AI might be that it doesn’t have one fixed personality. Instead, its ability to figure out what a person needs at a given moment emerges as the killer app.

Theo, emotionally desolate in the midst of a hard divorce, is having a hard time meeting people, so Sam goads him into going on a blind date. When Theo’s friend Amy splits up with her husband, her own artificially intelligent OS acts as a sort of therapist. “She’s helping me work through some things,” Amy says of her virtual friend at one point.

In our own world, we may be a long way from computers that are able to sense when we’re blue and help raise our spirits in one way or another. But we’re already making progress down this path. In something as simple as a responsive web layout or iOS 7′s “Do Not Disturb” feature, we’re starting to see designs that are more perceptive about the real world context surrounding them–where or how or when they’re being used. Google Now and other types of predictive software are ushering in a new era of more personalized, more intelligent apps. And while Apple updating Siri with a few canned jokes about her Hollywood counterpart might not amount to a true sense of humor, it does serve as another example of how we’re making technology more human–a preoccupation that’s very much alive today.

You hear the squeals of the pigs long before reaching a set of long buildings set in rolling hills in southern China.

Feeding time produces a frenzy as the animals strain against the railings around their pens. But this is no ordinary farm.

Run by a fast-growing company called BGI, this facility has become the world's largest centre for the cloning of pigs.

The technology involved is not particularly novel - but what is new is the application of mass production.

The first shed contains 90 animals in two long rows. They look perfectly normal, as one would expect, but each of them is carrying cloned embryos. Many are clones themselves.

This place produces an astonishing 500 cloned pigs a year: China is exploiting science on an industrial scale.

To my surprise, we're taken to see how the work is done. A room next to the pens serves as a surgery and a sow is under anaesthetic, lying on her back on an operating table. An oxygen mask is fitted over her snout and she's breathing steadily. Blue plastic bags cover her trotters.

Two technicians have inserted a fibre-optic probe to locate the sow's uterus. A third retrieves a small test-tube from a fridge: these are the blastocysts, early stage embryos prepared in a lab. In a moment, they will be implanted.

The room is not air-conditioned; nor is it particularly clean. Flies buzz around the pig's head.

My first thought is that the operation is being conducted with an air of total routine. Even the presence of a foreign television crew seems to make little difference. The animal is comfortable but there's no sensitivity about how we might react, let alone what animal rights campaigners might make of it all.

I check the figures: the team can do two implantations a day. The success rate is about 70-80%.

Dusk is falling as we're shown into another shed where new-born piglets are lying close to their mothers to suckle. Heat lamps keep the room warm. Some of the animals are clones of clones. Most have been genetically modified.

The point of the work is to use pigs to test out new medicines. Because they are so similar genetically to humans, pigs can serve as useful "models". So modifying their genes to give them traits can aid that process.

One batch of particularly small pigs has had a growth gene removed - they stopped growing at the age of one. Others have had their DNA tinkered with to try to make them more susceptible to Alzheimer's.

Back at the company headquarters, a line of technicians is hunched over microscopes. This is a BGI innovation: replacing expensive machines with people. It's called "handmade cloning" and is designed to make everything quicker and easier.

The scientist in charge, Dr Yutao Du, explains the technique in a way that leaves me reeling.

"We can do cloning on a very large scale," she tells me, "30-50 people together doing cloning so that we can make a cloning factory here."

A cloning factory - an incredible notion borrowed straight from science fiction. But here in Shenzhen, in what was an old shoe factory, this rising power is creating a new industry.

The scale of ambition is staggering. BGI is not only the world's largest centre for cloning pigs - it's also the world's largest centre for gene sequencing.

In neighbouring buildings, there are rows of gene sequencers - machines the size of fridges operating 24 hours a day crunching through the codes for life.

To illustrate the scale of this operation, Europe's largest gene sequencing centre is the Wellcome Trust Sanger Institute near Cambridge. It has 30 machines. BGI has 156 and has even bought an American company that makes them.

BGI's chief executive, Wang Jun, tells me how they need the technology to develop ever faster and cheaper ways of reading genes.

Again, a comparison for scale: a recently-launched UK project seeks to sequence 10,000 human genomes. BGI has ambitions to sequence the genomes of a million people, a million animals and a million plants.

Wang Jun is keen to stress that all this work must be relevant to ordinary people through better healthcare or tastier food. The BGI canteen is used as a testbed for some of the products from the labs: everything from grouper twice the normal size, to pigs, to yoghurt.

I ask Wang Jun how he chooses what to sequence. After the shock of hearing the phrase "cloning factory", out comes another bombshell:

"If it tastes good you should sequence it," he tells me. "You should know what's in the genes of that species."

Species that taste good is one criterion. Another he cites is that of industrial use - raising yields, for example, or benefits for healthcare.

"A third category is if it looks cute - anything that looks cute: panda, polar bear, penguin, you should really sequence it - it's like digitalising all the wonderful species," he explains.

I wonder how he feels about acquiring such power to take control of nature but he immediately contradicts me.

"No, we're following Nature - there are lots of people dying from hunger and protein supply so we have to think about ways of dealing with that, for example exploring the potential of rice as a species," the BGI chief counters.

China is on a trajectory that will see it emerging as a giant of science: it has a robotic rover on the Moon, it holds the honour of having the world's fastest supercomputer and BGI offers a glimpse of what industrial scale could bring to the future of biology.

The long, torturous death throes of the PC grew more excruciating than ever last year. Several research firms say PC shipments plunged a record 10 percent last year, the steepest drop ever. But, as smartphones and tablets take over our world, it’s not just laptops and desktops that could wind up on the pile of dead tech. They may drag the web down with them.

Keith Rabois thinks so, and he’s not alone. PayPal mafioso Rabois has a good tech track record. After his early involvement in PayPal, he helped bootstrap LinkedIn. He backed social photo startup Slide, which Google bought for nine figures. And he was chief operating officer at Twitter co-founder Jack Dorsey’s mobile payments company Square. Now, he works as an investor at Khosla Ventures, and he believes the end of the PC also means the web is over:

The gist of the argument is this: as app-happy mobile devices become the primary way we compute, the good old browser becomes irrelevant. The hyperlinked, free-flowing, egalitarian, and ubiquitous world wide web will fade away. Instead, digital existence will mostly transpire within the more self-contained domains of individual apps, which offer their creators the flexibility and power of building right into the mobile operating systems. We will still have the internet, but it won’t be the same wherever you use it. And some will have more power over it than others.

In the WIRED cover story “The Web Is Dead,” former editor-in-chief Chris Anderson said much the same thing in August, 2010:

More than three years later, the trend has only accelerated. Facebook CEO Mark Zuckerberg famously disavowed his company’s effort to build a cross-platform mobile app in HTML5 — the lingua franca of the web — in favor of OS-specific native apps. And many places in the developing world are experiencing a phenomenon known as “leapfrogging,” moving straight from no internet at all to the web-shy world of mobile, due mainly to the lower cost of entry and the absence of the heavy physical infrastructure required to support broadband PC use.

Large swaths of the world — and just about every elementary schoolkid today — experiences computing mainly through native mobile apps. The browser is — at most — just one of many ways of connecting. This does not bode well for the web as a central venue of the interconnected life.

What we lose is openness. On the web, anyone can publish anything for almost nothing, and it will look pretty much the same across a world of machines. This was the original appeal that drove the web’s early growth. Maybe as a new generation discovers that joy, the web will gain new life. It may not be as popular as it once was, but maybe it will become as cool as it was in the halcyon days of the mid-1990s, when anyone in the know felt like they had a whole new world all to themselves.

The Command Line Murders

.OOOOOOOOOOOOOOO @@ @@ OOOOOOOOOOOOOOOO.
OOOOOOOOOOOOOOOO @@ @@ OOOOOOOOOOOOOOOO
OOOOOOOOOO'''''' @@ @@ ```````OOOOOOOOO
OOOOO'' aaa@@@@@@@@@@@@@@@@@@@@""" """""""""@@aaaa `OOOO
OOOOO,""""@@@@@@@@@@@@@@"""" a@"" OOOA
OOOOOOOOOoooooo, |OOoooooOOOOOS
OOOOOOOOOOOOOOOOo, |OOOOOOOOOOOOC
OOOOOOOOOOOOOOOOOO ,|OOOOOOOOOOOOI
OOOOOOOOOOOOOOOOOO @ THE |OOOOOOOOOOOOOI
OOOOOOOOOOOOOOOOO'@ COMMAND OOOOOOOOOOOOOOb
OOOOOOOOOOOOOOO'a' LINE |OOOOOOOOOOOOOy
OOOOOOOOOOOOOO'' MURDERS aa`OOOOOOOOOOOP
OOOOOOOOOOOOOOb,.. `@aa``OOOOOOOh
OOOOOOOOOOOOOOOOOOo `@@@aa OOOOo
OOOOOOOOOOOOOOOOOOO| @@@ OOOOe
OOOOOOOOOOOOOOOOOOO@ aaaaaaa @@',OOOOn
OOOOOOOOOOOOOOOOOOO@ aaa@@@@@@@@"" @@ OOOOOi
OOOOOOOOOO~~ aaaaaa"a aaa@@@@@@@@@@"" @@ OOOOOx
OOOOOO aaaa@"""""""" "" @@@@@@@@@@@@"" @@@|`OOOO'
OOOOOOOo`@@a aa@@ @@@@@@@"" a@ @@@@ OOOO9
OOOOOOO' `@@a @@a@@ @@"" a@@ a |@@@ OOOO3
`OOOO' `@ aa@@ aaa""" @a a@ a@@@',OOOO'

There's been a murder in Terminal City, and TCPD needs your help.

To figure out whodunit, you need access to a command line.

Once you're ready, download this repo (or download clmystery.zip and unzip it).

Open a Terminal, go to the location of the files, and start by reading the file 'instructions'.

One way you can do this is with the command:

cat instructions

To get started on how to use the command line, open cheatsheet.md or cheatsheet.pdf (from the command line, you can type 'nano cheatsheet.md').

Don't use a text editor to view any files except these instructions, the cheatsheet, and hints.

Editors note: This is a guest post by Nick Meyer, who is currently working on the Napwell, the world’s first Napping Mask. The Mask is currently running a Kickstarter campaign here. 

It's popular these days to make the claim that napping is good for you. This author has even built an entire startup on the premise that we should nap more and better. But what data is this conclusion based on? One important study by NASA for the most part.

In the 1980s and 1990s, NASA and the FAA were studying whether or not in-cockpit napping could improve the job performance and safety of pilots flying long haul routes. The results are somewhat technical, but almost all contemporary news articles citing a measurable increase in on-job performance due to napping are actually based on this data.

In the study, NASA teams first picked out a group of commercial airline flight pilots flying a standard itinerary between Hawaii, Japan and Los Angeles. They then divided the pilots into two groups: A Rest Group (RG) that was allowed a 40 min cock-pit nap during the cruise portion of each flight and a No Rest Group (NRG) that was not allowed a mid-flight nap. Over the course of a six day study, the pilots flew four (4) flights during which NASA teams analyzed them for wakefulness before, during and after their flights. The teams even brought along EEG and EOG machines to measure the pilots’ brain activity during the tests to confirm whether or not the pilots sleeping, and how alert they were.

The most interesting results were as follows:

Reaction Time - Using a measure of reaction time called a "PVT Trial" the teams found that the naps helped pilots maintain their baseline reaction speed over the course of the flight. The data below show that over the course of a flight (from pre-flight to post-flight) the napping pilots maintained their reaction speed versus their non-napping colleagues, who tended to grow slower over the course of the flight. More importantly, the napping pilots maintained that reaction speed on subsequent flights, whereas the non-nappers pilots suffered accumulate fatigue from previous flights.

Performance Lapses – Using the same measure of reaction speed, the teams found that the number of performance “lapses,” was decreased following a nap. A “lapse” is a very slow reaction versus normal response time, and is basically the pilot freezing for a brief moment in the cockpit. The napping pilots showed 34% fewer performance lapses during later stages of the flight than their colleagues. Contextualizing this a bit more, a nap would basically help a pilot maintain their reaction speed and prevent overly slow responses during the later stages of a 7-9 hour flight, when fatigue would normally set in.  

Figure 16 – Finally, using EEG and EOG devices to measure brain activity, the teams checked for sleepiness during the last 90 minutes of all flights, including the crucial period prior to landing a plane (TOD to Landing). Sleepiness is indicated by the number of “microevents” that occur—brief periods when brain activity changes, and the brain enters the first stages of falling asleep. There are 3 flavors of microevents: Theta, SEM and Alpha, depending on brain waves and eye movements.

Across the board, the napping pilots had significantly fewer microevents, or were much more alert, than the non-napping pilots. A statistical analysis done by the team showed that the non-napping pilots were roughly twice as likely to register a microevent, or were ~100% sleepier than their napping colleagues.

The NASA team concluded that naps provided a 34% increase in pilot performance and 100% increase in physiological alertness. These basic facts are often cited by health and wellness magazines as benefits of napping, but few actually dive into the details of where the metrics came from, and what they mean.

Good data can generate tremendous value when put in the hands of consumers, companies and organizations to improve their decision making. A traditional government study can seem to be a bit of a throwback in today’s highly digitized world, but such studies are often essential in underpinning our understanding of sleep, naps and in this case how napping can help sustain on-the-job performance.

So, if you're looking for a justification for taking a nap, now you have it.

This is a guest post by Nick Meyer, who is currently working on the Napwell, the world’s first Napping Mask. The Mask is currently running a Kickstarter campaign here. 

* JavaScript is moving to server-side and is going to have classes and modules and other stuff coming soon
* Java is moving towards JavaScript with lambda and some dose of type inference and due to (also) Avatar, enabling to execute efficiently JavaScript code on the Java server-side platform.

And PHP ? IMHO, PHP sits into the middle, that is, between JavaScript and Java.

A JavaScript++ (that JavaScript+classes+modules) will be as good as PHP: no type, classes, modules, lambda... (but, today, JavaScript is not as widely used on the server-side as PHP).

But PHP does not have (out of the box) the "typing" insurance and the performance of the Java platform.

Then: JavaScript++ < PHP < Java.

When, JavaScript and Java are moving towards each other, what will be the place left to PHP ? IMHO, it will be not so big...

Today is undoubtedly an exciting day for all of us at Nest, but it’s also meaningful for you, our customers. I’m sure you have questions about what today’s news means for you, the Nest Thermostat, and Nest Protect. While Tony answered many of the broader questions regarding our new partnership with Google, I thought we’d talk product – my favorite!

Before we dig in, I want to acknowledge what I consider the first great Nest partnership. I’m not talking about Google. I’m talking about the one between you and the team here at Nest. Some of you have been with us since day one while others learned about us more recently, maybe while watching The Ellen DeGeneres Show or hitching a ride on the Nest Fire Truck. One thing’s for sure – there’s no way our products would be what they are today without you, so thank you.

We’re looking forward to continuing our great partnership and remain devoted to you above all else. We know you entrust your homes and information to us and are committed to protecting that the same way we’ve always done.

Now onto the Q&A. I’m sure you’ll notice that not much will change – that is our intent!

Will Nest continue to support iOS so I can have the Nest app on my iPhone or iPad?
Yes, absolutely. We’ll continue supporting iOS, Android and modern web browsers so you can check in on your home and control the temperature from wherever you are.

Will Nest and Google products work with each other?
Nest’s product line obviously caught the attention of Google and I’m betting that there’s a lot of cool stuff we could do together, but nothing to share today.

What will happen to the Nest warranties on products?
No change there – we stand behind our products like we always have.

Will I still be able to find Nest products at my local retailer?
You bet. We intend to continue selling through the same partners in the US, Canada and the UK.

Will Nest customer data be shared with Google?
Our privacy policy clearly limits the use of customer information to providing and improving Nest’s products and services. We’ve always taken privacy seriously and this will not change.

Hopefully I’ve covered all of your questions related to the Google partnership, but if not, feel free to hit me up on Twitter (@nestmatt). I’ll answer what I can.

Let’s do this!
-Matt