Last night Andrea Shepard, a core Tor developer living in Seattle, posted this message to Twitter:

The image she linked to shows the shipment tracking details for a computer Shepard ordered from Amazon, the global internet superstore and cloud computing giant that in late 2013 secured a $600 million contract with the CIA. Here's the image; click to enlarge it.

As you can see, the tracking details are highly unusual. Instead of shipping the computer directly from the Amazon storage facility in Santa Ana, California, to Shepard in Seattle, the package was first dispatched to Dulles, Virginia. From Dulles, it moved another four times around the military and intelligence belt in suburban Washington DC, finally landing in Alexandria at 11:03 am on January 23.

Contrary to Amazon's shipment tracking summary, Virginia is not the package's final destination. Shepard does not live in Alexandria and told Amazon to ship the computer to a Seattle, Washington address. You can see this for yourself in the top right hand corner of the image.

At the end of December 2013, journalists working for the German newspaper Der Spiegel published information about a top-secret arm of the NSA, called the Tailored Access Operations division. TAO does highly targeted surveillance, a world apart from the indiscriminate, mass surveillance that happens under other NSA and FBI programs. One of the more alarming things we learned in the TAO story is that the NSA intercepts computers ordered online and installs malware on them, before sending them on to their final destination.

Could this be what happened to Shepard's computer, ordered on Amazon and delivered to Alexandria, instead of to Seattle? Could Amazon have made a mistake in notifying Shepard about this extra journey, which was likely meant to stay a secret? If this really is an example of the TAO laptop-interception program in action, does this mean that companies like Amazon are made aware of the government's intention to "look after" consumer products ordered by their customers? Or did Shepard receive this weird notice only after some sort of glitch in the NSA's surveillance matrix?

If this indeed is evidence of the NSA intercepting a laptop to install spyware on it, it's yet more proof that, even when the spying is highly targeted and precise, the NSA isn't necessarily using its powers to only go after terrorists or dangerous criminals. Shepard is neither a criminal nor a terrorist. She's a developer, an activist, and a free speech supporter.

Is that all it takes to become a victim of the NSA's targeted spying? Someone should ask Amazon and the NSA what happened here, so we can get to the bottom of this bizarre situation. If in fact the NSA is installing malware on the computers of activists and coders working to protect internet anonymity, the government has a lot of explaining to do. If this is the case, it shows that even highly-targeted surveillance can be wildly abused, if agencies are left to fester in the dark to do whatever they want.

Introducing: Map

Map is a cool little function, and the key to easily injecting parallelism into your Python code. For those unfamiliar, map is something lifted from functional languages like Lisp. It is a function which maps another function over a sequence. e.g.

urls = ['http://www.yahoo.com', 'http://www.reddit.com']
results = map(urllib2.urlopen, urls)

This applies the method urlopen, on each item in the passed in sequence and stores all of the results in a list. It is more or less equivalent to

results = []
for url in urls: 
 results.append(urllib2.urlopen(url))

Map handles the iteration over the sequence for us, applies the function, and stores all of the results in a handy list at the end.

Why does this matter? Because with the right libraries, map makes running things in parallel completely trivial!

Parallel versions of the map function are provided by two libraries: multiprocessing, and also its little known, but equally fantastic step child: multiprocessing.dummy.

Digression: What’s that? Never heard of the threading clone of multiprocessing library called dummy? I hadn't either until very recently. It has all of ONEsentence devoted to it in the multiprocessing documentation page. And that sentence pretty much boils down to “Oh yeah, and this thing exists.” It’s tragically undersold, I tell you!

Dummy is an exact clone of the multiprocessing module. The only difference is that, whereas multiprocessing works with processes, the dummy module uses threads (which come with all the usual Python limitations). So anything that applies to one, applies to the other. It makes it extremely easy to hop back and forth between the two. Which is especially great for exploratory programming when you’re not quite sure if some framework call is IO or CPU bound.

Getting Started

To access the parallel versions of the map functions the first thing you need to do is import the modules that contain them:

from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool 

and instantiate their Pool objects in the code:

pool = ThreadPool() 

This single statement handles everything we did in the seven line build_worker_pool function from example2.py. Namely, It creates a bunch of available workers, starts them up so that they’re ready to do some work, and stores all of them in variable so that they’re easily accessed.

The pool objects take a few parameters, but for now, the only one worth noting is the first one: processes. This sets the number of workers in the pool. If you leave it blank, it will default to the number of Cores in your machine.

In the general case, if you’re using the multiprocessing pool for CPU bound tasks, more cores equals more speed (I say that with a lot of caveats). However, when threading and dealing with network bound stuff, things seem to vary wildly, so it’s a good idea to experiment with the exact size of the pool.

pool = ThreadPool(4) # Sets the pool size to 4

If you run too many threads, you’ll waste more time switching between then than doing useful work, so it’s always good to play around a little bit until you find the sweet spot for the task at hand.

So, now with the pool objects created, and simple parallelism at our fingertips, let’s rewrite the url opener from example2.py!

Look at that! The code that actually does work is all of 4 lines. 3 of which are simple bookkeeping ones. The map call handles everything our previous 40 line example did with ease! For funzies, I timed both approaches as well as different pool sizes.

Results:

Pretty awesome! And also shows why it’s good to play around a bit with the pool size. Any pool size greater than 9 quickly lead to diminishing returns on my machine.

Real World Example 2:

Thumbnailing thousands of images

Let’s now do something CPU bound! A pretty common task for me at work is manipulating massive image folders. One of those transformations is creating thumbnails. It is ripe for being run in parallel.

The basic single process setup

A little hacked together for example, but in essence, a folder is passed into the program, from that it grabs all of the images in the folder, then finally creates the thumbnails and saves them to their own directory.

On my machine, this took 27.9 seconds to process ~6000 images.

If we replace the for loop with a parallel map call:

5.6seconds!

That’s a pretty massive speedup for only changing a few lines of code. The production version of this is even faster by splitting cpu and io tasks into their own respective processes and threads — which is usually a recipe for deadlocked code. However, due to the explicit nature of map, and the lack of manual thread management, it feels remarkably easy to mix and match the two in a way that is clean, reliable, and easy to debug.

So there it is. Parallelism in (almost) one line.

The top 1 percent aren't killing the American Dream. Something else is—if you live in the wrong place.

Here's what we know. The rich are getting richer, but according to a blockbuster new study that hasn't made it harder for the poor to become rich. The good news is that people at the bottom are just as likely to move up the income ladder today as they were 50 years ago. But the bad news is that people at the bottom are just as likely to move up the income ladder today as they were 50 years ago.

We like to tell ourselves that America is the land of opportunity, but the reality doesn't match the rhetoric—and hasn't for awhile. We actually have less social mobility than countries like Denmark. And that's more of a problem the more inequality there is. Think about it like this: Moving up matters more when there's a bigger gap between the rich and poor. So even though mobility hasn't gotten worse lately, it has worse consequences today because inequality is worse.

But it's a little deceiving to talk about "our" mobility rate. There isn't one or two or even three Americas. There are hundreds. The research team of Raj Chetty, Nathaniel Herndon, Patrick Kline, and Emmanuel Saez looked at each "commuting zone" (CZ) within the U.S., and found that the American Dream is still alive in some parts of the country. Kids born into the bottom 20 percent of households, for example, have a 12.9 percent chance of reaching the top 20 percent if they live in San Jose. That's about as high as it is in the highest mobility countries. But kids born in Charlotte only have a 4.4 percent chance of moving from the bottom to the top 20 percent. That's worse than any developed country we have numbers for.

You can see what my colleague Derek Thompson calls the geography of the American Dream in the map below. It shows where kids have the best and worst chances of moving up from the bottom to the top quintile—and that the South looks more like a banana republic. (Note: darker colors mean there is less mobility, and lighter colors mean that there's more).

So what makes northern California different from North Carolina? Well, we don't know for sure, but we do know what doesn't. The researchers found that local tax and spending decisions explain some, but not too much, of this regional mobility gap. Neither does local school quality, at least judged by class size. Local area colleges and tuition were also non-factors. And so were local labor markets, including their share of manufacturing jobs and those facing cheap, foreign competition. But here's what we know does matter. Just how much isn't clear.

1. Race. The researchers found that the larger the black population, the lower the upward mobility. But this isn't actually a black-white issue. It's a rich-poor one. Low-income whites who live in areas with more black people also have a harder time moving up the income ladder. In other words, it's something about the places that black people live that hurts mobility.

2. Segregation. Something like the poor being isolated—isolated from good jobs and good schools. See, the more black people a place has, the more divided it tends to be along racial and economic lines. The more divided it is, the more sprawl there is. And the more sprawl there is, the less higher-income people are willing to invest in things like public transit. 

That leaves the poor in the ghetto, with no way out for their American Dreams. They're stuck with bad schools, bad jobs, and bad commutes if they do manage to find better work. So it should be no surprise that the researchers found that racial segregation, income segregation, and sprawl are all strongly negatively correlated with upward mobility. But what might surprise is that it doesn't matter whether the rich cut themselves off from everybody else. What matters is whether the middle class cut themselves off from the poor.

3. Social Capital. Living around the middle class doesn't just bring better jobs and schools (which help, but probably aren't enough). It brings better institutions too. Things like religious groups, civic groups, and any other kind of group that keeps people from bowling alone. All of these are strongly correlated with more mobility—which is why Utah, with its vast Mormon safety net and services, is one of the best places to be born poor.

4. Inequality. The 1 percent are different from you and me—they have so much more money that they live in a different world. It's a world of $40,000 a year preschool, "nanny consultants," and an endless supply of private tutors. It keeps the children of the super-rich from falling too far, but it doesn't keep the poor from rising (at least into the top quintile). There just wasn't any correlation between the rise and rise of the 1 percent and upward mobility. In other words, it doesn't hurt your chances of making it into the top 80 to 99 percent if the super-rich get even richer.

But inequality does matter within the bottom 99 percent. The bigger the gap between the poor and the merely rich (as opposed to the super-rich), the less mobility there is. It makes intuitive sense: it's easier to jump from the bottom near the top if you don't have to jump as far. The top 1 percent are just so high now that it doesn't matter how much higher they go; almost nobody can reach them.

5. Family Structure. Forget race, forget jobs, forget schools, forget churches, forget neighborhoods, and forget the top 1—or maybe 10—percent. Nothing matters more for moving up than who raises you. Or, in econospeak, nothing correlates with upward mobility more than the number of single parents, divorcees, and married couples. The cliché is true: Kids do best in stable, two-parent homes.

It's not clear what, if any, policy lessons we should take from this truism. As my colleague Jordan Weissmann points out, we don't really have any idea how to promote marriage. We can try telling people how great it is to get hitched. We can even get rid of the marriage penalties some low-income couples face. But these won't, and haven't, been making more people exchange till-death-do-us-parts. And should we even want to? Steve Waldman points out that poor women know better than upper-middle-class people yelling at them to get married whether they should or not. They know whether their boyfriend would make a good husband, a good father, a good teacher. And they know that marriage is important. That they're not getting married tells us something. Sometimes no match is better than a bad match.

***

Flat mobility is the defining Rorschach test of our time. Conservatives look at it, and say, see, we shouldn't worry about the top 1 percent, because they're not making the American Dream any harder to achieve. But liberals look at it, and say see, we should care about inequality, because it can make the American Dream harder to achieve—and it raises the stakes if you don't. But both want to increase upward mobility. It's not enough to keep it where it was 50 years ago. We need to actually become the land of opportunity. 

The American Dream is alive in Denmark and Finland and Sweden. And in San Jose and Salt Lake City and Pittsburgh. But it's dead in Atlanta and Raleigh and Charlotte. And in Indianapolis and Detroit and Jacksonville. Fixing that isn't just about redistribution. It's about building denser cities, so the poor aren't so segregated. About good schools that you don't have to live in the right (and expensive) neighborhood to attend. And about ending a destructive drug war that imprisons and blights the job prospects of far too many non-violent offenders—further shrinking the pool of "marriageable" men.

Because the American Dream is dead in too much of America.

Thoreau Reader: Home - Life without Principle Intro - Life without Principle - 2

[2]    So now I would say something similar to you, my readers. Since you are my readers, and I have not been much of a traveller, I will not talk about people a thousand miles off, but come as near home as I can. As the time is short, I will leave out all the flattery, and retain all the criticism.

[3]    Let us consider the way in which we spend our lives.

[4]    This world is a place of business. What an infinite bustle! I am awaked almost every night by the panting of the locomotive.(1)  It interrupts my dreams. There is no sabbath. It would be glorious to see mankind at leisure for once. It is nothing but work, work, work. I cannot easily buy a blank-book to write thoughts in; they are commonly ruled for dollars and cents. An Irishman, seeing me making a minute in the fields, took it for granted that I was calculating my wages. If a man was tossed out of a window when an infant, and so made a cripple for life, or scared out of his wits by the Indians, it is regretted chiefly because he was thus incapacitated for — business! I think that there is nothing, not even crime, more opposed to poetry, to philosophy, ay, to life itself, than this incessant business.

[5]    There is a coarse and boisterous money-making fellow in the outskirts of our town, who is going to build a bank-wall under the hill along the edge of his meadow. The powers have put this into his head to keep him out of mischief, and he wishes me to spend three weeks digging there with him. The result will be that he will perhaps get some more money to board, and leave for his heirs to spend foolishly. If I do this, most will commend me as an industrious and hard-working man; but if I choose to devote myself to certain labors which yield more real profit, though but little money, they may be inclined to look on me as an idler. Nevertheless, as I do not need the police of meaningless labor to regulate me, and do not see anything absolutely praiseworthy in this fellow's undertaking any more than in many an enterprise of our own or foreign governments, however amusing it may be to him or them, I prefer to finish my education at a different school.

[6]    If a man walk in the woods for love of them half of each day, he is in danger of being regarded as a loafer; but if he spends his whole day as a speculator, shearing off those woods and making earth bald before her time, he is esteemed an industrious and enterprising citizen. As if a town had no interest in its forests but to cut them down!

[7]    Most men would feel insulted if it were proposed to employ them in throwing stones over a wall, and then in throwing them back, merely that they might earn their wages. But many are no more worthily employed now. For instance: just after sunrise, one summer morning, I noticed one of my neighbors walking beside his team, which was slowly drawing a heavy hewn stone swung under the axle, surrounded by an atmosphere of industry, — his day's work begun, — his brow commenced to sweat, — a reproach to all sluggards and idlers, — pausing abreast the shoulders of his oxen, and half turning round with a flourish of his merciful whip, while they gained their length on him. And I thought, Such is the labor which the American Congress exists to protect, — honest, manly toil, — honest as the day is long, — that makes his bread taste sweet, and keeps society sweet, — which all men respect and have consecrated; one of the sacred band, doing the needful but irksome drudgery. Indeed, I felt a slight reproach, because I observed this from a window, and was not abroad and stirring about a similar business. The day went by, and at evening I passed the yard of another neighbor, who keeps many servants, and spends much money foolishly, while he adds nothing to the common stock, and there I saw the stone of the morning lying beside a whimsical structure intended to adorn this Lord Timothy Dexter's(2) premises, and the dignity forthwith departed from the teamster's labor, in my eyes. In my opinion, the sun was made to light worthier toil than this. I may add that his employer has since run off, in debt to a good part of the town, and, after passing through Chancery,(3) has settled somewhere else, there to become once more a patron of the arts.

[8]    The ways by which you may get money almost without exception lead downward. To have done anything by which you earned money merely is to have been truly idle or worse. If the laborer gets no more than the wages which his employer pays him, he is cheated, he cheats himself. If you would get money as a writer or lecturer, you must be popular, which is to go down perpendicularly. Those services which the community will most readily pay for, it is most disagreeable to render. You are paid for being something less than a man. The State does not commonly reward a genius any more wisely. Even the poet laureate would rather not have to celebrate the accidents of royalty. He must be bribed with a pipe of wine; and perhaps another poet is called away from his muse to gauge that very pipe. As for my own business, even that kind of surveying which I could do with most satisfaction my employers do not want. They would prefer that I should do my work coarsely and not too well, ay, not well enough. When I observe that there are different ways of surveying, my employer commonly asks which will give him the most land, not which is most correct. I once invented a rule for measuring cord-wood, and tried to introduce it in Boston; but the measurer there told me that the sellers did not wish to have their wood measured correctly, — that he was already too accurate for them, and therefore they commonly got their wood measured in Charlestown before crossing the bridge.

[9]    The aim of the laborer should be, not to get his living, to get "a good job," but to perform well a certain work; and, even in a pecuniary sense, it would be economy for a town to pay its laborers so well that they would not feel that they were working for low ends, as for a livelihood merely, but for scientific, or even moral ends. Do not hire a man who does your work for money, but him who does it for love of it.

[10]    It is remarkable that there are few men so well employed, so much to their minds, but that a little money or fame would commonly buy them off from their present pursuit. I see advertisements for active young men, as if activity were the whole of a young man's capital. Yet I have been surprised when one has with confidence proposed to me, a grown man, to embark in some enterprise of his, as if I had absolutely nothing to do, my life having been a complete failure hitherto. What a doubtful compliment this is to pay me! As if he had met me half-way across the ocean beating up against the wind, but bound nowhere, and proposed to me to go along with him! If I did, what do you think the underwriters would say? No, no! I am not without employment at this stage of the voyage. To tell the truth, I saw an advertisement for able-bodied seamen, when I was a boy, sauntering in my native port, and as soon as I came of age I embarked.

[11]    The community has no bribe that will tempt a wise man. You may raise money enough to tunnel a mountain, but you cannot raise money enough to hire a man who is minding his own business. An efficient and valuable man does what he can, whether the community pay him for it or not. The inefficient offer their inefficiency to the highest bidder, and are forever expecting to be put into office. One would suppose that they were rarely disappointed.

[12]    Perhaps I am more than usually jealous with respect to my freedom. I feel that my connection with and obligation to society are still very slight and transient. Those slight labors which afford me a livelihood, and by which it is allowed that I am to some extent serviceable to my contemporaries, are as yet commonly a pleasure to me, and I am not often reminded that they are a necessity. So far I am successful. But I foresee that if my wants should be much increased, the labor required to supply them would become a drudgery. If I should sell both my forenoons and afternoons to society, as most appear to do, I am sure that for me there would be nothing left worth living for. I trust that I shall never thus sell my birthright for a mess of pottage. I wish to suggest that a man may be very industrious, and yet not spend his time well. There is no more fatal blunderer than he who consumes the greater part of his life getting his living. All great enterprises are self-supporting. The poet, for instance, must sustain his body by his poetry, as a steam planing-mill feeds its boilers with the shavings it makes. You must get your living by loving. But as it is said of the merchants that ninety-seven in a hundred fail, so the life of men generally, tried by this standard, is a failure, and bankruptcy may be surely prophesied.

[13]    Merely to come into the world the heir of a fortune is not to be born, but to be still-born, rather. To be supported by the charity of friends, or a government pension, — provided you continue to breathe, — by whatever fine synonyms you describe these relations, is to go into the almshouse. On Sundays the poor debtor goes to church to take an account of stock, and finds, of course, that his outgoes have been greater than his income. In the Catholic Church, especially, they go into chancery, make a clean confession, give up all, and think to start again. Thus men will lie on their backs, talking about the fall of man, and never make an effort to get up.

[14]    As for the comparative demand which men make on life, it is an important difference between two, that the one is satisfied with a level success, that his marks can all be hit by point-blank shots, but the other, however low and unsuccessful his life may be, constantly elevates his aim, though at a very slight angle to the horizon. I should much rather be the last man, — though, as the Orientals say, "Greatness doth not approach him who is forever looking down; and all those who are looking high are growing poor."

[15]    It is remarkable that there is little or nothing to be remembered written on the subject of getting a living; how to make getting a living not merely honest and honorable, but altogether inviting and glorious; for if getting a living is not so, then living is not. One would think, from looking at literature, that this question had never disturbed a solitary individual's musings. Is it that men are too much disgusted with their experience to speak of it? The lesson of value which money teaches, which the Author of the Universe has taken so much pains to teach us, we are inclined to skip altogether. As for the means of living, it is wonderful how indifferent men of all classes are about it, even reformers, so called, — whether they inherit, or earn, or steal it. I think that Society has done nothing for us in this respect, or at least has undone what she has done. Cold and hunger seem more friendly to my nature than those methods which men have adopted and advise to ward them off.

[16]    The title wise is, for the most part, falsely applied. How can one be a wise man, if he does not know any better how to live than other men? — if he is only more cunning and intellectually subtle? Does Wisdom work in a tread-mill? or does she teach how to succeed by her example? Is there any such thing as wisdom not applied to life? Is she merely the miller who grinds the finest logic? It is pertinent to ask if Plato(4) got his living in a better way or more successfully than his contemporaries, — or did he succumb to the difficulties of life like other men? Did he seem to prevail over some of them merely by indifference, or by assuming grand airs? or find it easier to live, because his aunt remembered him in her will? The ways in which most men get their living, that is, live, are mere makeshifts, and a shirking of the real business of life, — chiefly because they do not know, but partly because they do not mean, any better.

[17]    The rush to California,(5) for instance, and the attitude, not merely of merchants, but of philosophers and prophets, so called, in relation to it, reflect the greatest disgrace on mankind. That so many are ready to live by luck, and so get the means of commanding the labor of others less lucky, without contributing any value to society! And that is called enterprise! I know of no more startling development of the immorality of trade, and all the common modes of getting a living. The philosophy and poetry and religion of such a mankind are not worth the dust of a puffball. The hog that gets his living by rooting, stirring up the soil so, would be ashamed of such company. If I could command the wealth of all the worlds by lifting my finger, I would not pay such a price for it. Even Mahomet(6) knew that God did not make this world in jest. It makes God to be a moneyed gentleman who scatters a handful of pennies in order to see mankind scramble for them. The world's raffle! A subsistence in the domains of Nature a thing to be raffled for! What a comment, what a satire, on our institutions! The conclusion will be, that mankind will hang itself upon a tree. And have all the precepts in all the Bibles taught men only this? and is the last and most admirable invention of the human race only an improved muck-rake? Is this the ground on which Orientals and Occidentals meet? Did God direct us so to get our living, digging where we never planted, — and He would, perchance, reward us with lumps of gold?

[18]    God gave the righteous man a certificate entitling him to food and raiment, but the unrighteous man found a facsimile of the same in God's coffers, and appropriated it, and obtained food and raiment like the former. It is one of the most extensive systems of counterfeiting that the world has seen. I did not know that mankind were suffering for want of gold. I have seen a little of it. I know that it is very malleable, but not so malleable as wit. A grain of gold will gild a great surface, but not so much as a grain of wisdom.

[19]    The gold-digger in the ravines of the mountains is as much a gambler as his fellow in the saloons of San Francisco. What difference does it make whether you shake dirt or shake dice? If you win, society is the loser. The gold-digger is the enemy of the honest laborer, whatever checks and compensations there may be. It is not enough to tell me that you worked hard to get your gold. So does the Devil work hard. The way of transgressors may be hard in many respects. The humblest observer who goes to the mines sees and says that gold-digging is of the character of a lottery; the gold thus obtained is not the same same thing with the wages of honest toil. But, practically, he forgets what he has seen, for he has seen only the fact, not the principle, and goes into trade there, that is, buys a ticket in what commonly proves another lottery, where the fact is not so obvious.

[20]    After reading Howitt's account(7) of the Australian gold-diggings one evening, I had in my mind's eye, all night, the numerous valleys, with their streams, all cut up with foul pits, from ten to one hundred feet deep, and half a dozen feet across, as close as they can be dug, and partly filled with water, — the locality to which men furiously rush to probe for their fortunes, — uncertain where they shall break ground, — not knowing but the gold is under their camp itself, — sometimes digging one hundred and sixty feet before they strike the vein, or then missing it by a foot, — turned into demons, and regardless of each others' rights, in their thirst for riches, — whole valleys, for thirty miles, suddenly honeycombed by the pits of the miners, so that even hundreds are drowned in them, — standing in water, and covered with mud and clay, they work night and day, dying of exposure and disease. Having read this, and partly forgotten it, I was thinking, accidentally, of my own unsatisfactory life, doing as others do; and with that vision of the diggings still before me, I asked myself why I might not be washing some gold daily, though it were only the finest particles, — why I might not sink a shaft down to the gold within me, and work that mine. There is a Ballarat, a Bendigo for you, — what though it were a sulky-gully?(8) At any rate, I might pursue some path, however solitary and narrow and crooked, in which I could walk with love and reverence. Wherever a man separates from the multitude, and goes his own way in this mood, there indeed is a fork in the road, though ordinary travellers may see only a gap in the paling. His solitary path across lots will turn out the higher way of the two.

[21]    Men rush to California and Australia as if the true gold were to be found in that direction; but that is to go to the very opposite extreme to where it lies. They go prospecting farther and farther away from the true lead, and are most unfortunate when they think themselves most successful. Is not our native soil auriferous? Does not a stream from the golden mountains flow through our native valley? and has not this for more than geologic ages been bringing down the shining particles and forming the nuggets for us? Yet, strange to tell, if a digger steal away, prospecting for this true gold, into the unexplored solitudes around us, there is no danger that any will dog his steps, and endeavor to supplant him. He may claim and undermine the whole valley even, both the cultivated and the uncultivated portions, his whole life long in peace, for no one will ever dispute his claim. They will not mind his cradles or his toms. He is not confined to a claim twelve feet square, as at Ballarat, but may mine anywhere, and wash the whole wide world in his tom.

[22]    Howitt says of the man who found the great nugget which weighed twenty-eight pounds, at the Bendigo diggings in Australia: "He soon began to drink; got a horse, and rode all about, generally at full gallop, and, when he met people, called out to inquire if they knew who he was, and then kindly informed them that he was 'the bloody wretch that had found the nugget.' At last he rode full speed against a tree, and nearly knocked his brains out." I think, however, there was no danger of that, for he had already knocked his brains out against the nugget. Howitt adds, "He is a hopelessly ruined man." But he is a type of the class. They are all fast men. Hear some of the names of the places where they dig: "Jackass Flat," — "Sheep's-Head Gully," — "Murderer's Bar," etc. Is there no satire in these names? Let them carry their ill-gotten wealth where they will, I am thinking it will still be "Jackass Flat," if not "Murderer's Bar," where they live.

[23]    The last resource of our energy has been the robbing of graveyards on the Isthmus of Darien,(9) an enterprise which appears to be but in its infancy; for, according to late accounts, an act has passed its second reading in the legislature of New Granada,(10) regulating this kind of mining; and a correspondent of the "Tribune" writes: — "In the dry season, when the weather will permit of the country being properly prospected, no doubt other rich 'Guacas' [that is, graveyards] will be found." To emigrants he says: — "do not come before December; take the Isthmus route in preference to the Boca del Toro one;(11) bring no useless baggage, and do not cumber yourself with a tent; but a good pair of blankets will be necessary; a pick, shovel, and axe of good material will be almost all that is required": advice which might have been taken from the "Burker's Guide."(12) And he concludes with this line in Italics and small capitals: "If you are doing well at home, STAY THERE," which may fairly be interpreted to mean, "If you are getting a good living by robbing graveyards at home, stay there."

[24]    But why go to California for a text? She is the child of New England, bred at her own school and church.

1. The Fitchburg Railroad opened to Concord on June 17, 1844, then to Acton, past Walden Pond, on October 1, 1844. - back
2. "Lord" Timothy Dexter (1748-1806), of Newburyport, Mass, known for his eccentricities - back
3. The English Chancery Court developed from the Lord Chancellor's jurisdiction, in which judges decide the outcome of a case, as in a property dispute, by their interpretation of the law - back
4. Plato (c.427–c.347 BC) Greek philosopher, a student of Socrates, writer, founder of the Academy in Athens, in the year 835, where Aristotle studied - back
5. Reference to the California gold rush that began in 1848 - back
6. An early spelling of Muhammad - back
7. Alfred W. Howitt (1830-1908) Australian anthropologist, naturalist, geologist, explorer - back
8. Ballarat, Bendigo and Sulky Gully are places in Australia where gold was discovered - back
9. Early name of the Isthmus of Panama, a narrow strip of land linking North and South America. - back
10. Republic of New Granada (1831-1856), included modern Colombia and Panama - back
11. Alternate crossings of Panama, to reach the Pacific Ocean - back
12. Referance to William Burke (1792-1829), a serial killer in Edinburgh, Scotland, who sold bodies to a School of Anatomy in the 1820's - back

The South Korean ministry of Science, ICT and Future Planning has banned the common practice of mobile manufacturers and networks putting un-removable apps on smartphones.

Telcos will now be required to make all pre-installed apps deletable, except for those that enable wi-fi connectivity, near-field communication, customer service and an app store.

"The move aims to rectify an abnormal practice that causes inconvenience to smartphone users and causes unfair competition among industry players," said the ministry in a Korean-language press release.

It added that users will also benefit from the regulations in terms of battery life and data storage.

We're unlikely to see similar legislation in the UK. Apple has famously never permitted carriers or networks to pre-install apps on the iPhone, and Android users can remove these apps by rooting and swapping out their handset software for a clean version, though the technical skills required make this a somewhat rare practice.

On a future road trip, your robot car decides to take a new route, driving you past a Krispy Kreme Doughnut shop. A pop-up window opens on your car’s display and asks if you’d like to stop at the store. “Don’t mind if I do,” you think to yourself. You press “yes” on the touchscreen, and the autonomous car pulls up to the shop.

Wait, how did the car know that you might want an original glazed doughnut? Because it has data on your driving habits, and you’re a serial offender when it comes to impulsive snacking. Your car is also linked to your online accounts at home, and you had recently “liked” Krispy Kreme’s Facebook page and visited its website. 

Is this future scenario convenient—or creepy? It’s one thing if a car’s driver-drowsiness detection system (which exists today) sees that you’re nodding off and suggests coffee. But to make your automated car divert from its usual course because some advertiser paid it to do so, well, that sounds like a mini-carjacking.

Whatever you think of it, this future may be coming up on the road ahead. At the Consumer Electronics Show (CES) earlier this month in Las Vegas, automakers announced deals to deliver online services or in-car apps to web-enabled cars of tomorrow. And where there are free or cheap online services, there’s online advertising—that train is never late. 

We don’t know what that advertising might look like: It could literally steer your future car, or it could be more familiar, such as streaming ads across your windshield in auto-driving mode (maybe too distracting in manual-driving mode). But because ad revenue is still the dominant e-business model, it’s a safe bet that advertising will be coming to a future car near you. After all, Google’s acquisition of Nest—maker of “smart” thermostats and other appliances—last week appears to be its first step toward owning the Internet of Things. If the technology giant is leaping the firewall of your personal computer to the rest of your home, why not also your car? Apple co-founder Steve Jobs reportedly had hoped to bring an “iCar” to market, essentially a huge iPhone with wheels. 

Could advertisers really influence the route taken by a self-driving car? It seems plausible, and legal, in at least some circumstances. Say there are multiple routes to your destination. Some may be shorter in terms of distance but longer in terms of travel time, or some routes are equidistant. In those cases, there’s no obviously “right” route to take, but advertiser money could be a “plus factor” that’s just enough to tip driving algorithms in their direction. 

This practice doesn’t seem to be a big inconvenience for the car’s passengers, as long as the detour doesn’t add much extra time or distance to their trip. Some taxi drivers and hotel concierges are known to accept kickbacks from restaurants, casinos, strip clubs, and other establishments to steer business toward them. So this already happens today. But even if not illegal, it raises ethical questions and the need for transparency in a world run by algorithms most of us don’t understand.

More Ethical Potholes

Privacy is already a chief worry about in-car apps and robotics more generally, which some predict to be the next battleground for civil liberties. The doughnut scenario above speaks to that fear. Distracted driving could be made worse with in-car apps, as this hilarious video predicts. But there are other, less obvious problems to think about too:

A couple of weeks ago, a Massachusetts man was arrested when allegedly his Google+ account automatically emailed invitations to everyone in his address book, including his ex-girlfriend who had a restraining order against him, without his knowledge. Something similar could happen with robot cars, such as driving a registered sex offender right by a school when he isn’t supposed to be within 2,000 feet of them. Who would be to blame: the human behind the wheel, or the self-driving car?

As one automotive vice-president unwisely pointed out at CES, “We know everyone who breaks the law, we know when you’re doing it. We have GPS in your car, so we know what you're doing. By the way, we don’t supply that data to anyone.” This raises the issue of whether capability implies responsibility: Are you morally obligated to act on information that could prevent serious harm to someone?  For instance, if an intelligence agency collects data that strongly suggest an impending terrorist attack, it seems wrong not to warn the public or try stopping the attack.

As this applies to automated cars and certain people, it could be the duty of manufacturers to not only figure out where a car’s driver should go, but also where he or she should not go. In some distant future, if the locations of most people can be pinpointed through GPS and other methods, a robot car could tell when a driver is about to violate a restraining order and then refuse to travel there. If they have the data to connect the dots, they probably should do it when it matters.

And it doesn’t just matter for legal reasons, but other factors could be important to users of future wired cars. An owner of a shiny new robot car probably wouldn’t appreciate being deliberately driven—because of advertisers—past fast-food restaurants if she’s on a diet, or by a cluster of bars if she’s a recovering alcoholic, or toward maternity stores if she hasn’t publicly revealed her pregnancy.    

It could be that drivers and passengers can instruct cars to avoid certain destinations. Putting aside the question of why we should be imposed upon like this at all, if the car were to drive to those verboten destinations anyway, that’s probably wrong. Recall in Isaac Asimov’s novels that the second law of robotics is to always obey human orders (where they don’t violate the first rule to not cause or allow harm to humans). 

However, resisting humans is a major point of autonomous cars: We humans are often error-prone and reckless, while algorithms and unblinking sensors can physically drive better than us in most if not all cases. An automated vehicle is designed precisely to disregard our orders where they are imminently risky. That’s to say, refusing human orders is sometimes a feature, not a bug. It’s unclear, then, whether opting-out of certain destinations (or opting-in) is reason enough for cars to comply with those commands.

* * *

The app itself is becoming the new killer app. The latest Windows 8 machines mimic the app dashboards on Apple OS and Android mobile phones. And we can expect online applications to be part of future cars, robotic or not. As existing apps on our mobile phones and computers are already doing now, in-car apps will raise a host of legal and ethical dilemmas, from privacy and beyond.

The problem I discussed at the beginning was related to advertising, but advertising itself isn’t the problem. At their best, advertising could be helpful video clips or images that educate you about products and solutions you truly might be interested in. At their worst, they’re annoyances that interrupt your concentration while you’re absorbed in an essay, video, podcast, or video game.  Ads can push you to vote one way, or buy this thing you don’t need.  They could make you into a worse person—or a better person. 

So while advertising gets a lot of criticism, ads seem to be a necessary evil if the consumer wants to pay as little as possible. That’s neither here nor there in our discussion now, but the decision to allow a car to be controlled by third-parties—directing the route for an advertiser’s interests and not the car owner’s—is the real problem.  Advertising inside a wired car is not just about showing you tantalizing stuff, but it could be about driving you physically to that stuff.  This paradigm shift would make ads even more invasive than critics today might imagine.

More seriously, manufacturers will also need to make hard life-and-death choices in programming autonomous cars, and these decisions should be considered thoughtfully, openly, to ensure a responsible product that millions will buy, ride in, and possibly be injured with. That’s all the more reason to focus on ethics—not just on law, as we’re doing at the Center for Automotive Research at Stanford (CARS)—in steering the future of transportation in the right direction.

Google, Microsoft, Cisco, and Apple lost a significant chunk of workers to startups in 2013.

Jobvite released a report today on hiring patterns in the tech industry, and it found that tech giants have lost their luster for many employees.

“Jobseekers do not hesitate to ditch their cushy jobs to move on to growing, trendy, fast-paced startups with the expectation to work on cool new things and increased financial prospects,” Jobvite chief technical officer Adam Hyder told VentureBeat. “Bigger companies lose their luster and culture over a period of time, and employees hate to work on products that generate high revenue but are stagnant.”

The report found that Twitter attracted the largest number of applications from Google and Microsoft employees. Dropbox and Box were also prime destinations for these workers, and former Apple folk primarily “clustered” at Airbnb, Hulu, Nest, Spotify, Square, Uber, and Yelp.

Leadership changes have a powerful impact on employee movement. Outbound applications from Yahoo employees to tech startups declined by 44 percent after Marissa Mayer became its CEO. Following Microsoft chief Steve Ballmer’s resignation announcement, it suffered a 46 percent increase in employees leaving.

Major migrations are also happening from employees in finance and the aerospace and defense industry who are now heading to the tech world. Citigroup and Wells Fargo employees applied in large numbers to LinkedIn, Twitter, Gilt Groupe, LivingSocial, Workday, Square and Yelp. Boeing and Lockheed Martin lost talent to SpaceX.

“Bigger companies need to create startup culture and offer challenging work to their employees, otherwise they risk losing their employees to fast paced startups that offer challenging work,” Hyder said. “Startups can see a surge in applications from big companies when big companies go through leadership changes or product stagnation. They should seize such opportunities and quickly hire the top talent that they would not be able to attract otherwise.”

Jobvite is a recruiting solution it claims was used by eight of the last 11 top tech companies to IPO, and 25 percent of the U.S. workforce. Hyder said that since Jobvite manages candidate applications processes as well as employer’s hiring processes, it can analyze this data to draw connections and identify trends.

“We can correlate the applicants’ current and past employers, education, experience and other trends,” Hyder said. “We can further correlate these trends with temporal and event information.”

This data was collected over the course of 2013.

ByDUSTIN CURTIS

When I first started working on Svbtle, I was building it out of frustration. I felt that publishing platforms had become too complicated by focusing on the wrong things. What I was looking for was a platform that rolled up its sleeves and worked hard to get at the core of what writing is really about–sharing ideas, naturally.

Svbtle is designed to highlight the things that matter; it’s an extremely simple platform for collecting and developing ideas, sharing them with the world, and reading them. That’s it. We’ve focused all of our energy into designing the simplest interface possible for accomplishing these goals. Svbtle is blogging with everything else stripped away.

Until now, we’ve been an exclusive platform open only to approved users. We took this initial approach because we wanted to ensure that the software worked, first of all, and that the platform was seeded with great content by seasoned and experienced authors. Now we’re finally ready to let more people try Svbtle. Today, we’re opening sign up for everyone.

It works like your brain.
Svbtle’s dashboard is designed to work the same way your brain works. It encourages you to dump ideas, links, and thoughts into a flow of draft posts, and then makes it easy to slowly sculpt those ideas into publishable articles. It just feels natural.

It gets out of the way.
When we’re writing, we like to have no distractions, so we removed all of them. Only a few essential styling options remain, but articles can be written using shorthand formatting with Markdown, for more control. (Don’t worry; it’s easy to learn.)

Svbtle’s writing interface was designed to get out of the way, and it lets you focus on writing.

It cares about your identity.
Writers shouldn’t be defined by the brands of their publishers or platform, but rather by their own personal presence on the web. So we’ve built features that enable you to own your space inside Svbtle: Your full name appears by everything you write, you can use a custom domain, and you can choose an avatar and accent color. We have plans for more personalization and branding features soon, too.

We want to make the best place for writing, reading, and building a personal presence on the web. And we’re just getting started. Give Svbtle a try; I hope you enjoy what we’ve built.

Sign up →

Security is obviously one of the most important aspects of Avatar design. We’ve tried to follow best practices and minimize as many different threats as we can come up with, but it is our sincere hope that this section is closely inspected by the crypto community and questions are asked.

We have tried to keep everything related to security as simple as possible for two reasons. The first reason is that we don't want to create complex protocols prone to either implementation, or logical errors. Sticking with the best practices and fundamentals will provide a solution for almost all security challenges. The second reason is that we want everything to be as easy as possible for others to understand and review properly. This way possible errors in thinking are found fast and fixed even faster.

If you are reading this chapter, you are probably familiar with Alice and Bob. In case you aren't, all you need to know is that they are two friends using Avatar.

Design goals and threat models

The design goals for the Avatar OS are summarized in Avatar Principles. The overarching goal is to connect and enable people to interact over the Internet securely and privately.

The threat model for Avatar OS is an adversary that can seize and access the device but is not able to constantly monitor the user using the device. There's very little that we can do against an adversary who is running a keylogger or spying on the screen in real-time.

The threat model for Avatar Network is based on an adversary not being able to control all nodes in the routing path. We also assume the adversary isn't able to intercept 100% of the packets. Due to the internet's global nature it's unlikely that an adversary has the capability to intercept 100% of traffic all over the world. The decentralization fundamentals provide the mental framework about how Avatar OS was designed to work with Avatar Network.

Encryption schemes used

All Objects are encrypted with AES in CTR mode using 256bit keys with Encrypt-then-MAC composition. Avatar uses the Forge library as a generic crypto library.

For messaging and identification/signing purposes we use elliptic curve DSA with secp256k1 (Koblitz curves). The first iteration of the design used RSA with 2048bit keys but tests showed that performance is noticeably poor with today's mobile devices. Please note that NSA backdoor applies only to elliptic curve DSA with secp256r1 curves. The library for ECC is JSBN extended with ECDSA_JSBN for ECDSA and secp256k1. We are aware of theoretical weaknesses in secp256k1 and are evaluating Curve25519 as a replacement.

Avatar OS code delivery protocol

Secure code delivery within the browser is a challenging problem because, due to the nature of the web, the browser was designed to treat all code as potentially malicious and to survive from running it without any external authority saying what code is good and what code is bad. This is of course fantastic from the freedom point of view but it means browsers don't have any methods to verify the validity of any code it attempts to execute. There has been few propositions in recently published whitepapers that would provide this functionality but at the moment there's no mechanism that enables browser to ask "is this code 100% identical to what foo.com/code.js is supposed to be".

Traditionally websites use SSL (TLS) connection to secure the connection and believe it's enough protection. However even if the connection between web server and visitor is secured, a properly motivated adversary can always compromise the server itself, replacing the necessary files with malicious ones. For this reason we believe it's not optimal to serve Avatar OS code in avatar.ai without an external verification mechanism making sure the files on the server have not been tampered with. The current solution is still not optimal because files are stored at avatar.ai. The optimal solution would be to use Avatar Network to store Avatar OS code but due to various challenges it's not possible yet.

Our code delivery protocol is based on storing a proof of validity in Namecoin blockchain and doing near real-time checks for validity of the live code in avatar.ai.

Our thinking behind this is that an adversary would have to either get enough computing power to replace the hash in a Namecoin blockchain, or create a very expensive attack separating requests that validate to content from requests from normal users. We know some governments have the capabilities to divert traffic before it even touches the target server but the current assumption is that traffic diverting is possible only on a very small scale and can't be done on a large enough scale required here.

In the Avatar OS release packaging phase, Avatar OS code will be base64 encoded and hashed with sha256. The hash is then stored into a Namecoin blockchain. Because Avatar is packaged as one HTML file for portability, we can verify it just by requesting it over the Internet, computing hash from the received file and comparing it with the hash in the Namecoin blockchain.

The checks are done in 5-15 second intervals via Tor network. Tor helps us to protect the location of servers doing the checks thus making the attack surface smaller. It also makes it harder for anybody monitoring the traffic to know which requests are doing the validation. Checker servers access the Namecoin blockchain directly to get the hash and make the comparison with the live code. If the live code verification fails, the checker server will alert the admins and attempt to replace the compromised live code with a verified one.

Everything is an encrypted Object

All communication between Avatars happens with encrypted JSON objects. All messages, files and basically everything you produce with Avatar OS are Objects. These Objects are stored either locally or in the Network. To access any Object in Avatar, even the locally stored, you need to know the unique id (UID), the access key (AKEY) and the key for encryption (EKEY). UID, AKEY and EKEY are explained in more detail further on in the Object address protocol chapter.

The main design goal with Objects is that if somebody is able to obtain an Object, there should be no way for them, without a correct EKEY, to say which type of Object it is, what data it has, who created it and who has accessed it. By requiring both UID and AKEY to fetch an Object we make it much harder to bulk request random objects by generating random 64 strings. The secondary goal was to provide more security when Alice uses a shared device. This is achieved by storing meta data inside Alice's Avatar instead of Objects themselves. Even if Alice doesn't securely log out and leaves her Objects in the shared devices they can't be opened, or analyzed, because of the missing EKEY that only Alice's Avatar knows. This helps with the inadequate access controls of browser technologies like IndexedDB.

Anatomy of Avatar

We have many different Avatar parts in the system but when we are saying "Avatar", we are talking about a user's Avatar Object which contains all the important data for that user. Each Avatar is composed of an Object Registry, Personal Information Registry and Contact List. Everything is stored in one JSON object which is treated just like any other Object in the system. This makes it very hard for an adversary to separate high-value targets like Avatar Objects from any other Objects in the Network.

When a new Avatar is created it will generate a unique Avatar ID (AID), a public address (APA) and – for cryptographic signing purposes – a private key (APRK) and a public key (APK).

Object Registry

Object Registry is the place where all the knowledge of all Objects which Avatar knows about is kept. Essentially it's one long list of Object UIDs and each record contains enough information to reach the Object. Object Registry will be explained in detail later in this document.

Personal Information Registry

PIR is a place where Avatar stores all data about the user. The user can store any arbitrary data and if any external entity wants to request information from PIR, Avatar will request permission from the user. Nothing in PIR is available to any external entity without the user's explicit permission.

The idea behind PIR is to provide a way for users to store important data about themselves so that it's protected but at the same time they can give permission for others to access it.

Contact List

Contact List has all the information related to relationships with different Avatars. Each Contact shares its unique address (shared address) with a user's Avatar enabling one-to-one communication. Messaging and friending protocols are explained later in the document.

Authentication protocol

When Avatar OS has been executed successfully it expects the user to provide a username and password to authenticate. The user's Avatar may or may not exist on the device. If UID and AKEY don’t return anything from the local cache, Avatar OS will ask Avatar Network to deliver the correct Object.

The user's Avatar consists of two separate objects: Buffer Object and Avatar Object. Like everything else in Avatar, Objects are encrypted JSON objects.

Buffer Object’s sole purpose is to give the user the ability to change password without encrypting Avatar Object again and the ability to see a password reminder in case they have forgotten their password. Because Avatar is a decentralized system, there is no one who can reset the user’s password. If the password is forgotten, there is no way to retrieve the data from the user's Avatar. Another reason for this design was to make it more expensive to steal objects from Avatar Network and try to find Avatar Objects by bruteforcing weak passwords.

When the user starts the authentication process, the only information available are the username and the password. Because the whole point of Buffer Object is to give the user a chance to recover the forgotten password from their memory, we can’t use user's password to derive UID and AKEY for Buffer Object.

Buffer Object UID and AKEY are derived by computing a 128 character hash from the username with scrypt and then splitting the hash into two 64-character long strings. Avatar uses sjcl-scrypt library for scrypt, however this will change before the first version to another implementation without SJCL.

The Buffer Object consists of two layers. The first layer is encrypted by using username as EKEY. The first layer contains the reminder question as a plain text and the second layer as an encrypted string. The first layer isn't expected to be secure. The second layer is encrypted with the user's password and contains UID, AKEY and EKEY for the user’s Avatar Object. If the user decides to change the password, only the second layer needs to be changed and encrypted again.

Once Avatar Object has been successfully fetched from the Network and decrypted with EKEY from the Buffer Object it can be imported into Avatar OS. Avatar Object contains all user information from personal information to references to stored data. Essentially it's the heart of Avatar OS and what makes it yours.

Object address protocol

Storing data in Avatar Network is a challenge because our assumptions dictate that we can't trust any nodes to behave properly. This makes it difficult to create a system that offers the features users are expecting for any messaging or online storage platform. The first versions of Avatar will focus on providing message storing via Avatar Network and the DHT item size will be limited at the protocol level. Once we have more production data to verify everything works as designed, this limitation will be removed.

Avatar Network uses a modified R5N DHT (whitepaper) as a distributed storage layer on top of Avatar Network. Avatar Network Protocol also utilizes R5N DHT for its internal purposes.

In DHT terminology UID would be the equivalent of a key for a data item. DHT has been modified so that to fetch a data item you also need to send a password which is AKEY. If AKEY is missing or wrong, DHT will drop the request. UID and AKEY are both 64-character strings.

Object address protocol for non-chainable Objects

Non-chainable Objects are individual Objects that exist without a previous or next Object. For example individual chunks from a bigger Object are non-chainable Objects because they are meant to exist without any means to connect them to other chunks.

For non-chainable Objects both UID and AKEY are randomly generated by using Fortuna PRNG and the current timestamp.

Object address protocol for chainable Objects

Some Manifest Objects need to have chainability capabilities to provide a chronological ordering where needed. For example discussion threads and status updates. We can solve this by creating a UID/AKEY scheme that provides a chronologically ordered list of Objects. Chainable Objects use a deterministic scheme to create UID and AKEY.

Chainable Object's UID is derived from the following deterministic scheme: sha256(seed + (n+1)). The seed can be anything but most likely it will be a shared address which will be covered later on. When Avatar wants to add a new Object to the chain, it will simply keep adding to n until it finds a free slot. AKEY is derived by taking the first 32 characters of the seed: sha256(32 characters + (n+1)). This isn't optimal but makes bulk fishing the Network for Objects very expensive and difficult.

Object address protocol for multi-level chainable Objects

Multi-level chainable Object means that there are two or more seed values and an iteration number. At the moment these are only used in messaging protocols to provide conversation-like mechanics.

In multi-level UID is derived from the following deterministic scheme: sha256(seed1 + seed2 + (n+1)). Similar to chainable Objects AKEY is derived by taking the first 32 characters of the seed: sha256(32 characters + seed2 + (n+1)).

Because Avatar Network can also expire items for storage efficiency purposes, expired keys are not deleted but instead show up as empty. When Avatar follows the chain it will keep trying until it finds a key that doesn't exist. For efficiency purposes we can also utilize other Avatars (with permission) for knowledge about Objects and their chain lengths.

Data handling

Avatar OS wants to provide a truly easy way to store and share files online while keeping maximum security.

When the user imports a file, the OS automatically converts it to an Object. An Object can store any data, binary or ascii. It has a meta data partition so the OS can determine whether it can open/edit it by itself or if it is data that requires an external application. We use the file's content type to determine this. By converting everything into Objects we hope to create a standardized way to share data over the Internet without having to worry about file formats. In the future Avatar Bridge will support basic transcoding tasks which opens up a lot of opportunities. For example Avatars could sell their computing power for transcoding.

Object Registry

Each Avatar Object has an Object Registry which is a list of all Objects it knows about. Objects can be either available in the local cache or fetched from Avatar Network when required. Being able to choose what to keep locally is necessary to support devices with lower storage capabilities and to make it possible to quickly log in to Avatar on any device to check messages without downloading everything.

Object Registry Item

When a new Object appears Object Registry creates an Object Registry Item for that specific Object. Object Registry Item only carries UID, AKEY and EKEY for an Object's Manifest Object. This is because we want to keep Object Registry as small as possible so Avatar Object's size remains relatively low even with a lot of files. If Avatar has been given a permission secrets for an Object, those are also stored in Object Registry Item. Permission secrets will be covered later on.

Manifest Object and Chunks

Manifest Objects are basically small documents listing the contents of an Object. Their main purpose is to separate security details from Objects containing data and to provide a lightweight way to share Objects.

One of the main challenges for a voluntary-participation-driven network is how to balance network load, fairly distribute bandwidth usage, and guarantee availability of the data. These challenges are even harder when it comes to mobile clients. For these reasons, Avatar splits all files bigger than 32 kilobytes into chunks of varying sizes between 32KB and 2MB. The size of the chunk is mainly determined by the size of the file and randomized to a degree. Small chunk size will help availability challenges with unreliable infrastructure where individual Bridges are expected to drop out frequently. Random sizes also make it harder to analyze chunks to identify more high value Objects like Object Registries or Avatar Objects for example.

Each data chunk will be assigned a unique and random UID, AKEY and EKEY. All chunk sizes, UIDs, AKEYs and EKEYs are stored in Manifest Object in order. Each chunk will be hashed with sha256 and the hash will be stored in the Manifest. The first chunk is always the meta data partition of the Object so Avatar doesn't have to download every chunk to access the meta data. This also makes it possible to stream Objects.

When you want to send data to another Avatar you send the UID, AKEY and EKEY to the Manifest Object instead of sending the Object.

Object permissions and secure Object modifying

Updating anything in a distributed environment is hard from a security point of view. The challenge can be distilled into a question: How do you guarantee that nobody other than the owner is able to replace data? If you rely on the standard merchanism of the owner sending a new data and update command, how do you make sure the receiving node doesn't use this information to issue another update command replacing the data with its own?

One common everyday work scenario is that Alice creates a document and she wants to give Bob editing rights. However the document will also be sent to Charlie but Charlie shouldn't be allowed editing rights.

With centralized architecture it would be as simple as restricting what Charlie can do at the server level, but we don't have that luxury with Avatar. In a decentralized architecture we can't restrict what Charlie does with the Object if he has received it, but we can control that he doesn't replace Alice's and Bob's version in the Network without permission from Alice or Bob.

Permission info is kept in a separate permissions object for each chunk inside the Manifest Object. The Permission object contains scopes for various actions like changing permissions, updating and deleting the Object from the Network. Each scope contains three items: shared secret (PSEC), public key hash (PHASH) and meta data (PMETA). Having to compute each chunk separately is not very efficient but we haven't found any other solutions that would provide a way for Alice to update her Objects in the Network without giving the malicious node a way to reuse Alice's update credentials and make malicious updates in her name.

PSEC is the first secret derived with Sharmir's secret sharing algorithm from 256bit ECDSA private key. Please note that anybody who has access to permission change scope has access to variables used to generate shared secrets, and therefore can add secrets without anybody else's permission. This happens because if we want to be able to potentially add shared secrets (for new users) later on, we must lock in the parameters used to generate the secrets in the first place, in order to be able to keep the existing shared secrets while adding new ones. So allowing other users to change permissions should be decided with care.

By using Sharmir's secret sharing we can also build m-of-n permissions. This means that you can create an Object that can be modified only if, for example, 3 out of 5 agrees and uses their shared secret to get the private key. The default setting is 1 out of X.

PHASH is sha256 hash computed from the public key counterpart to the private key in PSEC. Because Shor's algorithm will provide quantum computers a very, very fast way to bruteforce private keys from public keys, we want to avoid exposing any keys unless absolutely necessary. Instead of storing the public key in the chunk's permission scope, we only store a hash of it. This way the public key is only exposed if the Object needs to be updated. Quantum computing doesn't provide any meaningful threat to cryptographic hash functions.

PMETA offers a way to store other arbitrary data related to permission scope. For example when generating new shared secrets Bob needs the original random number used for Sharmir's secret sharing algorithm. PMETA is encrypted with AES using the scope's ECDSA private key as an encryption key.

Updating example

Alice has decided to give Bob permission to update the data portion of her Object. First she computes new shared secrets for the update permission scope for all the chunks, and sends them to Bob. Bob stores his shared secrets to the Object Registry Item corresponding to the Object. Bob makes some changes and wants to publish them. First he will chunk the Object with chunk sizes specified in the Manifest. After that he checks which chunks have changed due to his updates by hashing the old and the new and comparing the hashes.

For each changed chunk he takes his shared secrets and derives ECDSA private keys from PSECs stored in the update permission scopes. Each private key will give him the public key for that specific chunk. For each chunk Bob will now encrypt the data with the chunk's EKEY, hash the encrypted data with sha256 and use a private key and hash to get a signature consisting of two values, checksum 1 (PCHK1) and checksum 2 (PCHK2).

Bob will now do an update call to the Network. With the call he sends new chunk data, UID/AKEY for access, a public key, PCHK1 and PCHK2. The network will route Bob's call to the nodes that are holding the chunk with that specific UID. The nodes will verify access by matching AKEY. If Bob's AKEY matches with the one on the chunk, the node will then compute sha256(public key). The hash of the public key will be matched against the requested permission scope's PHASH value. If the hashes match, Bob's public key is valid and the node now knows that Bob has the correct private key.

Now the node needs to make sure that Bob's new data hasn't been altered in transit and do one more check that Bob has the right to change the data. The node will now compute a sha256 hash from the new data chunk. Then it will feed the hash, PHCK2 and public key through ECDSA. If the result matches with PHCK1 then the node knows that Bob truly has the private key and the data is identical with the one where PCHK1 and PCHK2 came from.

Because the node doesn't know the private key, it can't compute PCHK1 and PCHK2 to create a malicious update. The only thing a malicious node can do is to corrupt the data and store that, or drop the update. A malicious node can do both of these actions anyway. Either way, the node would soon be flagged and dropped from the Network. However, having to expose the public key creates a weakness which quantum computing could exploit. A malicious node can store all public keys it encounters and derive private keys later on. The attack surface is limited in a sense that if the private key is compromised, the node can't read data, it can only forge a certain action provided by the permission scope. This could be theoretically fixed by creating some sort of changing key scheme.

"Frending" and following protocol

Avatar's friending mechanism operates on shared addresses. A shared address is a 64-character string that is shared between two Avatars, so it’s essentially a relationship ID. When Alice wants to friend Bob and give Bob a chance to send her something, Alice's Avatar will create a new unique address for that relationship, and then send that address to Bob's APA. We are assuming that Bob has shared his public address with Alice. When Bob receives Alice's friend request, he can then either accept the request or decline it. Bob can assign Alice to a certain group or mark Alice as an "acquaintance", so that unless Bob specifically views Alice's profile, he won't see Alice's public updates but he will still receive any private messages Alice sends. From his Avatar's Contact List Bob can configure exactly what type of communication he will allow Alice to send him. The idea is that you can store contacts in Avatar that you don't feel are really part of your life but you still want to possibly reach in the future.

Because Avatar always defaults to private, your Avatar is not visible to anybody else unless you post a public update or you give out your AID. When you post an update you can limit it to a certain group you have created, or make it public. Public updates are available to everyone in Avatar Network who knows your AID or APA.

If you've published a public update or shared your AID, another Avatar may "friend" you and add your APA to their Contact List. Your APA enables people to only read your public updates, nothing more. This is like a personal RSS feed. Other people can read the feed but you can't approach them unless they let you know you have their permission.

You can follow somebody's public updates with or without giving them permission to send you anything other than their public updates. It's also optional to share that you are following their updates. Following somebody's updates should not be an all-inclusive right to send anything else. This mechanism allows you to decide how visibly you want to follow somebody. You might want to follow, for example, a person representing a certain ideology but wouldn't want to share your interest publicly or with the person in question.

Messaging protocol

Avatar wants to offer easy-to-use, anonymous and secure messaging with store-and-forward capabilities (like email). Avatar Network protocol takes care of anonymization so the messaging protocol focuses on storing and delivering secured messages in one-to-one, one-to-many and many-to-many scenarios.

The biggest challenge with a decentralized messaging solution is to offer secure store-and-forward capabilities without going back to centralized servers. Store-and-forward capabilities are essential for anybody who would prefer to send a message than chat.

Message security

Messages are Objects just like everything else and enjoy the same protection as other Objects. All messages themselves are signed by the sender with their APK.

One-to-one messaging

When Alice wants to send a message to Bob, she first generates a random 64-character string as a new conversation ID. The conversation ID is used as a secondary seed in a multi-level chainable Object. Then her Avatar follows data handling and Object address protocols and ends up with a multi-level chainable Manifest Object. She stores the Manifest Object in the Network and tries to establish a connection to Bob's Avatar over the Network and send him a notification about the message. The notification includes the last 10 characters of the shared address, conversation id and AKEY. This way we minimize the risk of compromising identities, or the shared address. By looking at his Contact List for shared addresses Bob's Avatar can easily identify which Contact it came from. Bob's Avatar then follows multi-level Object chaining protocol and gets the correct UID and AKEY for fetching the Manifest Object.

If Bob is not online when Alice tries to notify him, Alice will find the next free key in Bob's shared address and store the notification there. When Bob successfully connects to Avatar Network, his Avatar will go through all Contacts and try to follow each shared address' Object chain. Alice's notification will be waiting for Bob and Bob's Avatar now knows to get the message.

If Bob wants to reply, he uses multi-level Object chaining protocol to generate possible UIDs and AKEYs and keeps trying until he finds a free slot. Then he stores a Manifest Object and tries to notify Alice, just like Alice did before.

One-to-many messaging

In one-to-many messaging Alice sends a message to multiple recipients. These are usually status updates. One-to-many messaging works exactly like one-to-one messaging but instead of notifying one receiver, Avatar notifies multiple. Here using Manifest Objects pays off because sending the whole Object, which could be text, video or any random data, would be very expensive.

Many-to-many messaging

Many-to-many messaging is essentially how discussion boards works. All participants see everyone else's messages. Many-to-many messaging can be created by automatically sharing relevant Manifest Objects to participating users.

There are few efficiency concerns in many-to-many messaging which require further research. The default behaviour in Avatar OS is that new messages would be propagated by the author when published, essentially notifying all other participants. However with many-to-many messaging this is very inefficient so a better option is to have Avatar manually pulling new messages when the user views the discussion. Automated notifications for new messages in "followed" discussions would be nice though.

Push or pull

Avatar uses both. Wherever possible, pushing is used for performance reasons. There are Twitter accounts with nearly 50 million followers and follower counts are constantly increasing. If an Avatar user were that popular and all of those 50 million followers checked at frequent intervals for potential updates it would create a huge amount of unnecessary stress on Avatar Network. This can be avoided by pushing an update notification and the follower's Avatar OS would pull the update when the user logs in.

Off-the-record (OTR)

Avatar messaging doesn't support Off-the-Record messaging in its current form. The essential part of OTR is to provide a private, face-to-face type of conversation. With OTR the problem is store-and-forward mechanism which can't exist without compromising OTR's basic principles. Theoretically it is possible to create OTR chat between Avatars over Avatar Network but it's not currently in our roadmap.

We feel that it's more important to offer a secure alternative to an email type of messaging than to provide OTR functionality that already exists in many IM applications.

Introduction

I recently discovered an interesting game called VimGolf. The objective of the game is to transform a snippet of text from one form to another in as few keystrokes as possible. As I was playing around with different puzzles on the site, I started to get curious about my text editing habits. I wanted to better understand how I manipulated text with vim and to see if I could identify any inefficiencies in my workflow. I spend a huge amount of time inside my text editor, so correcting even slight areas of friction can result in worthwhile productivity gains. This post explains my analysis and how I reduced the number of keystrokes I use in vim. I call this game Vim Croquet.

I started my analysis by collecting data. All my text editing on a computer is done with vim, so for 45 days I logged every keystroke I used in vim with the scriptout flag. For convenience, I aliased vim in my shell to record all my keystrokes into a log file:

aliasvim='vim-w~/.vimlog"$@"'

Next, I needed to parse the resulting data. Parsing vim is complicated. vim is a modal editor where a single command can have different meanings in different modes. Commands can also have contextual effects where the behavior of certain actions can be different depending on where they are executed within a buffer. For example, typing cib in normal mode moves the user into insert mode if the command is executed between parentheses, but leaves the user in normal mode if executed outside of parentheses. If cib is executed in insert mode it has an altogether different behavior; it writes the characters cib into the current buffer.

I looked at several candidate tools for parsing vim commands including industrial parser libraries like antler and parsec as well as a vim-specific project called vimprint. After some deliberation, I decided to write my own tool. I don’t do a lot of language processing, so investing the time to learn a sophisticated parser seemed unwarranted.

I wrote a crude lexer in Haskell to tokenize the keystrokes I collected into individual vim commands. My lexer uses monoids to extract normal mode commands from my log for further analysis. Here’s the source code for the lexer:

importqualifiedData.ByteString.Lazy.Char8asLCimportqualifiedData.ListasDLimportqualifiedData.List.SplitasLSimportData.MonoidimportSystem.IOmain=hSetEncodingstdoututf8>>LC.getContents>>=mapM_putStrLn.processprocess=affixStrip.startsWith.splitOnMode.modeSub.capStrings.splitmark.preprocesssubs=appEndo.mconcat.map(Endo.sub)sub(s,r)lst@(x:xs)|s`DL.isPrefixOf`lst=sub'|otherwise=x:sub(s,r)xswheresub'=r++sub(s,r)(drop(lengths)lst)sub(_,_)[]=[]preprocess=subsmeta.DL.intercalate" ".DL.words.DL.unwords.DL.lines.LC.unpacksplitOnMode=DL.concat$map(\el->splitmodeel)startsWith=filter(\el->mark`DL.isPrefixOf`el&&el/=mark)modeSub=map(subsmtsl)splitsr=filter(/="")$s`LS.splitOn`raffixStrip=clean.concat.map(\el->splitmarkel)capStrings=map(\el->mark++el++mark)clean=filter(not.DL.isInfixOf"[M")(mark,mode,n)=("-(*)-","-(!)-","")meta=[("\"",n),("\\",n),("\195\130\194\128\195\131\194\189`",n),("\194\128\195\189`",n),("\194\128kb\ESC",n),("\194\128kb",n),("[>0;95;c",n),("[>0;95;0c",n),("\ESC",mark),("\ETX",mark),("\r",mark)]mtsl=[(":",mode),("A",mode),("a",mode),("I",mode),("i",mode),("O",mode),("o",mode),("v",mode),("/",mode),("\ENQ","⌃e"),("\DLE","⌃p"),("\NAK","⌃u"),("\EOT","⌃d"),("\ACK","⌃f"),("\STX","⌃f"),("\EM","⌃y"),("\SI","⌃o"),("\SYN","⌃v"),("\DC2","⌃r")]

Here’s a sample of the data in its unprocessed form and its structure after lexing:

cut-c 1-42 ~/.vimlog|tee>(cat-v;echo)|./lexer
`Mihere'ssometext^Cyyp$bimore ^C0~A.^C:w^M:q
`Myyp$b
0~

My lexer reads from stdin and sends processed normal mode commands to stdout. In the above example pipe, I use a process substitution to print a representation of the unprocessed data on the second line and the resulting output of the lexer on subsequent lines. Each line in the output of the lexer represents a grouping of normal mode commands executed in sequence. The lexer correctly determined that I started in normal mode by navigating to a specific buffer using the `M mark, then typed here's some text in insert mode, then copy and pasted the line and moved to the start of the last word on the line using yyp$b, then entered additional text, and finally navigating to the start of the line and capitalizing the first character using 0~.

After lexing my log data, I forked Patrick Wied’s awesome heatmap-keyboard project and added my own custom layout to read the output of my lexer. Patrick’s project does not detect most meta-characters like escape, control, and command, so it was necessary for me to write a data loader in JavaScript and make some other modifications so the heatmap would accurately depict key usage in vim. I translated metacharacters used in vim to unicode representations and mapped these onto the keyboard. Here’s what my key usage looked like based on $\approx 500,000$ normal mode keystrokes processed by my lexer. Increasing wavelengths denotes more prevalent key usage:

A prominent feature of the heatmap is the prevalent usage of the control key. I use control for numerous movement commands in vim. For example, I use ⌃p for Control P and I cycle forward and backward through open buffers with ⌃j and ⌃k, respectfully. Control is an efficient movement on my Kinesis Advantage because I remap it to left thumb delete.

Another pattern in the heatmap that jumped out at me was my heavy use of ⌃E and ⌃Y. I routinely use these commands to navigate up and down through source code, but moving vertically with these commands is inefficient. Each time one of these commands is executed, the cursor only moves a few lines at a time. A more efficient pattern would be to use larger vertical movements with ⌃U and ⌃D. These commands move the cursor up or down a half screen at a time, respectively.

The heatmap gives a good overview of how I use individual keys, but I also wanted to learn more about how I used different key sequences. I sorted the lines in the output of my lexer by frequency to uncover my most used normal commands using a simple one-liner:

$ sort normal_cmds.txt | uniq -c | sort -nr | head -10 | \
 awk '{print NR,$0}' | column -t
1 2542 j
2 2188 k
3 1927 jj
4 1610 p
5 1602 ⌃j
6 1118 Y
7 987 ⌃e
8 977 zR
9 812 P
10 799 ⌃y

Seeing zR rank as my 8th most used sequence was unexpected. After pondering this, I realized a huge inefficiency in my text editing. My .vimrc is setup to automatically fold text. The problem with this configuration is that I almost immediately unfold all folded text, so it makes no sense for my vim configuration to use automatically fold text by default. Therefore, I removed this setting so that I would no longer need to repeatedly use the zR command.

Another optimization I wanted to looked at was normal mode command complexity. I was curious to see if I could find any commands that I routinely used which also required an excessive number of keystrokes to execute. I wanted to find these commands so that I could create shortcuts to speed up their excution. I used entropy as a proxy to measure command complexity using a short script in Python:

#!/usr/bin/env pythonimportsysfromcodecsimportgetreader,getwriterfromcollectionsimportCounterfromoperatorimportitemgetterfrommathimportlog,log1psys.stdin=getreader('utf-8')(sys.stdin)sys.stdout=getwriter('utf-8')(sys.stdout)defH(vec,correct=True):"""Calculate the Shannon Entropy of a vector """n=float(len(vec))c=Counter(vec)h=sum(((-freq/n)*log(freq/n,2))forfreqinc.values())# impose a penality to correct for sizeifall([correctisTrue,n>0]):h=h/log1p(n)returnhdefmain():k=1lines=(_.strip()for_insys.stdin)hs=((st,H(list(st)))forstinlines)srt_hs=sorted(hs,key=itemgetter(1),reverse=True)forn,iinenumerate(srt_hs[:k],1):fmt_st=u'{r}\t{s}\t{h:.4f}'.format(r=n,s=i[0],h=i[1])printfmt_stif__name__=='__main__':main()

The entropy script reads from stdin and finds the normal mode command with the highest entropy. I used the output of my lexer as input for my entropy calculation:

$ sort normal_cmds.txt | uniq -c | sort -nr | sed "s/^[ \t]*//" | \
 awk 'BEGIN{OFS="\t";}{if ($1>100) print $1,$2}' | \
 cut -f2 | ./entropy.py
1 ggvG＄"zy 1.2516

In the command above, I first filtered all the normal mode commands that I executed more than 100 times. Then, among this subset, I found the command with the highest entropy. This analysis precipitated the command ggvG$"zy, which I executed 246 times in 45 days. The command takes an unwieldy 11 keystrokes and yanks the entire current buffer into the z register. I typically use this command to move the contents of one buffer into another buffer. Since I use this sequence so frequently, I added a short cut to my .vimrc to reduce the number of keystrokes I need to execute:

nnoremap<leader>ya ggvG$"zy

My Vim Croquet match revealed three optimizations to decrease the number of keystrokes I use in vim:

• Use coarser navigation commands like ^U and ^D instead of ^E and ^Y
• Prevent buffers from automatically folding text to obviate using zR
• Create shortcuts for verbose commands that are frequently used like ggvG$"zy

These 3 simple changes have saved me thousands of superfluous keystrokes each month.

The code snippets above are presented in isolation and may be difficult to follow. To help clarify the steps in my analysis, here’s my Makefile, which shows how the code presented in this post fits together:

SHELL:= /bin/bashLOG := ~/.vimlogCMDS := normal_cmds.txtFRQS := frequencies.txtENTS := entropy.txt
LEXER_SRC := lexer.hs
LEXER_OBJS := lexer.{o,hi}
LEXER_BIN := lexer
H := entropy.pyUTF := iconv -f iso-8859-1 -t utf-8
.PRECIOUS: $(LOG)
.PHONY: all entropy clean distclean
all: $(LEXER_BIN)$(CMDS)$(FRQS) entropy$(LEXER_BIN): $(LEXER_SRC)
 ghc --make $^$(CMDS): $(LEXER_BIN)
 cat $(LOG) | $(UTF) | ./$^> $@$(FRQS): $(H)$(LOG)$(CMDS)
 sort $(CMDS) | uniq -c | sort -nr | sed "s/^[ \t]*//" | \
 awk 'BEGIN{OFS="\t";}{if ($$1>100) print NR,$$1,$$2}'> $@
entropy: $(H)$(FRQS)
 cut -f3 $(FRQS) | ./$(H)
clean:
 @- $(RM)$(LEXER_OBJS)$(LEXER_BIN)$(CMDS)$(FRQS)$(ENTS)
distclean: clean

For the past few months I've been working on a project with a fairly complex interactive web interface. This required me to venture into the wild and unpredictable jungle of Javascript development. I was totally unprepared for what I would find. Soon after starting the project it became clear that just using JQuery would not be sufficient for my project. I needed a higher level Javascript framework. After a doing a little research I settled on Ember.js.

The Zombie Code Apocalypse

Ember was definitely a big improvement over straight JQuery, and allowed me to get some fairly complex UI behavior working very quickly. But recently I've run into some problems. The other day I had a UI widget defined like this:

App.FooController = Ember.ObjectController.extend({
 // ...
});
App.FooView = Ember.View.extend({
 // ...
});

It was used somewhere on the page, but at some point I decided that the widget was no longer needed, so I commented out the widget's markup. I wasn't sure whether we would ultimately keep the widget or not, so I opted to keep the above javascript code for the controller and view around for awhile so it would be easily available if I later decided to re-enable that UI element.

Everything seemed to work fine until a few days later when I noticed that another one of my controls, Bar, was not being populated with data. After spending hours trying to figure out the problem, I finally happened to comment out the unused code for the Foo widget and the problem went away. WTF?!? Why should this have anything to do with the functioning of a completely unrelated widget? This makes absolutely no sense to me, and it completely violates the natural assumption that the controller and view for two completely unrelated controls would have no impact on each other. I would have liked to know the underlying cause, but I didn't want to waste time with it, so I just removed the code and moved on.

Spontaneously Changing Values

Maybe a week later I ran into another problem. Some data was changing when I didn't expect it to. I looked everywhere I could think of that might affect the data, but couldn't find anything. Again, I spent the better part of a day trying to track down the source of this problem. After awhile I was getting desperate, so I started putting print statements all over the place. I discovered that the data was changing in one particular function. I examined it carefully but couldn't find any hint of this data being impacted. Eventually I isolated the problem to the following snippet:

console.log(this.get('foo'));
this.set('bar', ...);
console.log(this.get('foo'));

The first log line showed foo with a value of 25. The second log line showed foo with a value of 0. This is utter madness! I set one field, and a completely different one gets changed! In what world does this make any shred of sense? This time, even when I actually figured out where the problem was happening I still couldn't figure out how to solve it. At least the first time I could just comment out the offending innocuous lines. Here I narrowed down the exact line that's causing the problem, but still couldn't figure out how to fix it. Finally I got on the #emberjs IRC channel and learned that Ember's set function has special behavior for values in the content field, which foo was a part of. I was able to fix this problem by initializing the bar field to null. WAT?!?

I was in shock. This seemed like one of the most absurd behaviors I've encountered in all my years of programming. Back in the C days you could see some crazy things, but at least you knew that array updates and pointer arithmetic could be dangerous and possibly overwrite other parts of memory. Here there's no hint. No dynamic index that might overflow. Just what we thought was a straightforward getter and setter for a static field in a data type.

Blaming Systems, Not People

Before you start jumping all over me for all the things I did wrong, hear me out. I'm not blaming the Ember developers or trying to disparage Ember. Ember.js is an amazing library and my application wouldn't exist without it or something like it. I'm just a feeble-minded Haskell programmer and not well-versed in the ways of Javascript. I'm sure I was doing things that contributed to the problem. But that's not the point. I've been around long enough to realize that there are probably good justifications for why the above behaviors exist. The Ember developers are clearly way better Javascript programmers than I will ever be. There's got to be a better explanation.

Peter Senge, in his book The Fifth Discipline, talks about the beer distribution game. It's a game that has been played thousands of times with diverse groups of people in management classes all over the world. The vast majority of people who play it perform very poorly. Peter points out that we're too quick to attribute a bad outcome to individual people when it should instead be attributed to the structure of the system in which those people were operating. This situation is no different.

Like the beer distribution game, Javascript is a complex system. The above anecdotes demonstrate how localized well-intentioned decisions by different players resulted in a bad outcome. The root of the problem is the system we were operating in: an impure programming language with weak dynamic typing. In a different system, say the one we get with Haskell, I can conclusively say that I never would have had these problems. Haskell's purity and strong static type system provide a level of safety that is simply unavailable in Javascript (or any other mainstream programming language for that matter).

The Godlike Refactoring

In fact, this same project gave us another anecdote supporting this claim. The project's back end is several thousand lines of Haskell code. I wrote all of the back end code, and since we have a pretty aggressive roadmap with ambitious deadlines the code isn't exactly all that pretty. There are a couple places with some pretty hairy logic. A few weeks ago we needed to do a major refactoring of the back end to support a new feature. I was too busy with other important features, so another member of the team worked on the refactoring. He had not touched a single line of the back end code before that point, but thanks to Haskell's purity and strong static type system he was able to pull off the entire refactoring single-handedly in a just a couple hours. And once he got it compiling, the application worked the first time. We are both convinced that this feat would have been impossible without strong static types.

Conclusion

I think there are a couple of interesting points worth thinking about here. First of all, the API chosen by Ember only hid the complexity, it didn't reduce it. What seemed to be a simple get() method was actually a more complex system with some special cases. The system was more complex than the API indicated. It's useful to think about the true complexity of a problem compared to the complexity of the exposed API.

The second point is that having the ability to make categorical statements about API behavior is very important. We use this kind of reasoning all the time, and the more of it we can do, the fewer the number of assumptions we will have to question when something isn't behaving as we expect. In this case, I made the seemingly reasonable categorical assumption that unused class definitions would have no effect on my program. But for some reason that I still don't understand, it was violated. I also made the categorical assumption that Ember's get() and set() methods worked like they would work in a map. But that assumption didn't hold up either. I encounter assumptions that don't hold up all the time. Every programmer does. But rarely are they so deeply and universally held as these.

So what can we learn from this? In The Fifth Discipline, Senge goes on to talk about the importance of thinking with a systems perspective; about how we need to stop blaming people and focus more on the systems involved. I think it's telling how in my 5 or 6 years of Haskell programming I've never seen a bug as crazy as these two that I encountered after working only a few months on a significant Javascript project. Haskell with it's purity and strong static type system allows me to make significantly more confident categorical statements about what my code can and cannot do. That allows me to more easily build better abstractions that actually reduce complexity for the end user instead of just hiding it away in a less frequented area.

This is the week of the SolidWorks World 2014 in San Diego, Ca, and so far there have been quite a few pretty groundbreaking announcements from the convention. Last night we got the announcement from Stratasys, pertaining to  their multi-material, multi-color printers, and this afternoon we got to take a look at a printer which is the first ever carbon fiber extruding 3D printer on the market.

It’s called the “Mark One,” and is manufactured by MarkForged. Gregory Mark, the President of MarkFoged, also co-owns Aeromation, which is another tech company responsible for manufacturing computer controlled race car wings. The wings are typically made out of carbon fiber because of its lack of weight, and durability. Mark found that it is quite a daunting task to manufacture parts out of carbon fiber because of the time and expense in laying the fiber down, piece by piece, in the production process. That’s what sparked the initial drive for him to pursue a 3D printer which could simply print the material.

The Mark One printer can print in four different materials, one at a time, which includes carbon fiber, fiberglass, PLA, and nylon.

Here are the specs of the Mark One printer, announced this afternoon:

Printing Technology: Fused Filament Fabrication (FFF) / Composite Filament Fabrication (CFF)
Build Size: 305mm x 160mm x 160mm (12″ x 6.25″ x 6.25″, 486ci)
Material Compatibility: Carbon Fiber, Fiberglass, Nylon, PLA
Highest Layer Resolution: FFF Printing: 100 Microns / CFF Printing: 200 Microns
Extruders: Dual Quick Change
Filament Sizes: FFF: 1.75mm, CFF: MF4
Pause / Resume Prints: Yes
Software: Cloud Enabled
Supported OS: Mac OS 10.7 Lion +, Win XP+, Linux*
Supported Browser: Chrome 30+, Firefox 10+, Safari 6+*
Supported Files: STL, .OBJ
Connectivity: WiFi, USB, SD Card 

The printer itself is very simple and elegant. Produced mainly for manufacturers, the price tag of $5000 may put it within range of some do-it-yourselfers. There could be hundreds or thousands of applications for this new technology, especially within the prosthetic industry, as carbon fiber is the perfect material for prosthesis.  Pre-orders begin next month, and the first shipments will go our before the second half of this year.

Take part in the discussion around the Mark One Printer, at 3DPrintBoard.com here:  http://3dprintboard.com/showthread.php?1551-Mark-One-Carbon-Fiber-3D-Printer

Share and Enjoy

Couple weeks ago we published a short article about cookies vs tokens in the context of single page applications, in particular applied to AngularJs apps. It seems the community is interested in this topic, so we published a second article on token based authentication in realtime frameworks like socket.io. There is a great interest in this subject so we decided to continue with an article that explores in more detail some of the most common questions around token-based authentication. So here we go...

1. Tokens need to be stored somewhere (local/session storage or cookies)

In the context of tokens being used on single page applications, some people have brought up the issue about refreshing the browser, and what happens with the token. The answer is simple: you have to store the token somewhere: in session storage, local storage or a client side cookie. Most session storage polyfills fallback to cookies when the browser doesn't support it.

If you are wondering "but if I store the token in the cookie I'm back to square one". Not really, in this case you are using cookies as a storage mechanism, not as an authentication mechanism (i.e. the cookie won't be used by the web framework to authenticate a user, hence no XSRF attack)

2. Tokens can expire like cookies, but you have more control

Tokens have an expiration (in JSON Web Tokens is represented by exp property), otherwise someone could authenticate forever to the API once they logged in at least once. Cookies also have expiration for the same reasons.

In the world of cookies, there are different options to control the lifetime of the cookie:

In the tokens world, once the token expires, you simply want to get a new one. You could implement an endpoint to refresh a token that will:

You can even store in the token the original issue date, and enforce a re-login after two weeks or so.

app.post('/refresh_token', function (req, res) {
 // verify the existing token
 var profile = jwt.verify(req.body.token, secret);
 // if more than 14 days old, force login
 if (profile.original_iat - new Date() > 14) { // iat == issued at
 return res.send(401); // re-logging
 }
 // check if the user still exists or if authorization hasn't been revoked
 if (!valid) return res.send(401); // re-logging
 // issue a new token
 var refreshed_token = jwt.sign(profile, secret, { expiresInMinutes: 60*5 });
 res.json({ token: refreshed_token });
});

If you need revocation of tokens (useful if tokens are long-lived) you will need to have some sort of registry of issued tokens to check against.

3. Local/session storage won't work across domains, use a marker cookie

If you set a cookie's domain to .yourdomain.com it can be accessed from yourdomain.com and app.yourdomain.com, making it easier to detect from the main domain (where you probably have, let's say, your marketing site) that the user is already logged in and redirect her to app.yourdomain.com.

Tokens stored in local/session storage, on the other hand, can't be accessed from different domains (even if these are subdomains). So what can you do?

One possible option is, when the user authenticates on app.yourdomain.com and you generate a token you can also set a cookie set to .yourdomain.com

$.post('/authenticate, function() {
 // store token on local/session storage or cookie
 ....
 // create a cookie signaling that user is logged in
 $.cookie('loggedin', profile.name, '.yourdomain.com');
});

Then, in youromdain.com you can check the existance of that cookie and redirect to app.yourdomain.com if the cookie exists. The token will be available on app subdomain, and from there on, the usual flow applies (if the token is still valid use it, if not get a new one unless last login was more than the threshold you set up).

It could happen that the cookie exists but the token was deleted or something else happened. In that case, the user would have to login again. But what's important to highlight here is, as we said before, we are not using the cookie as an authentication mechanism, just as a storage mechanism that happens to support storing information across different domains.

4. Preflight requests will be sent on each CORS request

Someone pointed out that the Authorization header is not a simple header, hence a pre-flight request would be required for all the requests to a particular URLs.

OPTIONS https://api.foo.com/bar
GET https://api.foo.com/bar
 Authorization: Bearer ....
OPTIONS https://api.foo.com/bar2
GET https://api.foo.com/bar2
 Authorization: Bearer ....
GET https://api.foo.com/bar
 Authorization: Bearer ....

But this happens if you are sending Content-Type: application/json for instance. So this is already happening for most applications.

One small caveat, the OPTIONS request won't have the Authorization header itself, so your web framework should support treating OPTIONS and the subsequent requests differently (Note: Microsoft IIS for some reason seems to have issues with this).

5. When you need to stream something, use the token to get a signed request

When using cookies, you can trigger a file download and stream the contents easily. However, in the tokens world, where the request is done via XHR, you can't rely on that. The way you solve this is by generating a signed request like AWS does, for example. Hawk Bewits is a nice framework to enable this:

Request:

POST /download-file/123
Authorization: Bearer...

Response:

ticket=lahdoiasdhoiwdowijaksjdoaisdjoasidja

This ticket is stateless and it is built based on the URL: host + query + headers + timestamp + HMAC, and has an expiration. So it can be used in the next, say 5 minutes, to download the file.

You would then redirect to /download-file/123?ticket=lahdoiasdhoiwdowijaksjdoaisdjoasidja. The server will check that the ticket is valid and continue with business as usual.

6. It's easier to deal with XSS than XSRF

Cookies have this feature that allows setting an HttpOnly flag from server side so they can only be accessed on the server and not from JavaScript. This is useful because it protects the content of that cookie to be accessed by injected client-side code (XSS).

Since tokens are stored in local/session storage or a client side cookie, they are open to an XSS attack getting the attacker access to the token. This is a valid concern, and for that reason you should keep your tokens expiration low.

But if you think about the attack surface on cookies, one of the main ones is XSRF. The reality is that XSRF is one of the most misunderstood attacks, and the average developer, might not even understand the risk, so lots of applications lack anti-XSRF mechanism. However, everybody understands what injection is. Put simply, if you allow input on your website and then render that without escaping it, you are open to XSS. So based on our experience, it is easier to protect against XSS than protecting against XSRF. Adding to that, anti-XSRF is not built-in on every web framework. XSS on the other hand is easy to prevent by using the escape syntax available by default on most template engines.

7. The token gets sent on every request, watch out its size

Every time you make an API request you have to send the token in the Authorization header.

GET /foo
Authorization: Bearer ...2kb token...

vs.

GET /foo
connect.sid: ...20 bytes cookie...

Depending on how much information you store in that token, it could get big. On the other hand, session cookies usually are just an identifier (connect.sid, PHPSESSID, etc.) and the rest of the content lives on the server (in memory if you just have one server or a database if you run on a server farm).

Now, nothing prevents you from implementing a similar mechanism with tokens. The token would have the basic information needed and on the server side you would enrich it with more data on every API call. This is exactly the same thing cookies will do, with the difference that you have the additional benefit that this is now a conscious decision, you have full control, and is part of your code.

GET /foo
Authorization: Bearer ……500 bytes token….

Then on the server:

app.use('/api', 
 // validate token first
 expressJwt({secret: secret}), 
 // enrich req.user with more data from db
 function(req, res, next) {
 req.user.extra_data = get_from_db();
 next();
 });

It is worth mentioning that you could also have the session stored completely on the cookie (instead of being just an identifier). Some web platforms support that, others not. For instance, in node.js you can use mozilla/node-client-sessions.

8. If you store confidential info, encrypt the token

The signature on the token prevents tampering with it. TLS/SSL prevents man in the middle attacks. But if the payload contains sensitive information about the user (e.g. SSN, whatever), you can also encrypt them. The JWT spec points to the JWE spec but most of the libraries don't implement JWE yet, so the simplest thing is to just encrypt with AES-CBC as shown below.

app.post('/authenticate', function (req, res) {
 // validate user
 // encrypt profile
 var encrypted = { token: encryptAesSha256('shhhh', JSON.stringify(profile)) };
 // sing the token
 var token = jwt.sign(encrypted, secret, { expiresInMinutes: 60*5 });
 res.json({ token: token });
}
function encryptAesSha256 (password, textToEncrypt) {
 var cipher = crypto.createCipher('aes-256-cbc', password);
 var crypted = cipher.update(textToEncrypt, 'utf8', 'hex');
 crypted += cipher.final('hex');
 return crypted;
}

Of course you can use the approach on #7 and keep confidential info in a database.

UPDATE: Pedro Felix correctly pointed out that MAC-then-encrypt is vulnerable to Vaudenay-style attacks. I updated the code to do encrypt-then-MAC.

9. JSON Web Tokens can be used in OAuth

Tokens are usually associated with OAuth. OAuth 2 is an authorization protocol that solves identity delegation. The user is prompted for consent to access his/her data and the authorization server gives back an access_token that can be used to call the APIs acting as that user.

Typically these tokens are opaque. They are called bearer tokens and are random strings that will be stored in some kind of hash-table storage on the server (db, cache, etc.) together with an expiration, the scope requested (e.g. access to friend list) and the user who gave consent. Later, when the API is called, this token is sent and the server lookup on the hash-table, rehydrating the context to make the authorization decision (did it expire? does this token has the right scope associated for the API that wants to be accessed?).

The main difference between these tokens and the ones we've been discussing is that signed tokens (e.g.: JWT) are "stateless". They don't need to be stored on a hash-table, hence it's a more lightweight approach. OAuth2 does not dictate the format of the access_token so you could return a JWT from the authorization server containing the scope/permissions and the expiration.

10. Tokens are not silver bullets, think about your authorization use cases carefully

Couple of years ago we helped a big company implement a token-based architecture. This was a 100.000+ employees company with tons of information to protect. They wanted to have a centralized organization-wide store for "authentication & authorization". Think about "user X can read field id and name of clincial trial Y for hospital Z on country W" use cases. This fine-grained authorization, as you can imagine, can get unmanageable pretty quickly, not only technically but also administratively.

• Tokens can get really big
• Your apps/APIs gets more complicated
• Whoever grant these permissions will have a hard time managing all this.

We ended up working more on the information architecture side of things to make sure the right scopes and permissions were created. Conclusion: resist the temptation of putting everything into tokens and do some analysis and sizing before going all the way with this approach.

Disclaimer: when dealing with security, make sure you do the proper due dilligence. Any code/recommendation that you get here is provided as-is.

Please leave a comment or discuss on HN.

Happy tokenizing!

Photo taken from: http://alfanatic.webs.com/

Now when you search Google, a nice number of search filters have been removed, including the discussions filter. The number of filters or vertical options have been drastically reduced. Here is an example.

But if you look at an article from six months ago, the options for that search were much more vast.

Now we have web, images, shopping, maps, news, videos, books, flights and applications. We are now missing places (which is maps I guess), blogs, discussions, recipes, and patents.

All Google Testing also notes that the order of the filter options have changes based on your query. Interesting.

For some, the removal of these additional filters is very upsetting. A Google Web Search Help thread has one searcher who said "very annoying as I rely on this function a lot."

Forum discussion at Google Web Search Help.

Update: Here is a response from a Google spokesperson:

The Ukrainian parliament has voted overwhelmingly to annul controversial anti-protest legislation.

The decision comes less than two weeks after the measures were introduced.

The law, which banned the wearing of helmets by protesters and the blockading of public buildings, had helped fuel continuing anti-government demonstrations.

In another move to appease the protesters, Ukraine's Prime Minister Mykola Azarov has offered to quit.

In a statement, he said the offer of his resignation to President Viktor Yanukovych was intended to create "social and political compromise".

Parliament - holding an emergency debate on the crisis - voted by 361 to 2 to repeal the protest law.

President Yanukovych had already agreed to scrap the legislation in a concession to the opposition.

But up until the vote, it was unclear how MPs from his governing Party of the Regions would cast their ballots as they were allowed a "free vote" - to vote as they saw fit.

Aside from the ban on helmets and blockades, the legislation had outlawed unauthorised tents in public areas and the slandering of government officials.

MPs applauded as the result was announced.

There was a similar response in Kiev's Independence Square - the focus of the demonstrations.

A BBC correspondent who went to the square described it as relatively quiet with no sign of the recent violence which has affected parts of central Kiev.

Activists were chatting with police and listening to the progress of the debate in parliament.

The protests have spread in recent days across Ukraine, even to President Yanukovych's stronghold in the east. Official buildings in several cities have been occupied.

The interior ministry says protesters stabbed and wounded three policemen in the southern city of Kherson, one of whom later died.

At least five people have been killed in violence linked to the protests.

In his resignation statement, Prime Minister Azarov said: "To create additional opportunities for social and political compromise and for a peaceful solution to the conflict, I made a personal decision to ask the president of Ukraine to accept my resignation as prime minister of Ukraine."

The government had "done everything to ensure the peaceful resolution of the conflict" and would do "everything possible to prevent bloodshed, an escalation of violence, and violation of citizen's rights", he said.

If the president signs the decree for the resignation, then the whole cabinet resigns, says the BBC's David Stern in Kiev. But they can remain in their posts for 60 days until a new government is formed.

President Yanukovych had already offered Mr Azarov's job to the opposition at the weekend, proposing that Fatherland leader Arseniy Yatsenyuk take the post. Mr Yatsenyuk declined the offer.

Parliament adjourned after the vote on the protest law and is due to discuss the issue of granting an amnesty to convicted protesters on Tuesday afternoon.

Mr Yanukovych offered an amnesty only if protesters cleared barricades and stopped attacking government buildings.

The president made the concessions during talks on Monday with Mr Yatsenyuk, Udar (Punch) chief Vitali Klitschko, and nationalist leader Oleg Tyahnybok.

Meanwhile, top EU diplomat Catherine Ashton has brought forward a planned visit to Ukraine by 48 hours and will now arrive on Tuesday for meetings with Mr Yanukovych and opposition leaders.

She said she was "alarmed" by reports on Monday that the government was preparing to introduce a state of emergency. Officials have denied any such plan.

Ms Ashton arrives from Brussels where she, with other senior EU leaders, will have met Russian President Vladimir Putin at an EU-Russia summit.

Differences over Ukraine were expected to be high on the agenda.

The crisis in Ukraine was sparked when Mr Yanukovych pulled out of a planned trade deal with the EU last November in favour of a $15bn (£9bn) bailout from Russia.

Send your pictures and videos to yourpics@bbc.co.uk or text them to 61124 (UK) or +44 7624 800 100 (International). If you have a large file you canupload here.

Read the terms and conditions