As part of the Django 1.5 release process, today we've released Django 1.5 release candidate 1, a preview/testing package for Django 1.5. As with all pre-release packages, this is not for production use, but if you'd like to try out some of the new goodies coming in 1.5, or if you'd like to pitch in and help us fix bugs before the final 1.5 release, feel free to grab a copy and give it a spin.

In particular, we ask users of this release candidate to watch for and report any major bugs they encounter; if no release-blocking bugs are reported, Django 1.5 will be released in approximately one week.

You can get a copy of the 1.5 release candidate from our downloads page, and we recommend you read the release notes. Also, for the security conscious, signed MD5 and SHA1 checksums of the 1.5 release candidate are available.

Also, please note that Django 1.5 now requires a minimum of Python 2.6; Python 2.5 is no longer supported. Python 3.x releases, starting with Python 3.2, are experimentally supported in this release. For more information on Python 3 support, and the testing Django 1.5 will still need, see this blog post.

Posted by James Bennett on January 4, 2013

Disclaimer: This post is Meteor & Backbone beef. Both Meteor and Backbone are absolute genius, and far beyond anything I could dream to create. But IMO there are better tools. Prepare yourselves *gulp*, I need to get this off my chest.

First, Backbone. Why people? It revolutionized JavaScript, did wonderful things for the world, and served its purpose well. But now we have better tools, so let’s move on. It’s like Gentoo users proselytizing Gentoo to the masses, perpetuating it as most common Distro; where all this time, Ubuntu would have saved everyone countless hours. Not everyone wants to build everything from scratch - use Angular (or Ember, Knockout, etc).

Ok, Meteor v Derby. Meteor doesn’t use NPM, and has no REST support. Am I wrong? That’s huge! That means you can’t have an API (read mobile app) unless you write a second server accessing that MongoDB store. Derby is the exact same thing as Meteor, plus REST and NPM. So why do people use Meteor? For legitimate reasons: it has a better public face: more PR& press, tutorials, and resources. Derby has no press, behind-the-curve documentation & major version releases, and always-down live examples (see HabitRPG, PhishVids, and Bibliaovaso for functioning production examples). However, the codebase is impeccable and well-maintained, the architecture a work of genius, and its future bright. But to use Derby you have to use HEAD, and survive the sparse docs. Learn it and you’ll go further than you’d have gotten with Meteor. Derby should get its act together, and Meteor people should migrate.

Angular is awesome. That is all. Derby gets you further for the web app because you get real-time + data persistence automatically. However, as Angular is a client MVC, you can detach it from the server if you want to recycle your code on PhoneGap or App.js. So here is my recommendation:

{IF} The web app is the most important part, and you want it fast. The mobile app will come later only after the web app proves itself.

• First Derby
• Then Angular + PhoneGap, exposing your backend via REST (Derby static routes, Express)

{ELSE} You want mobile & web at roughly the same time, they’re equally important

• Angular for both Web & PhoneGap, + Server
• Server = Express (+ Socket.io / Sock.js for real-time), or even Parse / Firebase

FWIW, Meteor and Backbone rock - I’d be promoting the hell out of them if Derby & Angular didn’t exist. But they do, so I </rant>.

Note: original post here, but my Drupal server has been miserable lately and I'm currently migrating. Go there to post comments.

WRECK- IT RALPH - Walt Disney Animation Studios announces the debut of the hit arcade-game-hopping adventure "Wreck-It Ralph" marking a Disney first with the early release of the HD Digital and HD Digital 3D on February 12, 2013, followed by the 4-Disc Blu-ray Combo Pack, 2-Disc Blu-ray Combo Pack, DVD, SD Digital and On-Demand release on March 5, 2013.

From Walt Disney Animation Studios, "Wreck-It Ralph" takes viewers on a hilarious journey. For decades, Ralph (voice of John C. Reilly) has played the bad guy in his popular video game. In a bold move, he embarks on an action-packed adventure and sets out to prove to everyone that he is a true hero with a big heart. As he explores exciting new worlds, he teams up with some unlikely new friends including feisty misfit Vanellope von Schweetz (voice of Sarah Silverman). The film is directed by Emmy®-winner Rich Moore.

Featuring an all-star voice cast including Jack McBrayer as the voice of Fix It Felix, Jr. and Jane Lynch as the voice of Sgt. Calhoun, plus breakthrough bonus features that take viewers even deeper into the world of video games, Disney's "Wreck-It Ralph" has something for every player. Over an hour of all-new bonus material is featured on the Digital and Blu-ray Combo Pack, including deleted and alternate scenes, the theatrical short "Paperman," plus much more.

The home entertainment debut of "Wreck-It Ralph" will be available in multiple ways, containing exciting all-new bonus features that extend the fun-filled movie experience.

Bonus Materials Overview for These Products:
HD Digital
SD Digital
4-Disc Blu-ray Combo Pack (Blu-ray 3D + Blu-ray +DVD + Digital Copy)
2-Disc Blu-ray Combo Pack (Blu-ray +DVD)

Includes:
* Bit by Bit: Creating the Worlds of "Wreck-It Ralph" – Fans of the film will get a look at five new worlds created for "Wreck-It Ralph." The short takes viewers into Game Central Station with the artists who brought Sugar Rush, Hero's Duty and Fix It Felix Jr. to life.

* Alternate & Deleted Scenes – Four separate scenes are highlighted with an introduction and optional audio commentary from director Rich Moore.

* Video Game Commercials – Viewers can check out the commercials created for the video games featured in the film - Fix It Felix Jr., Sugar Rush, Hero's Duty and Fix It Felix Hammer.

* "Paperman" – This animated short film played in theaters before "Wreck It Ralph." It tells the story of a young man in an office who sees the girl of his dreams in a skyscraper window across the street. But how can he get her attention?

Blu-ray Exclusive Bonus Materials

Includes:
* Disney Intermission: The Gamer's Guide to "Wreck-It-Ralph" – When the film is paused, host Chris Hardwick appears on screen to guide viewers through a series of 10 video segments offering an inside look at the many video game references, Disney references and other hidden surprises featured in the film.

Suggested Retail Prices:

4-Disc Blu-ray Combo Pack = (Blu-ray 3D + Blu-ray +DVD + Digital Copy)
$49.99 US and $56.99 Canada

2-Disc Blu-ray Combo Pack = (Blu-ray +DVD)
$39.99 US and $46.99 Canada

DVD = $29.99 US and $35.99 Canada

Digital and On-Demand = Consumers should check with their television provider or preferred digital retailer for pricing and additional information

The Pacific Northwest may be the epicenter of U.S. coffee culture, and now a new study shows the region's elevated caffeine levels don't stop at the shoreline.

The discovery of caffeine pollution in the Pacific Ocean off Oregon is further evidence that contaminants in human waste are entering natural water systems, with unknown consequences for wildlife and humans alike, experts say.

(Read National Geographic magazine's "Caffeine: What's the Buzz?")

Scientists sampled both "potentially polluted" sites—near sewage-treatment plants, larger communities, and river mouths—and more remote waters, for example near a state park.

Surprisingly, caffeine levels off the potentially polluted areas were below the detectable limit, about 9 nanograms per liter. The wilder coastlines were comparatively highly caffeinated, at about 45 nanograms per liter.

"Our hypothesis from these results is that the bigger source of contamination here is probably on-site waste disposal systems like septic systems," said study co-author Elise Granek.

The difference may be due to more stringent monitoring in more developed areas.

"Wastewater-treatment plants, for the most part, have to do regular monitoring to ensure they are within certain limits," added Granek, a Portland State University marine ecologist. Granek noted, though, that caffeine is unregulated, and so is not specifically monitored.

By contrast, for on-site waste-disposal systems, "there is frequently not much monitoring going on."

The big sewage plants may also be at an advantage because Oregon cities are relatively small. The plants don't have to process the sheer volume of waste associated with a major city such as Boston, which one study has found to be pumping fairly high levels of caffeine into its harbor.

(Related: "Cocaine, Spices, Hormones Found in Drinking Water.")

"Contaminant Soup" Has Unknown Impacts

Hydrologist Dana Kolpin welcomed the new research, saying caffeine concentrations in water have been documented before but more often in freshwater than marine environments.

"Caffeine is pretty darn ubiquitous, and there is growing evidence that this and other understudied contaminants are out there,"  said Kolpin, of the USGS's Toxic Substances Hydrology Program in Iowa City, Iowa.

In our waste "there is a whole universe of potential contaminants including pharmaceuticals, hormones, personal-care products like detergents or fragrances, even artificial sweeteners."

Caffeine is something of a canary in a coal mine for elevated levels of human contaminants in water, said Kolpin, who wasn't part of the new study.

In other words, if caffeine's in the water, chances are there are other contaminants too.

"What does this mean?" he asked. "Aquatic organisms are getting hit with a soup of low-level contaminants.

"Are there environmental or human-health consequences from exposure to these compounds or different mixtures of compounds? Obviously that's the million-dollar question."

(Infographic: How Coffee Changed America.)

Caffeine and Cellular Stress in Animals

Caffeine has been documented in waters around the world, including Boston Harbor, Puget Sound, the Mediterranean, and the North Sea. It might persist for up to 30 days in marine waters, study co-author Granek noted.

But the stimulant's impact on natural ecosystems is unknown. Nonlethal effects may be invisible but could have repercussions up and down the food chain and from generation to generation.

Granek and colleagues have shown in lab experiments that caffeine at the levels found offshore does affect intertidal mussels, causing them to produce specialized proteins in response to environmental stress.

The levels found in the remote study areas, for example, "did cause these mussels to exhibit cellular stress," she said. "If we expose them to higher concentrations or longer terms, do we see changes in growth rates, or changes in reproductive output?" The team hopes to find out with future experiments.

Kolpin said some studies of other contaminants have shown more drastic effects, including one at a remote Ontario Lake, which concluded that estrogen from birth control pills can cause wild fish populations to collapse.

"With caffeine, we're not yet sure about its environmental effects," he said. "But it's a very nice tracer, even if it doesn't have a large effect, because in most parts of the world, you know that this is coming from a human waste source."

The Pacific Northwest caffeine research was published in the July 2012 edition of the Marine Pollution Bulletin.

I used to drink more caffeine than you do.

Note the bared teeth, the delighted eyes, and the paw wrapped protectively wrapped around that frail cup.

That is almost certainly true.  From my college days (1985ff) through the end of 2011, my caffeine consumption was extravagant.  Epic.  Other people speak of the number of cups of coffee they drink per day; they are pikers.  For me, the number was pots.  Two, three, or four pots between grimly waking up and falling asleep.  From the first thing in the morning through multiple after-dinner imbibings, the blessed black bean brew was my brain’s constant companion.

Ah, Jolt.

Along with coffee, soda was my parallel caffeine delivery system.  I still recall the glorious days of Jolt Cola (more sugar, and twice the caffeine!), two-liters of which saw me through my sophomore and junior years.  Coke was too basic for me, but doable when nothing else was available.  Mello Yellow was fine, but hard to obtain.  Although it had a splendidly lying name: lots of caffeine, so nothing mellow; green, not yellow color.

Mountain Dew was my drink of choice, sweet and fiercely caffeinated.  One year my housemates and I purchased enough Mountain Dew cans in bulk to make a six foot tall stack.  It nearly replaced water for us. I was quaffing a can with breakfast, bottles during the day, cups in the evening, plus a final can in bed, just to relax.

Gunpowder tea, preferably.

Other caffeine mechanisms also supplied my needs.  Chocolate, especially chocolate-covered espresso beans, helped.  Black tea sometimes sufficed when I was among Brits, or just wanted the taste.  Hot chocolate was fine in winter.  But Turkish coffee, ah, that was the sublime caffeine delivery system.  I fell in love with the potent stuff in Bosnia during the 1990s war, and sought it out ever afterwards.  I visited an academic in Mostar whose house had taken a hit from a shell or missile.  In its ruins, on a half-shattered gas-powered stove, the prof and his wife brewed Turkish coffee every day.  I recognized my fellows, members of the worldwide society of caffeine devotees.  That concentrated bolt of coffee was like neutronium, or anti-Kryptonite for Superman, an outrageously heavy distillate for my gleeful brain.

The ur-coffee.

I could also combine caffeination systems.  During a long drive I’d load up with Mountain Dew and a giant cup of coffee.  After a couple of hours I’d stop to replenish those sources, buy some Water Joe, then add a couple of doses of Stok to the steaming coffee.  (At home my wife forbid me from brewing coffee with Water Joe, lest my chest simply explode)

Once a chemist friend gave me a small container of pure caffeine.  She warned me not to just snarf the white power straight down, so I took to dabbing a finger in it.  That was peppy.

Too Much Coffee Man, my ideal superhero.

Why did I drink so much caffeine?  it wasn’t simply chemical or behavioral addiction.  My habit began in college as a way of providing enough energy to do both my studies and jobs.  I took heavy courseloads (double or triple majoring), while working.  After graduation that overload of work never went away.  I did my M.A. in a single year, while working at a bookstore.  For my PhD I was teaching nearly the entire time.  As a professor I taught four (4) classes per term, while conducting research, plus doing lots of service (committees, technology, advising, etc), plus consulting on the side.  I also married, and we had two children.  With such long days (and nights), the caffeine was essential.

After a while caffeine no longer provided stimulus.  Instead it became a way of recovering some basic energy level from a pit of exhaustion.  A strong dose stopped making me sparky and manic, as far as I can tell, but powered me up just enough to get things done over the course of a very long, but well fueled, day.  I suppose this is another way of saying “maintenance level”.

I would have continued along this glorious, bean-strewn path until my body failed, and it nearly did.  Throughout 2010 and 2011 I suffered frequent bouts of gut pain.  This wasn’t indigestion, but fiery shocks, enough to wake me up at night or knock me off track during the day.  The pains increased in frequency, duration, and intensity, ultimately coming several times a day, and leading to regular nausea.  Besides being painful and disgusting, these attacks were debilitating.  Ultimately I decided to seek medical advice.  Well, “I decided” really means “I gave into my wife’s patient, well-informed concern”.

On December 22, 2011, we arrived at the family clinic we saw for most medical questions.  I described my symptoms to the doctor, who looked concerned.  He asked me to describe my caffeine intake, and his facial expressions were quite entertaining.  He demonstrated incredulity, dismay, outrage, amusement, followed by iron determination.  When I finished, the doc laid it out for me.

“Either I hospitalize you tomorrow, or you go cold turkey on caffeine.  Immediately.”

To be continued.

(photos by Cogdog, 7 Bits of Truth, Akuppa, Nate Steiner, Wikipedia)

The Bulletin of the Atomic Scientists tracks the nuclear capabilities of every country. We’re going to use their data to demonstrate Haskell’s HLearn library and the usefulness of abstract algebra to statistics. Specifically, we’ll see that the categorical distribution and kernel density estimates have monoid, group, and module algebraic structures.  We’ll explain what this crazy lingo even means, then take advantage of these structures to efficientlyanswer real-world statistical questions about nuclear war. It’ll be a WOPR!

Before we get into the math, we’ll need to review the basics of nuclear politics.

The nuclear Non-Proliferation Treaty (NPT) is the main treaty governing nuclear weapons. Basically, it says that there are five countries that are “allowed” to have nukes: the USA, UK, France, Russia, and China. “Allowed” is in quotes because the treaty specifies that these countries must eventually get rid of their nuclear weapons at some future, unspecified date. When another country, for example Iran, signs the NPT, they are agreeing to not develop nuclear weapons. What they get in exchange is help from the 5 nuclear weapons states in developing their own civilian nuclear power programs. (Iran has the legitimate complaint that Western countries are actively trying to stop its civilian nuclear program when they’re supposed to be helping it, but that’s a whole ‘nother can of worms.)

The Nuclear Notebook tracks the nuclear capabilities of all these countries.  The most-current estimates are from mid-2012.  Here’s a summary (click the warhead type for more info):

I’ve consolidated all this data into the file nukes-list.csv, which we will analyze in this post.  If you want to try out this code for yourself (or the homework question at the end), you’ll need to download it.  Every line in the file corresponds to a single nuclear warhead, not delivery method.  Warheads are the parts that go boom!  Bombers, ICBMs, and SSBN/SLBMs are the delivery method.

There are three things to note about this data.  First, it’s only estimates based on public sources.  In particular, it probably overestimates the Russian nuclear forces. Other estimates are considerably lower.  Second, we will only be considering deployed, strategic warheads.  Basically, this means the “really big nukes that are currently aimed at another country.”  There are thousands more tactical warheads, and warheads in reserve stockpiles waiting to be disassembled.  For simplicity—and because these nukes don’t significantly affect strategic planning—we won’t be considering them here.   Finally, there are 4 countries who are not members of the NPT but have nuclear weapons: Israel, Pakistan, India, and North Korea.  We will be ignoring them here because their inventories are relatively small, and most of their weapons would not be considered strategic.

Programming preliminaries

Now we’re ready to start programming. First, let’s import our libraries:

>import Control.Lens>import Data.Csv>import qualified Data.Vector as V>import qualified Data.ByteString.Lazy.Char8 as BS> >import HLearn.Algebra>import HLearn.Models.Distributions>import HLearn.Gnuplot.Distributions

Next, we load our data using the Cassava package.  (You don’t need to understand how this works.)

>main = do> Right rawdata <- fmap (fmap V.toList . decode True) $ BS.readFile "nukes-list.csv">        :: IO (Either String [(String, String, String, Int)])

And we’ll use the Lens package to parse the CSV file into a series of variables containing just the values we want.  (You also don’t need to understand this.)

> let list_usa = fmap (\row -> row^._4) $ filter (\row -> (row^._1)=="USA" ) rawdata> let list_uk = fmap (\row -> row^._4) $ filter (\row -> (row^._1)=="UK" ) rawdata > let list_france = fmap (\row -> row^._4) $ filter (\row -> (row^._1)=="France") rawdata > let list_russia = fmap (\row -> row^._4) $ filter (\row -> (row^._1)=="Russia") rawdata > let list_china = fmap (\row -> row^._4) $ filter (\row -> (row^._1)=="China" ) rawdata

NOTE: All you need to understand about the above code is what these list_country variables look like. So let’s print one:

> putStrLn $ "List of American nuclear weapon sizes = " ++ show list_usa

gives us the output:

List of American nuclear weapon sizes = fromList [335,335,335,335,335,335,335,335,335,335 ... 1200,1200,1200,1200,1200]

If we want to know how many weapons are in the American arsenal, we can take the length of the list:

> putStrLn $ "Number of American weapons = " ++ show (length list_usa)

We get that there are 1951 American deployed, strategic nuclear weapons.  If we want to know the total “blowing up” power, we take the sum of the list:

> putStrLn $ "Explosive power of American weapons = " ++ show (sum list_usa)

We get that the US has  516 megatons of deployed, strategic nuclear weapons.  That’s the equivalent of 1,033,870,000,000 pounds of TNT.

To get the total number of weapons in the world, we concatenate every country’s list of weapons and find the length:

> let list_all = list_usa ++ list_uk ++ list_france ++ list_russia ++ list_china> putStrLn $ "Number of nukes in the whole world = " ++ show (length list_all)

Doing this for every country gives us the table:

Now let’s do some algebra!

Monoids and groups

In a previous post, we saw that the Gaussian distribution forms a group. This means that it has all the properties of a monoid—an empty element (mempty) that represents the distribution trained on no data, and a binary operation (mappend) that merges two distributions together—plus an inverse. This inverse lets us “subtract” two Gaussians from each other.

It turns out that many other distributions also have this group property. For example, the categorical distribution.  This distribution is used for measuring discrete data. Essentially, it assigns some probability to each “label.”  In our case, the labels are the size of the nuclear weapon, and the probability is the chance that a randomly chosen nuke will be exactly that destructive.  We train our categorical distribution using the train function:

> let cat_usa = train list_usa :: Categorical Int Double

If we plot this distribution, we’ll get a graph that looks something like:

A distribution like this is useful to war planners from other countries.  It can help them statistically determine the amount of casualties their infrastructure will take from a nuclear exchange.

Now, let’s train equivalent distributions for our other countries.

> let cat_uk = train list_uk :: Categorical Int Double> let cat_france = train list_france :: Categorical Int Double> let cat_russia = train list_russia :: Categorical Int Double> let cat_china = train list_china :: Categorical Int Double

Because training the categorical distribution is a group homomorphism, we can train a distribution over all nukes by either training directly on the data:

> let cat_allA = train list_all :: Categorical Int Double

or we can merge the already generated categorical distributions:

> let cat_allB = cat_usa <> cat_uk <> cat_france <> cat_russia <> cat_china

Because of the homomorphism property, we will get the same result both ways. Since we’ve already done the calculations for each of the the countries already, method B will be more efficient—it won’t have to repeat work we’ve already done.  If we plot either of these distributions, we get:

The thing to notice in this plot is that most countries have a nuclear arsenal that is distributed similarly to the United States—except for China.  These Chinese ICBMs will become much more important when we discuss nuclear strategy in the last section.

But nuclear war planners don’t particularly care about this complete list of nuclear weapons.  What war planners care about is the survivable nuclear weapons—that is, weapons that won’t be blown up by a surprise nuclear attack.  Our distributions above contain nukes dropped from bombers, but these are not survivable.  They are easy to destroy.  For our purposes, we’ll call anything that’s not a bomber a survivable weapon.

We’ll use the group property of the categorical distribution to calculate the survivable weapons.  First, we create a distribution of just the unsurvivable bombers:

> let list_bomber = fmap (\row -> row^._4) $ filter (\row -> (row^._2)=="Bomber") rawdata> let cat_bomber = train list_bomber :: Categorical Int Double

Then, we use our group inverse to subtract these unsurvivable weapons away:

> let cat_survivable = cat_allB <> (inverse cat_bomber)

Notice that we calculated this distribution indirectly—there was no possible way to combine our variables above to generate this value without using the inverse! This is the power of groups in statistics.

More distributions

The categorical distribution is not sufficient to accurately describe the distribution of nuclear weapons. This is because we don’t actually know the yield of a given warhead. Like all things, it has some manufacturing tolerances that we must consider. For example, if we detonate a 300 kt warhead, the actual explosion might be 275 kt, 350 kt, or the bomb might even “fizzle out” and have almost a 0kt explosion.

We’ll model this by using a kernel density estimator (KDE).  The KDE basically takes all our data points, assigns each one a probability distribution called a “kernel,” then sums these kernels together.  It is a very powerful and general technique for modelling distributions… and it also happens to form a group!

First, let’s create the parameters for our KDE.  The bandwidth controls how wide each of the kernels is.  Bigger means wider.  I selected 20 because it made a reasonable looking density function.  The sample points are exactly what they sounds like: they are where we will sample the density from.  We can generate them using the function genSamplePoints.  Finally, the kernel is the shape of the distributions we will be summing up.  There are many supported kernels.

> let kdeparams = KDEParams> { bandwidth = Constant 20> , samplePoints = genSamplePoints> 0 -- minimum> 4000 -- maximum> 4000 -- number of samples> , kernel = KernelBox Gaussian> } :: KDEParams Double

Now, we’ll train kernel density estimates on our data.  Notice that because the KDE takes parameters, we must use the train’ function instead of just train.

> let kde_usa = train' kdeparams list_usa :: KDE Double

Again, plotting just the American weapons gives:

And we train the corresponding distributions for the other countries.

> let kde_uk = train' kdeparams list_uk :: KDE Double> let kde_france = train' kdeparams list_france :: KDE Double> let kde_russia = train' kdeparams list_russia :: KDE Double> let kde_china = train' kdeparams list_china :: KDE Double>> let kde_all = kde_usa <> kde_uk <> kde_france <> kde_russia <> kde_china

The KDE is a powerful technique, but the draw back is that it is computationally expensive—especially when a large number of sample points are used. Fortunately, all computations in the HLearn library are easily parallelizable by applying the higher order function parallel.

We can calculate the full KDE from scratch in parallel like this:

> let list_double_all = map fromIntegral list_all :: [Double]> let kde_all_parA = (parallel (train' kdeparams)) list_double_all :: KDE Double

or we can perform a parallel reduction on the KDEs for each country like this:

> let kde_all_parB = (parallel reduce) [kde_usa, kde_uk, kde_france, kde_russia, kde_china]

And because the KDE is a homomorphism, we get the same exact thing either way.  Let’s plot the parallel version:

> plotDistribution (genPlotParams "kde_all" kde_all_parA) kde_all_parA

The parallel computation takes about 16 seconds on my Core2 Duo laptop running on 2 processors, whereas the serial computation takes about 28 seconds.

This is a considerable speedup, but we can still do better. It turns out that there is a homomorphism from the Categorical distribution to the KDE:

> let kde_fromcat_all = cat_allB $> kdeparams> plotDistribution (genPlotParams "kde_fromcat_all" kde_fromcat_all) kde_fromcat_all

(For more information about the morphism chaining operator $>, see the Hlearn documentation.) This computation takes less than a second and gets the exact same result as the much more expensive computations above.

We can express this relationship with a commutative diagram:

No matter which path we take to get to a KDE, we will get the exact same answer.  So we should always take the path that will be least computationally expensive for the data set we’re working on.

Why does this work? Well, the categorical distribution is a structure called the “free module” in disguise.

Modules and the Free Module

R-Modules (like groups, but unlike monoids) have not seen much love from functional programmers. This is a shame, because they’re quite handy. It turns out they will increase our performance dramatically in this case.

It’s not super important to know the formal definition of an R-module, but here it is anyways: An R-module is a group with an additional property: it can be “multiplied” by any element of the ring R. This is a generalization of vector spaces because R need only be a ring instead of a field. (Rings do not necessarily have multiplicative inverses.)  It’s probably easier to see what this means by an example.

Vectors are modules.  Let’s say I have a vector:

> let vec = [1,2,3,4,5] :: [Int]

I can perform scalar multiplication on that vector like this:

> let vec2 = 3 .* vec

which as you might expect results in:

[3,6,9,12,15]

Our next example is the free R-module. A “free” structure is one that obeys only the axioms of the structure and nothing else. Functional programmers are very familiar with the free monoid—it’s the list data type. The free Z-module is like a beefed up list. Instead of just storing the elements in a list, it also stores the number of times that element occurred.  (Z is shorthand for the set of integers, which form a ring but not a field.) This lets us greatly reduce the memory required to store a repetitive data set.

In HLearn, we represent the free module over a ring r with the data type:

:: FreeMod r a

where a is the type of elements to be stored in the free module. We can convert our lists into free modules using the function list2module like this:

> let module_usa = list2module list_usa

But what does the free module actually look like? Let’s print it to find out:

> print module_usa

gives us:

FreeMod (fromList [(100,768),(150,200),(300,250),(335,249),(340,50),(455,384),(1200,50)])

This is much more compact! So this is the take away: The free module makes repetitive data sets easier to work with. Now, let’s convert all our country data into module form:

> let module_uk = list2module list_uk> let module_france = list2module list_france> let module_russia = list2module list_russia> let module_china = list2module list_china

Because modules are also groups, we can combine them like so:

> let module_allA = module_usa <> module_uk <> module_france <> module_russia <> module_china

or, we could train them from scratch:

> let module_allB = list2module list_all

Again, because generating a free module is a homomorphism, both methods are equivalent.

Module distributions

The categorical distribution and the KDE both have this module structure. This gives us two cool properties for free.

First, we can train these distributions directly from the free module.  Because the free module is potentially much more compact than a list is, this can save both memory and time. If we run:

> let cat_module_all = train module_allB :: Categorical Int Double> let kde_module_all = train' kdeparams module_allB :: KDE Double

Then we get the properties:

cat_mod_all == cat_all
kde_mod_all == kde_all == kde_cat_all

Extending our commutative diagram above gives:

Again, no matter which path we take to train our KDE, we still get the same result because each of these arrows is a homomorphism.

Second, if a distribution is a module, we can weight the importance of our data points.  Let’s say we’re a general from North Korea (DPRK), and we’re planning our nuclear strategy. The US and North Korea have a very strained relationship in the nuclear department. It is much more likely that the US will try to nuke the DPRK than China will. And modules let us model this!  We can weight each country’s influence on our “nuclear threat profile” distribution like this:

> let threats_dprk = 20 .* kde_usa> <> 10 .* kde_uk> <> 5 .* kde_france> <> 2 .* kde_russia> <> 1 .* kde_china>> plotDistribution (genPlotParams "threats_dprk" threats_dprk) threats_dprk

Basically, we’re saying that the USA is 20x more likely to attack the DPRK than China is.  Graphically, our threat distribution is:

The maximum threat that we have to worry about is about 1300 kt, so we need to design all our nuclear bunkers to withstand this level of blast.  Nuclear war planners would use the above distribution to figure out how much infrastructure would survive a nuclear exchange.  To see how this is done, you’ll have to click the link.

On the other hand, if we’re an American general, then we might say that China is our biggest threat… who knows what they’ll do when we can’t pay all the debt we owe them!?

> let threats_usa = 1 .* kde_russia > <> 5 .* kde_china>> plotDistribution (genPlotParams "threats_usa" threats_usa) threats_usa

Graphically:

So now Chinese ICBMs are a real threat.  For American infrastructure to be secure, most of it needs to be able to withstand ~3500 kt blast.  (Actually, Chinese nuclear policy is called the “minimum means of reprisal”—these nukes are not targeted at military installations, but major cities.  Unlike the other nuclear powers, China doesn’t hope to win a nuclear war.  Instead, its nuclear posture is designed to prevent nuclear war in the first place.  This is why China has the fewest weapons of any of these countries.  For a detailed analysis, see the book Minimum Means of Reprisal.  This means that American military infrastructure isn’t threatened by these large Chinese nukes, and really only needs to be able to withstand an 800kt explosion to be survivable.)

By the way, since we’ve already calculated all of the kde_country variables before, these computations take virtually no time at all to compute.  Again, this is all made possible thanks to our friend abstract algebra.

Homework + next Post

If you want to try out the HLearn library for yourself, here’s a question you can try to answer: Create the DPRK and US threat distributions above, but only use survivable weapons.  Don’t include bombers in the analysis.

In our next post, we’ll go into more detail about the mathematical plumbing that makes all this possible. Then we’ll start talking about Bayesian classification and full-on machine learning. Subscribe to the RSS feed so you don’t miss out!

Why don’t you listen to Tom Lehrer’s “Song for WWIII” while you wait?

The fastest scripting language on Earth.

Download |Overview: Slideshow |Tutorial |Reference |Community

Why do we need a new programming language?

Existing languages have too many faults to support modern requirements.

Goals.

• high performance
• rapid prototyping and a scripting language deployment model
• safety and correctness
• scalability
• adaptability
• platform independence

Performance

The ability to obtain high performance in a wide variety of domains is fundamental. Light-speed behaviour was a primary goal of C++ and it is for Felix too. In fact, we aim for hyper-light performance: faster than C.

Network performance matters too. This means we need platform independent asynchronous I/O. To support a large number of clients, we need lightweight threads with fast context switching.

Rapid Prototyping and scripting harness

C++ is typically very hard to build. Make files, macros, compiler and OS feature tests, switches, paths and a huge array of poorly integrated non-portable tools make development a nightmare compared with the easy of deploying Python or Perl scripts. IDEs fix these problems in a clean way, but only handle boilerplate workflows.

Correctness

Unfortunately dynamic typing is not amenable to reasoning about program correctness, and traditional imperative programming also presents obstacles.

We need:

• static typing for low level safety
• contract programming model for high level safety
• automatic memory management without compromising performance
• overloading for convenience
• parametric polymorphism for data types
• functional programming for correctness
• imperative programming for performance
• platform independence for portability
• platform dependence for performance and low level programming
• shared-memory concurrency for real-time performance
• message passing for distributed concurrent cloud computing
• low level optimisations for tight calculations
• high level optimisations for overall application speed
• extensible grammar in user space for implementing Domain Specific Sub-Languages
• easy use of existing C and C++ libraries and code bases
• built-in support for web development
• built-in support for high performance computing
• built-in support for game development

How Felix meets these goals.

Felix is designed to address all these issues. It is a C++ code generator and thereby can provide compatibility with existing C and C++ code bases. We let the native C++ compiler do the hard work of low level optimisation whilst Felix does high level optimisations. The resulting code is very fast, sometimes "faster than the speed of light (C)", but can be platform independent and is simply to deploy: just distribute the source files and run them, like a scripting language.

However Felix has its own type system based on a combination of Ocaml and Haskell. Like Ocaml it provides strong support for functional programming, whilst also supporting imperative programming. The type system is strict.

First order polymorphism is core, not a bolt-on as in Java and C++. Felix also provides open overloading like C++, but only allows exact matches. It also provides Haskell style type classes as an alternative way to obtain genericity.

To overcome syntactic impedence mismatching with the wide number of application domains, the Felix grammar is defined in user space. It can be extended by the end user to provide a suitable Domain Specific Sub-Language. The parser is GLR and the user actions are written in R5RS Scheme.

A rich set of shortcuts makes programming a breeze. Built-in regular expression support and other features provide string handling on par with Perl. Web programming is enabled by built-in asynchronous socket I/O combined with cooperatively multi-tasked fibres that would support millions of http clients if only the server could supply enough sockets. Context switching is achieved by a pointer swap, and state is maintained by a spaghetti stack.

By torsten.grote, on January 3rd, 2013

Music: Angels and Airwaves – Diary

I’m always a little late on the 2012 lists, stick with me, this’ll be worth it.

Executive Summary of 2012:

I started 2012 by taking a bit of a risk careerwise, where before I had been the enterprise huge-company guy, I ventured into mid-sized startupish culture, pushing myself out of my personal comfort zone in terms of commute and direction. I was getting into mobile (IOS) development in my spare time, but still wanted to fill in some experience gaps I’d missed in previous opportunities. I had a good 10 month run at a great ‘startup’ in downtown Austin, was laid off, felt my world was upside down for a little while there, and came out of the tailspin – righted myself and found myself with a really great career path/opportunity for 2013.

2012 wasn’t all career & code, but it was certainly too much of it, and it shows. We’ll get to the finer points on that angle as we get through the list.

So, here’s what I learned:

I learned to take risks.

When I quit my job at PayPal a few years ago, I had interviewed around a bit, but didn’t find anything quite perfect. Meanwhile, I was steadily growing more and more disenchanted with code and slow moving big companies. At some point I decided to just quit, taking the risk to take a little time off to take a break, and fall back in love with code in the process.

The day I quit, I was scared to tell my wife what I’d done, and I was shocked when she gave me one of the best hugs of my life and simply told me “What took you so long?”. – Only days later she would not stop telling me that there was an immediate change in my mood, my sense of peace, and my happiness.

The thing that happened when I quit PayPal was, I took my destiny into my own hands. I decided firmly that PayPal wasn’t it for me, and that the next great thing was out there somewhere.

I spent that next year consulting on a cloud infrastructure product at Dell, making some really great friends in the process, and learning all about datacenter innards that I’d never had a chance to learn about back at PayPal.

When the Dell gig was up a year later, I interviewed around again, and found myself pushing my personal envelope of comfort and safety even further – trying a startup-like mid-size company with an open-atmosphere office layout and 90% all-star employees. I wasn’t a cultural fit for the new place, but I was surprised to find myself very comfortable in my own skin – despite philisophical clashes with the culture at large.

The point is, the downside to risk is the unknown, but the upside, if you can handle it, is also the unknown. Each time I switched jobs I learned a little more about what’s always the same in software, and what can be, and should be different. Each time I jumped from one ship to the next, I accepted a little bit more risk and discomfort, and learned a lot about myself, because the opportunity ahead of me always offered 1,000 new variables.

While I was at that ‘startup’ with the all-stars, I learned quite a few valuable lessons, such as:

I learned that sales guys really matter.

As a hot-shot self-involved software engineer/coder, I’ve always had trouble understanding what it is that the sales guys actually *do* at a company. It always seemed the sales guys were the guys selling more than we could deliver, and generally mucking things up – this may still be a fair assessment for a giant cant-do-wrong company such as a PayPal or Dell where the money is pouring in the doors no matter what happens. … It would not be unfair to say I had contempt and disdain for the marketing guys.

Then I worked at a startup.

When you personally meet the 3 or 4 guys who have those sales calls week to week to keep the 3 or 4 big clients sending the checks in, the contempt fades. When you hear the war stories of those 3 guys working in a shitty little closet of an office, and see a team of 20+ engineers around you in a posh downtown office making great salaries 3 years later – the disdain goes too.

Seeing and hearing those sales guys in action was, I think, *critical* to my understanding of how the world works. And let me tell you, Mr. Hotshot Engineer or Designer, the part you do, doesn’t mean shit. The world of money goes around based on relationships and a little bit of luck. The guy in your office who’s non-replacable is the guy who looks and acts like a walking parody out of a ralph lauren ad – so long as he can sell.

When you see VP so and so from client X lose their shit for some random reason and threaten your company’s bottom line on a whim, then see your sales guy have that same VP offering more money next month after a 5 minute phone conversation, that’s when the light bulb clicks on, and you realize that really, the code matters a little, but not very much.

I learned that assholes matter.

The thing about jerks is, they run their mouths. The thing about the right kind of asshole is, they’ve got charisma, and backbone. There are certainly standard run-of-the-mill wannabe jerks who don’t have the right, and I’d argue we could all stand being nicer to one another day to day in general, but man, assholes make the world actually move.

Working in the big company, you’ll see an executive here or there who wasn’t the original big idea guy or gal, and wasnt the nepotism stick-it-out-long-enough ladder climber – frankly, this odd executive is the asshole. And, like it or not, you and I need these people.

When you work in the smaller company, and see the next engineering VP hired cold off the street, along with half a dozen other hires – you’ll know which one he is. He’s going to run his mouth. He’s going to be friendly to everyone, but trash talk everyone too. He’s cunning, he’s smart as a whip, and if he’s rude, he’ll get you if you cross him. Most everyone will agree that this new guy is going to ruin the company, except.. he doesn’t, he does the exact opposite. It’s the nice polite guys who are too scared to risk their necks who ruin the company.

The thing about the asshole is, he gets shit done, he has drive. He has a fault of running his mouth, and along with that comes a life full of lessons of how to get himself out of the jams his mouth gets him in over and over – that means, this guy has a spine, this guy can make tough decisions, and this guy will deliver when it counts.

I’m not advocating ladder climbers, I’m not advocating jerks being jerks for the sake of jerktitude, I’m just saying, they have a place, and when you find the right asshole, they’re going to deliver and kick ass while doing it. The delicious irony will be, 5 years from now when your midsize is larger than midsize, the asshole who everyone hates will be the only executive of the lot who arguably deserves his merit badge title. Think on that.

I learned the value of having lunch with others.

One of the perks at my job last year was paid lunches. This is a really great thing for developers, because developers are idiots in many ways. First, they’re anti-social, and secondly, they’re cheap. So, if they can pretend to be a robot and work 8 hours straight with a hotpocket “meal” in the middle, they will, like idiots.

The downside to this is that it takes your developers a year to make the friendships your marketing team will make with each other in a week. The solution to this problem is to get your developers to go have lunch together.

After having lunch with my new peers for only a few weeks, it became immediately obvious to me how stupid I’d been in my career until that point. Previously I’d always opted for a 15 minute or 30 minute lunch, microwaving something, and getting back to it. In only a few weeks with the new group, I knew more about several of those guys than I did people I’d worked beside for 3+ years at PayPal.

Lunch matters, take it, and have lunch with friends and colleagues, often. Happy hours, too.

I learned that open environments can work for coders, within reason.

Open environment offices are an in-thing. Facebook did it, so everyone else must now do this too.

I disagree with the open environment if it’s done the wrong way. Doing it the wrong way includes: forbidding telecommuting; having more than ten people in any one open space; having sales guys in the same open office as the coders; not providing quiet spaces (couches, little closet hotel cubes) for people to go to to talk or work; and test-piloting your open-space idea on an executive sales team, then mandating it for the world (PayPal..).

Doing it the right way means doing the opposite of everything above, and providing nice noise cancelling headphones for your employees.

Coders need quiet, and time to think, and noise-cancelling headphones are not enough. The place I worked this past year had a great open-environment layout and telecommuting policy, but even still there were more than a few days where I had unending headaches caused by the choice of music or office chatter.

I learned that telecommuting is really awesome, within reason.

Let’s see, this past year I accomplished many things while telecommuting, I watched the entire 5 seasons of the wire (amazing, perception altering show, btw), many movies, did a lot of laundry, took my dog on many walks, and was more productive than I’ve ever been at any job in the meantime.

Telecommuting isn’t for everyone, you’ve got to be driven, on-task, and have a list to constantly feed on when you finish the task before. But, for coders, who need the quiet, the peace, and the space to play This Will Destroy You at deafening volumes from time to time, telecommuting is great.

The thing to remember about telecommuting is that it’s lost face time with your colleagues and boss, and you’ve got to make time while in the office to make up for that lost time.

When 2012 started, I thought telecommuting was something you do on a day without real work, in a startup-like mid-size, those days don’t exist – and that’s a good thing, because by the end of 2012 I can safely say telecommuting days are the days you take to really go heads down (even with This Will Destroy You or The Wire blaring in the background) and get shit done. Asking for a telecommuting day no longer carries a guilty connotation with me, and as a person who consistently delivers, I actually *need* the freedom to just go do the right thing from time to time.

I learned to track myself better.

I’ve always been a list maker, and from time to time I burn myself out with the lists. There’s a balance between the lists and actually letting life just happen, and I suck at that balance.

It turns out, I had a fatal flaw in the way I managed lists. That is, I deleted items after I finished them.

Don’t delete items after you finish them, grey them out, in place, and make a new list each week.

When you grey the items out, you start to get a feel for how much you actually accomplish.

Going into 2012 I was constantly feeling stressed out that I was never on top of my list, and that the list was constantly growing.

After a year of greying items out I feel worlds different, I now have great pride in how much I accomplish, and yet I also finally understand about how many items I can truly knock out per week. I also, for better or for worse, realize what a shitty friend and flake of a person I can be sometimes, because the lists show me these things where before the giant list that never ends with deleted items did not.

I learned that tracking myself does not matter.

One thing I’ve always done with my lists of tasks at work is track what I did each day, because I thought I could cover my ass with the paper trail. It turns out, that matters somewhat, but .. well, not really ..

I learned to be fired (and, when to quit).

I wasn’t fired, I was layed off, budgets got tight, last in first out, etc, but really – I was fired. I was on the top of the boss’ list to axe for a while, and I knew it. I wasn’t a great cultural fit, and had fundamental philosophical differences with prioritization and feel-good-about-ourselves-rewriting-endlessly-for-the-hell-of-it wankery.

When I started working at the place, it was an uncomfortable risk in several ways, and at first I gave myself 6 months to decide if I liked it – at some point that changed to a year mark, and a little after that it turned into a “2 or 3 years, I guess..” kind of thing from my side. I was into it, having fun, but cultural friction was perpetually upsetting. Nothing quite like being 1 of 5 “platform” engineers never invited to the endless feel-good-about-ourselves wankery standards meetings that never went anywhere, perhaps the fact that it was wankery in my mind had something to do with it

To the point about covering your ass with your paper trail, the thing there is, that doesn’t matter. If someone has a target on your back for whatever reason, the paper trail won’t help you. What will help you is spending more time getting to know people and working things out by communicating more, if anything.

Communication helps, but also, life is short, if you’re unhappy with your lot in life, even a little bit, consider changing your lot. For me, I had to be laid off to have the wakeup call that I was settling in several ways to work at the place. I had convinced myself that the settling was part of the discomfort/risk experiment, and honestly I’d probably still be working there today had I not had the not-so-gentle push out that I needed.

Saying I “Learned to be fired” sounds funny, but truly, learning to not compromise 100%, have a little backbone, and be myself mostly was worthwhile. Had I not been myself, I would have hated myself for capitulating to philosophical differences I couldn’t get behind, and I probably still would have been first on the list to go. There’s a stigma to the thought of being fired, especially if you’re someone who’s fired for really bad reasons – such as not actually doing your job. In my case, I did my job to the best of my ability, and kicked ass while doing it, all without compromising my character or beliefs in the process. Being let go for philosophical differences is a lot like being that hard-won asshole VP – being let go b/c you give a damn, and stood up for something, but fell on the wrong side of the dice. (In this case, I just gave a different Damn than the rest of the team.. )

I’m not advocating getting fired, but truly, politely contributing to the cause without making a scene of your philosophical differences too often, is worth it, even if you’re fired over it. I suppose a secondary lesson to learning that assholes are needed is that you can’t be everyone’s friend. You win some, and you lose some. That’s really all there is to it.

No slight to the people I worked with or for, to each his own, truly – I wasn’t a cultural fit at the place, end of story, and that’s one of the beautiful things about software – there’s a dozen or more overarching cultural styles you’ll encounter depending on the shop – if you don’t fit at one place, you’ll fit at another.

I learned to communicate with my spouse, regularly.

The layoff could have been a lot worse, had I not been able to communicate openly and work through the topsy-turvy tailspin with my wife by my side.

Earlier in the year my wife and I had gone to some couples therapy together. As a non-religious person who values reason and actually doing something to change yourself for the better, I highly recommend therapy when the time is right.

We skipped marriage counseling, luckily having enough wits about ourselves to already talk about and understand each other’s thoughts on money, babies, family, etc before actually tying the knot. I’m glad we skipped marriage counseling b/c honestly we wouldn’t have enjoyed it or been ready for it – we were on a high that didnt really dip from the day we met until late 2011 – the counseling would have caused needless turmoil, or been wasted on deaf ears.

So we had communication breakdown, and went to speak with a counselor, a great one.

It’s funny how even the greatest relationships still have amazing amounts of built-in fear and reservation. There were so many downright silly and stupid things that my wife and I were so scared to talk about with each other, nitpicks on character or even habits and whatnot that didnt matter in the big scheme of things. The thing is, the nitpicks build into a mountain at somepoint and will kill you if you can’t talk about them. Having a third party intermediary person there to listen to us and encourage us to talk about the scary things really helped.

At first, there were many tears and deep breaths while we vocalized things that were bothering us, and sometimes voices were raised, but the counselor kept draining reason into our ears and showing us how to handle these communications on our own. A few weeks later, even the most intimate fears or new worries were voiced easily without any fear at all – we didn’t even need the counselor anymore. That’s how you can tell you’ve got a great counselor, when there’s an end game and you can see yourself clearly in a better place of understanding than before.

Another tidbit the counselor gave us was a really great, if corny sounding tool: relationship talks.

A relationship talk is a weekly meeting (no shit, like a business meeting) that lasts 20 minutes. Spouse A has ten minutes to talk, uninterrupted, if time is left at the end of their ten minutes, Q&A can happen. Then Spouse B takes a turn. That’s it, the end. Next week, you switch who goes first. After the meeting, you do something fun together, which in our case usually wound up being a walk around the neighborhood b/c we’d find there was so much stuff to talk about that the relationship talk would open up. You do the relationship talk *every* week, no exceptions. Sometimes the talk’s a tear jerker, most times it’s boring, but doing it every week is essential.

The relationship talk keeps the communication lines open. When they’re wide open and you’re humming along happily, the talks may be a bit bland, but even then they’ll suprise you with news you had no idea of, and when the comm lines are shut down or atrophying, the relationship talk will save you. I promise.

We also started doing ‘Wonderful Wednesdays’, which, more or less, is date night. No computers, no tv, something wholesome and fulfilling and rejuvinating, together, no exceptions. In practice, it’s sometimes a ‘Wonderful Tuesday’ because something really can’t budge, but making time in your life, together, to chill the fuck out is important.

I learned to read myself, and trust my gut.

I am what, and who I am. I’m driven, impatient at times, overly dramatic, and extremely emotional when everything’s just right.

Otherwise, like most of this last year, I’m dead inside.

Early on at the startup, my boss informed me that he didn’t like the way I communicate, I’m too wordy, imprecise, yatta yatta. – It was a fair assessment in part, but I took it all the wrong way and used it as a vice rather than a tool to grow with. Long story short, it fucked me up.

My natural style is to be transparent, open, and accomodating, so hearing that I was expected to be more precise and careful with my words sounded a lot like I’d have to be someone different while I worked at the place if I was going to cut it. In the end, I probably WOULD have had to be a different person to jive well there longterm, but the point my boss was making wasnt that I should shut down and be a different person, he was saying I should strive to be slightly less sloppy.

I told myself I was becoming a respectable little adult by shutting down and towing the line, and when I was finally laid off a good while later – it all came pouring out. My writing became prolific again, as did my creative whims. I was staying up through the night one or two nights per month, I was even honest-to-god crying fairly often at even the littlest beautiful or horrible things – I was me, again.

Perhaps I’m both of these people, the emotional creative dreamer, and the tight-ass conservative who’s dead inside, but I prefer the dreamer.

Time and time again I’ve noticed these moments where everything synchronizes into this chaotic yet perfectly ordered moment where some huge chapter in life immediately makes sense – as if my subconscious has been working on it all the while, for months. The backside of those events is always the same – I’m writing, I’m loving my life, I’m creating all sorts of crazy little mementos, I’m taking it easy more often, and I’m, yes, occasionally crying at the immense overcoming beauty and tragedy of it all.

2012 gave me metrics, if I’m not blogging for the fuck of it at 5am on a work night once or twice a month, something’s off. If I’m not occasionally trading sleep for another one-night shot at something great, something’s wrong – and it’s time to really assess what my gut is telling me.

I learned to be brave, and patient.

Life is funny. Often the most important stories and lessons of your life are ticking right along, slowly, very slowly, in the background – completely hidden by silly shit you tell yourself matters more. Here’s what I mean, when I was pulling photos from the past 3 years out of my archives for the Freddie Book, I had to scroll past thousands, literally thousands of photos of my dog. I’m a dog guy, not a cat guy, and though I think my dog’s pretty fucking amazing, the story to tell for recent years is not hers, it’s the cat’s.

Here’s a picture of the awesome dog anyway:

Anyway, after being laid off, I took time off, again like after PayPal, because I could, and because I knew I needed to. TumbleOn seemed to be taking off, so I distracted myself with dreams about that for a while, and I visited my good friend in Seattle for a bit, and so on. After the predictable this-is-going-nowhere burnout phase that followed shortly, I was finally at ease for a moment or two. This was one of those moments where the chaos distilled into clarity.

One morning I woke up, and decided to write the story of Freddie. There are perhaps 3 days in the past year like the day I wrote that story, but Freddie’s story was the best of the three, easily.

Freddie came through me like they say the Bible did the prophets, it just flowed, as if from somewhere else. I didnt plan it 3 days previous, I didnt even know what the damn point of the thing was as I was writing it, and yet, as the story flowed out it turned into this really incredible, amazing thing – an allegory for my own personal story of 2012 – of learning to take larger and larger risks, be brave, and welcome change – all while giving credit to the one person who deserves all the credit for any betterment in my life ever, my wife.

I won’t bother you with the details of Freddie’s story here, you can go read the book yourself (online, for free).

I learned that success is not what capitalism says it is.

Working at the ‘startup’ this past year was really a great experience to be walking through while simultaneously seeing our personal side project, TumbleOn, grow.

Until you’ve worked at a startup, or a ‘startup’ that’s really a decently-funded mid-size pretending to be a ‘startup’, you don’t really ‘get’ what success is. Success, arguably, in American or capitalistic terms, is striking it rich. But, frankly, striking it rich is a stroke of luck, whether you win the lottery or win the right-time-right-place lottery like Bill gates.

This past year redefined my personal measure of success. Success is not a certain amount of money, or a title, success is being one of those initial three guys in a closet-sized office, seeing 20 well-salaried employees loving life 3 years later. Success is making something great that brings joy to other people’s lives.

Would I like TumbleOn to make me and my friends independently wealthy? Sure, but to me, TumbleOn is already the largest personal success of my short little career, and the sweet and sad fact is, it may be the largest success of my long term career as well. In my own little corner of the world, I’ve been a part of a small team who made something really amazing that thousands of people use regularly, viewing more than 300 million photos in 2012, and more than a billion in 2013.. laughing, gasping, giggling, and being inspired along the way.

To me, as I said earlier this year, success is not the money, success is making something bigger than myself, that makes the world just a little bit better than it would be without me.

I learned to take detours.

A stronger relationship between my wife and I resulted in a bolder wife, who’s more uppity lately, in all the right ways. This year she really put the time in to repeat the message to me that I should live life a bit more rather than plan so much.

For example, I immediately thought to take a long-planned-anyway trip to Seattle after being laid off, but kept the thought secret and was ashamed – it seemed extravagant given the circumstances. A few days after the layoff, my wife independently suggested and insisted that I go to Seattle. I hemmed and hawwed about it, listing the cons of the idea (money), and she wouldn’t have it. She knew what was good for me, and sitting at home in the familiar, moping, driving myself crazy wasn’t it.

Months later, I was visiting some friends, absent mindedly returning from lunch with a good buddy, when I got lost.

I wasn’t lost lost, but I was way off the beaten path. My coder brain immediately detoured me to the most efficient route home – still a different route than that intended, but efficient – when all of the sudden half way home I slammed on my brakes and nearly caused a wreck – swerving into the drive way of this great botanical gardens in Fort Worth. I’d completely forgotten the place was there, and decided that for once I would take my wife’s advice, and take the detour, throwing efficiency out the window.

On my way into the Japanese Gardens, the ticket clerk asked how my day was going, I told him “it was going alright, but it’s about to get a whole lot better”. The clerk laughed. I took a few steps inside and something primal overtook me. I felt as if I were outside myself, floating through this long-familiar place, truly soaking in the beauty and moment at hand – it was a feeling I really don’t have often.

As I walked past the zen garden I was congratulating myself on taking a detour, excited to tell my wife, sure that she’d be so proud of me for actually experiencing something rather than running the numbers beforehand.

I walked and savored the moment, being outside myself as in a dream, and conscious of it all the while. As I continued on, I saw something amazing, or I thought it was.

There was this crane, sunning himself, wings spread wide to wholly accept the sun. He too, was savoring the moment, and he just stood there, zen-like in this serene scene for a full 10 minutes, motionless. For a while I was convinced he was some poetic statue painted convincingly real, and then he blinked.

Seeing that crane, on my floating detour, was one of those weird things I don’t think I’ll ever forget for the rest of my life. Now, two months later, I can barely recall many of the particulars of that visit to Fort Worth at all, but that moment in the sun, watching that crane fully accepting the simultaneous chaos, beauty, and fragility of life – that scene and moment will stick with me for life.

Thank you, wife, for telling me to take detours.

I learned that time is precious.

The flipside to living a happy and full life is that there isn’t enough time in life.

There’s something they call “FU Money”, or, “Fuck You Money”. Loosely, this is the amount of money it would take for you to quit working and start living your life the way you want, and deserve, to live it.

Something flipped in Steve Jobs’ brain one day, and he realized the best way he could live his life was to live every day as if it were his last.

This year, I learned to set my “Fuck You Money” value to $0. I’m no zen master like Jobs, but for the first time since college I can honestly say that I’m working at a place where even if I were independently wealthy, I’d still be showing up for work the very next day.

That is what I, and you, and everyone, deserves.

You deserve to work with kick ass people who both inspire you and celebrate with you. You deserve to take time on wednesdays to be with your family and turn the computers and calendars off. You deserve to take time in the middle of the night to forego sleep in favor of some crazy undending blog rant (like this) or new project idea. You deserve to take vacation, and you deserve a wake-up call from time to time.

—

There’s this great White Stripes documentary of their last tour. There’s a segment in there where Jack White is talking about living on the edge, manufacturing wake-up calls. He says that when he goes on stage, he measures the comfortable distance to set the mic stand so he can easily grab his next guitar pick. Then, he moves the mic stand a few feet further, so he has to really jump and make himself get that pick in time. Sometimes he misses, sometimes he knocks the damn stand over, but every time he gaurantees he’s living – because he’s pushing his own personal envelope of comfort in even the silliest esoteric ways – and, he says, it works.

Late last month, Intel dropped by my office with a power engineer for a rare demonstration of its competitive position versus NVIDIA's Tegra 3 when it came to power consumption. Like most companies in the mobile space, Intel doesn't just rely on device level power testing to determine battery life. In order to ensure that its CPU, GPU, memory controller and even NAND are all as power efficient as possible, most companies will measure power consumption directly on a tablet or smartphone motherboard.

The process would be a piece of cake if you had measurement points already prepared on the board, but in most cases Intel (and its competitors) are taking apart a retail device and hunting for a way to measure CPU or GPU power. I described how it's done in the original article:

Measuring power at the battery gives you an idea of total platform power consumption including display, SoC, memory, network stack and everything else on the motherboard. This approach is useful for understanding how long a device will last on a single charge, but if you're a component vendor you typically care a little more about the specific power consumption of your competitors' components.

What follows is a good mixture of art and science. Intel's power engineers will take apart a competing device and probe whatever looks to be a power delivery or filtering circuit while running various workloads on the device itself. By correlating the type of workload to spikes in voltage in these circuits, you can figure out what components on a smartphone or tablet motherboard are likely responsible for delivering power to individual blocks of an SoC. Despite the high level of integration in modern mobile SoCs, the major players on the chip (e.g. CPU and GPU) tend to operate on their own independent voltage planes.

A basic LC filter

What usually happens is you'll find a standard LC filter (inductor + capacitor) supplying power to a block on the SoC. Once the right LC filter has been identified, all you need to do is lift the inductor, insert a very small resistor (2 - 20 mΩ) and measure the voltage drop across the resistor. With voltage and resistance values known, you can determine current and power. Using good external instruments (NI USB-6289) you can plot power over time and now get a good idea of the power consumption of individual IP blocks within an SoC.

Basic LC filter modified with an inline resistor

The previous article focused on an admittedly not too interesting comparison: Intel's Atom Z2760 (Clover Trail) versus NVIDIA's Tegra 3. After much pleading, Intel returned with two more tablets: a Dell XPS 10 using Qualcomm's APQ8060A SoC (dual-core 28nm Krait) and a Nexus 10 using Samsung's Exynos 5 Dual (dual-core 32nm Cortex A15). What was a walk in the park for Atom all of the sudden became much more challenging. Both of these SoCs are built on very modern, low power manufacturing processes and Intel no longer has a performance advantage compared to Exynos 5.

Just like last time, I ensured all displays were calibrated to our usual 200 nits setting and ensured the software and configurations were as close to equal as possible. Both tablets were purchased at retail by Intel, but I verified their performance against our own samples/data and noticed no meaningful deviation. Since I don't have a Dell XPS 10 of my own, I compared performance to the Samsung ATIV Tab and confirmed that things were at least performing as they should.

We'll start with the Qualcomm based Dell XPS 10...

Mega Man 2 Password Generator

Overview

Mega Man 2, like many games of its era, utilized a password system in order to continue your progress between game sessions. This removed the need for a battery in the game catridge and allowed gamers to share passwords (and thus progress) with others.

In Mega Man 2, the password is represented as a 5x5 grid in which the columns are labled 1-5 and the rows A-E. Each password is composed of 9 cells which are 'set', indicated by a red dot.

Thus, a password can be communicated as A5, B2, B4, C1, C3, C5, D4, D5, E2.

Put another way, this 5x5 grid represents 25 bits in which a password always has exactly 9 bits set. Using this representation, the password algorithm can be expressed succinctly in terms of these bits and using basicbitwise operations.

In the 25 bits, there are 5 words of 5 bits each where each word represents a row in the grid. The entire 25-bit password is thus comprised of the words A E D C B (using little endian). So, the first word (lowest 5 bits) are the 5 bits of the row B and the last word (bits 20-25) are the 5 bits of row A.

Words 1-4 (bits 1-20)

A Mega Man 2 password has exactly 9 bits set. 8 of these bits represent the alive/defeated status of each of the8 bosses in the game. The follow table illustrates the bit values for the alive/defeated status for each of the 8 bosses.

Thus, if both Bubble Man and Air Man were defeated but all other bosses were still alive, this is represented as the following bits (1-20) 01111 10001 01000 10000.

E-Tank Word (bits 21-25)

The last bit (9th) represents the number of E-Tanks Mega Man has. This is stored in the 5th word (row A) and represents bits 21-25, the most significant bits of the password. Mega Man can have between 0-4 E-Tanks and this is encoded simply per the bit position in this last word. Thus, if Mega Man has 0 E-Tanks the word is 00001, 1 E-Tank is 00010, 2 E-Tanks is 00100, and so forth. Unlike the other words, the 5th word (row A) will thus only ever have a single bit set.

The E-Tank word (row A) is important in that it encodes bits 1-20 by performing arotate left operation on bits 1-20 by the number of E-Tanks that Mega Man has. Thus, if Mega Man has 2 E-Tanks, bits 1-20 are rotated left by 2 positions. If Mega Man has 0 E-Tanks, this is effectively a no-op. The table above illustrating the bits set for each of the 8 bosses represent the bits prior to the rotate left operation. The bits of the E-Tank word are not included in the left rotation.

Algorithm

The algorithm for calculating a password can be summarized as follows:

Setup

To run mm2pwd you will need Ruby installed. mm2pwd has been tested with Ruby 1.9.3p362 which is the latest version as of this writing.

Running

To run mm2pwd, simply open a terminal session in the root directory and execute the following command:

$ ./mm2pwd.rb

Without an modification, this will generate a password in which Mega Man has all 4 E-Tanks and has defeated all 8 bosses. If you want to modify this, simply change the values in the initialize method.

License

Copyright 2013 Kevin Shekleton

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

+++
Update: Shortly after publishing this article here, it became #1 on Y Combinator’s Hacker News. Subsequently, GIZMODO approached me and asked to re-publish it. Finally, ReadWriteWeb’s Editor-in-Chief (Dan Lyons) wrote a great follow-up. As it is difficult to keep up with responding to all the comments, I mainly focus on responding on Google+.
+++

First things first, I’d love to get in touch on Google+ and Twitter (@ralf).

Over the past few years I’ve invested a lot into Apple products and services.

If you’d come by my house, you’d find four of the latest Apple TVs, two iMacs, the latest MacBook Air, a MacBook Pro, more than five AirPort Express stations and Apple’s Time Capsule. You could touch every single iPhone, from the first up to the iPhone 5, iPads ranging from first generation to fourth and we’ve lately added two iPad minis.

My iTunes Library comprises well over 8.000 songs – all purchased via the iTunes Store. No matter whom you would ask, everybody will confirm that I’m what some folks call an Apple fanboy.

The reach of Apple’s products goes beyond my personal life.

As the co-founder of Germany’s largest mobile development shop, I’m dealing with apps – predominantly iOS powered – in my daily professional life.

Driven primarily by the business I run, I tried to give Android a chance more than once.

In various self-experiments, I tried to leave my iPhone at home for the Motorola Droid, the Nexus One, the Samsung Galaxy S II and S III – and always switched straight back to the iPhone. None of those Android devices have worked for me – yet.

And then I got the Nexus 4.

When the latest Google flagship Android device shipped, I almost expected it to turn out as yet another “take-a-look-and-sell-it-on-ebay” experience. Little did I know.

It’s now almost two weeks since I switched the Nexus 4 on for the first time – and meanwhile I completely moved to it, leaving my iPhone 5 at home. Do I miss anything? Nope. Except iMessage. More to that later.

In this somewhat lengthy post, I’ll try to explain why.

My motivation is not to bash Platform A over Platform B. On the contrary: I will try to summarize my very personal findings and experience based on years of using iOS. I’ve seen the Apple platform evolve while Android was playing catch-up for so long. When iOS 6 came out, for the first time I complained about the lack of innovation in this major new release. I asked myself, whether we might see Apple beginning to lose its leading position in mobile platforms.

Before you read on, it’s important to emphasize that I’m a pro user.

I’m not the average smartphone owner, who makes just a couple of calls every now and then or runs an app once in a while. By the nature of my job and out of curiosity, I deal a lot with social media outlets, social networks and constantly try new services. With that said, my judgement might not be suitable for everyone. In case you consider yourself being a demanding power user, though, you might find this helpful.

At the time of this writing, I used Android Jelly Bean 4.2.1 on an LG Nexus 4.

Summary

Putting it into a single line: The latest version of Android outshines the latest version of iOS in almost every single aspect.

I find it to be better in terms of the performance, smoothness of the rendering engine, cross-app and OS level integration, innovation across the board, look & feel customizability and variety of the available apps.

In the following paragraphs, I try to explain why.

Performance and Smoothness of the Rendering Engine

I know there are benchmarks which measure all kinds of technical performance on a very detailed level. That’s not what I’ve done and, honestly, I’m not interested into that much. I’m talking about the performance I feel in my daily use.

Using the Nexus 4 with Android 4.2.1 is a pure pleasure when it comes to performance. I don’t exactly know what Google has done with “Project Butter” in Jelly Bean, but the result is astonishing. In the past, Android felt laggy, sometimes even slow and responses to gestures didn’t feel half as immediate as on iOS.

This has changed completely.

I’d say both platform are at least even. In some cases, Android even feels a bit ahead of iOS 6. I especially got this impression when it comes to rapidly switching between apps – which I constantly do now – and scrolling through a huge number of more complex content. (I’m not talking just tables with text here.)

While Android still doesn’t give you bouncing lists and scroll views – primarily, because Apple has a patent for this specific behavior – every transition between views has been reworked, polished and modernized. In most cases, it feels more modern, clean and up-to-date than its iOS counterpart.

Cross-app and OS level integration

One of the biggest advantages I found during my daily use is the level of cross-app and OS level integration.

This also is the area where I got most disappointed when Apple introduced iOS 6.

In fact, I think iOS has reached a point, where usability starts to significantly decrease due to the many workarounds that Apple has introduced. All of these just to prevent exposing a paradigm like a file system or allowing apps to securely talk to each others. There is a better way of doing this. Apples knows about it but simply keeps ignoring the issues.

On Android, it’s quite the opposite. One can see the most obvious example when it comes to handling all sorts of files and sharing.

Let’s assume, I receive an email with a PDF attachment which I’d like to use in some other apps and maybe post to a social network later.

On iOS, the user is forced to think around Apple’s constraints. There is no easy way to just detach the file from the email and subsequently use it in what ever way I want. Instead, all iOS apps that want to expose some sort of sharing feature, do have to completely take care for it themselves. The result is a fairly inconsistent, unsatisfying user experience.

On iOS, you might use the somewhat odd “Open in…” feature – in case the developer was so kind to implement it – to first move the file over to Dropbox, which gives you a virtual, cloud based file system. If you’re lucky, the other app, from which you want to use the file next, offers Dropbox integration, too, so you can re-download it and start from there. All because Apple denies the necessity of basic cross-app local storage.

On Android, it’s really simple.

I can detach the file to a local folder and further work with it from there. Leveraging every single app that handles PDF files. In case I receive a bunch of mp3 files, I can do the same. And every app, that somehow can handle audio playback, can reuse those mp3 files.

Another great example: Sharing stuff on social networks. On iOS, I have to rely on the developers again. Flipboard, as one of the better examples, gives me the ability to directly share with Google+, Twitter and Facebook. On my Nexus 4, I have 20+ options. That is, because every app I install can register as a sharing provider. It’s a core feature of the Android operating system.

But it goes even further: On Android, I can change the default handlers for specific file types – much like I’m used to from desktop operating systems.

If, for example, you’re not happy with the stock Photo Gallery application, that shows up whenever an app wants you to pick an image, you can simply install one from over a hundred alternatives and tell Android to use it as its new default. The next time, you post a photo with the Facebook app – or have to pick an image from within any other app – your favorite gallery picker shows up instead of Android’s own.

All of this is entirely impossible on iOS today. I’ve stopped counting how often I felt annoyed because I clicked a link to a location in Mobile Safari and would have loved the Google Maps app to launch. Instead, Apple’s own Maps app is hardcoded into the system. And there’s no way for me to change it.

The customizability is simply stunning

Let me make this very clear: Gone are the days where home screens on Android phones almost always looked awful.

If you don’t believe me, hop over to MyColorscreen and see for yourself.

Also note that all of those are real Android home screens, not just concepts provided by designers. They are not beautifully photoshopped wallpapers, but fully functional screens with app icons and active widgets.

And all of those can be configured pretty easily just by installing a couple of apps and tweaking settings. Here is an album showing my current configuration, which I was able to achieve after just a couple of days using Android as an absolute newbie.

Getting inspired? Here are some more of my favorites:

Now, iPhone lovers might argue, that the average Joe doesn’t want to deal with widgets, icons and custom animations.

I’ve used the same argument for years. Well, guess what, you don’t have to. The default Jelly Bean home screen looks beautiful already. But in case you want a somewhat more individual phone, the possibilities are endless.

For years, what you could do with Android, simply yielded awful looking home screens. This has changed. Significantly so.

And believe me or not, but after having configured my Nexus 4 just the way I always wanted – providing me with the fastest access to my most frequently used apps along with the most important information on a single screen – whenever I grab my iPhone for testing purposes, iOS feels pretty old, outdated and less user friendly. For me, there currently is no way of going back. Once you get used to all of these capabilities, it’s hard to live without them.

App quality and variety

Yes, there are still lots of really ugly apps on Google Play.

In my opinion, this has two primary reasons.

First, the obvious one: The lack of a centralized quality control and review. It’s great for encouraging variety, but obviously it also allows for some really cheap productions to be published to the store. Usually, you can spot those immediately from the screenshots on Google Play.

The second reason is more low-level: The way developers declare user interfaces (it’s primarily done in an XML configuration file) allows for rapidly hammering together dirty UIs. That’s what happens a lot and users can see and feel it. iOS developers tend to be more aware to involve designers and iOS UIs cannot be crapped together as easily.

However, gone are the days where the apps I use most greatly fall behind their iOS counterparts.

The Facebook app is identical in terms of the look and feel and its features. As a plus, it has better cross-app integration. The Google+ app is better on Android, but that’s to be expected. Flipboard is fantastic on Android, plus better integration. The same is true for Pulse News. The list goes on: Instagram, Path, LinkedIn, WhatsApp, Quora, Pocket, Amazon Kindle, Spotify, Shazam and Google Talk. They are all great on Android. Plus better integration. Plus home screen widgets. You sense a scheme here?

And if you want to experience some real UI magic – even if you just need an argument when you’re bumping into an iPhone owner the next time – install Zime, a highly addictive calendar for Android which features a smooth 3D animation and really innovative UI.

Talking about variety. This is, where Android’s openness pays off.

On iOS, many things I always wished to see being developed, simply cannot be done because of the strict sandbox Apple enforces around apps. On Android, I use an app to block unwanted calls. To auto-respond to incoming short messages. And to lock some specific apps with an extra passcode, so my customers don’t play with my Facebook profile, when I hand over my Nexus 4 for demos.

I also have apps that give me great insight into the use of mobile data across the device and all apps. Or the battery consumption. Or which apps talk home and how frequently.

None of it is available for iOS. And possibly won’t be at any time in the near future.

What I miss

I said this earlier: The only thing I miss is iMessages. I’m not kidding. Letting go iMessages was difficult, as many of my friends are on iPhones and used to text me via iMessage. While there are perfect alternatives (Facebook Messenger, Google Talk, WhatsApp, to name only a few), from time to time I still find a couple of unread iMessages, when I switch on my iPhone 5.

My most frequently used apps

I’m an Android newbie. During the last couple of days, I had to ask many questions and received hundreds of recommendations for apps. I installed, tried and uninstalled. And kept the great ones. My sincere thanks go out to the great Nexus and Android communities over at Google+.

In case you decided to give Android a try before you read this article, or got inspired here, I’d like to save you some of my journey. Here is a list of the apps I found most useful (and beautiful, given the high standards set by years as an iPhone addict):

• Nova Launcher Prime– a must have if you want to get creative with your home screen(s)
• Google+, Facebook, Facebook Messenger, LinkedIn, Quora, Dropbox, Evernote, Foursquare
• Twitter, or as alternatives Falcon Pro and Plume Premium
• Pulse News, Flipboard, Google Currents and Press (a handsome Google Reader) for your daily dose of news
•  Spotify, Google Play Music, Shazam (which you don’t necessarily need anymore, because Google Search on Android has this build in), doubleTwist for everything audio
• MX Player Pro as a really versatile video player that handles almost all formats seamlessly
• TubeMate to make YouTube videos available for offline viewing
• QuickPic as a replacement for the stock photo picker (Gallery)
• Pixlr Express as the most amazing photo editor I’ve seen on a smartphone. Forget about Camera+ on your iPhone.
• Instagram, Snapseed and Flickr for photo sharing
• Wifi Analyzer if you ever want to fine tune your wireless LAN
• Zedge for free access to hundreds of thousands of ringtones, wallpapers and notification sounds
• Power Toogles for home screen level access to toggling bluetooth, WiFi and other settings
• Remote for iTunes as a 100% replacement for the Remote app on your iPhone
• Minimalstic Text to create beautifully simplistic text widgets for your home screen
• ASTRO File Manager if you want to fiddle on file system level
• AirDroid to remotely manage all aspects of your phone from a browser on your desktop PC
• Battery Widget Reborn! Pro
• BeyondPod if you’re an avid podcast listener as I am
• ConnectBot, a really capable SSH client
• Eye In Sky weather widget for beautiful weather on your home screen
• Square Glass Jelly Bean Blue icon theme

Note: I always use the paid / pro version of apps, if one is available. Coming from iOS, I simply cannot adjust my eyes to in-app-ads and probably never will. Google Play now offers credit cards, PayPal and some other payment alternatives. Plenty of choice. I encourage everybody to give back to the developer economy and not just go for the free versions.

In case you’re wondering why I took the burden to include all of the links to the apps above, well, here is another advantage over iOS: Google Play allows the complete remote install via the Web. If you’re logged into your account you click the install button after visting one of the links in any browser, and wherever your phone is, the respective app will be installed silently.

My Android Wish List

Let me finish this post with a couple of wishes I’ve got for the next major version of Android, hopefully made available at this year’s Google I/O:

• More and centralized settings for notifications, or, a notification center.The rich notifications introduced in Jelly Bean and the overall usability of the notification bar and drawer are already far better than those on iOS. (On a side note, I never understood why usability masters like the Apple engineers decided to make the “clear” button so tiny, that you can hardly hit it without using a magnifying glass.)However, the level of customization you get for Android notifications is currently 100% up to the developers.This means, even though Android offers a great variety of possibilities, they are not consistently available in all apps. In fact, some apps barely let you switch notifications on and off, while others allow you to customize every aspect, from notification sound to the color of the notification LED to do-not-disturb times. These should be made available globally and enforced through the APIs.For example, I’d love to be able to receive notifications on Facebook messages, but don’t want them to show the full message preview in the notification bar.There are some apps, which let you chose whether you want a complete preview, or just a standard “you’ve got mail” message, without revealing its content. But it’s up to the developer whether you’ve got the choice or not.Or: Android has support for a notification LED that can flash in different colors. I configured the LED on my Nexus 4 to blink green on new WhatsApp messages. Incoming stuff from Facebook notifies in blue and new business mail causes the LED to flash in white. What sounds like a tiny feature is really valuable: While sitting in a meeting, you can grasp immediately whether you might want to check your phone right away or not. Unfortunately, not all apps let you customize the LED color. Again, it’s up to the developer to provide these settings as part of their application. This belongs into a centralized notification center.Options I’d like to see centralized: LED color, notification sound, content preview. They could also be exposed on app level, but the Android Notification Center should allow for overrides.
• Support for multiple accounts in Google Now.I’d love to see Google Now taking advantage of multiple configured Google accounts. On my device, I’d like my Google Apps for Businesses account to drive the calendar based cards but my private one for everything else (location and browsing history, etc.). Currently, Google Now can only leverage a single account. I therefore had to switch browsing and location history on for the Google Apps for Business account I use professionally. This should be a no-brainer for Google and I keep wondering, why the folks at Google tend to forget these multiple-account scenarios.
• Solving the inconsistencies grouped around the back button.I’ve actually found this on many lists and from what I’ve read it has already gotten better in Jelly Bean. However, at times I still get confused about the multiple navigation hierarchies that are caused by the native back button which is part of the OS and a second back button available within apps. Oddly enough, the mostly fantastic Google+ Android app suffers from this issue, too. Sometimes I end up on my home screen just because I “went back to far”. It’s not a big issue, but one which needs to be addressed.As a starter, how about giving the damn back button a different color if the next time you hit it, you’ll be taken out of the app.
• Indicate whether an app uses Google Cloud Messaging or some other technology  stay connected.I believe this one to be huge: On iOS, there are essentially no long-running background processes, except for VoIP or Navigation apps. This means, all apps that notify users of incoming data while they are inactive, make use of a centralized service operated by Apple, called Push Notifications. It has a great advantage with respect to battery life, as there is only a single process on the OS level, that monitors all incoming messages and distributes them to the targeted apps, instead of potentially many apps doing whatever they want to do to stay connected.Android has a similar service, named Google Cloud Messaging.Unfortunately, there is no obvious way to differentiate apps that leverage this service from those, that constantly poll or even keep a socket connection to their home servers.I’d love to see the ones making use of Google Cloud Messaging identified in Google Play and on the OS level, maybe in the already available App Info screen. That way, I could dramatically increase battery life by stopping those that constantly talk back home and encourage developers to make use of Google’s Cloud Messaging service.

One last word

At the beginning I stated, that I tried Android many times before and it never worked for me. I figured, there are two main reasons for this. First, Android has made a major step forward with Jelly Bean. It just wasn’t on pair with iOS before. Second, and more important, I found the stock Android experience provided by Google the best you can get. After switching to the Nexus 4, I tried my Samsung S III again, and it did not work for me.

What Samsung does with its TouchWiz modifications and many of the other tiny changes – and other non Nexus vendors, too – totally ruins the experience for me. If you’re coming from iOS I highly recommend choosing one of the Nexus devices with guaranteed updates and a clean Android environment the way Google envisioned it.

Closing it off

This was rather lengthy. I figured, switching the mobile OS platform should be worth an in-depth view. Hence this post. I hope you’ve enjoyed it.

Will I sell my iPhone 5? No. No. No. I never sold one. I’ll keep it. Maybe it’ll manage to win me back with iOS 7.

Looking forward to your feedback in the comments. Or on Google+, Facebook and Twitter.

IRC (Internet Relay Chat) has been around since 1988, which makes it ancient in Internet terms.

And although it’s still used by hundreds of thousands of users around the world, IRC has seen a dramatic downturn in usage.

We have talked to the creator of IRC, and others, about why the once so widely used technology has seemingly fallen out of favor with so many users.

The origins of IRC

We connected with Jarkko Oikarinen, the creator of IRC, who works at Google in Sweden, and he told us the story of how IRC was born.

The first IRC Server. Photo courtesy of Wikipedia.

Oikarinen says that he created IRC during three to four months in 1988 when he was a summer intern at the University of Oulu in Finland.

At the time, Oikarinen was maintaining a local BBS (Bulletin Board System) called OuluBox and the chat system there needed refreshing. While working on the updated chat system he also wanted to allow participants from the Internet who didn’t need to be logged in to OuluBox to participate in chat.

Thus, IRC was born.

Since then, IRC has served as an invaluable way of communicating for scores of users around the world. For almost whatever you’d like to discuss or get help with, there’s been an IRC network and channel that would serve your interests.

But since the arrival of the new century, IRC has dropped in popularity, with users moving to other forms of communication like the web and social media. We took a look at the numbers to see just how bad it is for IRC.

IRC has lost 60% of its users since 2003

It’s clear that overall IRC usage, both in terms of users as well as channels, has been in steady decline for many years. In fact, IRC has lost 60% of its users since 2003, a dramatic fall in numbers for any service.

Oikarinen attributes the decline in IRC to a trend of commercialization on the Internet.

“Companies want to bring users to their walled gardens,” he says, to ”keep the user’s profiles locked there and not make it easy for users to leave the garden and take their data with them.”

IRC’s distributed nature does not fit with the walled garden approach, says Oikarinen. So instead of supporting open communication tools like IRC, companies invest money in making their own solutions, he claims.

Christian Lederer, also known as “phrozen77,” is the webmaster of IRC-Junkie.org and he’s had his pulse on the IRC community for many years. According to Lederer, the decline in IRC usage has many possible reasons behind it:

• Lederer lists large and prolonged DDoS attacks in the early 2000s as one main reason behind the decline. The attacks disrupted many IRC networks, including the most popular ones, and crippled the chat experience for users. When the networks were back up again, many users had migrated elsewhere or abandoned IRC completely.

• Software piracy and the spreading of warez, is another reason Lederer points to. In the early days of IRC, finding such content was a major reason why some users connected to IRC networks. Over the years, users have found new and easier ways of obtaining warez, like P2P, resulting in less of a need for IRC.

• Social networking also played an important part of users fleeing IRC. With services like Facebook, Twitter, Linkedin, and others, users found it more convenient to communicate through the social network rather than logging in to an IRC channel.

• Finally, Lederer points to declining costs and increasing availability of cheap and reliable hosting. If someone disagreed with the way a network was run, they could suddenly start his or her own. In doing so, they could potentially take the channel they operated and their users with them, thereby decimating the numbers of the bigger networks even further.

So the decline in IRC usage is a complex issue with no straightforward answers. But it’s not all bad news, as we’ll see next.

Among top IRC networks, Freenode bucks the trend

If we look more closely at the top six IRC networks and chart their development since 2003, it’s clear there are winners and losers.

As you can see, QuakeNet, EFnet, and IRCNet have lost a lot of users, while DALnet and Rizon are floundering without moving much up or down.

But it’s not all doom and gloom in the world of IRC, however. Freenode.net is not following the typical trend. Rather it seems to be growing by leaps and bounds, up 486% from 2003 to 2012 in terms of users. According to Freenode’s blog, the network passed 80,000 concurrent connections on April 2, 2012.

In fact, Freenode has, according to these numbers, just become the number one IRC network in the world, just bypassing QuakeNet.

Christel Dahlskjaer, President of Peer-Directed Projects Center(PDPC), the organization that operates Freenode, explains the network’s growth with its focus on free and open source software.

“Freenode has indeed grown and continues to grow,” Dahlskjaer explains. “Freenode has never been a ‘traditional’ IRC network though. Our users tend to come to Freenode because they contribute to or use a free and open source (or other peer-directed) project that has a channel or more on the network. Then in turn other projects come to Freenode because there is a lot of overlap when it comes to users and contributors across the various projects.”

“In short, Freenode isn’t growing because it is Freenode or because it is IRC. Freenode is growing because of the projects that chose to make Freenode ‘home’ are growing,” Dahlskjaer summarizes.

On the question of whether Freenode’s current good fortune is sustainable, Dahlskjaer is direct. “I see no reason to think that growth is likely to stall anytime soon. For the last six years at least, Freenode has been very steady,” she says.

Where is IRC headed?

It’s clear that IRC is declining in overall usage but growing in certain niche areas. Perhaps that’s where the future of IRC lies.

Lederer says that IRC has to innovate to compete with easy-to-use solutions such as Facebook. This, in turn, is driven by a change in mindset of developers of IRC-related software, who have to drive this innovation, client-wise as well as protocol and server-wise.

To Oikarinen,“more interoperability” with other systems such as 3D virtual worlds, multimedia, etc. is one “interesting path forward.” Oikarinen is no longer actively involved in the development of IRC, but he says that it’s up to individuals now.

“It does not necessarily require a large team to make significant progress. Just one person can make a huge difference,” Oikarinen says.

Lederer makes a similar point, saying that some stand-alone clients are already pushing the boundaries of what is possible on IRC. He points to projects like KVIrc, which brought video chat to traditional IRC, as well as Konversation, with which several IRC users can share a virtual whiteboard.

Long live IRC

Although there’s no reason to think that IRC will disappear anytime soon, there’s also cause for concern about the future of the once so popular technology. Although Freenode can serve as an example of a growing IRC community, that, in and of itself does not mean the future is secure for IRC.

We at Pingdom recognize the tremendous value that IRC has brought to users around the world for many years, and hope that IRC will keep being widely used. In fact, we’ve just set up our own IRC server, which we have some exciting plans for.

Note about the data: We used the Internet Archive’s Wayback Machine to go back in time and look at how IRC has developed over recent years, using the data from SearchIRC.com. We’d like to point out that not all IRC networks are indexed by SearchIRC.com, the service we used to create the charts for this article. For example, Undernet is not a part of the SearchIRC index. We believe, however, that the general trend displayed by the SearchIRC data is correct.Top image via Shutterstock.

Text Editor Icons