How a banner ad for H&R Block appeared on apple.com—without Apple’s OK | Ars Technica

The ghost of Steve Jobs will not be pleased to see this. Zack Henkel

Robert Silvie returned to his parents' home for a Mardi Gras visit this year and immediately noticed something strange: common websites like those beloning to Apple, Walmart, Target, Bing, and eBay were displaying unusual ads. Silvie knew that Bing, for instance, didn't run commodity banner ads along the bottom of its pristine home page—and yet, there they were. Somewhere between Silvie's computer and the Bing servers, something was injecting ads into the data passing through the tubes. Were his parents suffering from some kind of ad-serving malware infection? And if so, what else might the malware be watching—or stealing?

Around the same time, computer science PhD student Zack Henkel also returned to his parents' home for a spring break visit. After several hours of traveling, Henkel settled in with his computer to look up the specs for a Mac mini before bedtime. And then he saw the ads. On his personal blog, Henkel described the moment:

But as Apple.com rendered in my browser, I realized I was in for a long night. What I saw was something that would make both designers and computer programmers wince with great displeasure. At the bottom of the carefully designed white and grey webpage, appeared a bright neon green banner advertisement proclaiming: “File For Free Online, H&R Block.” I quickly deduced that either Apple had entered in to the worst cross-promotional deal ever, or my computer was infected with some type of malware. Unfortunately, I would soon discover there was a third possibility, something much worse.

The ads unnerved both Silvie and Henkel, though neither set of parents had really noticed the issue. Silvie's parents "mostly use Facebook and their employers' e-mail," Silvie told me, and both those services use encrypted HTTPS connections—which are much harder to interfere with in transit. His parents probably saw no ads, therefore, and Silvie didn't bring it up because "I didn't want [them] to worry about it or ask me a lot of questions."

Henkel's parents had noticed the ads but assumed that companies like Apple and Walmart had put them there on purpose. "They were very surprised" to find out that the truth, Henkel told me.

Neither Silvie and Henkel were going to let a mystery like this go without solving it. Each embarked on a separate investigation and each came to the same conclusion: their parents' Internet provider was somehow involved in slapping ads onto webpages as they moved over the network.

Enlarge / How did that ad get there? Zack Henkel

Paging Sherlock

Both homes subscribed to Internet access from CMA Communications, a rural cable TV, Internet, and phone provider serving southern states like Texas and Louisiana. (CMA is owned by ETAN Industries; according to Bloomberg Businessweek, ETAN does business as "Credit Protection Association, LP" and "provides collection services.") But it was possible that CMA wasn't involved with the ads; locally installed adware might have been responsible, or the two sets of parents might have had their routers infected by a rare breed of malware.

To rule out the various options, Henkel isolated each link in the chain between his devices and the remote Web servers he was contacting. After seeing the ads appear on multiple websites, Henkel switched to his Android-powered phone to see if some kind of malware was affecting his personal Mac; the ads also appeared on the phone. He accessed websites from a Surface tablet; the ads were there. "I am not great at statistics, but I was fairly certain the probability of identical malware on all of these devices was low," he wrote on his blog.

He turned off Wi-Fi on his phone in order to force its data connection to route through the separate cellular network; the ads went away. He turned Wi-Fi back on and the ads reappeared. Local malware wasn't causing them, and they didn't exist when accessed through a different network, so they certainly weren't meant to be on the sites at all. The ads appeared to be injected either by his parents' router or by their ISP.

"I pulled up the Web inspector in Chrome and examined the source of a page which had the ad," Henkel wrote. "Appended to the very end of the HTML file for the webpage was a single line which called to r66t.com for a JavaScript file."

That single line of code read <script type="text/javascript" src="http://nodes.r66t.com/node_api/eeb77492-852f-11e2-af82-12313d316a64/entry/main.js"></script> and appeared to be the source of the issue. And it turned out that the R66T code didn't just add banner ads to sites that had none; it even overwrote its own ads onto high profile sites like the Huffington Post, which had plenty of ads of their own.

To see if his parents' router had somehow been compromised, Henkel plugged in a spare router and ran his connection through it. Same result. He then ran a series of traceroutes to see every hop his packets took on the way to sites like Bing.com. He was able to compare the results to traceroutes he had run before, since "poor performance has been a recurring issue" with the connection, he said. He found an extra hop in the connections now, one that passed Web requests through a Squid proxy server run by a R66T, where they were apparently altered to include the extra ad code.

"Wow, this is really wrong and crazy," Henkel told me, since it suggested that companies felt free to operate as a "man in the middle," one free to inject code of their own choosing into webpage requests that were—so users believed—simply between themselves and the websites they were trying to reach.

Silvie had a similar reaction. He used the traffic inspection tool Fiddler to examine his packets and "saw that the ads were coming from r66t.com only when the website was not being served over [the encrypted] HTTPS," he told me. But who or what was R66T?

Get your kicks on R66T

R66T, pronounced "Root 66" and intended as a play on the famous American highway Route 66, describes itself as "one of the nation's leading publisher of targeted content, information and advertising to private Wi-Fi and High-Speed Internet Access (HSIA) networks, conducting tens of millions of individual user sessions—approaching one-billion user-minutes per month." The company says that it supports Wi-Fi networks at places like airports, hotels, coffee shops, and malls, often providing free access in exchange for showing "hyperlocal" advertisements.

One of their product pitches, for instance, describes a hotel Wi-Fi system that creates an "ever-present toolbar or frame around the page that can display relevant content and information" and can handle "insertion of property related ads and promotional messages, as well as blacklisting competitors' ads throughout the entire Web session." A further item says that property managers can "feel secure in knowing that your competitors are not able to poach your customers while they are at your property."

This sounded similar to, but importantly different from, what Silvie and Henkel were seeing. For one thing, both of their parents were paying for Internet access and had for some time. For another, the ads they were seeing actual sat right on top of actual content, not in a frame or toolbar.

In its privacy policy, R66T says that it partners with Internet providers and that end users should love the service it provides. "This greatly enhances each user's online activity by providing an enhanced Internet services experience with advertisement overlays," says the policy.

Silvie quickly blocked all access to R66T domains on his machine and the ads disappeared; Henkel did likewise, but his anger drove him further. He suspected that CMA had partnered with R66T in an attempt to make more money from each Internet connection, but he couldn't prove it.

Enlarge / Part of R66T's pitch to Internet providers.

Henkel called CMA tech support and talked to several people. "None of them really believed what I was saying," he told me. CMA's privacy policies do give the company broad latitude to collect information such as "Web sites you view," the time that you view them, and "other information about your 'electronic browsing.'" They do not appear to say anything explicit about altering Internet traffic in flight, however.

Henkel filed a complaint with the Federal Communications Commission (FCC). On March 19, he received a polite note back from the agency, telling him that the issue "does not come under the jurisdiction of the FCC" and that he might want to contact the Federal Trade Commission instead.

Frustrated, he took to his personal blog and then to reddit to generate interest in the story, but even on reddit he could only find three other people who had noticed the ad issue and had Internet access through CMA.

On CMA's "About us" page, the company helpfully provides an e-mail address for comments and says that each message "is routed directly to CMA’s Vice President for initial review. Yes, we do read every email received." I reached out to them for comment; they have not responded.

In response to some direct questions about whether R66T was working with CMA to inject ads over common websites, R66T boss Mick Hall replied by e-mail that he was currently in Mexico City, had a heavy meeting schedule, and had to deal with a "current poor Internet and telecommunications infrastructure" that made getting me more information impossible at the moment. However, he promised to speak more next week.

R66T's logo... with a pretty girl.

A question of trust

ISP-sanctioned ad injection directly into webpages—if that's what this is—has a long history, but it has been fairly rare to find it happening on pay connections in the US.

Back in 2008, computer scientists at the University of Washington built a "Web tripwire" that could detect in-flight changes to content served over HTTP, and they recruited 50,000 unique users to help them test it out. At the time, 1.3 percent of the connections altered webpages in some way, though most of these were caused by local software, such as popup blocking tools. Only 46 of the 50,000 IP addresses showed modifications that appeared to be "intentionally caused by their ISP." A mere 16 of these were actually performed by an "ad injector" like NebuAd somewhere in the network path. (Notably, systems like NebuAd proved so unpopular with both government and consumers that the company went out of business.)

The practice of code injection attained new public prominence in January 2011 when it it became apparent that the Tunisian government, facing a countrywide revolt, was injecting a bit of password-grabbing code onto Facebook's login page whenever it was requested by a user inside the country. Facebook eventually addressed the issue by making the site accessible over HTTPS—though, as the authors of the 2008 paper note, HTTPS can be a "rigid and costly" solution.

Early this year, one angry Comcast customer claimed that the giant Internet provider was injecting its own code into webpages—but only for the purpose of displaying a "courtesy notice" that a data usage threshold was near.

Are CMA and R66T reviving ad injection on pay connections to residential users? Only those two companies can explain exactly what the situation is, and neither has been over-anxious to respond immediately to questions.

Whatever is happening, the situation has caused a bit of concern for Henkel's parents. "They were surprised that this could even be done," he told me. "They don't know who's sending them what now."