The Internet mostly survived the leap second two days ago. I’ve seen three confirmed problems. Cloudflare DNS had degraded service; they have an excellent postmortem. Some Cisco routers crashed. And about 10% of NTP pool servers failed to process the leap second correctly.

We’ve had a leap second roughly every two years. They often cause havoc. The big problem was in 2012 when a bunch of Java and MySQL servers died because of a Linux kernel bug. Linux kernels died in 2009 too. There are presumably a lot of smaller user application failures too, most unnoticed. Leap second bugs will keep reoccurring. Partly because no one thinks to test their systems carefully against weird and rare events. But also time is complicated.

Cloudflare blamed a bug in their code that assumed time never runs backwards. But the real problem is POSIX defines a day as containing exactly 86,400 seconds. But every 700 days or so that’s not true and a lot of systems jump time backwards one second to squeeze in the leap second. Time shouldn’t run backwards in a leap second, it’s just a bad kludge. There are some other options available, like the leap smear used by Google. The drawback is your clock is off by as much as 500ms during that day.

The NTP pool problem is particularly galling; NTP is a service whose sole purpose is telling time. Some of the pool servers are running openntpd which does not handle leap seconds. IMHO those servers aren’t suitable for public use. Not clear what else went wrong but leap second handling has been awkward for years and isn’t getting better.

  2017-01-03 00:38 Z

I ran into an awkward problem in Europe; I couldn’t get SMS messages. It’s a design flaw in Apple’s handling of text messages, its favoring of iMessage over SMS. If you turn data roaming off on your phone when travelling, you may not be able to get text messages reliably.

If you have an iPhone suitably logged in to Apple’s cloud services, other iPhones (and Apple stuff in general) will prefer to deliver text messages via iMessage instead of SMS. You see this in the phone UI: the messages are blue, not green. In general iMessage is a good thing. It’s cheaper and has more features.

The problem is Apple’s iMessage delivery requires the receiving phone have an Internet connection via WiFi or cellular data. If you have no WiFi at the moment and have data roaming turned off, your phone is offline. And so Apple can’t deliver to you via iMessage. They seem to buffer sent messages for when you come back online. Which is too bad, because your phone could still receive the message via SMS. Unfortunately iMessage doesn’t have an SMS delivery fallback.

In practice this design flaw meant I had to leave data roaming turned on all the time because I needed to reliably get messages from another iPhone user. Which then cost me about $30 in uncontrollable data fees from “System Services”. Some $15 was spent by Google Photos spamming location lookups (a bug?), another $15 receiving some photo iMessages from a well-meaning friend. Admittedly the SMS fallback I’d prefer would also cost some money, but I think significantly less in my case.

There’s a broader problem with iMessage which is that once a phone number is registered with it, iPhones forever more will not send SMS to that number. Apple got sued over this, so now they have a way to deregister your number.

  2016-10-13 14:29 Z

The world has had its first self-driving car fatality: a Tesla autopilot failed. So far the world hasn’t freaked out. I think self-driving cars will be way safer than human-driven cars. But there’s a lot of shaping the truth in Tesla’s announcement.

(Fair warning: this blog post is uninformed hot take territory. I’m reacting to Tesla’s description of the crash, published two months after the death. We’ll know a lot more after an independent investigation.)

Tesla’s press release is masterful. It characterizes the cause of the accident like this:

the vehicle was on a divided highway with Autopilot engaged when a tractor trailer drove across the highway perpendicular to the Model S. Neither Autopilot nor the driver noticed the white side of the tractor trailer against a brightly lit sky, so the brake was not applied.
A truck pulled out in front of the car on the highway. It may well have been an unavoidable accident. We’ll know eventually.

But note the facility of claiming the “driver” didn’t notice the truck. How do we know that? The man is dead, we have no idea what he saw. I don't know about you, but I've never once failed to spot a white truck against a bright sky, particularly when I'm driving towards it at 70mph. I could see how a computer vision system would fail that test though.

“The brake was not applied”. It takes time to apply the brakes after you see your death coming at you. Doubly so if you’re not actually driving. The passenger-behind-the-wheel was almost certainly not having his foot hovering gently near the accelerator / brake like an engaged driver would. That slows reaction time. I do this all the time with my simple cruise control and it scares the hell out of me when some slow jerk pulls in front of me and I don’t react quickly.

(I also admire the comfort of “he never saw it coming”. Sort of takes the sting out of the next sentence, which describes the unfortunate’s grisly decapitation.)

The real problem here is Tesla’s autopilot is a half measure, “driver assist”. It doesn’t fully drive the car. This design is the most dangerous of all worlds. I had this experience with my airplane’s autopilot all the time. At some point when the automation does enough work, you can’t help but check out mentally, let the machine take over. But if the machine isn’t capable of taking over entirely you can end up dead.

That’s why I’m in favor of fully autonomous vehicles. No steering wheel, no accelerator, maybe just a single brake or other emergency cutout. Of course in this situation the software has to work reliably. Let's say a fatality rate of 50% of human drivers. And insurance and the law have to adapt to this shift of control to software. I believe the technology nerds are very close to having systems that can fully drive a car with no “driver assist” ever needed, at least in clear weather. It will be a better future. And those robot cars will kill some of their passengers. Far fewer than humans are killing now.

  2016-07-01 16:10 Z

The Economist infected readers with malware. The vector? PageFair, a technology for web publishers for circumventing ad blockers. The payload was a remote access tool. Goodbye bank accounts!

This is outrageous. I install software on my computer to block ads, a clear statement of user preference. The Economist colludes with PageFair to ignore my choice, to run software on my computer that I explicitly don’t want. That software they run turns out to be installing malware.

The folks who write things like PageFair need to be sued into oblivion. Not just the company; stop the people who built this abusive technology from ever creating software again.

  2015-11-07 18:35 Z

The addition of ad blocking capability to iOS has brought on a lot of hand-wringing about whether it’s ethical to block ads in your web browser. Of course it is! Blocking ads is self preservation.

Ad networks act unethically. They inject huge amounts of garbage making pages load slowly and computers run poorly. They use aggressive display tricks to get between you and the content. Sometimes negligent ad networks serve outright malware. They violate your privacy without informed consent and have rejected a modest opt-out technology. Ad systems are so byzantine that content providers pay third parties to tell them what crap they’re embedding in their own websites.

Advertising itself can be unethical. Ads are mind viruses, tricking your brain into wanting a product or service you would not otherwise desire. Ads are often designed to work subconsciously, sometimes subliminally. Filtering ads out is one way to preserve clarity of thought.

I feel bad for publishers whose only revenue is ads. But they and the ad networks brought it on themselves by escalating ad serving with no thought for consumers. The solution is for the ad industry to rein itself way in, to set some industry standards limiting technologies and display techniques. Perhaps blockers should permit ethical ads, although that leads to conflicts of interest. Right now Internet advertisers are predators and we are the prey. We must do whatever we can to defend ourselves.

  2015-09-21 02:24 Z

The Apple Watch, that fancy new high-tech product, had a XXE vulnerability. That’s a security flaw in XML parsers that dates back to at least 2002. It keeps popping up in new products because apparently safe XML parsing is too hard for anyone to get right.

  2015-05-19 22:12 Z

I love the Internet service I get from Astound, but not so much their online account system. The billing login is pretty screwed up. Things to know:

  • The billing credentials are separate from your “Internet Account Manager” credentials.
  • Username and password will both be forced to lowercase.
  • The password recovery system will mail you your password in plaintext.
  • Parts of the online system ask for a registration password. That password is “astound”.
  • Astound formats account numbers in three fields, like 005 0123456 01. The form presents two boxes: try 005 012345601
  2015-01-20 18:25 Z

Riot’s hugely popular game League of Legends is still installing malware, some five months after saying they don't use it, players can delete it, and they planned to remove it.

The malware in question is Pando Media Booster. A few years ago this software was arguably useful, it allowed games like LoL to distribute patches via a peer-to-peer network. But Pando was discontinued in August 2013. Then in February 2014 someone used Pando to install malware on any suckers who still had the software. The software Riot is still distributing. And all of Riot’s customers who clicked “yes” on the update dialog had their browsers hijacked.

Riot has millions of users all over the world. I’m sympathetic to how hard it is to make software changes; they’re famously behind on a whole lot of development projects. But continuing to distribute malware to customers is unacceptable.

Update: a Riot employee said on Reddit that the problem was "the amount of work it takes to hand update new installers for every language" and offered the idea that the previous Pando owners might help them prevent the malware. That was five months ago.
  2014-08-02 18:03 Z

Interesting report of stolen Bitcoins, a phishing scam involving a Google ad. I just confirmed that the phishing ad is still running on Google on a search for blockchain.

The ad says the URL goes to The URL displayed on mouseover on the link is to a Google redirector, But when you click the link you go through a few redirectors and end up at, which is allegedly a phishing site. It sure looks suspicious; that page goes straight to a “type in your password” page, which the real site hides behind several clicks.

I don’t much care about the Bitcoin part of this, but Google should really not be selling ads with fake URLs on display.

  2014-06-07 15:45 Z

There are two terrible web properties out there that everyone hates, Scribd and Quora. Please don’t use them. Instead of Scribd just host a PDF anywhere, or upload text to pastebin or make a nice blog on WordPress or Medium or something. And instead of Quora use Ask MetaFilter or StackExchange.

Scribd’s business model is to host documents in formats that are unusable. For instance, here’s a copy of the Declaration of Independence. Or rather, the free preview; you have to download it to read the rest and a one-day guest pass costs $9. Here’s a copy of Elliot Rodger’s insane manifesto. It starts “This is the story of how I, Elliot Rodger, came to be.” Only I had to retype that phrase; if I copy-and-paste I get “]fjs js tfh stgry gl fgw J, Hccjgt Tgmahr, eknh tg dh” because Scribd uses some stupid DRM font. Easy enough for a pirate to reverse engineer but impossible for normal use. They also broke “Find”; there’s some Javascript thing overriding the browser that doesn’t seem to work.

Quora’s business model is to trick people into sharing information for free, then put it behind a login. It’s like Experts Exchange 2.0! For instance, on Quora you can read Who owns the copyright on content contributed to Quora? Only you can’t just read the text. Depending on your history with the site and the way you got there you may see a giant popup demanding you log in obscuring the page, or the first answer clear and then the rest blurred, or if you're lucky just the page. It appears nondeterministic.

Both businesses are deliberately trying to lock up text content to make it harder to access, to force users to pay or share advertising data or some such bullshit. The part that kills me is some engineer actually wrote code to deliberately break document sharing on the web. It’s terrible.

Update: the Quora CEO responded on Hacker News to correct me that Quora neither runs ads nor charges users. At the moment, they apparently have no revenue.
  2014-06-02 20:23 Z