I had a bit of email drama this week; Gmail started classifying half of my incoming legitimate email as spam. I got some great help from Gmail support who explained the problem and taught me how to properly forward email.

In detail, what happened… I get all my email to nelson@monkey.org, which I forward via procmail and SMTP to my gmail account. For some reason monkey.org recently got branded a possibly spammy domain. Because of my forwarding Gmail was under the impression that all my email was coming from monkey.org, so a bunch of it started getting marked as spam. The Gmail UI is a bit buggy in this circumstance; it was misidentifying which domain was the problem, telling me “we’ve found that lots of messages from gmail.com are spam” and the like when the real problem was monkey.org.

I fixed the problem by forwarding my mail properly. Gmail doesn’t just use the From: email header to identify the sender, it also uses the (normally invisible) From⎵ SMTP envelope. And because I misconfigured procmail, that header was always being set to nelson@monkey.org (since I was sending the mail). You can spoof the envelope too via -f, you just have to set it up that way. (Which makes me wonder why the spam filter pays any attention to it.)

It's a subtle problem; I only noticed it after several years. If you use procmail to forward to Gmail, you may want to look into your configuration. I believe most more ordinary forwarding mechanisms don’t have the envelope problem. Procmail is weird in that it’s generating new emails, not forwarding existing ones.

Check out this amazing popup Citibank gives you when you’re creating an account:

According to Citibank, a person’s name:

• Can’t be too long or too short
• Can’t contain a bunch of specific numbers
• Can’t contain spaces, apostrophes, hyphens

In other words, your name at Citibank can’t be anything like a person’s real name. Really it needs to be more like a password. But not too much like a password.

Welcome to our website. Your designation is THX_1138.

Yesterday’s leap second killed half the Internet, including Pirate Bay, Reddit, LinkedIn, Gawker Media and a host of other sites. Even an airline. Any Linux user processes that depends on kernel threads had a high chance of failing. That includes MySQL and many Java servers like webapps, Hadoop, Cassandra, etc. The symptom was the user process spinning at 100% CPU even after being restarted. A quick fix seems to be setting the system clock which apparently resets the bad state in the kernel (we hope).

The underlying cause is something about how the kernel handled the extra second broke the futex locks used by threaded processes. Here’s a very detailed analysis on the failing code but I’m not sure it’s correct. According to this analysis the bug was introduced in 2008, then fixed in March 2012. But it may be the March fix is part of the problem. OTOH most of the systems that failed will be running kernels older than March so the problem must go further back. There's a kernel fix and also a detailed analysis. Time is hard, let’s go shopping.

It’s frustrating that these bugs keep popping up; the theory is not so difficult. The NTP daemon tells the kernel a leap second is coming via adjtime(), the kernel should handle it by slewing or holding the clock, all is well. But it didn’t work in 2012. Didn’t work in 2009 either; a logging bug caused kernels to crash on the leap second. 2005 was better. Google’s solution of giving up on the kernel entirely and having the NTP daemon lie about what time it is seems more clever now.

I got hit by this bug myself, the CrashPlan backup daemon runs Java and got caught in a spin. And none of my machines really kept time right because POSIX does not account for leap seconds. Both Ubuntu boxes just ran 23:59:59 twice, so time went backwards on a subsecond basis. My Mac was even worse, it actually flipped over to 00:00:00 before going backwards to 23:59:59 briefly. No doubt my GPS devices are off by a second now; most consumer devices have no facility to update the leap second database. (Correction: GPS satellites broadcast the UTC offset.) The only thing that worked right was NIST’s clock widget pictured above, showing 23:59:60.

I have an increasingly bad feeling about Apple’s efforts to control what software runs on the computers they sell. Not just the business implications, but technical issues.

There’s two ways to get software onto a Mac: buy it directly from the developer or buy it via Apple’s Mac app store. App store versions are almost always worse; fewer features, more awkward updates. And now, with the sandbox requirements, total contortions.

I bought NZBVortex via the app store. Version 2.7 was recently released to be sandbox compatible. But for it to work completely you have to download a “helper application” that runs outside the sandbox. It’s a clever workaround, but it’s needless complexity and subverts the whole purpose of sandboxing.

I also bought Alfred, an awesome app launcher. At first I got the free version via the app store but that turned out to be a mistake, because you really want the (for pay) Powerpack but because of the sandbox restrictions the Powerpack is not available via the app store. The Alfred developers optimistically say “the Mac App Store is one of many ways to buy software for your Mac,” and encourage you to buy directly from them. Which works today.

But what if the Mac App Store becomes the only easy way to buy software for your Mac? Gatekeeper is a step down that road; you’ll still be able to run software from other sources, but only if it’s signed by an Apple-issued developer key or you personally disable a security setting. So far Apple’s not dictating policy on what key-signed code is allowed to do. But what if they decide to?

Meanwhile, over in Apple’s actual computer market, iPhones and iPads, total lockdown has been the rule since day one. And sometimes it goes badly. The most recent kerfuffle is over Dropbox signups; Dropbox has capitulated, removing some aspects of user signups to satisfy Apple’s market monopoly demands. And Apple has behaved poorly in the past, for example blocking Google Voice and Camera+ for ridiculous reasons.

I admire Apple. I sympathize with their desire to control the quality and security of software. But I don’t want to have to completely trust them to make the right decisions.

Just hit a weird problem on my iPad: trying to install an app or upgrade existing apps resulted in a mysterious "Error 1004". Apple's official help is useless and the user forums are full of amusing cargo cultism ("reset your time zone!")

The solution seems to have been flushing Safari's cookies. Or maybe its cache or history. One way or the other, the iPad App Store had the wrong idea about my login. And now I have to log in to everything again.

The Internet is at a dangerous inflection point. Facebook Connect is quickly creating a monopoly on identity. Sites are increasingly requiring Facebook logins now: Techcrunch comments and turntable.fm early access are two examples. And many more sites like TripAdvisor now promote Facebook over their own logins.

As a user the Facebook Connect experience is great. I see a familiar blue button, I click it, and I'm done. No creating an account, no coming up with a new username and password, no entering specific data. And it's not just a login, many Facebook integrated sites give me a better experience with access to my Facebook social network. For site owners the advantage of Facebook connect is clear: good user experience, less code to manage, and access to Facebook data.

The problem is that Facebook is creating a monopoly. That's a huge risk to every other company on the Internet. It's bad for users too, we're losing the ability to use pseudonyms online. And while Facebook's technical execution is excellent the company has demonstrated over and over again its willigness to act unethically towards their users. We don't want them controlling user identity.

There is a terrific technical alternative to Facebook Connect: OpenID. The tech works well and it's open, letting users and companies choose their identity provider. But despite some four years' headstart it's never succeeded in being adopted widely like Facebook Connect has. And while I like competing login systems like Sign in with Twitter, identity is too important on the Internet to let any proprietary solution dominate. Our ecosystem needs a productive open standard. I still think OpenID is sufficient, but I'm in a dwindling minority.

Apparently it's news to almost every web developer out there, but in the real world people's names have spaces in them. My name is "Nelson Minar". It is not "Nelson_Minar" or "NelsonMinar" or "NelsonM" or "Nelson397" or any of the other nonsense I have to use to work with some website who's decided to constrain names to some 1980s software-friendly character subset.

The hardest part of signing up for a new site these days is picking a unique user name. It's annoying to have to remember different names. And it's really obnoxious when my janked up UserName is also used as my display name. The right way to do logins right now on the Web is use email address as the login name and let the user choose their own display name which does not need to be unique. That's not ideal (email addresses can change) but it works pretty well. If you absolutely have to not use email as the login name, please at least let my login name have a space in it.

While I'm delivering the news, here's something for you ignorant American backwoods motherfuckers. Some people's names have "special characters" in them. Like François Rabelais or Björk Guðmundsdóttir or 艾未未. It's 2011; the only software that can't handle Unicode properly is Perl. (As if you needed another reason not to use Perl.) Stop limiting your code; there are only two languages that can even be written in ASCII.

Another Sony hack yielded a database of 1,000,000 plaintext passwords. Why does Sony have plaintext passwords? Because they're idiots and deserve to suffer a civil lawsuit. But Sony's negligence is security researchers' gain: check out this analysis of the password haul. The most astonishing result:

Two thirds of people with accounts at both Sony and Gawker reused their passwords.

Passwords are a broken mechanism of authentication. They are weak, dangerous, and difficult for naïve users to use correctly. It's time to end passwords.

The US military is on the Internet march. Obama said of two new military appointments "Between them they bring deep experience in virtually every domain ... Land, air, space, sea and cyber." Also today the WSJ reports "The Pentagon has concluded that computer sabotage coming from another country can constitute an act of war, a finding that for the first time opens the door for the U.S. to respond using traditional military force."

I used to think "cyberwarfare" was an inflated threat. But in the past couple of years my thinking has reversed. Stuxnet was a big deal, presumably a deliberate act of cyberwarfare by the US and Israel against Iran. China's hacking of Google was also a big deal; aimed at individual activists, not nations, but still important. The recent attacks against RSA SecurID and now Lockheed Martin are troubling. We're beyond script kiddies stealing some Warcraft accounts, this is focussed espionage against the US military.

One of the problems with cyberwarfare is it's not clear how to apply international law. What is "proportionate response" to a network break-in that disables a radar installation? How do you even identify an attacker when the attack was a virus that was planted six months ago on USB sticks? These statements says the Pentagon is moving to take these questions seriously. It's about time.