Hello from Singapore! It’s exactly like everyone said, a modern and somewhat synthetic city that’s Western in its administration and Eastern in its culture. I like it, but it’s odd being in a new city (1819) without a long history. Then again the syncretic culture that’s here now is neat, the mix of Chinese, Malay, Indian, British, all fused into something uniquely Singapore. And so commercially ambitious.

Every Singapore person I asked for tourist advice said “go eat”, particularly at the famous hawker centers. They’re like food carts but with guaranteed hygiene and centralized convenience. There’s three we found nearby: Maxwell Center near Chinatown, Tekka Centre in Little India, and Lau Pa Sat in the center. I definitely enjoyed them but it’s a pretty grimy and simple experience, like going to your favorite taqueria in San Francisco. Buy a beer from one stall, some dumplings from another, maybe a paratha or some spicy noodles from others. Bring your own napkins.

Hawkers are great for tasty food for cheap. Singapore also has an amazing deep restaurant culture reflecting its international position. Various kinds of Chinese and Malay are the main foods you see here, but there’s also lots of Indian and of course Singapore specialities like chili crab. And then a vast international mix, like the warren of French wine bistros on Ann Siang Hill.

The other remarkable thing I’ve seen in Singapore is the Singapore City Gallery near Maxwell Food Center, part of the urban planning department. The permanent display of a scale model of the city is interesting (and free, and air conditioned). But even better was the temporary display of the Draft Master Plan 2013, Singapore’s ambitious plan to develop their island more, building new communities and spaces. Singapore is in a unique position as a wealth city-state and they’re taking their development planning seriously. Interesting to see.

RSA Security (part of EMC) was one of America’s most respected security companies. Thanks to Edward Snowden, we now know the price of their reputation: $10 million. For that tiny sum RSA sold out their customers, deliberately installing a compromised random number generator in their core security library BSafe at NSA’s request. For $10M, a company’s reputation destroyed.

The nature of NSA’s sabotage is worth looking at in detail. We knew back in 2006 that Dual_EC_DRBG, a NIST standard crypto random number generator, was fishy. That algorithm has baked into it an arbitrary constant; two Microsoft researchers figured out that if an adversary had chosen that constant, then the numbers were predictable and any system built on it was insecure. Snowden’s leaks confirmed in Sep 2013 that this backdoor had been placed. And now in Dec 2013 we know the price: $10M. (Interestingly, one old-school cypherpunk knew the price back in September).

It’s worth noting that RSA’s complicity with NSA is not their only enormous security black eye. Back in 2010 their flagship SecurID two factor login system was also widely compromised, it’s assumed by the Chinese government trying to get military and commercial access to US and European interests.

Open source ends up looking good in all this mess. NSA has probably attacked other random number implementations. There was a weird push from Intel to get Linux to completely trust their undocumented hardware generator, something resisted by the Linux team (thankfully). And OpenSSL, the open alternative to RSA’s library, doesn’t use the compromised algorithm (although their code has had its problems).

I remain indignant that NSA is willfully going around deliberately sabotaging the security of core Internet components. Even if you believe it’s good for NSA themselves to be able to break all encryption, it is so dangerous to have back doors like this hiding in systems. NSA is actively undermining everyone’s security.

Hello from Bali! Ken and I have been here most of a week and are having a marvelous time. The combination of tropical beauty, friendly people, and deep culture makes Bali amazing. We’ve been staying in lovely resort hotels which is great but I regret not getting more into the villages and towns and experiencing more regular life. It’s very hot and difficult to get around, so far we’ve been taking private cars from the hotel.

Our first few nights were in Jimbaran Bay, down south near the airport. Beautiful bay, clear and gentle and great sunsets. The Bali Intercontinental was great, particularly the extra amenities with Club access. I’d characterize it as the beach + resort part of Bali, lots of emphasis on swimming and massages and relaxing. We did get out a little, particularly to enjoy the Kecak Dance in Uluwatu and to go down the road for a nearby resort’s excellent Indonesian restaurant.

Now we’re in Ubud, the arts & crafts center. Balinese culture has such depth in music, dance, decorative arts, fine painting, there’s just a huge amount to explore and happily it’s all vibrant and available. Our first day here was spent being taken from shop to shop, large warehouse-style galleries of stone carving, paintings, jewelry, etc. It’s definitely touristy, tourist money helps sustain the economy. But it’s also deep and rich and with an authenticity of hundreds of years that some vulgar visitors can’t disrupt.

Yesterday was more of a high arts experience. Through an American friend we met Dewa Alit, a gamelan composer from a family of musicians. We visited him for a lesson in the very basics of gamelan with me clumsily trying to learn to play a few patterns. He’s an internationally known musician so I feel a bit guilty spending his time on something so rudimentary, but he was generous and patient and I got a huge amount out of it. Alit doesn’t do this kind of thing regularly; some enterprising Balinese could make a fine business teaching gamelan workshops for tourists. We also visited the Agung Rai museum of Balinese painting, with collections mostly from the 1940s to contemporary art. Fantastic stuff and I know nothing at all about this genre and would love to know more.

So much more to see, we didn’t even get into the religious culture and temple festivals. Missed the cockfighting, too. I’m hoping to get out into Ubud today and just walk around the shops at my own pace. but then it’s pouring rain and will be 95° and the Four Seasons Sayan is awfully comfortable. There’s so much to learn about Bali, I could easily spend a month here.

Update: last day was a drive to an art gallery, a horrible traffic jam drive through the forgettable tourist dross on Monkey Forest road in Ubud, and then an amazing visit to the home of artist Ketut Soki. We’d seen his work in shops but it seemed awfully expensive without knowing more; he’s a master artist and the quality is visibly better than the cheap souvenir stuff. Our awesome driver Korta offered to take us to the artist’s home to buy a painting direct from the artist without the 100% gallery markup. Really great experience and I can’t wait to get this beautiful painting on my wall.

AOL, Facebook, Google, LinkedIn, Microsoft, and Yahoo joined today to ask for Global Government Surveillance Reform. While asking nicely isn’t likely to accomplish anything on its own, the public statement does move the discussion forward.

The request is sincere. I personally know people at Google and Twitter who’ve shaped their policies and they have just as much of a liberty / freedom of speech / Internet nerd background as you could hope. And all of these companies have a commercial interest in limiting spying to protect their international businesses. For example, NSA spying on Gmail is a significant threat to Google’s business plans in Europe. They have a self-serving reason to want to stop NSA overreach.

My concern is they don’t have any real hope of succeeding in what they’re asking. This request strikes me as particularly naïve: “governments should limit surveillance to specific, known users for lawful purposes, and should not undertake bulk data collection of Internet communications.”. Bulk data mining is a hugely valuable intelligence asset, there’s no way NSA is just going to stop collecting everything they can just because we ask nicely.

I think the requests for oversight and transparency are more realistic. Since the Snowden documents started coming out I’ve wavered between being appalled by NSA’s contempt for the Constitution and impressed by their technical capabilities. I accept at some level the US government needs an agency that, say, is hoovering up everyone’s cell phone movements as a way to track bad guys. The problem is that it’s illegal for NSA to collect that data against US citizens. And the current workaround for the Fourth Amendment is a ridiculous system with no meaningful judicial or legislative oversight.

The current state of surveillance threatens democracy; it has to change. The good news is some of America’s most powerful corporate interests also want to fix it.

YouTube Center is good software. It’s an unofficial browser extension to make YouTube work better. Works in most browsers; for Chrome you download the Opera .crx file and drag it into the Tools/Extensions page.

What does it fix? #1 thing is it lets you disable DASH playback, the nonsense YouTube implemented a couple of years ago. In theory DASH makes videos play faster and more efficiently; in practice it’s the crap that makes it impossible to pre-buffer a video or seek backwards while playing. YouTube Center also does a good job at resizing the video window to use more of the screen, so that a 720p video actually has a 720 row high window to play in. I also use it to prevent auto-play and to select the video resolution I prefer.

The main drawback is that there are too many configuration options, many of which you don’t need. Classic hackerware; the author lets you configure everything, so it’s up to the user to tune the few things they really need to set.

I’ve used a few “fix YouTube” extensions in the past that were flaky or broke when YouTube changed something. This one seems to be working for me. I don’t understand why Google’s let their video product get so crummy that it’s necessary to hack it like this.

Camino Restaurant in Oakland is one of my favorite restaurants in the Bay Area. I’ve been there a few times, I think every time with Marc, and every time the meal has been excellent. Worth a trip over the bridge for.

Last night’s dinner was typically great. Dungeness Crab legs, broiled on live fire with a lovely spice coating (alas, served in shell, but it’s literally the first crab of the season). Then a perfectly cooked bit of chicken three ways; moist breast, a sort of smoked leg, and a ballotine of delicious bits with strong seasoning. A little bitter greens, a little rustic grain (farro?) to catch the sauce, simple and refined. I even had dessert, a dense little persimmon pudding with just a bit of quince for sweetening, very savory and satisfying. Excellent cooking, well balanced.

Art of Eating had a profile of Camino a couple of years ago (issue #89) that I can loan you a copy of if you’re really curious. The article’s focus is on their cooking with live fire, which is indeed quite homey in the open kitchen. But while the technique impresses me I think its true value isn’t in the smoke but rather in forcing the chef to be attentive and careful to every single dish. Combine with excellent ingredients and a sense of what makes a delicious, restrained meal and it’s good dining.

Camino is run by Russell Moore and Allison Hopelain. It’s on Grand Ave in Oakland. You need a reservation.

A bit of nostalgia today for Netrek, one of the best online games ever. It’s from the early 1990s and is an important game design precursor to team based online games. Also its netcode was a huge breakthrough in real time Internet gaming.

The game design is brilliant. It’s an 8v8 team game. You mostly play in the upper left window, a Spacewars-like game where you fly your spaceship around and zap other players with your phasers and torpedoes. But the real game is in the upper right, the galactic overview map. The goal is to fly to planets and take them over by beaming down armies while fighting off the enemy players. That combination of high level strategy and local tactics is a hallmark of RTS games like Starcraft, MOBA games like League of Legends, and squad FPS games like Battlefield. I’m not saying Netrek invented that whole idea (Netrek itself was based on PLATO Empire), but it took 5–10 years before mainstream games became as interesting as Netrek. There were even classes in the game, different types of spaceships for different roles.

The network code was also hugely innovative, particularly the UDP code from 1992. Back then the Internet was overloaded and slow, 56 kbit/s links were common. Andy McFadden rewrote the original TCP netcode to use UDP and suddenly the game became way more playable on congested links. The key insight is UDP lets the game client decide what to do about packet loss rather than relying on TCP retransmits. Netrek could afford to lose the occasional packet; you might not see a torpedo coming your way but then again you didn’t have to wait 3 seconds for that packet before seeing the 25 other torpedoes launched afterwards. Weirdly most contemporary games use TCP (despite drawbacks), although League of Legends at least is UDP.

Netrek partly benefitted from the great community of the academic Internet of the early 90s. I’ve run into a few old Netrek buddies in our later careers as working software people: Andy McFadden and Jeff Nelson at Google and Stephen von Worley of DataPointed. I wonder if any of the Netrek folks went on to work in the gaming industry?

Here’s something ugly, the whois response for pirate book site readanybooks.net. Below is an extract of the interesting parts that both MacOS and Debian’s whois display.

$ whois readanybooks.net

   Domain Name: READANYBOOKS.NET
   Registrar: XIN NET TECHNOLOGY CORPORATION
   Whois Server: whois.paycenter.com.cn
   Name Server: RICK.NS.CLOUDFLARE.COM

Billing Contact:
  Name           : li xiaoing
  Email          : jr361005@126.com
<script src=
  "http://img2.xinnet.com/d/js/acmsd/thea178.js">
  </script>&nbsp;

Huh? What’s an HTML tag doing in this whois response? And under what circumstances might that script tag be executed? I can imagine a naïve Web interface just injecting that script wholesale into my browser. Every way I load the referenced script it seems benign (right now), but that’s an attack vector waiting to happen.

The Elgato Game Capture HD is good hardware. For $150 it captures HDMI video and audio from a game console and writes it to your computer’s hard drive. I bought it because Grand Theft Auto V was so astonishly beautiful I wanted to capture some of what I was seeing. There’s nothing particularly game-specific about the product, I think it’d work to record any unprotected video source.

The device is an HDMI passthrough. HDMI in, HDMI passed through (no delay), video also compressed and sent via USB to a computer with (few seconds delay). The native output format is an MP4 container with H.264 video and AAC stereo audio. The capture software is remarkably good; simple capture controls and live streaming to sites like Twitch. There’s even an easy little editor for extracting excerpts and uploading to YouTube or whatever.

There are a few drawbacks. The device doesn’t seem to support surround sound and only allows stereo input, so no surround sound is possible via HDMI. Also it has to be powered even to pass through video. Between those two hassles I don’t feel like I can leave my game console plugged into it all the time, so instead I’m swapping cables when I want to use it. Also it can’t quite do 1080p at 60fps, not a problem quite yet but soon to be one.

Still for $150 it’s a pretty capable video encoder. If you need a cheap way to capture HDMI, it’s worth a look.

NSA has been illegally copying all internal Google and Yahoo traffic. Apparently Google’s move to encrypt their internal traffic was well motivated. But what’s really astonishing is the slide from NSA explaining the program.

Yes, it’s a fucking smiley face. “Lol 4th amendment jk”.

There’s been so many revelations about NSA’s spying this summer it’s hard to make sense of the big picture. But for me, there’s two big things. One: NSA is wholly (and illegally) spying on US citizens in direct violation of their mandate and the Constitution. Two: NSA is remarkably competent at spying, using big data and Internet technologies effectively. A competent spy agency with no effective public oversight fundamentally threatens American freedoms.

Ken and I just took a nice trip to Germany, focussed mostly on the northeastern corner on the Baltic Sea. Lovely trip, very mellow, here’s a bunch of photos.

The biggest revelation for us was the Baltic sea resorts, 19th century spas and hotels. We started our trip on Rügen, a relaxing quiet island. The town of Binz has a terrific collection of nice hotels and restaurants. Also nearby is Nationalpark Jasmund with its famous chalk cliffs, the bizarre Prora (a facist beach resort built in the 30s), and the Rasender Roland beach steam train.

But the best Baltic sea experience was a last minute decision to go to the Grand Hotel Heiligendamm and its amazing gourmet restaurant Friedrich Franz. Really lovely overnight, fantastic cooking. Heiligendamm is interesting for being one of the first ever beach resorts, founded in 1793 and popular with various royalty. Up to and including the G8 summit in 2007. Also another steam train, the Molli Bahn. Just a terrific place all around, worth planning a stay if you’re in the area.

Beyond the resorts we visited various Hanseatic towns: Rostock, Wismar, Stralsund, Lübeck. And a trip down to the lake region to Schwerin. Lots of beautiful brick buildings dating from a wealthy past in the 15th-17th centuries. Of the places I liked Lübeck and Schwerin the best, combining charming town centers with some lively modern life.

The trip was bookended by visits to Berlin and Hamburg. Berlin is amazing, particularly right now since its relatively low cost of living has attracted a vital core of artists and entrepreneurs. I think we may try to go back to spend a month living there next year. Hamburg is also quite pleasant for a visit, I think it’s a city that would reward settling in and exploring a bit.

Every time I travel I refresh my apps designed to be used when the iPhone is offline. These apps all cache data so I can use Wikipedia or a map without a WiFi or cellular connection. I started doing this because international roaming data was so expensive but the apps are now good enough that I think I will use them even when I’m home. Cached data = fast! Here’s the best of the lot, I believe all these apps are available both for iOS and Android.

ForeverMap 2: OpenStreetMap. Download a few hundred megabytes and have a map of a whole country in your pocket. Routing too! Map data quality varies based on OSM coverage (it’s great in US and most of Western Europe). The rendering and usability of the app is fantastic. They also have a turn based navigation program I haven’t tried. I’m amazed Apple hasn’t yet bought Skobbler to help fix their maps problem.

Wiki Offline: Wikipedia. Download 4GB of English Wikipedia once, read forever. The formatting is finally good enough that most articles come through unscathed. Only thing missing is the pictures. Being able to wikidive without waiting for network is terrific.

Triposo: travel guides. Triposo scrapes open data sources like Wikipedia, Wikitravel, OSM, and Flickr and then compiles it into a usable offline travel guide on your phone. It’s great for answering the question “what are the three things I should see in this town, and where should I have lunch?”.

Ascendo dictionaries. There’s a zillion low quality free translation dictionaries out there, this one seemed to have a decent German database and work well offline.

I’ve been in Berlin for the past few days, having a great time. But also a bit bleak, it’s hard to be in Berlin without seeing the awful German history of the 20th century. The Holocaust, the division of Berlin, the Stasi, the people murdered trying to cross the Wall. It’s inescapable.

What I admire is how directly the German state seems to engage with its evil history. There are museums and monuments everywhere, from small plaques at the former homes of Jews who were deported and murdered to reminders of the Wall to state funded museums like the Topography of Terror.

The tone of the presentations (at least in English translation) is forthright and neutral. So all you see is an unvarnished explanation of how the Third Reich inventoried and killed millions of people. No attempt to explain or contextualize the act, certainly not to justify it, not even color commentary on how horrible it was. Just meticulous, detailed documentation of the terrible crimes of the Holocaust.

It feels like an honest attempt to understand and account for the past crimes of German governments. By presenting things so directly it becomes impossible to explain it away as some aberrant past, some temporary mania, the inexplicable actions of others. They are saying “Here are the facts of our history. Never forget.”

I’d like to see a similarly forthright American account of some the worst parts of our history. The genocide of the Native Americans, the importation and enslavement of Africans, the Civil War (on both sides). There’s too much explanation and justification in our historical narrative, not enough simple accounting of the evils in our past.

Another reason to end passwords as a method of authentication is the poor usability of strong passwords on mobile devices.

1. Typing a strong password like xry7s6Dx26Pz is nearly impossible on a mobile keyboard, particularly since for some dumb reason I can’t even see it when I type.
2. I can’t use a password agent like LastPass effectively on iOS because there’s no way for it to plug in to Mobile Safari. I’m stuck awkwardly copy and pasting passwords between the LastPass app and Safari, having to type my master password every time. LastPass does have its own baked-in browser but that’s far too limited on iOS.
3. Trying to log in to other apps pretty much requires copy-and-paste of the password, since there’s nothing like a pluggable authentication framework.

Sorry if this is stating the obvious, but the lack of usability of strong passwords on my iPhone and iPad is a big part of why I don’t log into sites on mobile devices.

Google has reduced itself to outright spamming users to promote its products. Here’s a screenshot of an email I got today about Google’s failing payments product, Google Wallet. Note the footer, the email is marked “You have received this mandatory email service announcement to update you about important changes to your Google Wallet account”. What are those important changes?

1. A call for me to use Google Wallet more
2. An ad for the Android version of Google Wallet
3. The Google Play logo
4. Logos for stores that accept Google Wallet
5. A request that I subscribe to more Google Wallet ads

In summary: four ads for Google products, one ad for random other companies that happen to use Google Wallet, and zero important changes. I guess I should block noreply@wallet.google.com?

It’s cliché now to point out how disappointing Google, Inc. has become. But this seems bad even for the trend. All that’s missing is the “+1 on Google+” button.

One of the great failures of the Internet era has been giving up on end-to-end encryption. PGP dates back to 1991, 22 years ago. It gave us the technical means to have truly secure email between two people. But it was very difficult to use. And in 22 years no one has ever meaningfully made email encryption really usable.

A big part of the problem is the architecture of Internet services. Most of us host our email on a third party server like Gmail or Lavabit or whatever. That makes true end-to-end encryption very difficult. Instead we have to trust our hosting service with access to our email, and as we find the government can compel them to rat you out (or simply break in).

We do have SSL/HTTPS, the only real end-to-end encryption most of us use daily. But the key distribution is hopelessly centralized, authority rooted in 40+ certificates. At least 4 of those certs have been compromised by blackhat hackers in the past few years. How many more have been subverted by government agencies? I believe the SSL Observatory is the only way we’d know.

The cypherpunks movement foresaw all of this surveillance risk. It outlined principles and technologies to protect individuals from both evil hackers and overreaching governments. It failed to actually implement it.

In your heart you know it’s flat

I love this phrase, the motto of the Discordian Flat Earth Society (early source). It neatly characterizes the problem of so much common sense knowledge, intuition, wisdom. Of course we know the Earth is flat; just look at it!

The older I get, the less patience I have with woo-woo people who praise intuition and folk wisdom. From child killing anti-vaccination superstition to muddled thinking about the dangers of cell towers, GMO foods, and nuclear power to the simple underinvestment in discovering how the universe actually works. In the words of Neil deGrasse Tyson, “The good thing about science is that it’s true whether or not you believe in it.”

Riot Games has found that in the League of Legends community, bad behavior comes mostly from people who are generally good. The problem with the LoL community isn’t that there’s a few jerks who spoil things for everyone; it’s that a lot of people act like jerks occasionally.

This factoid comes from a talk by Riot showing statistics from their player community. “Toxic” means raging during an online game, insulting and threatening other players. They use the word “toxic” because they’ve found bad behavior is contagious. One person acting badly can make other people angry, who then act badly in subsequent games. Dangerous problem.

The graph above shows Riot’s view of toxicity in the community. The graph on the left shows 1% of players are toxic, frequently acting badly; 78% of players are generally good. The graph on the right shows toxic behavior. And only 5% of toxic behavior comes from toxic people; 77% of it comes from people who are usually good.

That finding has all sorts of implications for how to stop toxic behavior in an online community. It’s not enough to just ban the jerks; good people have bad days too. Instead you have to teach the whole community what the community standards are. And quickly identify people who are having a bad day, intervene before their toxicity infects too many other people. I think it’s a hopeful finding; if you can just remind people of their better nature, you can prevent a lot of bad behavior.

Two really unfortunate stories of Internet bullying over the weekend. Indie game genius Phil Fish says he’s canceling Fez II after a bunch of focussed harassment. And the Call of Duty studio director has gotten a bunch of truly disgusting invective for a game balance change. All part of the Internet’s war on creatives.

Get lost you waste of talent. Shitbag telling others to kill themselves, you dont deserve your fame, money or attention. • Hahaha get fucked you pathetic fucking blowhard Fez was shit by the way • You may have made a decent game but you are still a terrible human being.
hey can you un nerf THE FUCKING SNIPER U FUCKING CUNT WTF WHY DID U DO THIS IM GOING 2 KILL YOU! DICK HEAD FUCKING DIE!!!!! • I will kill you for nerfing snipers you fag • omfg you fucking fat cunt i hope you die in a fatal car crash you fucking twat also your nan has cancer

The gaming “community” are mostly a bunch of monsters. I agree with Anil Dash that the community itself is responsible for fixing the problem.

I’ve been playing a lot of League of Legends lately, a team PvP game notorious for its toxic community. The developer Riot Games took a strong step towards solving the problem, The Tribunal, a way for the community to judge whether players violate the game’s code of good behavior. In my experience it works pretty well as a deterrent. Riot has stats showing that warnings and punishment are discouraging bad behavior.

It’s basically a community moderation system. After every game anyone can flag a player for bad behavior. Enough flags and a tribunal case (example) is created. Players randomly review cases (chat logs mostly) and vote on whether to punish. Mild penalties are automatic, severe ones are reviewed first by Riot employees.

There’s a lot more to say about the Tribunal, I hope to have some follow up blog posts. One particularly interesting aspect is that the review cases are public. Most moderation systems are private to avoid disputes but I think the open discussion makes the system more effective.

I got excited about Google’s new product Chromecast and quickly placed an order on the Google Play store. They said “will ship by August 2”; plenty of time to cancel the order if I found it elsewhere for cheaper or sooner. And so I did on Amazon; same price, delivers today for free. Yay!

Only there’s no way to cancel a Google Play store order. I see my order, Status “Pending”. I click “Cancel” and I’m told “We could not cancel your order at this time. Please try again later” with a link to this support page which says “it has already been packed”. Only without any promised delivery date, so I’m assuming it’s still weeks away.

Google has a decent phone support option. The nice American lady told me that there was no way to cancel the order and my best option was to refuse delivery. Except the package is being delivered at an unknown future date with instructions to leave without a signature. She had no explanation for why it was impossible to cancel an order that had not yet been billed or shipped. To be fair to Google I vaguely remember seeing a warning when I purchased saying “you may not be able to cancel this order”. Frankly, I ignored that, it seemed so improbable. Next time I just won’t order from Google Play.

Order fulfillment is hard. Google’s not making enough money on $35 hardware sales to be worth doing a good job of it. So why are they even trying? I guess it’s to compete with Apple. But they should consider how good the experience is buying hardware online from Apple.

My grandmother’s family is from here. “The blackest land. The whitest people.”

Terrific news today; the Supreme Court decided that even despised homosexuals like me deserve equal protection under the 5th Amendment and have overturned the odious Defense of Marriage Act. It’s a hugely important decision that establishes federal support for gay marriage.

But let’s not forget how we got here, in 1996, when DOMA was passed. The Democrats were fully in power; Congress had a majority of Democrats in both houses and the President was a Democrat. And those rat bastards voted to deny people like me our constitutional rights by overwhelming majorities: 85–14 in the Senate, 342–67 in the House. And Clinton cheerfully signed it into law.

Of course the Republicans are even worse, but that’s no surprise given their position as the party for bigots. Here’s a list of the Democrat senators who voted for DOMA. Many of them are still running the country.

Baucus, Biden, Bingaman, Bradley, Breaux, Bryan, Bumpers, Byrd, Conrad, Daschle, Dodd, Dorgan, Exon, Ford, Glenn, Graham, Harkin, Heflin, Hollings, Johnston, Kohl, Lautenberg, Leahy, Levin, Lieberman, Mikulski, Murray, Nunn, Reid, Rockefeller, Sarbanes, Wellstone

Some of these folks have since recanted, but I kind of feel like each one owes me and every other gay American a personal apology.

It’s been a good week for my river project. A lot of exposure on the Internet, 100,000+ viewers from Gear Junkie to a NASA Twitter account to that fine example of British tabloids, the Daily Mail (Online).

I owe the new attention to Jason Kottke. He ran a blog post on my map and the traffic exploded, not just directly from his site but into the minds of other people. I’d actually gotten a lot of attention on Reddit MapPorn last month, but it didn’t go further than that. Kottke has a lot more reach, from BoingBoing to Reddit again to Popular Science to Wired Design to some traditional print publications that haven’t come out yet (if they ever do). I’m guessing Jason found my map from Mike Bostock’s talk at Eyeo, that’s a real kind of legitimacy you can’t buy.

It’s a funny sort of brief fame for a little work print I did as a 45 minute aside on a much bigger project. I think people like that one picture because it’s easy to understand and looks cool, particularly the natural complexity of the earth. I feel bad I didn’t spend more time explaining this better. It’s not really a map of rivers at all, it’s flowlines, which may just as easily be seasonal streams or arroyos or drainage canals.

A bunch of folks have asked about a poster version. I’d like one too and it’s not too expensive, so I may give it a go. First I need to fix those nasty rectangular artifacts; apparently an artifact of digitization on USGS quads.

Rapportive is a good web service. It’s a browser extension for Gmail that puts information about correspondents in a sidebar. Here’s an example screenshot. It shows Tim’s face, his location, his jobs, and details from social media like Twitter, Facebook, etc.

The UI is quite nice, the way it sits next to my email without calling attention to itself. I regularly find helpful context on random people in my mailbox. The data mining is pretty good, I suspect they’re leaning heavily on LinkedIn for location, titles, etc. Gmail is the current application but the profiles they’re building on people could have enormous value in a variety of contexts.

Apparently I’m late to the party; they got all their press in 2010 and were bought by LinkedIn last year. Not sure why I hadn’t heard of it before. It’s a bit uncomfortable how deeply it links into Gmail, but it’s useful enough I’m giving it a try.

Last weekend I gave a talk about TopoJSON at State of the map US, the OpenStreetMap conference. TopoJSON is an extension of GeoJSON that encodes topology, enabling interesting visualizations and making for smaller files. The video of my talk is online, you can also see my slides.

The talk is an overview of what TopoJSON is. I also compared the sizes of TopoJSON files to the same data in GeoJSON and found TopoJSON files are about 25–50% the size of the equivalent GeoJSON after gzip. That’s without simplification and GeoJSON rounding comparable to TopoJSON quantization. You get space savings even when there are no shared boundaries, although obviously you get more with shared arcs.

One of the most exciting talks at SotM US was Dane Springmeyer’s talk on what MapBox is doing with their PBF vector tiles. They’ve done a lot of work on making high quality vector data available for cartography. They found they only need to prepare tiles to z=14 (about a square mile); at that scale you can just make the tile encode all features to full precision. They are able to render all of the OSM data for MapBox Streets into just 30GB of tile data in about 100 CPU hours. That’s quite manageable; very exciting.

Dane and I took a quick look and I think their PBF tiles are about the same size as TopoJSON tiles, maybe 15% smaller. OSM data doesn’t have many shared boundaries, so the main thing TopoJSON is doing is delta encoding of arcs. MapBox tiles also use delta encoding. Their PBFs also encode properties more efficiently than JSON, but after gzip I think the difference is less significant.

I like paying for digital movies. So I rented The Hobbit last night to watch on my Xbox. The movie was OK. The twelve times the streaming failed and the movie paused while it buffered was not. Amazon’s movie was about 3.5 gigabytes for 170 minutes, or 2700kbit/s. My download speed is a reliable 6000kbit/s. So what’s the problem?

The bandwidth graph above shows the problem; something terribly wrong with the streaming. First, the Xbox client doesn’t seem to buffer much, if at all. Playback would be a lot better if they used all 6000kbit/s and cached to disk. Second, their streaming server seems to have lost the connection ten times in three hours. Naturally they blame my ISP. At least they refunded the rental fee.

I like to pay for media, but maybe next time I’ll consider downloading an unlicensed copy. Pirate Bay offers a 2000kbit/s version that I could have downloaded and then watched uninterrupted for free. It was available two weeks before the official release.

If you’re looking for a mortgage, think twice before doing business with CitiMortgage. I refinanced my mortgage with them last year. Well, actually this year; it took them nine months. All along the way the process was incompetent and contemptuous of the customer.

I had an existing loan with Citi. They offered to refinance to a lower rate, no cost to me. All very simple: loan-to-value ratio wasn’t a problem, no question about us qualifying. I agreed to refinance in June 2012, gave them all documentation in July, confirmed all documents in place in August. And then nothing happened. For months. All I got was computer-generated letters saying my application had been canceled and empty verbal promises that “we’ll be underwriting soon”. And repeated requests for fresh documents, because the old W-2 copies, the old appraisal, etc, all “expired”.

Underwriting finally looked at the file in January, some six months after I completed my documentation. Then another comedy of incompetence and we finally signed in February. Mortgages usually take 30 days, 60 if it’s low priority. They promised 90 days; it took them 260. From what I’ve heard, that’s been pretty typical for CitiMortgage in the last year.

Maybe it’s all incompetence, but Citi ends up profiting. All told I paid an extra $2800 in interest waiting for them to get around to processing my refinance application. So basically I’m a sucker; I should have gone elsewhere, preferably through a mortgage broker.

Citigroup was one of the main culprits in the mortgage crisis of 2008 that nearly wrecked the US economy. And they were the brokers defrauding their clients in the investment bank scandal of 2003. The incompetence I encountered with my refinance is a different problem. But I really should stick with a policy of not doing business with companies that treat customers with such contempt.

I just finished reading American Nations: A History of the Eleven Rival Regional Cultures of North America, a history and cultural criticism book by Colin Woodward. It’s OK, not great. The map below is the thesis of the book.

Woodward argues North America is best understood as eleven separate distinct “nations” with unique cultural and political identities. The first half of the book gives the origins of these various tribes and argues for their inviolate coherence. This part of the book was insightful and interesting. The second half interprets various recent events in terms of a 400 year old conflict between Yankees and Deep Southerners. This part of the book was boring and ax grindy.

Related: I’m now an active GoodReads user and am trying to do a better job cataloging the books I read.

I no longer really use passwords to log into websites. Instead I use an authentication agent that lives in my browser and proves my identity to websites. Sadly, the authentication protocols of the Web require sending my secret token rather than doing some safer public key protocol. And the details of figuring out how to transmit the token to each website are needlessly complex.

To put it another way, passwords are completely broken; even strong passwords like “qeadzcwrsfxv1331” are crackable. With LastPass in my browser I literally do not know what my password is on pretty much every one of the 479 websites I log in to. I already run a complex authentication protocol. The stupid thing is that it’s a very bad protocol, involving stuffing secrets into random form elements on the web page.

Mozilla Persona is a strong proposal for how to end passwords in a better way, at least for desktop computers. And Tim Bray has lots of good notes on the authentication and identity. I still think OpenID is sufficient, or maybe the newer OpenID Connect system. Hell, at this point I’ll accept log in with Facebook or Google+ Sign-In. But whatever it is needs to be universal. And it really should be vendor neutral.

I love the idea that the JJ Abrams films are not really Star Trek; they’re really Star Trek fanfic. I don’t remember where I first read that idea, but it’s exactly right. I liked both movies, don’t get me wrong, but they are just ridiculous. Here’s the first movie script:

Kirk is this awesome 13 year old kid and he has a hot car and then he drives it off a cliff but he jumps out just in time. And then he gets in a fight in a bar and then he joins Starfleet and sneaks on board the Enterprise. And then Sulu has to space jump and he pulls out this sword and he’s, like, a killer ninja. And there’s a time traveling Romulan with special magical Red Matter. And Vulcan blows up but actually it’s a parallel Star Trek universe where all the same stories happen only totally different. And Spock and Uhura, they kiss.

Totally rad story, right? The new movie is just as ridiculous, if somewhat clever in what it does. I enjoyed it. Here’s hoping Abrams gives the same tawdry treatment to the Star Wars films, that’s a franchise ripe for self-parody.

I just completed a project I’ve been working on for a few weeks, a vector tile map of American rivers based on the NHDPlus dataset. It’s mostly a demo project with readable source, but it’s also kind of pretty.

There are three and a half products:

1. Web maps you can view in your browser. I made versions with Polymaps, Leaflet, and D3 (cribbing code from Mike Bostock). Jason Davies has an amazing Albers projection version and Ziggy Jonsson did a D3/Leaflet hybrid.
2. A GeoJSON vector tile server. It’s at http://somebits.com:8001/rivers/{z}/{x}/{y}.json You’re welcome to use it for light projects and demos, but it is not provisioned for heavy use.
3. A thoroughly documented tutorial on building a vector tile server. This is the real product, my goal was to learn about doing vector tiles in open source and share it with others. With this code it should be pretty easy for anyone to duplicate my map and adapt it to their own data.
4. Raster renders of every single river. It’s too much data to serve as a vector tile map, but it sure is pretty.

Vector maps are exciting. The proprietary map world is moving steadily towards vectors; pretty much all mobile maps are vector now and Google Maps is switching to vectors on the desktop. The open source and data world is getting there too. Thanks to Mike Migurski there’s now an experimental OpenStreetMap vector service that’s very promising. Also my personal thanks to Mike: the genesis of this project was getting an hour of his time.

The Tesla S hype has me interested. So now I’m curious, what does it really cost to run per mile? The Tesla site has a decent calculator, here’s some numbers derived from it.

Tesla says they get 283Wh/mile. Electricity in San Francisco costs $0.35/kWh. So that works out to $0.10/mile in a Tesla. Tesla compares itself to 22 MPG cars. Gas in San Francisco is roughly $4/gal, so it’s $0.18/mile in a gas car. By that math, a Tesla is roughly half the cost of a gas car in San Francisco.

San Francisco has outrageously high electricity costs. At the national average of $0.11/kWH a Tesla is more like $0.031/mile, or six times better than a gas car.

On the other hand, batteries wear out. Tesla is offering to replace the battery after the 8 year warranty at a prepaid cost of $10,000 – $12,000. Assuming 12,500 miles a year that adds $0.10/mi to the cost of driving a Tesla, dwarfing the cost of the electricity! The Tesla ends up being $0.13 – $0.20 / mile compared to $0.18/mi for the 22 MPG gas car (and roughly $0.12 – $0.20 / mile for gas cars in general).

If you think of the battery as another form of “fuel” that needs replacing every eight years, then the Tesla costs about the same per mile to drive as a gas car no matter what electricity rates you pay. But maybe the battery will last longer; no one really knows. Also, I suspect most Tesla customers think of the battery cost as depreciation and not a consumable.

Another argument for Tesla is that electricity is somehow more environmentally friendly than gas. Not really; a Tesla is metaphorically spewing 44% coal emissions out its tailpipe. It’s 20% nuclear though, I think that’s a win.

So it’s been long enough now I can tell this story about how I met Stewart Brand. Back in 1995 I was a fresh-out-of-college programmer at the Santa Fe Institute, a research place that attracted all sorts of interesting people. And one of the staff asked me if I could give a ride to Esther Dyson from the Albuquerque airport. “She’s quite interesting!” I was no dummy and said yes. I mean, my little Honda was big enough for two! And so I got the car washed and met her at the airport. And when we met she asked “could you give my friend Stewart a ride too? He’ll be here in about twenty minutes”. I had no idea who that’d be until he got into my car and I was just so pleased with myself. The three of us crammed in my little hatchback for the hour long drive with two of the most interesting, provocative technophilosopher types I’d ever met. Not bad for a 23 year old kid.

Needless to say I took advantage of every minute of having them trapped in my car with me. They were quite friendly and thoughtful and fun to talk to. At some point Stewart mentioned that he’d been at the MIT Media Lab for a while (was writing the book on it, actually) and I mentioned I was applying for grad school there. And so he kindly says “Nicolas owes me a favor, I’ll write a letter for you” and that’s part of how I got to go to the Media Lab for grad school.

I’m embarrassed posting this now because it seems so starfucker, but back in the mid 90s there just weren’t that many people talking like Dyson and Brand were. About the intersection of technology and culture, about the Internet, about building things with beauty and depth. That lucky hour had a big influence on me. And they were both so friendly and generous. I’ve met plenty of arrogant self-proclaimed pundits, maybe even acted like one myself on occasion, and I always try to remember Stewart Brand’s friendly humility.

I finally made good on last year’s New Year’s resolution and transferred domain names away from GoDaddy (registered via Google) to Hover. Hover is a humane registrar, the evolution of Tucows, and they have a good service. Getting out of the clutches of GoDaddy is not easy but Hover has put a lot of effort into helping you. Their docs are thorough and the webapp is good. Even so, I was starting to wish I’d used their free valet service. The steps are roughly:

1. Prepare the domain for transfer at GoDaddy. Disable privacy, unlock the name, and get the authorization code emailed.
2. Approve the transfer via an email from Hover.
3. Tell Hover to start the transfer.
4. Set up your new DNS records at Hover.
5. Accept the transfer at GoDaddy. The moment you accept, GoDaddy will remove your whois and DNS records and your site is offline.
6. Wait for Hover to receive your domain and publish it with new whois data. This takes a few minutes.
7. Edit your whois record at Hover to point to their DNS servers.

Step 7 has a race condition; Hover has to have received your domain name before you’re allowed to edit the name server authority in the whois data. And various things cache whois and root DNS information for minutes to hours. My site was offline for about 10 minutes while this sorted itself out. The right thing would be to edit the name server authority for your domain first, before initiating the transfer. Hover seemed happy to provide DNS service before the transfer was complete, I just couldn’t update the whois info.

Another glitch was that some of my names weren’t registered directly by me, but instead via Google Sites or AppEngine. That extra step causes a big mess; here’s a detailed description of the solution. In brief, you have to go to Google Admin Control Panel. That has a link for Domain Settings / Advanced DNS settings that gives you login credentials at GoDaddy that Google made and never told you about. There’s a “Sign in to DNS console” link right there that leads you to GoDaddy management, you can unlock the name and get the authorization code there. But that site has been broken for a year and you can’t disable domain privacy with it. Instead log in to this other GoDaddy site; you have to recover the username (a different random number), but the password Google gave you will work. The “cancel private registration” button works there. It’s almost like GoDaddy doesn’t want this transfer to be easy.

I’ve been having a grand time playing the Minecraft Feed the Beast Ultimate Pack. It’s a ginormous mod pack for Minecraft throwing together some 45–70 mods to extend Minecraft in various ways. It’s terribly complex, occasionally inconsistent, but surprisingly stable, balanced, and fun. I definitely recommend it if you like Minecraft and want to add more stuff to tinker with. I think it’d be particularly good for kids.

There’s a huge number of mods with varying degrees of documentation; part of the fun of FTB is figuring out how stuff works. The unofficial FTB wiki is a good place to start. Some of the big mods I like… BuildCraft and IndustrialCraft add machines, engines, pipes, pumps, all sorts of automation. Thaumcraft adds a beautifully designed magic system. ComputerCraft embeds a Lua scripting engine, letting you write programs for robots that mine and build structures and stuff. And Forestry adds a bunch of agricultural stuff including a crazy apiculture system of bee genetics.

If you want to play it, get the FTB Launcher and use it to install the Ultimate pack. It’s good about installing stuff in its own directory. Unfortunately Java on the Mac is a total mess; you have to set JAVA_HOME to run Java 6 and also configure the launcher to add the XX:PermSize flag when launching the game.

What do we do when Google shuts down Google Groups? I have no particular information that Groups is about to get the axe but I sure wouldn’t bet on it sticking around. Google is shutting down social products that don’t fit their Google+ strategy. And Groups has never gotten much love; it’s poorly staffed and the product keeps getting worse. (The site is still touting “the new Google Groups” that’s over two years old; some of the links documenting the “new” features don’t even work!)

The obvious casualty of a Groups shutdown are the communities that use Groups to communicate. But there’s plenty of alternatives: Yahoo Groups, Facebook, maybe even Google+. These products aren’t great; despite how lousy Google Groups is a lot of people still choose it. But I think the market will provide. Migration would be easier if Google offered data export: I think you can get list members but not messages.

But the unique thing Groups has, the thing that’s really important, is the historical Usenet archive that grew out of the Deja News purchase and later supplemented with other donated archives going back to 1981. Usenet is a pale shadow of its former self, but in the pre-Web days Usenet was the place on the Internet for people to communicate. A lot of science, culture, and community happened there and Google has the only easily accessible copy.

Google’s Usenet archive is important, but it’s not commercially valuable. And Google hasn’t been very trustworthy in keeping products like that around. I’d love to see a plan announced now, before there’s a fire drill, to gift a copy of the Google Usenet archive for preservation. The Internet Archive would be a good steward, or maybe the Library of Congress. Someone whose mission is to safeguard the world’s information, not just sell targeted advertisements on it.

I have no idea how I find new music anymore, but here’s two mix tapes I’ve been listening to a lot lately thanks to mentions on Metafilter.

Nicolas Jaar uses techno mixing techniques to work slow tempo music into lyrical, meditative pieces. His two hour set on BBC Essential Mix is absolutely amazing, an eclectic and fresh mix of various music that’s incredibly thoughtful. Jaar also runs the Clown & Sunset label. (MeFi thread).

DJ Shadow is justly famous for his crate digging and hip-hop derived mixing, although honestly other than Endtroducing I haven’t like much in his CD releases. But the All Basses Covered set is absolutely fantastic. He was infamously kicked off the decks after 20 minutes at a stupid South Beach club for being “too future”. Happily he cleaned up the set and put it online. It has a lot of depth and humor; the chopped & screwed Simpsons theme is particularly clever. (MeFi thread).

Ken and I went to Hawaiʻi for a week for my birthday. The big island, at a fancy tourist resort, my first time ever. It was lovely but also a bit boring, next time I go I’ll do it differently.

The great thing about Hawaiʻi is that it’s easy to visit and is absolutely beautiful. I totally get why people go there in the winter, to get some warmth and sun and relaxation. We stayed at the Four Seasons Hualālai which was excellent if outrageously expensive. The problem with a resort like that is it’s disconnected from the real place. And as nice as it is to have your big decision of the day be which of the four pools you hang out by, that’s not really my kind of vacation.

So we escaped The Village and drove all over the Big Island. Saw lots of things, honestly many not very exciting. I was particularly frustrated that the archaeological sites didn’t have more to see. My favorite things were the amazing botanical garden near Hilo, the town of Waimea, finding great macadamia nuts, and a helicopter tour whose highlight was flying into the narrow canyons west of the Waipiʻo Valley. The Kīlauea volcano would have been better if we spent more time.

But what I missed was seeing a real place, getting more in to local culture and food and history. I’m kicking myself that I didn’t plan to visit other islands, in particular to go to Oʻahu to see the Big City, go to Pearl Harbor, and to accept my anthropologist friend’s offer to tour the Bishop Museum. Next time. (Incidentally, the TSA security theater is an enormous burden to inter-island travel. 30 minute flight, 90 minute security.)

PS: the Hawaiian language is fascinating: only 8 consonants and one of them a glottal stop, but plenty of diphthong vowels. t and k are the same letter, so taboo becomes kapu. Only really lives on in place names. Hawaiian Pidgin is in active use, although I only heard it once.

Yahoo shut Delicious down. (Well, they sold it to a new owner who made a mess of it.) A bunch of Delicious users jumped ship and signed up for Pinboard which was a lot like Delicious only better, cleaner, faster. It costs $10 (once!) and now Maciej is making a nice living running this little service for his loyal users. He’s not rolling in VC dough, he doesn’t have a staff of hundreds, I’m guessing he grosses roughly $100,000 a year. But he runs a great service for a dedicated, smart community. Pinboard is a success.

Before Google Reader dominated the scene there were a lot of competing feed readers that were little one man shops. But then Google launched something really excellent, and free, and that was the end of the feed reader market.

Now Google is shutting down Google Reader. It doesn’t make them the hundreds of millions of dollars they measure products by. There’s a large, vocal community of distraught users who are looking for somewhere, anywhere to go. There’s a few products that might fill that niche. Commercial products, cost a few bucks, could pay for the living of a couple of developers. Google Reader shutting down may be the best thing that could happen for them. It could make them a success.

One of the reasons I like my dentist is the way they greet me when I come for an appointment. I walk in the door and the cheerful woman says “hello Nelson,” like I’m a welcome guest. It immediately sets me at ease, takes the edge off the tooth-scraping to come. I only come in twice a year and they’ve recognized me from my second visit. It seems so natural it never occurred to me that kind of greeting takes effort.

How do they do it? They took my photo my first visit. And they only have two people coming in at any given time. So the receptionist knows to look up the next appointments, and look at their pictures, and create a friendly moment. So simple, so pro.

I’ve never seen any other customer service do this simple thing. Not my accountant, not my lawyer, not my doctor; there I’m some anonymous schlub who has to identify himself. Opportunity lost.

A few tech companies try to create this sense of personal service. Uber is awesome this way, from the greeting from your private driver to the rock star moment you walk out without handling payment. Square Wallet creates this feeling too; buy coffee with just your name. Great way to create customer good will.

One of my favorite map visualizations is Ben Fry’s ZipDecode, a 2004 project that shows the hierarchical nature of zip codes with a simple interaction. I wanted to play with it and was frustrated at how hard it was to run a Java applet these days. So I rewrote it: ZipDecode in Javascript.

Well I rewrote the easy parts. I left out the zoom, although that would be pretty straightforward using D3 transitions. And I left out the color transitions; the simple way I’m doing this in SVG it’s too slow to animate. I should rewrite it using Canvas to match the speed of the 2004 Java applet.

D3 makes this kind of visualization very easy. Particularly the projection; Ben’s online example uses a janky Platte Carrée, I imagine because he didn’t want to do the math. (The version in his book is more georesponsible.) I don’t want to do the math either, but with D3 I can just use the provided AlbersUSA. The source is on GitHub and is quite readable, I think.

Here he comes.
Here comes Speed Racer.
He’s a demon on wheels.
He’s a demon.
And he’s going to be chasing after someone.

Proposed: Just Cause 2 is the perfect video game. I finished it soon after release, then played it through again about a year later, and am now playing it again a third time. I almost never replay old games, there’s so many new games to try. But Just Cause 2 is the perfect game when I just want to sit down with a beer for an hour and have fun blowing stuff up.

The game is a great combination of elements that all came together. Open world sandbox, fun blowing stuff up, and an amazing movement mechanic that lets you jump and fly and do all sorts of improbable stunts. The world is incredibly detailed and seamless. The rendering is fantastic, particularly the lighting. So much variety; snowy mountain towns, dense jungles, cities, the desert, it’s quite an impressive world design. And there’s a variety of emergent gameplay, particularly in the way you can make mayhem grappling things together and blowing stuff up creatively. I even like the story and voice acting: it’s not Shakespeare, but it’s certainly James Bond. And the thin veneer that somehow you’re a force for good despite the “we destroyed the village to save it” thing is hilarious. (Seriously; you blow up water towers to help The People.)

Just Cause 2 joins a few other open world games I’ve had serious fun in: Saints Row 3, Spiderman 2 (console), Crackdown 1, and Red Faction: Guerrilla. These games are all way more fun for me than any Grand Theft Auto has ever been. GTA games are amazing, particularly technically, but they are so ponderous. Just Cause 2 gets out of the way and just lets you fuck shit up. So much fun.

Bonus links: data analysis of where people died, including an awesome 3d point cloud of player deaths (emergent map!) as well as a heatmap. And the RockPaperShotgun feature Postcards from Panau.

I had a bit of email drama this week; Gmail started classifying half of my incoming legitimate email as spam. I got some great help from Gmail support who explained the problem and taught me how to properly forward email.

In detail, what happened… I get all my email to nelson@monkey.org, which I forward via procmail and SMTP to my gmail account. For some reason monkey.org recently got branded a possibly spammy domain. Because of my forwarding Gmail was under the impression that all my email was coming from monkey.org, so a bunch of it started getting marked as spam. The Gmail UI is a bit buggy in this circumstance; it was misidentifying which domain was the problem, telling me “we’ve found that lots of messages from gmail.com are spam” and the like when the real problem was monkey.org.

I fixed the problem by forwarding my mail properly. Gmail doesn’t just use the From: email header to identify the sender, it also uses the (normally invisible) From⎵ SMTP envelope. And because I misconfigured procmail, that header was always being set to nelson@monkey.org (since I was sending the mail). You can spoof the envelope too via -f, you just have to set it up that way. (Which makes me wonder why the spam filter pays any attention to it.)

It’s a subtle problem; I only noticed it after several years. If you use procmail to forward to Gmail, you may want to look into your configuration. I believe most more ordinary forwarding mechanisms don’t have the envelope problem. Procmail is weird in that it’s generating new emails, not forwarding existing ones.

Actual Facebook Graph Searches is a brilliant blog collecting creepy uses of Facebook’s new social graph search. Facebook’s product is controversial and I’m glad they launched it. It’s fascinating data. Facebook’s entire business since day one has been about taking semi-private data and making it semi-public. They blur privacy lines all the time. And they keep abusing their users in all sorts of terrible ways, proactively changing the rules (like the Beacon fiasco). But it doesn’t matter, there are still hundreds of millions of people using Facebook every day. Elite nerds complain and wring our hands and have no impact; Facebook just lurches on. They are shifting the definition of privacy.

Our society fundamentally doesn’t understand how to treat privacy in the age of databases. From reverse phone books to gun permit maps to scraped lists of political donors, we are continuously astonished when semi-public data seems creepy and privacy invading when aggregated.

So now we have Graph Search, a whole new way of looking at semi-private data that is public and can be creepy when remixed. It’s either going to fail fantastically or it’s going to be something great and useful. I don’t know which. But I’m glad to watch the experiment.

The zero width space is a useful Unicode character. It’s white space but renders with zero width. Useful for hinting where a line break could go if a browser needs to wrap a long line. It’s also good for faking out Twitter’s annoying URL rewriter; if you stick a ZWS in the middle of a domain name then Twitter won’t rewrite your text with a t.co redirect.

The zero width space is Unicode character U+200B. (HTML ​). It’s remarkably hard to type. On Windows you can type Alt-8203. On Linux you apparently can type Ctrl-Shift-U 8203. On a Mac you need Character Viewer; search for “zero” and double click the invisible character on row 4, column 1 to insert a ZWS.

Or you can just cut and paste it. I put one up there for you, between the left and right angle brackets. Of course being zero width you can’t easily select it; best bet is to copy the angle brackets too, all 3 characters. Then paste and delete the brackets. You can verify the ZWS is still there by using the arrow keys to move the cursor; it should get “stuck” on the ZWS and require two movements to pass.

Astronomy enthusiasts have an expression: first light. That’s the first view through a new telescope, the thrilling moment when something you’ve long anticipated, maybe built by hand, is finally real. First light is the beginning of a telescope’s life. It’s cherished for the excitement but it’s also a way to honor all the work, use, and joy to come.

I have a problem with first light. I love the experience of building something new, the moment when a bunch of abstract work and thinking results in the first tangible, visible product. I’m pretty good at achieving first light, at exploring a new idea or area and figuring out how to get something working.

But first light should be the beginning of an endeavour, not the end. Real products come from months or years of polishing, refining, tuning. Astronomers enjoy the thrill of first light through a new telescope, but real astronomy comes from folks like William Herschel or JLE Dryer spending years using those telescopes to systematically catalog the skies. Years of minute, careful work; ultimately rewarding, but terribly repetitive and fiddly.

I don’t have much patience for consistent finishing work. And since my last full time job (in 2006!) I haven’t had requirements to finish things, to turn random software experiments into real usable artifacts. And so I have a string of half-finished prototypes not worth showing people. I find that intensely frustrating.

A few months back we bought a house in Grass Valley, CA. We’re now got the house pretty well set up and have some idea what it’s like. Mostly it’s awesome. I really like the area. And I’m loving being in a big, spacious, quiet place with privacy and calm. San Francisco was really getting me down, so crowded and noisy and aggro. Nevada County is worlds away from all that and very comfortable. I’ve ended up spending a lot more time at the new place than I originally expected, mostly because it’s just so pleasant. My big decision yesterday evening was whether to watch the sunset from the hot tub or set up the telescope to look at Io’s shadow transiting Jupiter.

I’ve gotten to know the area pretty well and there’s a lot more depth than I originally expected. Grass Valley and Nevada City are a surprisingly sophisticated enclave for rural California. There’s a lot of independent tech oriented folks up here, some working full time (typically remotely) and some semi-retired. I was fortunate to be able to join the Nevada City Hackathon last month and met a lot of great people.

We’ve had a few guests come and stay for a couple of nights which has been great. We’ve got a really comfortable guest bedroom, a big social kitchen, and plenty of relaxing quiet. My hope is that more friends from the Bay Area will want to come up and spend some time with us.