Pastebins are incredibly useful. But most of the public pastebins are not suitable for sharing within a company (think code fragments, log messages etc.) and most private pastebins are either ugly (except hastebin!), hard to setup/maintain and usually forced to be behind the firewall (for security).

So I decided to scratch my itch and whipped up a pastebin that fit my needs. Features:

• Usability: it’s a pastebin — you create a paste, you view a paste. That’s it. No need to specify the language — we guess it, thanks to highlight.js. It also looks good (I think), thanks to Solarized.
• Easy of deployment: no installation, no maintenance. Just deploy to Google App Engine.
• Security: you need a Google account to create a paste. No login required to view a paste though. Bonus: if you use Google Apps at your company, you can have a private pastebin for you company by restricting the app to your domain.
• Open source: Fork it, baby!

Facts:

1. The Internet dominates our lives.
2. The Internet thrives on advertising (think Google, Facebook, Twitter, Hulu)

It therefore follows that our lives are dominated by Internet advertising.

The traditional model for advertising has been that publishers put out ads to catch the attention of consumers via some like of delivery network. In the past these delivery networks were in print, on radio, on television and now, on the Internet. It seems like a win-win for everyone: the delivery network gets paid by the publishers; the publishers make money because they get more customers; the consumers find out about publishers that they may not have otherwise known.

But a new trend is emerging now: consumers directly pay the delivery networks to NOT see ads from publishers.

Think Spotify. Netflix. Pandora. NYTimes. Dozens (hundreds) of mobile apps and games that have paid versions without ads.

Think about it. There seems to be near universal consensus that ads are a negative thing. There seem to be some (many?) success stories that individual consumers are willing to pay extra to avoid exposure to ads.

It seems like a fundamental shift in advertising to me. A world where the delivery networks become content providers and instead of charging publishers, they make money directly from the consumers.

In particular, I wonder how the Internet giants will evolve to take this trend into account. Would you be interested in a paid Google account that has no ads and offers full privacy (because they don’t need to make money off of your information anymore)?

Also interesting is that Apple is probably one of the few companies that are isolated from this changing nature of ads.

For instance, on my iPhone I read pretty  much everything via Flipboard. On Android, I’m still struggling to find a good Flipboard replacement and shuttle between Pulse, Google Currents and recently, Feedly. In either case, I rarely ever go to the actual website.

I still get some of my content fix from Google Reader (xkcd, Abstruse Goose etc).

Most of the “news” — that is, when I’m in “skim mode” — comes from social media, mostly G+ and a tiny bit from Twitter.

I remember the days (several years ago) when Techcrunch changing it’s site layout used to be a news in itself. Now I can’t remember the last time I visited Techcrunch (well, that could be partially attributed to the content quality…)

My point is, in all of the above cases, the app or service presents the content in an origin-agnostic manner. When you read something on Flipboard, it’s presented to be consumable via the Flipboard interface (in most cases), and not meant to preserve the look and feel of the origin website.

And such apps and services are just becoming more and more prevalent: Evernote Clearly; Readability; content-provider specific apps such as those from Time, CNN, NYT etc.

So, is website design becoming irrelevant? Especially for content-heavy sites?

(The Oatmeal is an exception — Matthew forces you to visit the website, and it’s always worth it)

I got my invite for the Bitcasa beta last week but only got around to installing it yesterday. I’ve only used it sparingly thus far. If you are in a hurry, here’s the TL;DR version:

• Users might find the “cloudify” model confusing
• Built using osxfuse (not to be confused with MacFUSE) and Qt
• Infinite storage sounds too good to be true. What’s the catch?
• Building trust with users will take time

Cloudification and Confusion

Here’s Bitcasa on what cloudify does:

When a folder is Cloudified, a corresponding virtual folder is created on the Bitcasa server and the contents of your local folder are copied up to the server. When Connected to the Bitcasa server, any changes or additions to the folder will live on the server. When not Connected to the Bitcasa server, any changes or addition to the folder will live locally.

Just think about that for a second. The “cloudify” model sounds great in principle, but it does add a lot of complexity in terms of how users interact with the system. For instance, when I’m offline and make changes to one of my cloudified folders, that change happens presumably locally. I would assume that when I come back online, these changes are synced back to Bitcasa ala Dropbox. But what if I accidentally disconnect a folder, make some changes and then reconnect — per the FAQ, the changes made locally won’t be synced.

The consumer cloud storage is fairly mature right now and one can learn a lot by looking at how people respond to other systems. This thread on Quora is particularly insightful: again and again, simplicity comes up as one of the key reasons behind Dropbox’s success.

My prediction is that Bitcasa’s cloudify feature will be leveraged primarily by power users and the rest would end up using the default Bitcasa folder, Dropbox style.

Nuts and Bolts

Bitcasa seems to be built primarily using Qt. This isn’t a surprise: Qt is a mature, open source and cross-platform library.

$ otool -L Bitcasa
Bitcasa:
 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
 /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
 /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 44.0.0)
 @executable_path/../Frameworks/libmacfuse_i64.2.dylib (compatibility version 10.0.0, current version 2.0.0)
 /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 44.0.0)
 /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 53.0.0)
 @executable_path/../Frameworks/QtWebKit.framework/Versions/4/QtWebKit (compatibility version 4.7.0, current version 4.7.4)
 @executable_path/../Frameworks/QtXml.framework/Versions/4/QtXml (compatibility version 4.7.0, current version 4.7.4)
 @executable_path/../Frameworks/QtGui.framework/Versions/4/QtGui (compatibility version 4.7.0, current version 4.7.4)
 @executable_path/../Frameworks/QtNetwork.framework/Versions/4/QtNetwork (compatibility version 4.7.0, current version 4.7.4)
 @executable_path/../Frameworks/QtCore.framework/Versions/4/QtCore (compatibility version 4.7.0, current version 4.7.4)
 /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 52.0.0)
 /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1105.0.0)

$ mount
Sample Videos on /Users/diwaker/Bitcasa/Sample Videos (osxfusefs, nodev, nosuid, synchronous, mounted by diwaker)
TryBitcasa on /Users/diwaker/TryBitcasa (osxfusefs, nodev, nosuid, synchronous, mounted by diwaker)
TryBitcasaDedup on /Users/diwaker/TryBitcasaDedup (osxfusefs, nodev, nosuid, synchronous, mounted by diwaker)

Note further that Bitcasa represents “connected” folders as mount points over the existing folders. This is why when you disconnect a folder and make changes, they won’t propagate to Bitcasa’s copy of that folder. They are using osxfuse which implies that Bitcasa is intercepting file system calls; this is in contrast to Dropbox-like systems that detect changes to the local filesystem asynchronously. I haven’t compared fine-grained read/write performance just yet.

Here’s a snapshot of the Bitcasa Folders UI:

Bitcasa also does some deduplication. Uploading 100MB of mostly random data took around 4 minutes on a pretty fat pipe which isnt’ bad at all. Copying that data back out took just as long, if not longer. A copy of the same folder took less than 10 seconds to cloudify!

Security

Much has been said about Bitcasa’s security. However, most of the articles are concerned with a specific dimension of security: encryption.

A detailed discussion of Bitcasa’s security in general and encryption, in particular, deserves a post of its own. For now, suffice to say that even after several years of user experience, Dropbox still hit some pretty nasty security snafus in 2011. Like a lot of you, I’m very concerned about security, especially with a service that is offering me infinite storage for free! It takes time to build trust with your users — there’s no short cut.

Overall, Bitcasa is definitely interesting. Dropbox was almost beginning to monopolize the consumer cloud storage market, so some good competition will hopefully benefit the end users in the long run.

Summary: the NDTV app is probably one of the best free apps. I didn’t consider paid apps.

First, the usual suspects:

Times of India: The ToI app’s UI is functional, but otherwise not remarkable at a first glance. In true ToI tradition, the “Entertainment” section is feature prominently on the home page, just under top news. Of course, readers of ToI know that “Entertainment” and “Photos” are just euphemisms for soft porn — ToI happily parlays all kinds of NSFW material under the guise of “news”. I’m really curious to know how much of their app traffic (indeed, their website traffic) goes to the entertainment section.

Thankfully, buried under the “Settings”, the app allows reordering the various sections. You can also optionally specify a home city. I haven’t really used the Video section of the app, so can’t comment on it.

Overall, the app is not bad, but it can’t compensate for ToI’s reporting.

NDTV: The NDTV app feels only slightly more polished than the ToI app; structurally they’re quite similar and most differences are cosmetic. Unlike ToI though, NDTV’s Photos section is closer to what I’d expect on a news app (there’s still a heavy entertainment bias, of course).

But perhaps the most killer aspect of the NDTV app is that you can watch various channels of the NDTV group live!!

The only downside of the NDTV app is that it shows a lot more ads than the other apps I looked at.

Hindustan Times: the HT app is probably not being actively developed — it still has a CWG section!! Other differentiators are a dedicated “Blogs” section. Compared to ToI and NDTV, this app offers basically no customization, no videos. The content is not as rich or fresh as the other apps.

There were a lot of other news apps but none of them felt credible. The IBN Live app looked interesting but it seems to focus mostly on live TV and not news articles. For now, I’m sticking with the NDTV app.

What apps do you use to get your does of Indian news?

As we all know by now (read: probably 1% of the world’s population), at WWDC earlier this week, Apple spilled the beans on the upcoming iCloud, among other things. In this post, I wanted to share some of my thoughts on the much hyped iCloud (not that there is any dearth of opinions and articles on the subject, thanks to the echo-chamber that is Twitterverse and Blogosphere)

First off, some quick bullets summarizing what it is:

• iCloud aims to make cloud storage painless, the idea being that your data should be available to you from all your devices, all the time.
• It’s automatic and transparent. Apple is baking iCloud support deep into 9 different applications: iTunes, Photo Stream, Apps, Books, Documents, Backup, Contacts, Calendar and Mail. And that’s just the beginning.
• It’s free. Upto 5GB — excluding purchased music, books, apps and photo stream.
• Sync over the air: iCloud can sync across devices over wireless. As a concrete example, you’ll no longer need a cable to sync and backup your iPhone with your laptop.

Here are some cool things about iCloud:

• Scan and skip upload (iTunes only): when dealing with large data sets (such as your movies and music collection), one of the main impediments to using cloud storage is the overhead of doing the initial import. With a 1Mbps uplink, a 10GB music collection will take a full day to upload. Of course, if the file you are trying to upload already exists somewhere in the cloud, you don’t need to upload it and this is exactly what iCloud does. Because of the iTunes store, Apple already has a library of 18 million songs (and counting) and detecting if two files are for the same song is a lot easier than for many other media types (say images or movies).
• Storage APIs for developers: APIs are all the rage these days. By exposing the right set of APIs, Apple could attract developers to build iCloud functionality on other platforms (Android, for example). Unfortunately, the API is fairly limited at this point (key-value store or documents).
• HP, Teradata, maybe EMC are rumored to have supplied bulk of the hardware in the spanking new datacenter that will be the backbone for iCloud.
• Despite all the hoopla around “cloud” recently, it was still grounded firmly within the tech circles. Apple has the ability, experience and motivation to take cloud computing truly mainstream with iCloud.

What is NOT so cool:

• Apple has a habit of exaggerating the novelty and efficacy of their features (remember Spaces?) Scan and skip upload is nothing new: it is just deduplication under the wraps — a well known technique in storage systems. Videos and photos will still have to be uploaded though — there’s no real shortcut for those. Of course, there are techniques to dedup arbitrary data and I hope Apple is leveraging them.
• In the same vein, syncing of Mail, Calendar and Contacts is just catch up. Ever used Google? Likewise for Docs and Books. The delivery model is different — Apple apps work with the local data and sync when there’s connectivity. They haven’t touched upon conflict resolution, disconnected clients etc.
• Implications for Dropbox: transparent, automatic sync across multiple devices is a phenomenally hard problem. Apple makes it sound like they’ve nailed it. It took Dropbox several years to address all the performance and security concerns. I’d wager Apple will run into its share of snags along the way.
• Apples all the way: despite their claims, iCloud is designed to lock you in. Sure you may be able to leverage some of the features by installing additional software on a PC. But unless you are using an Apple device, you won’t get the full experience or service. Want your “reading list” available on Android (or Chome, for that matter)? Tough luck. Want your music available to other music players (open source players like Banshee and Amarok, god forbid)? How about your photo stream in Picasa?

Finally, there’s no doubt that iCloud will drastically alter the cloud landscape. However, Apple is focused mainly on the personal cloud — which is a good thing, they are playing to their strengths. It is also a great opportunity because the enterprise cloud market is still wide open. The requirements, challenges and “killer apps” in that market are very very different than the personal/consumer cloud market. Should be fun!

To help sort out my thoughts, I drew a picture (don’t you dare judge me for my lack of creativity!):

Below I elaborate more on how I currently use each of the services.

Twitter

• I tend to use it for technical and/or non-personal content. Things that I would want to publicize.
• Unlike Buzz/Facebook, I don’t pay too much attention to who is following me. Most tweets are public anyways.
• The 140 character limit is sometimes amusing, but often irritating. Are people still using regular SMS with Twitter?
• Multiple startups devoted to managing Twitter “noise” is not encouraging.
• @ replies are bandaid. Twitter is a broadcast-and-forget medium — I can’t have (or follow) a conversation on it.

Facebook

• Use it for sharing random, personal updates (or things I find interesting :p)
• Mostly on because of network effect (read: don’t want to be left off the social bandwagon).
• Like that I can “Like” most things and actually follow the conversation via comments.
• Always worried if my privacy settings are working and if there’s a new “default” I need to worry about.
• Pay more attention to who I friend. The noise level is still quite high despite that.

Buzz

• Usage domain similar to that of Facebook. Unlike Facebook, can choose to make posts Public.
• Love the email integration. Conversely, API/clients still have to catch up to Twitter.
• Supports likes, comments and “resharing”.
• Privacy is modeled around my contacts (chat or otherwise), which seems natural.

I’m fine with using Twitter for all of my public posts. The main confusion lies between Buzz and Facebook. Facebook obviously has more social traction. That said, Buzz is just more convenient to use (because of the email integration mostly). Of course, all of the various connectors available (Twitter <-> Buzz, Twitter <-> Facebook, multicast via ping.fm or Chromedeck etc) make the whole thing even more confusing. At the end of the day, I might just go back to not using anything on a regular basis.

How are you using Twitter, Buzz and Facebook?

The Social Network is rather like a fast paced documentary. The content, production value and background scores were great. I really enjoyed the bit around the Harvard boat race — a nice piece of whitespace in the movie :) But this post is not about these aspects; rather I wanted to make a few observations about the several tiny tid-bits of open source sprinkled throughout the movie.

• wget makes several appearances in a short segment of the movie where Mark is scraping the Harvard intranet for the seed data for various precursors to Facebook. To my relief, everything I saw seemed very real and plausible unlike, say, the hackery mumbo-jumbo in Matrix or (gasp) Swordfish. Nonetheless, I did not see (and have not seen) any evidence that Mark Zuckerberg is the programming genius that most reviews and synopsis claim. Of course, programming genius has no correlation with being successful (read: being the youngest billionaire)
• The usage of Emacs, Perl and curl were also faithful. The emphasis should be on Zuck’s intuition about the idea and his ability to prototype quickly. The technology itself was something any script kiddy could have come up with.
• Zuck is shown running KDE 3 on his workstation. Again, the attention to detail is impressive. KDE 3 was around the same time as the early years of Facebook development.

There were a few more things, but I saw the movie several weeks ago and the details are fuzzy in my head. Meanwhile, if you are interested in the veracity of the movie’s substance, I found this Gigaom post useful.

var sys = require('sys');
var http = require('http');
var url = require('url');
var path = require('path');

function search() {
  stdin = process.openStdin();
  stdin.setEncoding('utf8');
  stdin.on('data', function(term) {
    term = term.substring(0, term.length - 1);
    var google = http.createClient(80, 'ajax.googleapis.com');
    var search_url = "/ajax/services/search/web?v=1.0&q=" + term;
    var request = google.request('GET', search_url, {
      'host': 'ajax.googleapis.com',
      'Referer': 'http://floatingsun.net',
      'User-Agent': 'NodeJS HTTP client',
      'Accept': '*/*'});
    request.on('response', function(response) {
      response.setEncoding('utf8');
      var body = ""
      response.on('data', function(chunk) {
        body += chunk;
      });
      response.on('end', function() {
        var searchResults = JSON.parse(body);
        var results = searchResults["responseData"]["results"];
        for (var i = 0; i < results.length; i++) {
          console.log(results[i]["url"]);
        }
      });
    });
    request.end();
  });
}

search();

This program (also available as a gist) reads in search terms on standard input, and does a Google search on those terms, printing the URLs of the search results.

I was quite surprised (and a bit embarrassed) at how long it took me to get this simple program working. For instance, it took me the better part of an hour to realize that when I read something from stdin, it includes the trailing newline (as the user hits ‘Enter’). Earlier, I was using the input as-is for the search term, and that was leading to a 404 error, because the resulting URL was malformed.

Debugging was also harder, as expected. Syntax errors are easily caught by V8, but everything else is still obscure. I’m sure some of the difficulty is because of my lack of expertise with Javascript. But at one point, I got this error:

events:12
        throw arguments[1];
                       ^
Error: Parse Error
    at Client.ondata (http:881:22)
    at IOWatcher.callback (net:517:29)
    at node.js:270:9

I still haven’t figured out exactly where that error was coming from. Nonetheless, it was an interesting exercise. I’m looking forward to writing some non-trivial code with node.js now.

If you follow the world of Javascript and/or high-performance networking, you have probably heard of node.js. If you already grok Node, then this post is not for you; move along. If, however, you are a bit confused as to exactly what Node.js is and how it works, then you should read on.

The node.js website doesn’t mince words in describing the software: “Evented I/O for V8 JavaScript.” While that statement is precise and captures the essence of node.js succinctly, at first glance it did not tell me much about node.js. I did what anyone interested in node.js should do: downloaded the source and started playing around with it.

So what exactly is node.js? Well, first and foremost it is a Javascript runtime. Think of your web browser; how does it run Javascript? It implements a Javascript runtime and supports APIs that make sense in the browser such as DOM manipulation etc. Javascript as a language itself is fairly browser agnostic. So node.js is yet another runtime for Javascript, implemented primarily in C++.

Because node.js focuses on networking, it does not support the standard APIs available in a browser. Instead, it provides a different set of APIs (with fantastic documentation). Thus, for instance, HTTP support is built into node.js — it is not an external library.

The other salient feature of node.js is that it is event driven. If you are familiar with event driven programming (ala Python Twisted, Ruby’s Event Machine, the event loop in Qt etc), you know what I’m talking about. The key difference though is that unlike all these systems, you never explicitly invoke a blocking call to start the event loop — node.js automatically enters the event loop as soon as it has finished loaded the program. A corollary is that you can only write event driven programs in node.js, no other programming models are supported. Another consequence of this design choice is that node.js is single-threaded. To exploit CPU parallelism, you need to run multiple node.js instances. Of course, there are several node.js modules and projects already available to address this very issue.

To implement a runtime for Javascript, node.js first needs to parse the input Javascript. node.js leverages Google’s V8 Javascript engine to do this. V8 takes care of interpreting the Javascript so node.js need not worry about syntactical issues; it only need to implement the appropriate hooks and callbacks for V8.

node.js claims to be extremely memory efficient and scalable. This is possible because node.js does not expose any blocking APIs. As a result, the program is completely callback driven. Of course, any kind of I/O (disk or network) will eventually block. node.js does all blocking I/O in an internal thread pool — thus even though the application executes in a single thread, internally there are multiple threads that node.js manages.

Overall, node.js is very refreshing. The community seems great and there is a lot of buzz around the project right now, with some big companies like Yahoo starting to use experiment with node.js. node.js is also driving the “server side Javascript” movement. For instance, Joyent’s Smart platform allows you to write your server code in Javascript, which they can then execute on their hosted platforms.

Finally, no blog post about node.js is complete without an example of node.js code. Here is a simple web server: