The past couple of days my feed reader has been chock full of posts about one of the following: the year in review, predictions for 2008, reflections and introspections. So much so that I got tired of reading about the “new year” and never got around to writing MY end of the year post, but I’m sure the world didn’t miss much. But I did run into an interesting problem as I was thinking about what could have been my end of the year post: exactly what all did I do last year?

So I started by writing down all the months, the idea being that I would put down all the significant events that happened in any given month next to it. The hope is that there aren’t that many of them so the list should be fairly manageable. Now, I have always known that my memory is not that great, and that is why I tend to rely on tools to do the dirty book keeping for me: calendars, todo lists, reminders etc. But it was still a little shocking when I couldn’t immediately recall what I did in lets say May of last year. Of course I did remember things once I thought about it a little bit, often relying on context (what happened before May, after May etc).

The bottom line is that it wasn’t as easy as I thought it would be. For some months, I actually had to go back to my email inbox and other digital archives to figure out the salient happenings. This got me thinking about **personal information analysis and visualization**. And the more I thought about it, the more excited I became.

I was actually surprised to find such little information on the web about this. With our increasing information overload, cheap storage, and tons of archived data (online and offline), I think this space has tremendous potential for both academic and commercial ventures. For instance, here’s a really simple thing I want to be able to do: for a given time period (say 2007), I want to analyze and visualize all of my emails so that I can quickly figure out:
* who did I communicate with the most?
* what were the main topics I wrote about?

I couldn’t find any open source tool to do even this. And my initial Googling hasn’t turned up much in commercial offerings either. The closest thing I could find was a project called [[http://alumni.media.mit.edu/~fviegas/projects/themail/study/index.htm|themail]] from MIT Media Labs, but there’s no code that I can download. Then there is [[http://carohorn.de/anymails/|Anymails]], but it seems just a cool visualization, and not a lot of information (specially the kind I want).

If you know about any free or paid tools that can do this kind of analysis, please drop a line in the comments. And while you are at it, try to think about what YOU did all of last year :-)

Gmail now allows you to specify alternate “From” addresses. In Gmail, go to Settings -> Accounts

This is //so// cool!

So why, oh why does Apple has to go around inventing proprietary formats for storing local mail? Why can’t they use the excellent [[http://cr.yp.to/proto/maildir.html|Maildir]] format, when almost every other email client in the world does it? The excuse that this format is //optimised// for [[http://www.apple.com/macosx/features/spotlight/|Spotlight]] is bullshit. They brag so much about their API for enabling Spotlight support in applications, then why couldn’t they just write a backend for Maildir files (the way [[http://beaglewiki.org/Main_Page|Beagle]] does!)

Also, apparently this change is new to Tiger, and so old Mac OSX users are affected too. Even if they decided to change the format, is it too much to ask that the software confirms with the user before doing the upgrade, or atleast //informs// the user that something drastic is happening to his/her files? So much for desiging good user interfaces. Sheesh! Why don’t these people ever learn?

A few years back, this wasn’t a problem at all. SMTP by itself didn’t have (still doesn’t) any notion of security. So you could fake any email address you want in the envelope, and SMTP would happily deliver it for you. However, with the proliferation of spam on the Internet, ISPs and SMTP servers have become very wary of faked from addresses.

What does a fake from address really mean? For most purposes, it simply means that the domain of the from address and the domain that the mail originated in don’t match up. A lot of ISPs only relay mails through their own mail servers (and block port 25 for any other mail servers). Their own mail servers frequently restrict from addresses to be the one that the ISP gave you. A large number of organizations have started using [[http://antispam.yahoo.com/domainkeys|DomainKeys]] or [[http://spf.pobox.com/|SPF]], which makes faking a sender address even harder.

Gmail allows sending messages through their SMTP servers using your gmail account, but no matter what sender address you put in there, it gets substituted by your gmail address. So thats no good is it?

Ideally, I think there should be a decoupling between owenership and authentication. Each email address has a unique owner, multiple email addresses can have the same owner. Then SMTP should authenticate the owner, not the email address. Of course its easier said than done. I’ll think about it a bit more and write something more thought out later on.

Thanks to Gmail, a lot of non-Gmail users are also going to be able to enjoy larger storage space for their inboxes. Yahoo! is deploying larger inboxes for both free and premium email members over the summers. A mostly obscure Aventure Mail is offering 2GB for free for the first 10000 customers.

I thought the way people were buying and selling and swapping Gmail accounts was almost ridiculous. I mean, Gmail is so hot simply because it symbolises “cool” — I’m not sure how many people are after Gmail from the point of view of its utility or functionality. But where is all this going to lead? I mean what after 1GB? And how does this impact spam?

With Yahoo’s anti-spam proposal gaining momentum, we’re hoping that the amount of spam will reduce in the future. But it seems to me that more storage for inboxes might also imply more spam as well. Though that relationship is quite obscure, its more like a gut feeling. As Gmail suggests, archive, don’t delete.

Most people I know hate spam because they have to delete the stuff manually. If spam filtering really becomes effective, at what point will spam stop being a PITA, if at all it ever will. If you look at the most popular networking applications through the decades, starting in the 1970′s, you’ll see that email has been a constant contender. And it still remains, and will probably remain for the next decade as well.