Linkstew

Okay, this thing is pretty cool, and most of the code is actually there in the javascript, but I'm still trying to wrap my mind around where it's coming up with those "links" between nodes.

Yeah, I'm a sucker for networks and graphs.

After only two nights of coding, the shiny and new (but not very different) Linkstew not only has all the features of the old Linkstew, but it's got many new features as well.

So what are these shiny new features of Linkstew? Well...

• Comments. My main use for comments is actually for personal annotation of entries (and to occasionally complain about bad posts), but it's an open system, and everyone is free to contribute. Just hit the "++" link in the corner to join the fray.
• More immediate access to The Stew. A few items of The Stew are now available on the front page, with a link to see other related entries. This is to make it easier to quickly browse around related ideas on the site.
• Related Topics. On the front page, for example, the topics which are most related to the recent entries are displayed in the navigation bar. On a topic page, the topics most related to the displayed topic are displayed. And on an entry page, the entry's topics are displayed. Once again, this hopefully allows a reader to jump around related ideas, much like a person thinks.

And of course, all the features that have always been available are still here. Read Random Entries, or Search for key words from an entry you remember reading months ago, or read by date or by topic. The one feature I know I still need to implement is comment previews, but other than that, I'm really happy with the way the shiny new Linkstew+The Stew is shaping up.

And in honor of all this, I've got a new email address: bsii@thestew.net. And if you don't want to leave a comment about it, that's how you should contact me.

Well, since no one really knew it was there, I've added links to the top three related entries for each post to the front page. I'm not overly happy with the way it looks, but at least now it's clear that it's there.

This morning I finally finished categorizing all of my old entries. Since the links to the old categories are now totally redundant, I've removed them from the page, and in their place there is a link to a list of all the new categories.

However, that list of new categories will be very much a work in progress. Some of the categories are way too broad and need to be split up, while others are too narrow. I need to analyze how The Stew is performing with this data, and adjust the data and The Stew as necessary, to make the system as useful as possible.

Instead of posting over the last couple of days, I've been working like mad towards realizing one of the major goals of LinkStew -- A living information network, which calculates relationships between entries based on a number of heuristics, in the hopes of presenting readers with links to entries directly relevant to the post that was just read. Think of it like Amazon.com's "Recomendations" feature -- "If you enjoyed reading this entry, you might also enjoy reading these other entries."

So far, only about 1/5 of all the entries in the Stew implement the feature, but almost all of them in the last couple of months work. To access The Stew, you just follow a link to a particular entry on the site (like this one or the ones in the corner of each post) and look at the list of titles following the entry.

Right now, the Stew isn't updated in real time, because I haven't yet worked in optimizations, forcing the system to work out the relationship between every node in the system. Also, it doesn't take into consideration a lot of the heuristics I plan to add in the future. For now, just take a look at the Stew and let me know if you have any suggestions or ideas, and I'll return to classifying more of my posts, hopefully making the Stew more accurate.

Vaguely in the spirit of AOLiza comes Search Engine Therapy, inspired by a disturbing search request which turned up In Passing... Not bad, only the first couple problems weren't so much personal problems as they were medical problems, which kind of hint at the medical smart agents vaguely suggested in this article by Tim-Berners Lee... At any rate, it's worth a glance. There are definately better articles discussing the possibilities of Medical Smart Agents, but this is the most current and most interesting one I can think of.

I needed a break from working on my compiler for class and doing some programming for work, so instead of doing something other than programming, I went and made a few tweaks to the site. But it really was a sufficient break, even if it's effectively the same as the other things I'm doing today.

Anyway, I made two changes today. First, I fixed Random so that it's faster and so that it really is random. Before, I was using mysql to "select ... order by rand() limit 1", but that was a horribly slow and not very effective approach, so now I'm doing it in code, and it's much faster and much more random. Rejoice!

And second, I flattened the categories for posts in preparation for some of the other changes I'm going to make to my categorization system. I also made the category show up in the title bar of each post again... I'll figure out a better place to put it again later, because if I go through with the plan I came up with a couple of days ago, this layout is going to cause some nasty wrapping.

Every time I load Linkstew, I look at that long list of categories on the right and I sigh. I have so many categories in place, and yet they don't seem to be effective. Every time I start to write a new entry, I'm faced with the question of "What category should this post go in?" And more often than not, I pick "Life :: Present". Why? Because it seems the most appropriate.

Tonight, I did some math (well, mysql did some math) and found that more than 30% of all of my posts are in "Life :: Present". Another 13% are in "Think" and another 10% are in "Life :: Synchronicity". Of my 20 categories, more than 50% of my posts fall in 3 of those categories. Does this sound wrong to you? It sure sounds wrong to me.

And so I find myself with an interesting set of problems, both on the conceptual level and the interface level.

The two conceptual failings of the current system are that some posts can't be pidgeonholed into one category, and my attempt at categorizing things hierarchically didn't pan out -- sometimes a sub-category belongs under one parent, sometimes another, other times no parent at all.

My ideas for resolving the problems of the current system would eliminate the conceptual problems, but would introduce a whole slew of interface problems. The solution would be simply to do away with the hierarchy and replace it with a wider set of categories (a la memepool), and allow posts to live in more than one category. A wide set of categories would take up even more screen space than my categories already do. I don't want a quick glance at that section of the page to overwhelm the random surfer who happens upon the site and scare them away.

Okay, so put the categories on their own page, right? Having seen how people browse my site by watching my logs, not having any categories on the front page would be foolish. You see, when a random surfer hits my page, if they stick around at all one of the first things they do is click on a category.

So here's what I'm thinking... I'll have a wide set of categories where a post can live in multiple categories, and each post will have a list of the categories it's in so that if someone reads an item on my page they like, they have a quick indication of where to look for similar posts. Then, on the front page, I'll take some sort of weighted average of the most highly populated categories and the most clicked on categories and display them on the front page, along with a link to "more categories..."

Sorry, I was just thinking out loud. I probably won't have time to implement any of this until... ::checks palm:: July, so... Hey, count your blessings, at least I'm just rambling about this instead of about how foul my day was!

A quick list of places where I have digital information which I find myself having to keep track of: Four computers (though only two are used regularly), a Palm Vx, a cell phone, a Diamond Rio, a web site, nine unix shell accounts (five of these I don't actively use. The other four cannot be consolodated.), three database servers (each with numerous sub-databases), one web page, and five email addresses (which I have to check in three different ways).

The items on the above list are used to manage my contacts, music, bookmarks, todo lists and calendar items, email, "work", and a broad category which can only be called "files". Note that the first list is more than twice as long as the second list. Wouldn't it be nice if my PDA also contained my cell phone and my mp3 player, and if that pda could effortlessly synchronize it's contact list with my email address book? There are a lot of things I could do to simplify my information management tasks, but a lot of them rely on my tools of choice talking to each other. Which they don't.

So when I read this article on news.com about .NET and what Microsoft wants to do, I felt myself wanting to give in. The idea of a person centric paradigm instead of a device centric paradigm made my head spin. "Imagine the possibilities!" I thought to myself. No more having three copies of my contact list, no more managing three different email accounts, etc. I felt myself wanting to give up and embrace Microsoft's person centric information management strategy. I was even convinced that this would be something worth paying for!

Even now, the idea sounds so appealing... All I have to do is stop resisting and agree to put my faith in Microsoft to manage my information for me, let my information live on Microsoft's hardware instead of my own, hand over my soul and embrace Microsoft as my one true lord and savour from the information overload which is apparently getting to me big time.

But then I remind myself that Mac OS X is coming out in 4 days. (Which is scary to remind myself of, because that means I have a term paper, a cs project, and two homework assignments due in less than 3 days) Then I tell myself that other companies will implement solutions that require less submission. I know that Mac OS X won't solve all of my problems out of the box, but I really hope that it will serve as a foundation on which tools will be built which can help me consolodate my information.

While browsing around today, I came across the University of Washington's Busview system. I'll forgive it being a java applet merely because the application is so cool -- real time tracking of busses around the university. In addition to that, if lets you set alarms for when a bus arrives at a certain point, or find out how late a bus is running. I'd love for an application like this for either the berkeley campus shuttles or AC Transit. Then I could find out if I'd be waiting a stinking half an hour by Tolman hall for those stupid campus shuttle which apparently take a lunch break -- "Hold on while I double park this bus and run into micky-d's and grab a big mac..." In addition to that, an application on my cell phone allowing me to look up all this data would make for a killer combination which would probably cost the bus system way more than the number of new riders it would win. Actually, no bus system that wanted to make business would ever implement such a tracking system -- I know if I was able to look up how late my bus was running, it would just make me more likely to hoof it to my destination.

So someone sent me a link to Guess the Dictator/Sit-Com Character, and instead of reading the directions (bah!) I just started answering questions as if they were directed at me. Am I female? No... Do I have a Job? Yes... I did this for awhile, and eventually asked me if my show was about nothing. It was about at this point that I started to catch on. Anyway, I guess if I were a sitcom character, I would be Seinfeld, though that wasn't what the page is supposed to do.

After that I did some characters from Friends, and then I was all out of sitcom characters so I played with changing the background color of the page, and idly wondered why the page doesn't have a domain name before I wandered back to my linguistics homework.

Today while reading use Perl; I noticed that there was a new module named DBIx::XML_RDB. Now, as my job is composed almost entirely of XML and Database work right now, this module sounded interesting enough for me to check it out.

And check it out I did, and interesting it looks, but I'd still dig something on the level of Oracle's XML Tools in an open source package, because what this particular Oracle document describes sounds like a very well integrated and robust package. Too bad I don't have it to play with. Maybe I'll get a chance to work with real XML/Database tools some day, and not be stuck with Perl's half assed implementations. (They're not all bad, but the best ones I've found are the ones that are simply perl bindings onto libraries in other languages.