link

A great article on the upcoming Google/Microsoft architecture war, by Charles Ferguson of High Stakes, No Prisoners fame.

link

link

I rather like these abstract, almost poetic postlets that the moose has been leaving on her blog.

link

RSS Digest will show or syndicate an Atom or RSS feed on any web page. This is a feature many people want for their blogs. Many more than know of services like RSS Digest.

Incidentally, this is also a feature that frassle offers. Although in the current version of frassle, it's mired within the difficult-to-use publisher.

link

What's been distracting me from frassle for a week or so: a new site for collaboratively building taxonomies, so you can use them in any search engine you want. For you inner list-making nerd. More info soon.

link

Hi Peter,

Thanks for your interest in my SQL optimizations. I'm gradually making the review of all frassle pages and timing the queries, so I can speed up the worst offenders and make the system snappier.

The main page was an easy place to start. Its slow query was the one that shows recent update times. Previously, it used a subquery to calculate the latest update time for each row. Something like:

SELECT feed_id, blog_title, (SELECT MAX(note_time) FROM fa_notes n WHERE n.feed_id = f.feed_id) as last_post_time FROM fa_feeds f ORDER BY last_post_time DESC LIMIT 15;

One problem was in the way the fake last_post_time column was being generated. Although the fa_notes table has an index on the feed_id column, PostgreSQL was doing a sequential scan of that table in order to find the maximum (latest) post time. But even with the use of the index on the subquery, it was uselessly slow to calculate the last_post_time for each feed and then throw away all but the top 15.

The solution was fairly simple: instead of recalculating it with every page load, store the last_post_time for each feed in the feeds table. I added this column, populated it once using something like the query above, and then added an update trigger to the fa_notes table. The trigger runs whenever a blog post is inserted or updated, and if that post's time is greater than its feed's last_post_time, the last_post_time is updated.

Lesson: In retrospect, this inefficiency seems obvious. But early in the design of a system, it's hard to know which parts of it will be too slow. Sometimes, the database is smart enough to optimize things that would take any reasonable piece of software 14.7 days. Other times, it does exactly the opposite of the smart thing and your clever design succeeds only in confusing the database (and probably yourself and your colleagues). On the balance, I think it's best to stay flexible and react once you have some actual data:

SELECT COUNT(1) FROM fa_notes;
(wait 5 minutes)
363488

Hmm… I guess optimization is never finished.

I messed around with some SQL, and I think the home page should load a lot faster now.

link

Sadly, frassle's autosave feature has not yet debuted for the public world. Perhaps I can get it out of the lab this weekend, with some encouragement. Would anyone like to not lose posts anymore?

link

Cool story, Jennifer! Thanks.

Do I think this would work? I'm not sure. It does seem that some Chinese programmers have immigrated to the US and work successfully as hackers in companies here, so there must be some individuals of sufficient creativity to do at least that work. It is a challenge of building and communicating corporate culture, but I think that creativity *can* be nurtured in the right kind of institution. You're right in saying that it can't be built and budgeted like other products, but I think leveraging the competitive structure that people might already be used to could help it work. Ultimately, though, the real assurance is that China will someday need creativity in its economy. Perhaps not as much as the US does, but globalization will push China a little bit in that direction—for sure.

link

Philip Greenspun asked, "what kind of business should one start given that one already has a person on the ground in Shanghai?" Here's my response.

You should start a firm that, rather than doing commodity IT outsourcing, generates solutions to hard problems using a high-volume competition model. The goal is to drive creativity through competition: a customer provides a high-level challenge (I need to improve the efficiency of our widget manufacturing process), and you bring many different minds to bear on that challenge.

This is contrary to the conventional wisdom that Chinese offshoring is for uncreative work, but I think that (1) it has a chance of actually working, and if does work it will be huge and (2) it can exploit the population size advantage. The key is the competition model. In a nation as populous yet per-capita poor as China, those who get as far as technical school have made the cut over and over again. They are ruthlessly competitive, expert in doing things exactly right.

This has the unfortunate consequence of making them somewhat less creative, focusing on efficiency and correctness of product rather than originality. Still, out of a hundred top Chinese programmers, there must be a handful of already creative ones. In the right environment, others around them can learn to appreciate creativity. And if you want to motivate those others to learn faster, why not use the same motivator that has pushed them through years of difficult schooling? You can even recruit by inviting solutions to open research problems and offering a cash prize. Within the corporation, give the most creative employees a higher status (but do not stratify too much within these ranks in order to encourage mixing, freeform discussion, and serendipity).

Think of it as the marriage of Silicon Valley risk-taking and Chinese labor costs.