Google Alerts For Fun and Profit

I had a request today from a customer who wanted to make a list of products he was waiting for, and to be notified when Mighty Ape had them on the site ready to order. I decided that it was going to be problematic to implement this as a Mighty Ape feature, but I did help him sort out something that works just as well, using free tools available already.

Say you wanted to be notified when Mighty Ape makes the Transformers 2: Revenge Of the Fallen Blu-Ray available, here’s how you could go about doing it.

First, construct a Google query that narrows down what you’re after.

Since you’re interested in a product, make sure you add the keyword “site:mightyape.co.nz/product” to the query. That will restrict Google to search within that directory on our site.

We’re only interested in the Blu-Ray version so add that in there, too as “blu-ray“.

Now add some keywords that describe the product. Try and stay away from being too specific, filler words, numbers and punctuation. Something like “transformers revenge fallen“.

Now, using that might turn up some strange results, from other product’s descriptions or similar. Luckily, Mighty Ape stores the product name and format all in the page title, so we can direct Google to match only that by using the “allintitle:” keyword.

So put that all together and we have:
allintitle:transformers revenge fallen “blu-ray” site:mightyape.co.nz/product

Nice. Now all we need to do is put that into a new Google Alert and Google will let you know when new documents turn up matching your query in your E-Mail inbox, or RSS feed.

Enjoy! This can easily be adapted to other sites or other queries. Anybody got any other cool examples of useful Google Alerts?

100 Years of Change Adversity

Ars.Technica has a fantastic article up titled “100 years of Big Content fearing technology—in its own words“.

The article basically touches over most major technological advances over the last centry, and digs up what “Big Content” had to say about it at the time. Big Content being Music, Film and Print industries.

Such business-busters as the Xerox machine:

“the day may not be far off when no one need purchase books”

The VCR:

We are going to bleed and bleed and hemorrhage, unless this Congress at least protects one industry that is able to retrieve a surplus balance of trade and whose total future depends on its protection from the savagery and the ravages of this machine.

And of course, MP3:

We’ve come full circle here, as this is the inverse of Sousa; a new technology won’t eliminate the amateurs, it will eliminate all the professionals and leave nothing but amateurs.

Of course, this seems all too familiar. The roadblocks and excuses being thrown up today are the same that were spun up all those years ago, those which failed to come to fruition. The same things are being threatened – the death of industries, the loss of jobs and the destruction of dreams.

However, what can be said of all of these advances is that they caused change. They caused a reshuffle of the industry in question, but eventually everything settled down again to get back to what they were doing: making money. In fact, if I remember correctly, most content industries are in boom, some making more money than they ever had (with the exception of newspapers, but I feel they’re the author of their own demise).

So “Big Content” is quite comfortable where it is, thank you very much, and is apparently quite happy to fight tooth and nail against anything that may cause change. The problem is, these industries have a lot of teeth and too many nails. We have the huge lobby groups, the MP/RIAA and the very prominent spokespeople with their voice in the ear of our politicians.

This is why so many people keep an eye on the laws being passed in our names, on behalf of these industries. We need to watch whats going on in the fields of Copyright and Intellectual Property (don’t even get me started about software patents) before we regulate ourselves out of evolution.

Speeding up INSERTs on MySQL

Was working on a problem today that required a lot of bulk inserts to a MySQL table. I was getting about 200 inserts/second on my development system, which is OK considering there was some minor processing going on.

Since some batches can contain well over a million rows, I started working on how to optimise these queries so we can get them in there faster.

Firstly, watch your indexes on the table. Obviously the more indexes you have on the table the more work the DB has to do on INSERT to maintain them.

Second, if possible, ditch your ORM. Instead of building and hydrating objects for each row in the table used direct/prepared queries. Most ORMs worth their salt can handle this. i.e. Propel can give you direct access to the underlying PDO connection to use your prepared statements.

/** @var $con PDO */
$con = Propel::getConnection();

Obviously if you have business logic tied up in your objects, it’s best to use them instead of duplicating code.

Third, lock the table for a batch of inserts. Here’s some sample code

$i = 0;
$con = Propel::getConnection;
$con->beginTransaction();
 
foreach ($rowsToInsert as $row) {
 
  if ($i % 100 == 0) {
    $con->commit();
    $con->beginTransaction();
  }
 
  TablePeer::insertUsingPreparedStatement($row);
  $i++;
}

If your using MyISAM you can lock the table using the LOCK TABLE statement

$con->exec('LOCK TABLES table_name WRITE');
$con->exec('UNLOCK TABLES');

Depending on how important your data is you can change the (100) value in the $i % conditional to higher or lower. The reason this speeds up Inserts so much is that MySQL won’t flush it’s write cache to the disk until the transaction is finished, as opposed to every INSERT statement. However, having unflushed data in your cache is dangerous because it may disappear if something happens to the DB server, or get rolled back if your script carks it. Also, since a transaction/lock table call will stop all other access to the table, if it’s frequently read from those queries will be waiting on locks, so it’s good to refresh them frequently.

Using these three methods I almost tripled the performance of my script which now inserts between 550-600 rows per second. Win!

There’s some more tips over here in the MySQL manual and some of the comments are quite helpful too.

Nevermind… Lives!

Uh, yeah, back from my blog-free stupor.

Google OS… Lives!

Some fairly big news today!

Google has announced that it’s going to compete with Microsoft in the OS wars. Their long-denied Linux-based Google OS is due to be released next year.

Is 2010 the year of the Linux desktop?

TechCrunch has an analysis here and, as usual, quality work from Ars.Technica.

Some interesting tidbits:

Google says the software architecture will basically be the current Chrome browser running inside “a new windowing system on top of a Linux kernel.” So in other words, it basically is the web as an OS.

You may not remember… but Microsoft actually tried this with Active Desktop and IE4. You don’t need to be told that it failed miserably, but if anybody can make it work I think it’s Google. Also interesting to note that this is the way that Palm has gone with their WebOS based smartphone, the Palm Pre. The proof will be in the pudding.

I think what’s going to make all this possible is the standardisation of HTML5. HTML5 has fairly extensive provisions for offline work in web applications (Web Storage) which Google also has in Gears. They also say that any application developed for the new Chrome OS will work on any standards-compliant browser. Microsoft seriously better pull finger with their HTML5 support or they will run the very real risk of being left behind…

All and all, a great development. Lets see where it goes.

New Server

Moved Nevermind to a new server in the US – because it’s cheaper, of course! Basic VPS so I can run some Rails apps I’ve been hacking on.Also has a decent control panel and DNS manager.

Some weirdness… hopefully not too much, let me know if you find anything strange…

Next step: Setting up automatic installs for the wordpress upgrade and plugins!

Worth-less John Key

National’s handling of this Richard Worth affair has been bad, both ethically and politically. Even worse for them, Labour leader Phil Goff has come out of it smelling like roses, ratcheting his profile up a few notches.

I assume Crosby/Textor are on holiday (celebrating a 2-point clarification from Nicky Hagar, perhaps?) because even John Key himself, who’s spin is usually water tight, is beginning to fall over.

For instance, this interview on RNZ Checkpoint last night is an absolute stunner. Key refuses to answer questions, attacks Goff, attacks the victim, defends Worth and gets caught out lying.

Also, on Breakfast, Paul Henry shakes off his bias and gives a Key hard hitting interview instead of a handjob. He’s been previously guilty of dishing up patsie but he can seemingly smell blood and really tears shreds off him.

Hmmm, as much as Paul Henry gets my goat, the reality is that he grills John Key better than anyone else currently in the media, it’s always as if as a former National Party candidate that Paul has been personally offended by the Prime Minister’s lack of a red neck. –Bomber Bradbury

Also… Textses? Piggy in the Middle? Was also funny to hear him saying “I don’t sack ministers on unproved allegations”, when that was what he was calling Clark to do to Winston last year. What happened to 1-strike you’re out?

Meanwhile, Kiwiblogh desperately tries to spin the issue off National and onto Goff – no surprises there.

Oprah, The New Religion?

Just a quick post but this is definitely worth a read.

Should people with this much influence be more careful about what they’re pushing? Reading a bad book she pushed is one thing, ruining your life with hormone therapy is another… Kudos to Newsweek for this one.

Newsweek: Live your best life ever

On the Contrary…

“I am tired of politicians who say it is probably inevitable we will become a republic at some stage, but who are unwilling to do anything to bring it about – that is extremely weak.”
– Peter Dunne, Time to have our say on republic issue – 26 April 2009

“As I have always said in the past one day it’s likely New Zealand will become a republic but, I don’t think anything is going to happen under my watch.”
– John Key, Key pours cold water on republic referendum – 27 April 2009

It must feel like being savaged by a rather wet goldfish.

Hmmm… “John Key plays into the hands of political genius Dunne”? No, that doesn’t sound right.

“Dunne calls out John Key as ‘weak’”

Can someone help me out here?!

Auto-tune the News

With all this doom and gloom on the news today, isn’t it much nicer when the anchors and bobble-heads sing?

Isn’t this great? And wouldn’t you like Katie Couric to carry a tune?

And while we’re laughing at Auto-tune, here’s Baby T-Pain: