A surprising? lesson in the speed of languages

Scenario: I had the need for a small tool that would parse logfiles of approximately 6 million lines. To each line apply two regexps to extract a few values, and using two separate dictionaries/hashmaps (choose your poison wrt terminology) calculate how many times each capture group int he regexp shows up.

All in all, this is not going to be a very performance critical run, as it will only parse about 4 times these 6 million lines per day, so even at worst case we're talking less than 100M regexp matches per day. Piece of cake for any language. And since it's a weekend and I don't have any time when I have to deliver this finished script, I set out to prove to myself that python is going to be fast enough for this.

Continue reading

www.postgresql.org - brand new, yet old and familiar

Most of the visitors to www.postgresql.org probably never noticed that a couple of weeks back, the entire site was replaced with a new one. In fact, we didn't just change the website - just days before, we made large changes to our ftp network as well (more about that in another post, from me or others). So in fact, we hope that most people didn't notice. The changes were mainly a technical refresh, and there hasn't been much change to the contents at all yet. We did sneak in a few content changes as well, that have been requested for a while, so I'm going to start with listing those:

• The developer version of the documentation (updated serveral times per day from the tip of the HEAD branch that will eventually become the next version of PostgreSQL) now live on the main website, and will use the same stylesheets to look a lot nicer than before.
• Anybody who submits content to our site (news, events, professional services, products, etc) will notice there is now a new concept of an Organisation. This means that it will finally be possible to have more than one person manage the submissions from a single company or group.
• Again for those that submit content, it is now possible to view which of your submissions are still in the moderation queue, and it's also possible to edit something after it's been submitted. In fact, you can edit your items even after they've been approved. Any such editing will be post-moderated, and if this is abused that organization will be banned from post-moderation - but we don't expect that to ever be necessary.
• And finally, for those that submit content again, we've switched to markdown to format your submissions, instead of a very random subset of allowed HTML tags.

The rest of the changes are under the hood, and it's mostly done for two reasons: * The technology powering the site was simply very old * The frameworks used were quite obscure, which severely limited the number of people who could (or wanted to) work with them

Hopefully these two changes will make it easier to contribute to the website, so if you're potentially interested in doing that, please read on!

Continue reading

Heading west, going east, better schedule, and more

(public service announcement further down!)

I'm starting to get ready for the most mis-named conference so far this year. I mean, seriously. It's called East 2010, and yet it's located approximately 6000 km (that's almost 4000 miles for you Americans) to the west of the prime meridian. That's not even close. Sure, it's to the east of where the West conference is usually held, but really, this reminds me of POSIX timezones...

This will be a somewhat different conference than previous PostgreSQL conferences. It's the first big commercial conference. This has enabled it to change venue from "local university or college where rooms are cheap or free" to a conference hotel in downtown Philadelphia. Whether this is actually good for the talks is yet to be seen, but it's likely going to make some things around the conference easier and more integrated. There will also be a exhibition area - something we have tried for pgday.eu without much interest, but it will hopefully be more successful in this surrounding.

The contents of the conference are also somewhat different, since there is now a clear focus also on "decision makers", something that many PostgreSQL conferences have been lacking in. We may all want it to be true that the decision makers are the people who are actually going to use the product, but we know that's not true. This gives a conference schedule that contains a broader range of talks than we're used to - this can only be good.

Myself, I will be giving an updated version of my security talk, focusing on authentication and SSL. The updates are mainly around the new and changed authentication methods in 9.0, and some minor updates on the SSL part. If you've seen it before it may be an interesting refresher, but you might be better off going to see Greg Smith's benchmarking talk if you haven't...

Now for the second part, which is the public service announcement...

If you're like me, you find the official schedule for East very hard to read and basically useless to get an overview. It's also horrible to use from a device like the iPhone, something I'm quite likely to use at the conference. But we solved this problem for the PostgreSQL Europe conferences, and that code is pretty simple. So some copy/paste of a couple of hundred lines of python, some glue code, and voilà, a much more (IMHO) readable schedule. As a bonus, it also generates an aggregate iCalendar feed, that you can plug directly into the calendar application on the iPhone (or I assume most other phones). Google calendar may be very nice to use to work on the schedule, but I find this a lot more user friendly for those reading it - particularly in the ability to get an overview.

The page and the feed will both update once per hour by pulling from the official calendar feeds. They will also adjust for the fact that all the official feeds are 3 hours off due to the calendars being set in the wrong timezone. So should they suddenly jump 3 hours that's because the official ones were fixed - just remind me and I'll take out the adjustment. Anybody is free to use them, but of course, usual disclaimers apply, double check with the official one, etc, etc, etc.

Finally, then mandatory before

and after shots.

If you're interested in the code for the aggregator itself, it's up on my github page. And of course, any patches for cool features or just making it look better are always appreciated - it's open source after all.