Magnus Hagander's PostgreSQL blog (Entries tagged as intrastructure)

www.postgresql.org now active over IPV6 by default

nospam@example.com (Magnus Hagander) — Fri, 25 May 2012 20:01:22 +0000

For those of you who have been trolling our DNS details, you know that www.postgresql.org has been available over IPV6 for a while - just not activated in DNS. As of 15 minutes ago or so, we have now activated IPV6 for the main DNS record www.postgresql.org. Just like the IPV4 version, we are distributing the load across multiple frontends, with DNS based failover in case one of them were to go down. Right now, we have two IPV6 capable frontends and three IPV4 capable ones (the difference comes from our infrastructure in Europe being 100% IPV6 enabled, and our infrastructure in the US being 0% IPV6 enabled due to lack of upstream availability).

So if you are experiencing connectivity issues that started recently, check your IP stack if you are perhaps trying to connect over a broken IPV6 connection. If you need assistance beyond that, you can usually find helpful people in #postgresql on FreeNode who can help you figure out if the problem is on your end or ours!

Hopefully, this should be a completely invisible event to all our visitors....

www.postgresql.org - brand new, yet old and familiar

nospam@example.com (Magnus Hagander) — Wed, 21 Dec 2011 13:33:00 +0000

Most of the visitors to www.postgresql.org probably never noticed that a couple of weeks back, the entire site was replaced with a new one. In fact, we didn't just change the website - just days before, we made large changes to our ftp network as well (more about that in another post, from me or others). So in fact, we hope that most people didn't notice. The changes were mainly a technical refresh, and there hasn't been much change to the contents at all yet. We did sneak in a few content changes as well, that have been requested for a while, so I'm going to start with listing those:

The developer version of the documentation (updated serveral times per day from the tip of the HEAD branch that will eventually become the next version of PostgreSQL) now live on the main website, and will use the same stylesheets to look a lot nicer than before.
Anybody who submits content to our site (news, events, professional services, products, etc) will notice there is now a new concept of an Organisation. This means that it will finally be possible to have more than one person manage the submissions from a single company or group.
Again for those that submit content, it is now possible to view which of your submissions are still in the moderation queue, and it's also possible to edit something after it's been submitted. In fact, you can edit your items even after they've been approved. Any such editing will be post-moderated, and if this is abused that organization will be banned from post-moderation - but we don't expect that to ever be necessary.
And finally, for those that submit content again, we've switched to markdown to format your submissions, instead of a very random subset of allowed HTML tags.

The rest of the changes are under the hood, and it's mostly done for two reasons:

The technology powering the site was simply very old
The frameworks used were quite obscure, which severely limited the number of people who could (or wanted to) work with them

Hopefully these two changes will make it easier to contribute to the website, so if you're potentially interested in doing that, please read on!

Continue reading "www.postgresql.org - brand new, yet old and familiar"

Yes, the mailinglists are down

nospam@example.com (Magnus Hagander) — Sun, 30 Jan 2011 10:37:00 +0000

and along with them, a few other services.

From what we can tell, what has happened is that the datacenter that hub.org hosts most of their servers in, in Panama, dropped completely off the Internet several hours back. The PostgreSQL mailinglists are managed by hub.org, and is tied into their main infrastructure. For this reason, there is nothing the rest of the sysadmin team can do other than wait for the situation to resolve, and we unfortunately have no chance to bring up any backup servers anywhere.

As an added unfortunate bonus, it seems at least one of the hub.org nameservers is still running an incorrectly configured DNS zone file. This means that while this server is geographically hosted elsewhere, like it should be, email will get delivered to that host and then bounce saying that the postgresql.org domain does not exist. This is incorrect - the domain itself exists and works perfectly well, and if it wasn't for this incorrect zone file mail would be queued up and delivered once the main datacenter is back up.

Along with the lists, a few other services hosted with hub.org are currently unavailable - pgfoundry.org, pugs.postgresql.org, the developer documentation, jdbc.postgresql.org and possibly some other minor services.

All other infrastructure services are operating properly, including the website and the download mirrors.

Please be patient as we wait for hub.org to resolve this issue. For any up-to-date status information your best bet is the #postgresql IRC channel on FreeNode - but people are unlikely to be able to provide any information beyond "it's down, and we're waiting for hub.org".

PostgreSQL infrastructure updates

nospam@example.com (Magnus Hagander) — Sat, 27 Mar 2010 13:53:00 +0000

At PG-east^H^H^H^H^H^H^HJDCon here in Philly yesterday, Dave announced the infrastructure changes that we decided on late last year, and have been working on and off to get the groundwork done for over the past months (mostly off, or we'd be done already). Since I've had a number of questions around it, I'm going to post a summary here.

The bottom line is, we're changing - over time - how we're going to run and manage the infrastructure behind the postgresql.org services. Things like website, source code repositories, etc. The reason is simple - the way we are doing it now has become too hard to maintain, and just requires too much manual work.

There are several parts to these changes. For one, we are building automated methods for a lot of things that previously required a little manual work. But even a little manual work rapidly becomes very time-consuming when you have a lot of machines - and the number of machines we have have been increasing fast over the past couple of years.

The biggest change is that we will be moving our services off FreeBSD, that we use now, onto Debian GNU/Linux. At the same time, we will switch virtualization solution from FreeBSD Jails to KVM. FreeBSD jails have served us very well over time, but now that we have access to a well working KVM product that uses hardware virtualization, the gains of having full virtualization are much easier to get at.

One of the main drivers for the change is that maintaining ports-based installs are just taking way too much time. The new system will be based heavily around using Debian packages and the apt system, to the point that everything being installed will always be done using "meta-packages" that will ensure that all our machines look the same way. This also plugs into our monitoring systems very well, making applying (security) updates across the many machines much easier. In particular, it should get rid of what's sometimes a multi-day operation to get everything security patched, due to dependencies and the slowness of updating ports.

It is quite possible - in fact, I'd say probable - that there are perfectly fine ways of doing this on FreeBSD. But this outlines another big reason for this change - it's simply a lot easier to find people who can do these things on a Linux based system today. Our efforts are entirely volunteer based, and in the end we just don't have people who know enough FreeBSD volunteering to do this work. In fact, we have had several people retire from the sysadmin team over the years specifically because they refuse to work with FreeBSD. We don't expect this change to magically create more volunteers, but it will make it easier to recruit in the future.

The system will also automate a lot of the configuration work that needs to be done manually today, as well as automatically configure our monitoring systems (primarily Nagios and Munin) and backups. And finally, the system will automatically provision (and un-provision) users and access permissions across all machines from a central repository.

The code is not entirely done yet, but we do have enough in place to automatically provision and configure basic virtual machines in just minutes, as well as much

of the configuration. We have prototype deployments online, but we haven't started moving production services yet. We'll hope to get started on that shortly. It will obviously take a long time before we have migrated all services - in fact, we may never migrate all services. The focus will obviously be on the most manpower-intensive ones first.

As usual the code that's used to build these services will be published under the PostgreSQL licence once it's finished. Some security sensitive parts may be removed, but the bulk of it is very generic and should be re-usable as a whole or in parts.

Another issue that's going to delay some changes is the fact that a few of our infrastructure servers are currently not VT-capable, and thus cannot run KVM. In the short term these machines will continue to run FreeBSD Jails, but eventually we want to replace these machines with more modern ones. So if you are sitting on one or more VT-capable machines with enough RAM to run a number of virtual machines, hosted in a datacenter somewhere that can provide us with multiple IP addresses and a decent availability of network and services, that you would like donate to the PostgreSQL project, please get in touch with our sysadmins team to see if this is something we can work with!