Entries tagged as postgresql
Related tags8.4 cte fsm window aggregates 8.5 win64 windows 9.0 conferences stockholm sweden talk 9.1 extensions fdw logging pgcrypto amazon build cloud ec2 git android lists mail mailman majordomo backup commit encryption fosdem jdcon keynote pgcon pgconfeu pgconfeusite pgconfeusite2012 pgday pgeast pgeu pgwest photos pictures python schedule security ssl core cvs ddl django intrastructure pgweb varnish hot standby munin ipv6 limit sql union mysql pgindent ubuntu
Monday, May 2. 2011
Remote log reading in PostgreSQL 9.1 Posted by Magnus Hagander in PostgreSQL at 19:05
PostgreSQL 9.1 beta1 now available - now is a great time to start testing it, and trying out all the great new features.
There have always been a number of ways to read your PostgreSQL logs remotely, over a libpq connection. For example, you can use the pg_read_file() function - which is what pgadmin does. PostgreSQL 9.1 adds a new and more convenient way (in some ways) to do this - using SQL/MED.
PostgreSQL 9.1 comes with SQL standard SQL/MED functionality. The MED in is short for "Managemend of External Data", and as the name sounds, it's about accessing data that's external to the PostgreSQL server. The SQL/MED functionality is not (yet) complete, but it's already very useful in it's current state.
In SQL/MED, there is something called a Foreign Data Wrapper, that can be compared to a driver. Using this FDW, we can create one or more Foreign Servers, which is a definition of how to connect to a specific instance of the service - if any. Finally, we can create one or more Foreign Tables on each of the Foreign Servers, giving us direct access to the remote data using SQL.Continue reading "Remote log reading in PostgreSQL 9.1"
Wednesday, April 27. 2011
Joining the PostgreSQL Core Team Posted by Magnus Hagander in PostgreSQL at 19:02
As has just been announced here, I was recently invited to join pgsql-core, and have accepted.
I guess the guys currently on it finally got tired of all my complaints, and figured out the way to make me stop was to suck me into the organization. The future will tell if their strategy will be successful or not...
For those who don't know (this hopefully doesn't include my readers on Planet PostgreSQL), pgsql-core is the "steering committee" for the PostgreSQL project. Exactly what they do seem to be somewhat up for debate both outside and inside of the group itself, but it at least has something to do with the leadership of the project...
Anyway, I'd like to thank the guys in the group for showing this trust in me, and shall do my best not to screw it up!
Tuesday, April 12. 2011
PGConf.EU 2011 will be held in ... Posted by Magnus Hagander in PostgreSQL at 08:13
It's time to mark your calendars: PostgreSQL Conference Europe 2011 (formerly known as PGDay.EU) will be held on October 18-21 at the Casa400 Hotel in Amsterdam, The Netherlands.
Like last year, the conference will be held in a hotel venue, combining both the conference rooms and guest rooms, so you don't have to waste any time finding your way around the city. As in previous years, the conference will include full catered coffee breaks and lunches, to make the most of the time. The first day of the conference will be a training day, and the following three days will be regular conference tracks. The conference will accept talks in English, Dutch, German and French, to benefit those attendees who prefer talks in their native language.
We are just starting our search for sponsors - if you are interested in sponsoring the conference, or know someone who is, please take a look at our sponsorship opportunities and don't hesitate to contact us if you have any questions or would like to propose an alternative arrangement.
We will also follow up with a call for papers later, and in due course open for registration and post a conference schedule. For now, mark the dates, and follow the news on our website and on our twitter stream @pgconfeu.
Monday, March 14. 2011
Training at the increasingly ... Posted by Magnus Hagander in PostgreSQL at 12:20
Next week it's time for PgEast: 2011, this time in New York City.
I've already outlined why the East part of "PostgreSQL Conference East" (as it was called at the time) is incorrect: as is obvious to anybody with a basic knowledge of geography, the conference is to the west. From what I can tell, it's approximately 74 degrees west of zero, which means it's more than 20% of the world to the west.
In expanding this scope, it seems JD has this year decided to get the rest of the name wrong as well, in a bid to get more people. Just like it's 20% of the world wrong in location, it's no longer a PostgreSQL conference. Instead it's more of a cross-database conference, with an entire track dedicated to MongoDB (incidentally, approximately 20% of the tracks, it seems). Is that bad? Absolutely not - I'm looking forward to sneaking in on one or two of those MongoDB talks. But I think it means we have to go back to the proper name for the conference - JDCon-East!
And I'm sorry JD, but whatever numbers you get, you will not be the biggest PostgreSQL conference around. We are going to have to leave that title where it belongs - with the Brazilians (for now).
This year, the conference is also running a full 7 parallel training sessions the day before the actual conference. As part of this, I'm giving a half-day training on Streaming Replication and Hot Standby. If you haven't registered for it already, there are still seats open! And tell your friends - since this is how my trip there gets funded, I'd really like to get a full session...
I will also be giving a talk during the regular conference, Data Driven Cache Invalidation.
There's plenty of PostgreSQL - and MongoDB - around for everybody at this conference, so if you're anywhere nearby New York City, there is no reason not to be there!
Sunday, January 30. 2011
Yes, the mailinglists are down Posted by Magnus Hagander in PostgreSQL at 10:37
and along with them, a few other services.
From what we can tell, what has happened is that the datacenter that hub.org hosts most of their servers in, in Panama, dropped completely off the Internet several hours back. The PostgreSQL mailinglists are managed by hub.org, and is tied into their main infrastructure. For this reason, there is nothing the rest of the sysadmin team can do other than wait for the situation to resolve, and we unfortunately have no chance to bring up any backup servers anywhere.
As an added unfortunate bonus, it seems at least one of the hub.org nameservers is still running an incorrectly configured DNS zone file. This means that while this server is geographically hosted elsewhere, like it should be, email will get delivered to that host and then bounce saying that the postgresql.org domain does not exist. This is incorrect - the domain itself exists and works perfectly well, and if it wasn't for this incorrect zone file mail would be queued up and delivered once the main datacenter is back up.
Along with the lists, a few other services hosted with hub.org are currently unavailable - pgfoundry.org, pugs.postgresql.org, the developer documentation, jdbc.postgresql.org and possibly some other minor services.
All other infrastructure services are operating properly, including the website and the download mirrors.
Please be patient as we wait for hub.org to resolve this issue. For any up-to-date status information your best bet is the #postgresql IRC channel on FreeNode - but people are unlikely to be able to provide any information beyond "it's down, and we're waiting for hub.org".
Sunday, January 23. 2011
Another step towards easier backups Posted by Magnus Hagander in PostgreSQL at 12:46
Today I committed the first version of a new PostgreSQL tool, pg_basebackup. The backend support was committed a couple of weeks back, but this is the first actual frontend.
The goal of this tool is to make base backups easier to create, because they are unnecessarily complex in a lot of cases. Base backups are also used as the foundation for setting up streaming replication slaves in PostgreSQL, so the tool will be quite useful there as well. The most common way of taking a base backup today is something like (don't run this straight off, it's not tested, there are likely typos):
And when you're setting up a replication slave, it might look something like this:
There are obvious variations - for example, I come across a lot of cases where people don't bother checking exit codes. Particularly for the backups, this is really dangerous.
Now, with the new tool, both these cases become a lot simpler:
That simple. -Ft makes the system write the output as a tarfile (actually, multiple tar files if you have multiple tablespaces, something the "old style" examples up top don't take into account). -Z enables gzip compression. The rest should be obvious...In the second example - replication - you don't want a tarfile, and you don't want it on the same machine. Again, both are easily handled:
That's it. You can also add -P to get a progress report (which you can normally not get out of tar or rsync, except on an individual file basis), and a host of other options.
This is not going to be a tool that suits everybody. The current method is complex, but it is also fantastically flexible, letting you set things up in very environment specific ways. That is why we are absolutely not removing any of the old ways, this is just an additional way to do it.
If you grab a current snapshot, you will have tool available in the bin directory, and it will of course also be included in the next alpha version of 9.1. Testing and feedback is much appreciated!There are obviously things left to do to make this even better. A few of the things being worked on are:
Tuesday, January 18. 2011
pgindent vs dash Posted by Magnus Hagander in PostgreSQL at 12:13
For those who don't know, pgindent is the tool used to indent the source code of PostgreSQL. dash is the shell that ships as /bin/sh on at least Ubuntu.
pgindent requires indent from BSD (we use a patched version from NetBSD, the source is available on the PostgreSQL ftp site), and specifically does not work with GNU indent. Guess what Ubuntu ships with.
The solution is of course a small script that runs BSD indent from a different directory, and also points out the typedefs.list file from the PostgreSQL git repo. Something like this:
Spot the error? Yeah, that calls /bin/sh, which is dash. Which gives some really interesting results with pgindent, none of which are what you expect.
So if you run pgindent through a script like this, be sure to use /bin/bash and not /bin/sh!
Tuesday, December 28. 2010
Feedback from PGDay.EU the final ... Posted by Magnus Hagander in PostgreSQL at 11:57
The big change for PGDay.EU this year really was the switch from a university venue (first Monash University in Prato, then ParisTech in Paris) to a hotel venue (The Millennium Hotel in Stuttgart). We believe that much of the rest of the conference was an improvement over previous years - but it was an incremental improvement, whereas the change of venue was rather drastic. Looking at the feedback on this, I think we can conclude that this change was in general a positive one:
We're seeing a total of 75% who rate the venue as a 4 or a 5. Looking at the freetext comments, a large majority of them are very positive, but there are a few ones that stand out:
Amongst the positive ones we find a large number of comments saying that the "integrated venue" or "all inclusive" venue was a great step up.
Closely related to the venue, is the food. Unlike the big north American conferences PGCon and PG-East/West, we have for the past two years tried to provide proper lunches and not just sandwiches/boxed lunches. This obviously costs more money, but we believe it's worth it, and we think our visitors do. Last year we had a catering firm bring us assorted food, mainly cold cuts, at the conference venue, and this year we got proper lunch buffets (including multiple choices for dessert, of course..) at one of the hotel restaurants. I think the ratings speak for themselves - I would encourage those other conferences to look into improving their lunches as well!
A full 82% rated the food as 4 or 5. In the end, the cost for paying for a lunch "on ones own bill" would probably have cost more than half the conference fee - so we think we managed to provide some very good value. In fact, several people rated the food as being the best part of the conference(!)
There was, however, one person who said the food was one of the worst things about the conference - if you recognize that was you, we would very much like to know exactly why (no details were included) - please send me an email or write a comment here!
A few people commented on the large amount of food left over from lunch on at least one of the days - it is up to the hotel to decide what to do about that, but it is our belief that they do something "reasonable" with it - and not just throw it away. We know that the caterers last year delivered all leftovers to a nearby homeless shelter, for example. For next year, we will attempt to again get a specification from the catering/restaurant as to what happens to leftovers.
We feel that the overwhelming majority of our visitors found the changes an improvement, and we will therefor pursue something similar as our primary option for next year. We are always interested in improving further, of course, so if you have any other ideas - let us know! The final question we asked about the venue was where to hold the conference next year. Many were quite ambiguous in their suggestions ("big city in Europe" is in, "Hawaii" is out because we want to stick to Europe). Summarizing what we could gave us the following:
The final part of our evaluation was considering the conference website and registration:
In general these are very good rates. I'm happy to see that more than 50% rate the website overall experience as 4 or 5 - that's a much better rating than it's being given by the people who edit the content on it! Same for registration, with very few people rating it really low. There's clearly some room for improvement though:
In summary, here are some reasons in graphical and textual forms why you should already put attendance to next years PostgreSQL Conference Europe in your budget:
That concludes my summaries of the feedback from this years PGDay.EU conference. If your specific comments haven't been called out here, don't worry - we still read them all and will consider them all for next year!
Finally, thanks again to all who helped make this conference great!
See you again next year!
Wednesday, December 22. 2010
Feedback from PGDay.EU - the speakers Posted by Magnus Hagander in PostgreSQL at 15:37
The next issue of my "pie-chart-overflow blog posts about PGDay feedback" is about our speakers. The speakers are, if that's not obvious, the reason that people come to the conference. Having good speakers is an absolute requirement if we want to keep up the quality of the conference. Other things like venue and price are certainly important, but nothing compares to the actual content of the conference - which is provided by our speakers.
I'm very happy to say that we seem to have manage to keep the very high numbers for Speaker Quality that we had from last year (differing less than 3% which is well within the margin of error). The same goes for the scores our speakers got on their knowledge of the topic - indicating that we've managed to attract some of the most skilled speakers in the world. Which is not surprising given that in many cases, we the person speaking about a feature is actually the guy who wrote it. What is more surprising is that these same people are rated as very good speaker - which we all know isn't always true about your stereotypical developer.
Just like last year, we're not going to post the complete list of speaker ratings, given that they are easy to read wrong. But here is a list of our top speakers, excluding any that had less than 5 ratings. Any speakers who have fewer than 10 should be considered a very uncertain number, and I've again included the standard deviation to determine the uncertainty. We had a lot more speakers this year, so I have only included those scoring 4 or above this time around. Each speaker has received his own detailed score, of course.
The list based on Speaker Knowledge looks slightly different, but not very much. Given that our speaker knowledge has been rated even higher than speaker quality, I've only included those who scored 4.6 or higher (which is a fantastically high cutoff)
A great big thanks to all our speakers - you did a fantastic job.
We will need to work hard to keep up our recruiting of speakers for next years. If you were considering but decided not to submit a talk for some reason - please let us know why, so we can improve! Or if you have any ideas in general on our processes around this. For example, we had no female speakers at all this year - we know you're out there, and we certainly want you there, so what do we need to change to make this more interesting for you as a potential speaker? The same goes for other groups that we were missing of course: now is the time to let us know so we have the time to change things before next year!
Wednesday, December 22. 2010
Feedback from PGDay.EU - the contents Posted by Magnus Hagander in PostgreSQL at 09:53
This blog seems to be turning into a PGDay blog rather than a general PostgreSQL blog. But I promise I'll get back to some more technical content soon - or at least that I'll try.
A couple of days ago we closed the feedback system from PGDay.EU 2010, and have been busy tallying the result. It turns out that my constant nagging on people to please fill out the feedback worked - we got a lot more feedback this year than last year. That also means there's a lot more work in going through mainly all the freetext comments - that's the price I have to pay, I guess. In total we had around 60 people who left "full conference feedback", which is almost double from last year. It's still only just over 25% of the attendees, so it could certainly be even better yet. We also had 86 people who left session feedback (this is around 40% and a much better number of course) for a total of 570 session feedback entries.
So what did the feedback say - time for some pie charts! We've actually seen a slight decrease in the ratings for topic importance. This may well be because we've broadened the topics more. We're still seeing very good grades for content quality, which reinforces my feeling that our speakers deliver very valuable content to the attendees, and that the conference is well worth attending. (As a note to readers - I've had several people point out to me that german people are used to rating 1 being the highest and 5 being the lowest, so there may be some skewing in the voting because of this. Even though the pages very clearly stated that 5 is the highest, this is something we need to make even more clear for next year)
We spent a lot of time trying to put together the puzzle that is the schedule for so many talks over so short time. It turns out that we did a good job in general, but there was a large amount of overlap where people wanted to go to many talks at the same time. We also received a lot of comments in the freetext fields about this, and this is definitely something that we will consider for next year. It would probably have been better content-wise to have three tracks spread over three days (maybe not entirely complete) rather than four tracks over two days, but that would also have increased many of the costs with 33% which is a lot of money...
Of course, the "Hallway track" is a very important part of any conference like this, and this year we collected specific feedback on this side. I'm very happy to see that more than two thirds of our attendees rated the learning part of the hallway track as 4 or 5, and well over half found it a good way to connect with other people in the community!
If these numbers don't make you interested in next years PostgreSQL Conference Europe then, really, you're reading them wrong...
That's enough pie-charts for one post. I will follow this up with more feedback summary on our speakers and on our venue once it's ready.
Matt Spencer about PostgreSQL vs 64-bit windows
Wed, 06.06.2012 05:23
I agree totally, the 64-bit ve rsion definitely solves the pr oblem if you need more memory. We need to gradually sh [...]
Jim Smithson about PostgreSQL vs 64-bit windows
Tue, 05.06.2012 07:04
Users who are challenged by th e memory taxing qualities of t he 64 bit should go back to us ing 32 bit. It offers sm [...]
Magnus Hagander about PostgreSQL vs 64-bit windows
Thu, 19.04.2012 08:22
You are missing the point - wh ich is exactly that the filesy stem level cache does not ha ve a lower cache hit rat [...]
Marcov about PostgreSQL vs 64-bit windows
Mon, 26.03.2012 19:57
Any VM base caching is limited to the maximal mappable addre ss space(which is about 4G in windows 64-bit, 3GB in 3 [...]
Magnus Hagander about Finding gaps in partitioned sequences
Mon, 06.02.2012 10:18
Hah, thanks for pointing that out. That's what I get for try ing to clean up the formatting when posting it...
mark about Finding gaps in partitioned sequences
Tue, 31.01.2012 19:01
thanks, sorry I should have no ticed that was the seq MINUS t he lag function. p.s. you example may have a typo [...]
Magnus Hagander about Finding gaps in partitioned sequences
Mon, 30.01.2012 17:25
No, there are no user defined functions in this. seq is the column name, then minus the o perator, then lag the bu [...]
mark about Finding gaps in partitioned sequences
Mon, 30.01.2012 17:23
what is seq-lag ? i assume thi s is a user defined func.
Magnus Hagander about www.postgresql.org - brand new, yet old and familiar
Thu, 22.12.2011 17:41
Yeah, unfortunately we are sti ll waiting on David to migrate the content over there.
Thu, 22.12.2011 16:55
The weekly news page (http://w ww.postgresql.org/community/we eklynews/) seems to be broken. I see only a couple of [...]