Viewing entries tagged with postgresql. Return to full view.

Connecting to Azure PostgreSQL with libpq 12 in a Kerberos environment

If you are using Azure PostgreSQL and have upgraded your client side libpq to version 12 (which can happen automatically for example if you use the PostgreSQL apt repositories), you may see connection attempts fail with symptoms like:

$ psql -hZZZZZZ.postgres.database.azure.com -dpostgres -UXXXXX_dba@ZZZ-db01
psql: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

With no log information whatsoever available. This can happen if your client is in a Kerberos environment and has valid Kerberos credentials (which can be verified with the klist command). In this case, PostgreSQL 12 will attempt to negotiate GSSAPI encryption with the server, and it appears the connection handler in Azure PostgreSQL is unable to handle this and just kills the connection.

When running the same thing against a local PostgreSQL server prior to version 12, a message like the following will show up in the log:

2020-02-20 10:48:08 CET [35666]: [2-1] client=1.2.3.4 FATAL:  unsupported frontend protocol 1234.5680: server supports 2.0 to 3.0

This is a clear indicator of what's going on, but unfortunately the information isn't always available when connecting to a managed cloud service, such as Azure PostgreSQL. The hard error from Azure also prevents libpq from retrying without GSSAPI encryption, which is what would happen when connecting to a regular PostgreSQL backend or for example through pgbouncer.

The fix/workaround? Disable GSSAPI encryption in the client:

$ export PGGSSENCMODE=disable
$ psql -hZZZZZZ.postgres.database.azure.com -dpostgres -UXXXXX_dba@ZZZ-db01
Password for user XXXXX_dba@ZZZ-db01:
psql (11.6 (Ubuntu 11.6-1.pgdg16.04+1), server 9.5.20)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=>

If you have this type of issue, it's probably worth putting this environment variable in your startup scripts. It can also be set using the gssencmode parameter as part of the connection string, in environments where this is more convenient.

When a vulnerability is not a vulnerability

Recently, references to a "new PostgreSQL vulnerability" has been circling on social media (and maybe elsewhere). It's even got it's own CVE entry (CVE-2019-9193). The origin appears to be a blogpost from Trustwave.

So is this actually a vulnerability? (Hint: it's not) Let's see:

Continue reading

PGConf.EU 2018 - the biggest one yet!

It's now almost a month since PGConf.EU 2018 in Lisbon. PGConf.EU 2018 was the biggest PGConf.EU ever, and as far as I know the biggest PostgreSQL community conference in the world! So it's time to share some of the statistics and feedback.

I'll start with some attendee statistics:

451 registered attendees 2 no-shows 449 actual present attendees

Of these 451 registrations, 47 were sponsor tickets, some of who were used by sponsors, and some were given away to their customers and partners. Another 4 sponsor tickets went unused.

Another 52 were speakers.

This year we had more cancellations than we've usually had, but thanks to having a waitlist on the conference we managed to re-fill all those spaces before the event started.

Continue reading

Tracking foreign keys throughout a schema

I recently ran into the need with a customer to track the usage of a specific key throughout the schema. Basically, "what are all the tables and columns referencing this key, directly or indirectly". Luckily, with a little bit of catalog query, that's not hard:

WITH RECURSIVE what (tbl) AS (
   VALUES ('public.tt')
),
t (oid, key, constrid) AS (
 SELECT tbl::regclass::oid, conkey, NULL::oid
  FROM what INNER JOIN pg_constraint ON (contype='p' AND conrelid=tbl::regclass)
UNION ALL
 SELECT conrelid, conkey, c.oid
 FROM pg_constraint c
 INNER JOIN t ON (c.confrelid=t.oid AND c.confkey=t.key)
 WHERE contype='f'
)
SELECT nspname, relname, key, ARRAY(
    SELECT attname FROM pg_attribute a WHERE a.attrelid=t.oid AND attnum=ANY(key)
  )
FROM t
INNER JOIN pg_class cl ON cl.oid=t.oid
INNER JOIN pg_namespace n ON n.oid=cl.relnamespace

The output can be similar to:

 nspname | relname | key | array 
---------+---------+-----+-------
 public  | tt      | {1} | {ttt}
 public  | foo1    | {1} | {a}
 public  | foo2    | {3} | {z}

for a single column key (tt being the table with the primary key in, and the foo1 and foo2 tables referencing it directly or through the other one), or:

 nspname | relname |  key  | array 
---------+---------+-------+-------
 public  | m1      | {1,2} | {a,b}
 public  | m2      | {1,2} | {a,b}

for a multi-column foreign key.

In this particular use-case, it was an efficient way to track down key usage where naming standards for using the key had not always been followed. And of course, we also found a couple of cases where the column had the correct name but lacked the actual FOREIGN KEY definition, but that was done by just looking at the column names.

Updates about upcoming conferences

Summer vacation times are over. Well, for some of us at least, clearly some are still lucky enough to be off, which is showing itself a bit (see below). But as both conference organisation and participation throughout the rest of the year is starting to be clear, I figure it's time to share some updates around different ones.

Postgres Open SV

First of all - if you haven't already, don't forget to register for Postgres Open SV in San Francisco in two weeks time! Registration for the main US West Coast/California PostgreSQL community conference will close soon, so don't miss your chance. I'm looking forward to meeting many old and new community members there.

PostgreSQL Conference Europe

Next up after Postgres Open will be pgconf.eu, the main European PostgreSQL community conference of 2018. The planning for this years conference is at full speed, but unfortunately we are slightly behind. In particular, we were supposed to be notifying all speakers today if they were accepted or not, and unfortunately our program committee are a bit behind schedule on this one. We had over 200 submissions this year which makes their work even bigger than usual. But speakers will be receiving their notification over the upcoming couple of days.

Hopefully once all speakers have been able to confirm their attendance, we will also have a schedule out soon. Until then, you can always look at the list of accepted talks so far. This list is dynamically updated as speakers get approved and confirm their talks.

We have already sold approximately half of the tickets that we have available this year, so if you want to be sure to get your spot, we strongly recommend that you register as soon as you can! And if you want to attend the training sessions, you should hurry even more as some are almost sold out!

PGConf.ASIA

Work on the program committee of PGConf.ASIA has also been going on over the late summer, and is mostly done! The schedule is not quite ready yet, but expected out shortly. You can look forward to a very interesting lineup of speakers, so if you are in the Asian region, I strongly recommend keeping an eye out for when the registration opens, and join us in Tokyo!

FOSDEM PGDay

As has been announced, PostgreSQL Europe will once again run a FOSDEM PGDay next to the big FOSDEM conference in Brussels in February next year. We hope to also run our regular booth and developer room during FOSDEM, but those are not confirmed yet (more info to come). The Friday event, however, is fully confirmed. Of course not open for registration yet, but we'll get there.

Nordic PGDay

Nordic PGDay has been confirmed for March 19th next year. The format will be similar to previous years, and we will soon announce the location. For now, mark your calendars to make sure you don't double book! And rest assured, the conference will take place somewhere in the Nordics!

Usergroups and PGDays

Then there are a number of smaller events of course. Next week, I will speak at the Prague PostgreSQL Meetup. We should be kicking off the Stockholm usergroup. PDXPUG runs a PGDay in Portland in September (which I unfortunately won't be able to attend). In general, it seems like usergroups are starting to get going again after the summer break, so check with your local group(s) what's happening!

What does it mean to be on the board of PostgreSQL Europe

With the upcoming elections in PostgreSQL Europe, I'm excited to see that we have more candidates than ever before. But during the FOSDEM conference we just finished in Brussels, that also lead to a fairly large number of people who asked me the simple question "what does it actually mean to be on the board of PostgreSQL Europe". So I think it's time to summarize that for both those standing for election, and for the members of the organisation in general.

For a TL; DR; version, being on the board basically means a lot of administrative work :) But read on for some details.

Continue reading

PGConf.EU 2017 - time for the statistics

For anybody following this blog, you'll know I do this every year. PGConf.EU completed several weeks ago, and we have now collected the statistics, and I'd like to share some numbers.

Let me start with some statistics that are not based on the feedback, but instead based on the core contents of our registration database. I've had several people ask exactly how we count our attendees when we say it's the largest PGConf.EU ever, so here are the numbers:

Our total number of attendees registered was 437. This includes all regular attendees, speakers, training attendees, sponsor tickets and exhibitor only tickets. Of these 437 people, 12 never showed up. This was a mix of a couple of sponsor tickets and regular attendees, and 3 training attendees. This means we had 425 people actually present.

We don't take attendance each day. Right after the keynote on the first day there were just over 20 people who had not yet shown up, and by the end of the conference the total that number was down to 12. There were definitely fewer than 400 people who remained on a late Friday afternoon for the closing sessions, but at lunchtime the crowd was approximately the same size.

On top of the 437 actual attendees, we also had 5 further sponsor tickets that were never claimed. And we had another 59 people still on the waitlist, since we were unfortunately up against venue limits and we not able to sell all the requested tickets.

Continue reading

ftp.postgresql.org is dead, long live ftp.postgresql.org

As Joe just announced, all ftp services at ftp.postgresql.org has been shut down.

That of course doesn't mean we're not serving files anymore. All the same things as before area still available through https. This change also has an effect on any user still accessing the repositories (yum and apt) using ftp.

There are multiple reasons for doing this. One is that ftp is an old protocol and in a lot of ways a pain to deal with when it comes to firewalling (both on the client and server side).

The bigger one is the general move towards encrypted internet. We stopped serving plaintext http some time ago for postgresql.org, moving everything to https. Closing down ftp and moving that over to https as well is another step of that plan.

There are still some other plaintext services around, and our plan is to get rid of all of them replacing them with secure equivalents.

Setting owner at CREATE TABLE

When you create a table in PostgreSQL, it gets assigned default permissions and a default owner. We can alter the default privileges using the very useful ALTER DEFAULT PRIVILEGES command (a PostgreSQL extension to the standard). However, there isn't much we can do about the owner, which will get set to the role that is currently active. That is, it's the main login role, or another role if the user has run the SET ROLE command before creating the table.

A fairly common scenario that is not well handled here is when a number of end-users are expected to cooperate on the tables in a schema all the way, including being able to create and drop them. And this is a scenario that is not very well handled by the built-in role support, due to the ownership handling. Fortunately, this is something where we can once again use an event trigger to make the system do what we need.

Continue reading

Logging transactions that dropped tables

In a previous post I discussed a way to find out which transaction dropped a table by examining the transaction log, in order to set a restore point to right before the table was dropped.

But what if we have the luxury of planning ahead (right? Well, let's call it the second time it happens?). Shouldn't we be able to log which transaction dropped a table, and use that? Of course we should.

The first thing one tries is then of course something like this in postgresql.conf:

log_statement='ddl'
log_line_prefix = '%t [%u@%d] <%x> '

to include the transaction id of the table. Unfortunately:

2017-02-12 12:16:39 CET [mha@postgres] <0> LOG:  statement: drop table testtable;

The 0 as a transaction id indicates that this command was run in a virtual transaction, and did not have a real transaction id. The reason for this is that the statement logging happens before the statement has actually acquired a transaction. For example, if I instead drop two tables, and do so in a transaction:

postgres=# BEGIN;
BEGIN
postgres=# DROP TABLE test1;
DROP TABLE
postgres=# DROP TABLE test2;
DROP TABLE
postgres=# COMMIT;
COMMIT

I get this interesting output:

2017-02-12 12:17:43 CET [mha@postgres] <0> LOG:  statement: DROP TABLE test1;
2017-02-12 12:17:45 CET [mha@postgres] <156960> LOG:  statement: DROP TABLE test2;

Which shows two different transaction ids (one real and one not) for statements in the same transaction. That's obviously not true - they were both dropped by transaction 156960. The transaction id just wasn't available at the time of logging.

So what can we do about that? Event triggers to the rescue!

Continue reading

Conferences

I speak at and organize conferences around Open Source in general and PostgreSQL in particular.

Upcoming

SCALE 18x
Mar 5-8, 2020
Pasadena, USA
Nordic PGDay 2020
Mar 24, 2020
Helsinki, Finland
pgDay.paris 2020
Mar 26, 2020
Paris, France
PGCon 2020
May 26-29, 2020
Ottawa, Canada
PGConf.EU 2020
Oct 20-23, 2020
Berlin, Germany

Past

FOSDEM PGDay 2020
Jan 31-Feb 2, 2020
Brussels, Belgium
PGDaySF 2020
Jan 21, 2020
San Francisco, USA
DevOpsSaar
Dec 5, 2019
Saarbr├╝cken, Germany
Stockholm PostgreSQL Meetup
Dec 3, 2019
Stockholm, Sweden
Berlin PostgreSQL Meetup
Nov 21, 2019
Berlin, Germany
More past conferences