Postgres

It’s been almost a year since I last posted an update on eugene, the CLI tool I’m building to help people write safer SQL migration scripts for postgres. I announced this tool in Careful with That Lock, Eugene: Part 2. At the time, eugene would execute a single SQL script, recording all the locks acquired and warn about possible downtime due to migrations. It could produce JSON suitable for automated tooling and Markdown suitable for human reading and using in CI comments/checks. That version was already good enough for me to start using in real projects — but it’s improved a lot since then, it’s now easy to run with almost no setup. ...

Last week I was made aware that we had some foreign keys not backed by indexes in the system we’re developing at work. Foreign keys in postgres must be backed by an index only on the side they refer to, not necessarily the side they refer from. Here’s an example: create table author( id bigint generated always as identity primary key, name text not null ); create table book( id bigint generated always as identity primary key, author bigint not null references author(id), title text not null ); In this example, there’s a foreign key from the book table to the author table. Since author refers to a primary key in the author table, inserts into book can validate very quickly. There’s no index on the author column in the book table though. The consequence of this is that delete on author must check every single row in book to check if it’s safe to actually delete. The really annoying part of this is that the scan does not show up in query plans: ...

The past few weeks I’ve been experimenting with DuckDB, and as a consequence I’ve ended up talking about it a lot as well. I’m not going to lie, I really like it! However, experienced programmers will rightly be skeptical to add new technology that overlaps with something that already works great. So why not just use postgres? Well, I really like postgres too, and I think you should consider just using it! But despite both of these technologies being all about tabular data, they’re not really for the same kinds of problems. I think DuckDB is primarily an analysis or ELT tool, and it really excels in this space. postgres can do a lot of the things that DuckDB can do, but not nearly as fast or easily. I wouldn’t want to use DuckDB for a transactional workload, so it’s not going to replace postgres for anything that I use it for. ...

It’s always an exciting day for me when I get access to the source code for an entirely new application I need to work on. How does it look inside, how does it work? Sometimes, there’s some design documentation along with it, or operational procedures, or maybe some developer handbook in a wiki. I do check all of those, but I don’t expect any of those things to accurately describe how the code works, because they tend to change less frequently. It’s also fairly low-bandwidth, it takes a ton of time to ingest technical text. ...

In my post about batch operations, I used the where id = any(:ids) pattern, with ids bound to a JDBC array. I’ve gotten questions about that afterwards, asking why I do it like that, instead of using in (:id1, :id2, ...). Many libraries can take care of the dynamic SQL generation for you, so often you can just write in (:ids), just like the array example. I would still prefer to use the = any(:ids) pattern, and I decided to write down my reasoning here. ...

Throughout a career as a software developer, you encounter many patterns. Some appear just often enough to remember that they exist, but you still need to look them up every time. I’ve discovered that writing things down helps me remember them more easily. This particular pattern is very useful for my current project. So, it’s time to write it down and hopefully commit it to memory properly this time. Although this post is specific to PostgreSQL, I’m sure other databases have the necessary features to achieve the same results efficiently. ...

Database servers are usually long-lived, and important parts of the infrastructure that we build on. We rarely set them up from scratch, because we have to take such good care of them over time. I think this causes a lot of people to think that setting up a database server is some mysteriously difficult ordeal. To be clear, that’s actually true, if you need high availability and a solid recovery point objective. But there are a lot of use cases where that’s overkill, for example short-lived test environments, or CI/CD pipelines. ...

I have been working quite a bit on picking up dangerous migration patterns in migration scripts over at the eugene repository lately. A major feature I’ve added is syntax tree analysis, so that we can pick up some patterns without having to run the SQL scripts. This isn’t quite as precise as running the scripts, but it’s a lot faster and can catch quite a few common mistakes. So let’s take a look at how it works! ...

A while back, I wrote Careful with That Lock, Eugene about an idea for how to check if a database migration is likely to disturb production. That post came about after having an inspiring chat with a colleague about the advantages of transactional migration scripts and the ability to check the postgres system catalog views before committing a transaction. Over the past few weeks, I’ve been experimenting with this idea to test if I can use it to build valuable safety checks for DDL migrations. Kind of like shellcheck, but for database DDL migrations. ...

It is rewarding to work on software that people care about and use all around the clock. This constant usage means we can’t simply take the system offline for maintenance without upsetting users. Therefore, techniques that allow us to update the software seamlessly without downtime or compromising service quality are incredibly valuable. Most projects I’ve worked on use a relational database for persistence, and have some sort of migration tool like flyway or liquibase to make changes to the database schema. This post is about a particular kind of migration situation that, in my experience, most developers who work on such projects will encounter at some point in their career. They will want to apply a simple, and seemingly innocent migration, like adding a column to a table and it’ll cause some number of requests to fail, or maybe even a small outage. There are some tricks we can use here to reduce risk and automatically detect some patterns that cause this problem. ...