Posts

If you’re developing a transactional application backed by postgres, there’s a pretty cool trick you can use to check if you’re missing indexes that could potentially cause serious performance issues or even outages. In particular, I mean foreign keys where the referencing side of the constraint does not have an index. The idea is very simple, we can select all of the columns that take part in a foreign key, then remove the ones that take part in a complete index, and the remainder should be the empty set, or possibly match a known allowlist....

I worked for a few years in the intersection between data science and software engineering. On the whole, it was a really enjoyable time and I’d like to have the chance to do so again at some point. One of the least enjoyable experiences from that time was to deal with big CSV exports. Unfortunately, this file format is still very common in the data science space. It is easy to understand why – it seems to be ubiquitous, present everywhere, it’s human-readable, it’s less verbose than options like JSON and XML, it’s super easy to produce from almost any tool....

For tests that need to touch the database, it is generally a really good idea to roll back transactions. That way, you can run lots of tests in parallell or in any arbitrary order and the tests won’t interfere with each other. But sometimes, that just isn’t possible. One reason for this could be that the code base handles transactions in a way that makes it really hard to get a handle on them in the right place, or it could be a legacy code base where everything is running with auto-commit or some other explanation....

There are 2 configuration options that every OLTP application that uses postgres should set, in order to protect the database from high load: statement_timeout idle_in_transaction_session_timeout These can both be set by client configuration and require no special permissions to set, and are easily overridden locally for transactions that have different requirements. They can be a bit scary to retrofit to existing applications, but we can activate two postgres extensions to help us measure our queries to find safe values to set:...