Delete an old Borg backup when the disk is full

I got a problem

When the disk is full, Borg can do no more backups. No problem, you think, let’s delete some old backups and we are ready to go again. But there is a roadblock ahead: You cannot even list the backups when the disk is full because Borg can’t create a lock on the backup device. Which means you cannot get the names of the backups to be deleted. You can also not guess them because they contain dates and even times. Not a chance.

This is annoying. The lock is just a small file, but it cannot be created anymore, so Borg won’t proceed. Which is a bit tragic, because if you could just remove one old backup, you would have plenty of space for the lock and could just continue happily ever after. So, we have a situation that nicely[1] fits the definition of a deadlock.

The way Borg backup repos are organized on disk, it’s hard to tell if you could delete something manually to get space for the lock. If you panic and randomly delete just something, you will most probably break the backup repo.

How to break the deadlock

But there’s a trick that you can use. It feels a bit like Houdini must have felt when he did his escapes. Only you are not under water, in a sealed milk can or some other highly uncomfortable situation and you have time to lean back and think, so don’t panic.

First, list the backups using the --bypass-lock option, like so:[2]

borg --bypass-lock list /media/nik/Borgbackup1/borgbackup/

Now, break the lock, then delete the oldest backup, so you have room to move (that is, to create a lock file):[3]

borg break-lock /media/nik/Borgbackup1/borgbackup/
borg delete /media/nik/Borgbackup1/borgbackup::home-2023-02-02T11:32:45

You have to break the lock here, using the --bypass-lock option with borg delete is not allowed. Which raises the question: Could you first break the lock, then list, then delete?

Note to self: Check this when the situation occurs next. Which will happen for sure.

Now that you can work with your backups again, use that to your advantage: If you have more old backups, you can delete them with this one liner that let’s you remove several backups at once:[4]

borg delete /media/nik/Borgbackup1/borgbackup/::$(borg list /media/nik/Borgbackup1/borgbackup/ | grep 2022 | cut -d ' ' -f 1)
Note
This solution may not be appropriate in a company environment. I have no idea how breaking locks may affect critical infrastructure. It was a solution that helped me out in a safe, private setup. I can provide no guaranty about the security implications of the steps shown here in a general IT environment!

1. Most probably, as you are in trouble and possibly even under pressure, with your boss looking over your shoulder, his or her neck veins already throbbing visibly, you may not consider the situation to be "nice". I can relate to that.
2. As always, adapt the name and the paths. Using nik as a name and my directory names won’t take you that far.
3. Or, to be more precise: Let Borg create a log file.
4. This is just an example, of course you have to adapt file paths and the year. But you get the picture.

links

social