This slipped away during the review of !11 (merged). By default, black only formats the code and returns exit code 0. We either need to use "--check" for non-zero exit code if changes are needed, or use the approach in this commit to get a diff along with the exit status.
example of failed job: https://gitlab.isc.org/isc-projects/gitlab-runner-scripts/-/jobs/2769965