Last Friday something funny happened at work. A colleague called me desperate in the afternoon, telling me that he did a major fuck up.

Just a little bit of context before I elaborate on what happened. We run a Marketplace, which is a bespoke solution developed in house using Django. We manage several sellers, and we provide an end-to-end solution for them. For our administrative tasks, we make heavy use of Django Admin.

So, back on the fuck up. My colleague works in the finance department, and one of his duties is to pay out our sellers. He has a daily routine where he searches for approved withdrawals, sends the money via bank transfer, attaches the receipts. By the end of the day, he marks all withdrawal requests as completed. He does this process via an Admin action named “complete the withdrawals.” It is a fundamental step because it finalizes the workflow and mark all the transactions as withdrawn and issues a debit transaction of the value of the withdrawal to net the transactions.

We also have another Admin action named “cancel the withdrawals” that happens to be placed right below the “complete the withdrawals.” That’s where the fuck up happened. Instead of completing the withdrawals, he canceled all withdrawals. And it is a pretty destructive action because it releases all transactions making it available for withdrawing again.

Now that was a rookie mistake on my side because such a destructive action should never go straight without asking for confirmation. Also, it was very poorly placed (right next to the action used to complete the process).

Thankfully I was able to revert the problem by making use of our heavy logging of the interactions. Every step is logged, and any change is recorded. I asked the operations and customer service to pause all reimbursements to avoid changing the current balance and re-created the state before the cancellation action.

The bottom line is, always ask for confirmation before destroying data. Also, always log information and keep track of important changes, it may be vital to fix a problem. Even though there was a problem in the process, we were able to go back and revert it thanks to the logs.