Alpina Gerätebau AG does not exist. I made it up in August for an App Store screenshot. Salzburg address, a VAT number in the right shape, a nice ring to it.
On Tuesday night at 21:12 one of my apps asked the European Commission whether Alpina Gerätebau AG is real. The Commission said no. Two seconds later the app asked again. It kept that up until Wednesday afternoon. 31,054 questions about one fake company.
Alpina Gerätebau AG has now had more contact with the EU tax system than most real Austrian companies.
How you build a loop without noticing
Zeppol makes e-invoices for Shopify merchants. Before it applies reverse charge on a cross-border sale, it checks the buyer’s VAT number against VIES, the EU’s registry. That is the correct and legally required thing to do, and it is the last correct thing in this story.
Two background jobs did the work. One builds the invoice. If the VAT number has not been checked, it parks the invoice and asks the second job to check it. The second job checks, writes down the answer and tells the first job to try again.
The first job was only ever written to accept a yes. Anything else put the invoice straight back in the parked state, which asked for a check, which got the same no, which parked the invoice. Nobody had written the exit, because nobody had imagined a company that does not exist asking to be invoiced.
I had. In August. For a screenshot.
My code’s reaction to being blocked by the EU
At 10:47 on Wednesday the Commission blocked the server. Fair enough. Their reply changed from “no” to “you are blocked”.
My code had a list of replies that mean “their system is temporarily down, try again shortly”. Blocked was on that list. Right next to “Austria’s tax office is offline”. So it read a block from the European Commission as the Commission having a busy morning, and carried on for another four and a half hours. Patiently.
Ten minutes after the block, Hetzner sent me a very polite abuse report with a 24-hour deadline. That email is how I found out. My hosting provider wrote to me before my own software did.
Why nothing on my side said a word
31,054 requests in eighteen hours is one every two seconds. On any server graph that is a flat line. A few kilobytes a second. No CPU. Had I been staring at a monitoring dashboard the whole time, I would have seen a healthy, idle machine and gone to bed.
What made it abusive was repetition. The same request, to the same place, forever. Repetition is a property of the words in the log. Nothing was reading the log.
The rule that was already there
Zeppol caches every VAT verdict for 30 days. That rule was in the code from day one. Tested. Correct.
It was wired to exactly one thing: the colour of a status badge in the admin. The job that opens the actual connection to VIES never consulted it. So the badge said “checked” while the worker, one folder over, asked the same question every two seconds all night.
One line of code, checking the cache before making the request, would have turned 31,054 into 1.
What I changed
The cache is consulted before the request. An invoice is only rebuilt when the answer has changed. A block from VIES stops all traffic for six hours and emails me. There is a rate limit and a hard ceiling of 500 requests a day, both asserted in tests, so whoever weakens them turns the build red.
Then the change I care about most. A small script on the server reads every container’s logs, every two minutes, and emails me when the same line appears 30 times in 5 minutes. It knows nothing about VAT. It covers the other two products on the same machine too, which I had not built any of this for.
I considered routing all outbound traffic through a proxy and capping it there. Decided against it. That puts one new thing in front of three products, and when it falls over, all three go with it. The watchdog sits beside the apps. If it dies, nothing else notices, which is the whole point.
Where it stands
Statement sent to Hetzner, then a follow-up once the fix was live. Their ticket closes on its own unless they come back. A separate request went to the Commission asking them to lift the block, with the timeline and the fix attached. VAT checks are switched off on my side until they answer. Sending more requests at a service that has told you to stop is not a good look, and I have used up my allowance for the year.
The loop was an ordinary bug. I will write more of those. The next one gets found in five minutes instead of eighteen hours, and by me rather than by an abuse desk.
I have apologised to the EU, to Hetzner, and in spirit to Alpina Gerätebau AG, who never asked for any of this.