Microsoft Exchange and the “Y2K22 Bug”

If you’re running Microsoft Exchange on-premises like some of our clients, you’ll probably be aware that a bug, triggered by the year changing to 2022, caused e-mails to queue should you have the Microsoft’s in-built Anti-Malware service running – a service that’s running by default on all Exchange 2016 and 2019 Mailbox servers.

After getting through the Year 2000 Millennium Bug, you’d be forgiven for thinking the Year 2022 Bug wouldn’t be a thing.. yet, in the Microsoft Exchange world at least, it was.

On updating to the latest definitions, which included “22” at the start of the version, the filtering management service would crash.  This is because Microsoft was trying to write “2,201,010,001” in to a variable that had a maximum of “2,147,483,647”.  The version starts with the format YYMMDD – so, 21 would have been fine… but, upon changing to 22, it all went wrong.  Not a great start to 2022.

What this resulted in was our on-call engineer being woken up at 3am, as Exchange servers automatically updated their malware definitions.  The Microsoft Filtering Management Service was seen to be crashing and then stopping, and the Submission Queue was seen to be growing as no e-mail could be scanned – when no scanning can occur, delivery is deferred until it can be.  It’s amazing how many e-mails are flying about at that time of night – our client’s submission queues were growing at a decent rate.

After working out the problem, and seeing a flood of other people on the internet with the same issue, it was then down to waiting for Microsoft for a fix, or at least a workaround.

We tweeted about it on our @deeserve Twitter account to let other people know that there is an issue.

Microsoft did finally put a post out on their Exchange Team blog, with a workaround and an automated solution.  Currently, it seems, they’ve invented the 33rd December 2021, to get around the date issue (211233… is a perfectly valid start to the file name).  You can read their post here.

The workaround was implemented for  our clients that were affected by this issue on New Year’s Day, so that e-mail would flow again.

Without adequate monitoring, you wouldn’t have really known that this was happening – New Year’s Day is hardly the busiest time for e-mails for most people; would you have noticed if you hadn’t received anything? And that’s before you realise a lot of people wouldn’t even be checking.

As an administrator, comprehensive monitoring is a must – as is a fast response.  Bearing in mind the time of year this happened, a lot of people are returning to work today (the 4th January), some 4 days after this hit – that’s a lot of e-mail.

Another win for our 24/7 monitoring, and our 24/7 IT support.  Not so much of a win for Microsoft on this occasion.

(Did you like our 2022 Bug Blog Photo…?)

Posted on: 4th January 2022.