How the UK heatwave affected data centre floor temperatures (July 2022)

We monitor everything for our clients – from specific server metrics (such as CPU usage) all the way to the environment in which they’re installed (including temperature and humidity).

On the 19th July 2022, the UK recorded its hottest-ever temperature of 40.3c in Lincolnshire – something that the Met Office once said was “virtually impossible” without climate change (Source: Independent)

There were reports of supermarkets having no operational fridges or freezers, as they’d all stopped working due to the heat. Restaurants were forced to close as they could no longer serve customers cold refreshments. Google Cloud had, and continues to have as I write this, many problems caused by their cooling system failing in europe-west2-a, one of their data centres in London. Ultimately, all servers in the facility shut down to protect themselves from the heat at around 5:20pm yesterday and some services remain down now (11:30am).

Air conditioning has it’s limits – and when you have a data centre floor of servers and network equipment all pushing out considerable amounts of heat, high ambient temperatures are not a welcome addition – especially when they’re a bit more out of the ordinary than usual.

I remember once I bought a fridge freezer – and put it in the garage. One cold winters day, I noticed that everything had defrosted as it had shut down. As it turns out, it’s operating range could not go low enough to function in my cold garage. A quick Google later, I realised I needed a “Garage and Outbuildings” freezer that would work at much lower temperatures – and I’ve had no problems since.

Air Conditioning is no different to my fridge freezer – looking at the system in our office, it can operate in cooling mode from -10c to 46c. The outdoor unit is sheltered away from the sun – but, if this was to be installed on a roof with the beating sun on it, you can see how there could be a problem.

We do, of course, have notifications on temperatures, where we set warning limits to let our engineers know when there is likely to be a problem. It’s an early warning system, which is enough time, as past experience tells us, to report and resolve the problem should there be one – for installations at a client site, this often may be as simple as someone checking the air conditioning is running.

Thankfully, we did not hit these warning limits at any site – although we did see increases. HPE recommend servers are not operated in ambient temperatures of more than 35c (Source: HPE) – typically, we’d want to see much lower than that; certainly less than 30c.

Every data centre floor is different – some have different baselines, so it’s important to read what “normal” would be, and base warnings on that.

Ambient Temperature of an HPE ProLiant Server

The graph above shows the ambient temperature being read on an HPE ProLiant Server. This server happens to be in the bottom of the rack. Temperatures can be seen to be pretty stable – with data going back to the 14th July, the reading was always between 21c and 22c. We would, therefore, say that 21c – 22c is the “normal range” we’d be looking for. However, this wasn’t the case on the 19th July… where it hit 28c in the middle of the day – a 33% increase.

Another data centre in which we operate also saw large increases – although their baseline starts off colder, and rises each day:

A temperature of 16c overnight is “cold” (it’s the coldest data centre we use), but it rises to around 19c each day during normal operation. On the 18th July, it rose to 20c (as we braced ourselves for the big one!).. and on the 19th July, a massive 23c. Looking at the data, a “normal” day rise is 3c, an 18.75% rise. However, rising up to 23c, means it saw a 43.75% increase over the night temperature.

These were considerable increases – you can certainly see why Google had a big problem when their cooling failed; if we’re seeing increases like the above when everything is functioning properly, imagine how quickly it’d rise if it wasn’t.

It raises an important point for air conditioning maintenance – get your systems checked regularly, and ensure there are failovers in place should one fail (n+1 typically for critical infrastructure). But, if temperatures rise outside of the the air conditioning systems operating range, you’re likely to be in trouble.

Posted on: 20th July 2022.