Last fall, MT texted me asking what I know about European datacenter power. I know better than to get into that kind of discussion on an iPhone so I chatted back via AIM. I was a little bit surprised by how much power MT’s company had provisioned to their racks, but the numbers worked out exactly as expected. After all, thermal breakers work the same in the Old Country as they do in the New World. Logs from the AIM discussion after the break.
MT: There we go. Is this thing on? It's pulling your status properly. RS: hola it thinks you are offline can you read this? MT: Yes Hilarious. Well, it works, mostly, I guess. RS: ok so the 100% rating is more typically used on feeder circuits (i.e. more aggregated or continuous load) - think thousands of amps. http://static.schneider-electric.us/docs/Circuit%20Protection/0600DB0101.pdf now, i recommended against running with that little headroom regardless of what the datacenter standard specifies. can you guess why? MT: spikes in power consumption by equipment in the rack? RS: that's the idea, but not spikes so much as intentional fluctuations do you have appropriate test equipment at hand to measure equipment draw? MT: well, this is run as a lights out DC and is in Amsterdam there aren't electical probes long enough RS: do you have a lab? MT: nope RS: you are specifying equipment to put there, yes? MT: No, I'm being asked to figure out how hot we can run these racks my initial advice was to not exceed 80% on common sense principles RS: not to exceed 80% based on what? MT: they wanted a more concrete answer about how hot we CAN run these racks based on the maximum draw of the devices as set out by the manufacturers RS: ah, based on nameplate on the power supplies? MT: nah, based on technical specifications from cisco, dell, hp, etc RS: that's "nameplate" MT: so what they mention in the specs ah right ok RS: that is a very conservative approach. $DAYJOB does it that way (breaker according to nameplate ratings) for chassis-based things like an mx960 it makes sense. since you have no idea what cards someone is going to stick in it. for servers, less so. MT: I'm not sure about. If you're running a compute cluster of DL360s real hot... RS: DL360s can come with a 450w, a 750w, or a 1200w power supply, x2 MT: I mean, I remember someone trying that on the theory that servers were intentionally marked up on how much max power they can draw RS: how are you counting the power supplies? MT: and how well that worked RS: do you have redundant power? MT: assumedly. that's the other thing I'm trying to figure out about $DATACENTER Just started digging into this this morning. RS: you get what you order. the power is delivered on one or two circuits (or more) you count redundant power supplies differently if they are plugged into the same circuit vs. different circuits. MT: so, time to go digging into the MSA/work orders again? RS: it won't be in the MSA, it will be in the specific orders for the racks MT: the work order for colocation services, then RS: anyway, the 100k foot message is that this is a complex problem, but physics works the same regardless of whether the temperature is quoted in fahrenheit or celsius. you will draw fewer amps with a server running on 230v than the same server running on 120, which is nice in a lot of ways. but the VA or watts are the same. perhaps we should have a discussion about this when you visit. MT: yes MT: nah, peering at the colocation work order. this says infrastructure PDU 2x (1A + 1B) infrastructure PDU per footprint. Two power feeds - A and B per rack, 32 amp, 400V AC three-phase power RS: that sounds wrong. when they talk about 400v three phase, that's how it leaves the PDU, but you have no three phase equipment. or at least i am fairly certain you don't. MT: we don't, it's all fairly standard RS: :) MT: the feeds are three phase is what the work order sez RS: yeah, i'm curious about what comes into your cabinet though. here's the difference 3Y-208 phase to ground is 120v. 3Y-400 phase to ground (or more properly neutral) is 230v MT: 230v is correct RS: so there's your european electricity however if they bring in all three phases, you would have 3 x 32a circuits on A and 3 x 32a circuits on B. which would be... 18 kva redundant per rack. not an impossible scenario, but (a) you'd know it if you'd ordered it, and (b) we wouldn't be having this discussion since the limiting factor is how many C7000 blade chassiae you can cram in a rack. MT: I didn't order it but that sounds right RS: does the order show the MRC? MT: MRC? RS: monthly recurring charge. MT: yes RS: how many euros? MT: something like eur 7500 per month RS: for power? MT: for 17.6 kW er, 17.5, sorry RS: you just answered your meta-question :) MT: I did? RS: they're not billing you as redundant and you can not load it up to 100% how many kw in 32 amps of 400? MT: fuck if I know, I'm crap at remembering power conversions RS: for your purposes kva and kw are the same. well, you'll need to buck up on that for sure. :) MT: humor me here, how do I know it's not redundant RS: ok, briefly if you ignore power factor, harmonics, and inductive load (which it is safe to do with modern power factor corrected power supplies; it will get you within 5%) then VA (volts times amps) is exactly the same thing as watts. so if they gave you one phase of 230 at 32a, how many va? MT: 7360 RS: or 7.3 kw, still with me? MT: ah. RS: but actually they are giving you three phases of that or _____ kw MT: 2.4? RS: three phases are like three different circuits... try again? MT: so 7.3 x 3? RS: equals... MT: 21.9 RS: ok now you are contractually allowed 17.5 kva but the breakers are specced at 21.9 kva to be delivered on that "A" circuit, correct? divide 17.5 by 21.9 and tell me what you get. MT: .799 RS: HMMMM 80% FASCINATING :) the pricing seems to suggest that it's not really entered into the system as redundant, but that could be something funky about european pricing. or could be reflective of highly local concerns such as no space for more UPSes. "sorry, no discount for you!" MT: answer is, we're likely only paying for 80% RS: you are only paying for 80% for a reason MT: so while theoretically we could soak up to 100%, the overage charges are going to be a bitch RS: they wrote the contract that way because it is dumb to go over 80% forget the overage charges, consider the outage charges now, how you count it is entirely different and nameplate is absolutely the wrong way in an independent a/b system. also, if you're going redundant, you should not load each side to more than 40%, for reasons which should be obvious. MT: nod RS: ok, so we will talk about how to measure draw when you are in VA i gotta run but you have your answer for now. MT: yes thank you threw math at manager his eyes glazed over and he accepted my answer