- Tue 25 February 2014
- misc
Last fall, MT texted me asking what I know about European datacenter power. I know better than to get into that kind of discussion on an iPhone so I chatted back via AIM. I was a little bit surprised by how much power MT's company had provisioned to their racks, but the numbers worked out exactly as expected. After all, thermal breakers work the same in the Old Country as they do in the New World. Logs from the AIM discussion after the break.
MT:
There we go.
Is this thing on? It's pulling your status properly.
RS:
hola
it thinks you are offline
can you read this?
MT:
Yes
Hilarious. Well, it works, mostly, I guess.
RS:
ok so the 100% rating is more typically used on feeder
circuits (i.e. more aggregated or continuous load) - think
thousands of amps.
http://static.schneider-electric.us/docs/Circuit%20Protection/0600DB0101.pdf
now, i recommended against running with that little
headroom regardless of what the datacenter standard
specifies. can you guess why?
MT:
spikes in power consumption by equipment in the rack?
RS:
that's the idea, but not spikes so much as intentional fluctuations
do you have appropriate test equipment at hand to measure equipment draw?
MT:
well, this is run as a lights out DC
and is in Amsterdam
there aren't electical probes long enough
RS:
do you have a lab?
MT:
nope
RS:
you are specifying equipment to put there, yes?
MT:
No, I'm being asked to figure out how hot we can run these
racks my initial advice was to not exceed 80% on common
sense principles
RS:
not to exceed 80% based on what?
MT:
they wanted a more concrete answer about how hot we CAN
run these racks based on the maximum draw of the devices
as set out by the manufacturers
RS:
ah, based on nameplate on the power supplies?
MT:
nah, based on technical specifications from cisco, dell, hp, etc
RS:
that's "nameplate"
MT:
so what they mention in the specs ah right ok
RS:
that is a very conservative approach. $DAYJOB does it
that way (breaker according to nameplate ratings) for
chassis-based things like an mx960 it makes sense. since
you have no idea what cards someone is going to stick in
it. for servers, less so.
MT:
I'm not sure about. If you're running a compute cluster of
DL360s real hot...
RS:
DL360s can come with a 450w, a 750w, or a 1200w power
supply, x2
MT:
I mean, I remember someone trying that on the theory that
servers were intentionally marked up on how much max power
they can draw
RS:
how are you counting the power supplies?
MT:
and how well that worked
RS:
do you have redundant power?
MT:
assumedly. that's the other thing I'm trying to figure out
about $DATACENTER Just started digging into this this
morning.
RS:
you get what you order. the power is delivered on one or two
circuits (or more) you count redundant power supplies
differently if they are plugged into the same circuit
vs. different circuits.
MT:
so, time to go digging into the MSA/work orders again?
RS:
it won't be in the MSA, it will be in the specific orders
for the racks
MT:
the work order for colocation services, then
RS:
anyway, the 100k foot message is that this is a complex
problem, but physics works the same regardless of whether the
temperature is quoted in fahrenheit or celsius. you will draw
fewer amps with a server running on 230v than the same server
running on 120, which is nice in a lot of ways. but the VA or
watts are the same. perhaps we should have a discussion about
this when you visit.
MT:
yes
MT:
nah, peering at the colocation work order. this says
infrastructure PDU 2x (1A + 1B) infrastructure PDU per
footprint. Two power feeds - A and B per rack, 32 amp, 400V AC
three-phase power
RS:
that sounds wrong. when they talk about 400v three phase,
that's how it leaves the PDU, but you have no three phase
equipment. or at least i am fairly certain you don't.
MT:
we don't, it's all fairly standard
RS:
:)
MT:
the feeds are three phase
is what the work order sez
RS:
yeah, i'm curious about what comes into your cabinet though.
here's the difference 3Y-208 phase to ground is 120v. 3Y-400
phase to ground (or more properly neutral) is 230v
MT:
230v is correct
RS:
so there's your european electricity however if they bring in
all three phases, you would have 3 x 32a circuits on A and 3 x
32a circuits on B. which would be... 18 kva redundant per
rack. not an impossible scenario, but (a) you'd know it if
you'd ordered it, and (b) we wouldn't be having this
discussion since the limiting factor is how many C7000 blade
chassiae you can cram in a rack.
MT:
I didn't order it
but that sounds right
RS:
does the order show the MRC?
MT:
MRC?
RS:
monthly recurring charge.
MT:
yes
RS:
how many euros?
MT:
something like eur 7500 per month
RS:
for power?
MT:
for 17.6 kW
er, 17.5, sorry
RS:
you just answered your meta-question :)
MT:
I did?
RS:
they're not billing you as redundant
and you can not load it up to 100%
how many kw in 32 amps of 400?
MT:
fuck if I know, I'm crap at remembering power conversions
RS:
for your purposes kva and kw are the same.
well, you'll need to buck up on that for sure. :)
MT:
humor me here, how do I know it's not redundant
RS:
ok, briefly if you ignore power factor, harmonics, and
inductive load (which it is safe to do with modern power
factor corrected power supplies; it will get you within 5%)
then VA (volts times amps) is exactly the same thing as watts.
so if they gave you one phase of 230 at 32a, how many va?
MT:
7360
RS:
or 7.3 kw, still with me?
MT:
ah.
RS:
but actually they are giving you three phases of that
or _____ kw
MT:
2.4?
RS:
three phases are like three different circuits...
try again?
MT:
so 7.3 x 3?
RS:
equals...
MT:
21.9
RS:
ok now you are contractually allowed 17.5 kva but the breakers
are specced at 21.9 kva to be delivered on that "A" circuit,
correct? divide 17.5 by 21.9 and tell me what you get.
MT:
.799
RS:
HMMMM 80% FASCINATING :) the pricing seems to suggest that
it's not really entered into the system as redundant, but that
could be something funky about european pricing. or could be
reflective of highly local concerns such as no space for more
UPSes. "sorry, no discount for you!"
MT:
answer is, we're likely only paying for 80%
RS:
you are only paying for 80% for a reason
MT:
so while theoretically we could soak up to 100%, the
overage charges are going to be a bitch
RS:
they wrote the contract that way because it is dumb to go over
80% forget the overage charges, consider the outage charges
now, how you count it is entirely different and nameplate is
absolutely the wrong way in an independent a/b system. also,
if you're going redundant, you should not load each side to
more than 40%, for reasons which should be obvious.
MT:
nod
RS:
ok, so we will talk about how to measure draw when you are in
VA i gotta run but you have your answer for now.
MT:
yes thank you threw math at manager his eyes glazed over and
he accepted my answer