A conversation about datacenter power


  • Tue 25 February 2014
  • misc

Last fall, MT texted me asking what I know about European datacenter power. I know better than to get into that kind of discussion on an iPhone so I chatted back via AIM. I was a little bit surprised by how much power MT's company had provisioned to their racks, but the numbers worked out exactly as expected. After all, thermal breakers work the same in the Old Country as they do in the New World. Logs from the AIM discussion after the break.

MT:
    There we go.
    Is this thing on? It's pulling your status properly.

RS:
    hola
    it thinks you are offline
    can you read this?

MT:
    Yes
    Hilarious. Well, it works, mostly, I guess.

RS:
    ok so the 100% rating is more typically used on feeder
    circuits (i.e. more aggregated or continuous load) - think
    thousands of amps.
    http://static.schneider-electric.us/docs/Circuit%20Protection/0600DB0101.pdf
    now, i recommended against running with that little
    headroom regardless of what the datacenter standard
    specifies.  can you guess why?

MT:
    spikes in power consumption by equipment in the rack?

RS:
    that's the idea, but not spikes so much as intentional fluctuations
    do you have appropriate test equipment at hand to measure equipment draw?

MT:
    well, this is run as a lights out DC
    and is in Amsterdam
    there aren't electical probes long enough

RS:
    do you have a lab?

MT:
    nope

RS:
    you are specifying equipment to put there, yes?

MT:
    No, I'm being asked to figure out how hot we can run these
    racks my initial advice was to not exceed 80% on common
    sense principles

RS:
    not to exceed 80% based on what?

MT:
    they wanted a more concrete answer about how hot we CAN
    run these racks based on the maximum draw of the devices
    as set out by the manufacturers

RS:
    ah, based on nameplate on the power supplies?

MT:
    nah, based on technical specifications from cisco, dell, hp, etc

RS:
    that's "nameplate"

MT:
    so what they mention in the specs ah right ok

RS:
    that is a very conservative approach.  $DAYJOB does it
    that way (breaker according to nameplate ratings) for
    chassis-based things like an mx960 it makes sense.  since
    you have no idea what cards someone is going to stick in
    it.  for servers, less so.

MT:
    I'm not sure about. If you're running a compute cluster of
    DL360s real hot...

RS:
    DL360s can come with a 450w, a 750w, or a 1200w power
    supply, x2

MT:
    I mean, I remember someone trying that on the theory that
    servers were intentionally marked up on how much max power
    they can draw

RS:
    how are you counting the power supplies?

MT:
    and how well that worked

RS:
    do you have redundant power?

MT:
    assumedly. that's the other thing I'm trying to figure out
    about $DATACENTER Just started digging into this this
    morning.

RS:

    you get what you order.  the power is delivered on one or two
    circuits (or more) you count redundant power supplies
    differently if they are plugged into the same circuit
    vs. different circuits.

MT:
    so, time to go digging into the MSA/work orders again?

RS:
    it won't be in the MSA, it will be in the specific orders
    for the racks

MT:
    the work order for colocation services, then

RS:

    anyway, the 100k foot message is that this is a complex
    problem, but physics works the same regardless of whether the
    temperature is quoted in fahrenheit or celsius.  you will draw
    fewer amps with a server running on 230v than the same server
    running on 120, which is nice in a lot of ways.  but the VA or
    watts are the same.  perhaps we should have a discussion about
    this when you visit.

MT:
    yes

MT:
    nah, peering at the colocation work order. this says
    infrastructure PDU 2x (1A + 1B) infrastructure PDU per
    footprint. Two power feeds - A and B per rack, 32 amp, 400V AC
    three-phase power

RS:

    that sounds wrong.  when they talk about 400v three phase,
    that's how it leaves the PDU, but you have no three phase
    equipment.  or at least i am fairly certain you don't.

MT:
    we don't, it's all fairly standard

RS:
    :)

MT:
    the feeds are three phase
    is what the work order sez
RS:
    yeah, i'm curious about what comes into your cabinet though.
    here's the difference 3Y-208 phase to ground is 120v.  3Y-400
    phase to ground (or more properly neutral) is 230v

MT:
    230v is correct

RS:
    so there's your european electricity however if they bring in
    all three phases, you would have 3 x 32a circuits on A and 3 x
    32a circuits on B.  which would be...  18 kva redundant per
    rack.  not an impossible scenario, but (a) you'd know it if
    you'd ordered it, and (b) we wouldn't be having this
    discussion since the limiting factor is how many C7000 blade
    chassiae you can cram in a rack.

MT:
    I didn't order it
    but that sounds right

RS:
    does the order show the MRC?

MT:
    MRC?

RS:
    monthly recurring charge.

MT:
    yes

RS:
    how many euros?

MT:
    something like eur 7500 per month

RS:
    for power?
MT:
    for 17.6 kW
    er, 17.5, sorry

RS:
    you just answered your meta-question :)

MT:
    I did?

RS:
    they're not billing you as redundant
    and you can not load it up to 100%
    how many kw in 32 amps of 400?

MT:
    fuck if I know, I'm crap at remembering power conversions

RS:
    for your purposes kva and kw are the same.
    well, you'll need to buck up on that for sure.  :)

MT:
    humor me here, how do I know it's not redundant

RS:
    ok, briefly if you ignore power factor, harmonics, and
    inductive load (which it is safe to do with modern power
    factor corrected power supplies; it will get you within 5%)
    then VA (volts times amps) is exactly the same thing as watts.
    so if they gave you one phase of 230 at 32a, how many va?

MT:
    7360

RS:
    or 7.3 kw, still with me?

MT:
    ah.

RS:
    but actually they are giving you three phases of that
    or _____ kw

MT:
    2.4?

RS:
    three phases are like three different circuits...  
    try again?

MT:
    so 7.3 x 3?

RS:
    equals...

MT:
    21.9

RS:
    ok now you are contractually allowed 17.5 kva but the breakers
    are specced at 21.9 kva to be delivered on that "A" circuit,
    correct?  divide 17.5 by 21.9 and tell me what you get.

MT:
    .799

RS:
    HMMMM 80% FASCINATING :) the pricing seems to suggest that
    it's not really entered into the system as redundant, but that
    could be something funky about european pricing.  or could be
    reflective of highly local concerns such as no space for more
    UPSes.  "sorry, no discount for you!"

MT:
    answer is, we're likely only paying for 80%

RS:
    you are only paying for 80% for a reason

MT:
    so while theoretically we could soak up to 100%, the
    overage charges are going to be a bitch

RS:
    they wrote the contract that way because it is dumb to go over
    80% forget the overage charges, consider the outage charges
    now, how you count it is entirely different and nameplate is
    absolutely the wrong way in an independent a/b system.  also,
    if you're going redundant, you should not load each side to
    more than 40%, for reasons which should be obvious.

MT:
    nod

RS:
    ok, so we will talk about how to measure draw when you are in
    VA i gotta run but you have your answer for now.

MT:
    yes thank you threw math at manager his eyes glazed over and
    he accepted my answer