[RITL] Cloud Costs for Research/HPC
Chuck Forsyth
charles.forsyth at ucr.edu
Wed Sep 14 13:55:58 PDT 2022
Hi All,
Not that "lift and shift" HPC to the cloud being expensive is any "new news", Just some more feedback reinforcing that below.
I mentioned during the last meeting while we were discussing cost challenges that; If one looks at research workflows differently, in perhaps a more modern way. Utilizing cloud-native technologies for access and scale could potentially be less expensive and a game-changing approach to research. This absolutely would not work for all research/research computing workflows but could work for some.
Regards,
Chuck Forsyth
Associate Director of Research Computing
Chairman of University of California Research IT Committee
XSEDE Campus Champion<https://www.xsede.org/community-engagement/campus-champions> / ACM<https://member.acm.org/~charles_forsyth>
Research Computing | Information Technology Solutions
University of California, Riverside
Website<https://its.ucr.edu/research-computing> | Research Computing Slack<https://ucr-research-compute.slack.com/>
https://calendly.com/forsythc
________________________________
From: Philip Michael Papadopoulos <ppapadop at uci.edu>
Sent: Wednesday, September 14, 2022 3:45 PM
To: Chuck Forsyth <charles.forsyth at ucr.edu>
Cc: uc-research-it at lists.berkeley.edu <uc-research-it at lists.berkeley.edu>
Subject: Re: RITC Meeting Reminder - Tomorrow, Thursday 9/15 @ 1-2pm
Hi All,
Somebody volun-'told' me to look at cloud costs, So I did.
The "rough but realistic" numbers for UCI are that -discounted- AWS equivalent storage and compute is
at least 3X our hardware costs. At RCIC's "mid-scale" this is more than $10M over 6 years.
* We operate on an approximate 6-year HW replacement cycle (5 year warranty + 1 year). Our equipment might last longer than that, but that's just a bonus. For storage, we generally don't run the extra year and replace much closer to the warranty cycle.
* Usually, RCIC has about $800K/year budget for hardware and software licenses. That purchases, storage, CPUs, GPUs, networking, bits-and-bobs. I've been at UCI a bit over 4 years, so we're still in a "grow footprint". Very roughly, RCIC's hardware+software budget is $5M/6 years. Faculty do add to that when they purchase hardware or lease storage space.
* AWS cost over 6 years, just shy of $16M. This is the most charitable version of costs for amazon.
* 3 year pre-paid upfront reserved based on our -mean- core/GPU utilization + 15% UC Discount
* No factoring of "transaction" or "egress" costs.
* No factoring of additional campus external networking costs if you really put everything in the cloud.
* If you thought about on-demand, don't. The costs are 2X that of 3-year prepaid.
* RCIC Cores are non-hyperthreaded. AWS cores ARE hyperthreaded. Costs are based upon non-hyperthreaded since we know that hyperthreaded and HPC-style computation don't really mix.
* AWS instances have NO low-latency network - our local clusters are all 10GbE + 100Gb IB.
* I looked at two AWS instances that were close - one had SSDs one had more memory. Pick your poison as to which is closer to an IB-connected, NVMe-local physical node
Here's a copy of spreadsheet cells in this early stage.
CRSP - Campus research storage (highly available, GPFS based storage)
DFS - BeeGFS cluster-local storage.
For both storage systems, all data is replicated (backed up).
Amazon Computing/GPU Instances (~ similar to RCIC Configs)
On Demand 1 year Reserved
3-year Reserved
Note: CPU 'cores' in AWS are hyperthreaded. Costs are normalized to non-hyperthreaded
Discount 15%
m5.12xlarge 48 4GB/Core 12Gbps (no IB) Hyperthreaded $0.08 $0.05 $0.03
c5d.12xlarge 48 2GB/core 12Gbps, Local SSD Hyperthreaded $0.08 $0.05 $0.03
p3.2xlarge 8 8GB/Core 10Gbps V100 GPU, 16GB $2.60 $1.69 $0.89
CPU COSTS - Annualized based on RCIC Actual Load*
CORE YEARS 6200 On Demand 1 year Reserved
3-year Reserved
m5.12xlarge $4,431,859.20 $2,606,410.25 $1,665,794.30
c5d.12xlarge $4,431,859.20 $2,606,410.25 $1,619,629.10
GPU COSTS - Annualized based on RCIC Actual Load
GPU YEARS 40
p3.2xlarge $911,390.40 $592,701.60 $312,732.00
S3 Storage Costs
GB/Mo Discounted TB/Year
S3 Standard 0.021 0.01155 141.9264
S3 Infrequent Access 0.0125 0.006875 84.48
Discount 45%
CRSP Utilization 600 TB
DFS 4500 TB
6 Year Costs (Nominal Lifetime for Hardware in HPC3)
m5.12xlarge $26,591,155.20 $15,638,461.50 $9,994,765.80
c5d.12xlarge $26,591,155.20 $15,638,461.50 $9,717,774.60
p3.2xlarge $5,468,342.40 $3,556,209.60 $1,876,392.00
CRSP-Equivalent
$510,935.04 $510,935.04 $510,935.04
DFS-Equivalent $3,832,012.80 $3,832,012.80 $3,832,012.80
AWS Totals $36,402,445.44 $23,537,618.94 $15,937,114.44
On Wed, Sep 14, 2022 at 11:58 AM Chuck Forsyth <charles.forsyth at ucr.edu<mailto:charles.forsyth at ucr.edu>> wrote:
Dear Colleagues,
The September Research IT Committee meeting will be tomorrow afternoon (Thursday 9/15) from 1-2 on Zoom. We'll come together to thank Lisa for her leadership and change Chairs, we'll then do some brief committee housekeeping (New calendar invite, verify member and email lists), we'll talk about the Research Data Backup Project and the Steering Committee, we'll touch on our Research in the Cloud Document<https://urldefense.com/v3/__https://docs.google.com/document/d/1_x-okBV1VhBHuN3pmpBGwwGsiVjqVNlaXhp_ldWI7cs/edit?usp=sharing__;!!CzAuKJ42GuquVTTmVmPViYEvSg!OsTRztHqGfaR6zjiF5wuk2j5jzPiYmXPaQ6K_yVmoEuxK0b7WxuaZXpDav3MBDjIj2jh_pYQXousGaN6h5atIakzWY8$>, and if we have time, we'll touch base on our UC Research Facilitators' efforts and UC Software Carpentries' efforts.
Agenda
1. Welcome and sign in<https://urldefense.com/v3/__https://spaces.ais.ucla.edu/display/UCRITC/2022-9-15*Meeting*Notes__;Kys!!CzAuKJ42GuquVTTmVmPViYEvSg!OsTRztHqGfaR6zjiF5wuk2j5jzPiYmXPaQ6K_yVmoEuxK0b7WxuaZXpDav3MBDjIj2jh_pYQXousGaN6h5atiFB9gIo$>
2. "Changing of the Chairs"
3. Committee housekeeping (New calendar invite, verify member and email lists)
4. Research Data Backup Project (Steering Committee questions)
5. Research in the Cloud Document<https://urldefense.com/v3/__https://docs.google.com/document/d/1_x-okBV1VhBHuN3pmpBGwwGsiVjqVNlaXhp_ldWI7cs/edit?usp=sharing__;!!CzAuKJ42GuquVTTmVmPViYEvSg!OsTRztHqGfaR6zjiF5wuk2j5jzPiYmXPaQ6K_yVmoEuxK0b7WxuaZXpDav3MBDjIj2jh_pYQXousGaN6h5atIakzWY8$>
6. Touch base on UC Research Facilitators' efforts and UC Software Carpentries' efforts
7. Anything else suggested by the committee
Zoom Details
https://berkeley.zoom.us/j/92251202810?pwd=WTY5ZzRtZ2tjUGFxZ3lsRHR6Qkt5QT09<https://urldefense.com/v3/__https://berkeley.zoom.us/j/92251202810?pwd=WTY5ZzRtZ2tjUGFxZ3lsRHR6Qkt5QT09__;!!CzAuKJ42GuquVTTmVmPViYEvSg!OsTRztHqGfaR6zjiF5wuk2j5jzPiYmXPaQ6K_yVmoEuxK0b7WxuaZXpDav3MBDjIj2jh_pYQXousGaN6h5atSPtYVz8$>
Meeting ID: 922 5120 2810
Passcode: TPu8elU6
One tap mobile
+16692192599,,92251202810#,,,,,,0#,,49546497# US (San Jose)
+16699006833,,92251202810#,,,,,,0#,,49546497# US (San Jose)
Meeting Notes
https://spaces.ais.ucla.edu/display/UCRITC/2022-9-15+Meeting+Notes<https://urldefense.com/v3/__https://spaces.ais.ucla.edu/display/UCRITC/2022-9-15*Meeting*Notes__;Kys!!CzAuKJ42GuquVTTmVmPViYEvSg!OsTRztHqGfaR6zjiF5wuk2j5jzPiYmXPaQ6K_yVmoEuxK0b7WxuaZXpDav3MBDjIj2jh_pYQXousGaN6h5atiFB9gIo$>
Committee Home Page
https://spaces.ais.ucla.edu/display/UCRITC/UC+Research+IT+Committee+Home<https://urldefense.com/v3/__https://spaces.ais.ucla.edu/display/UCRITC/UC*Research*IT*Committee*Home__;KysrKw!!CzAuKJ42GuquVTTmVmPViYEvSg!OsTRztHqGfaR6zjiF5wuk2j5jzPiYmXPaQ6K_yVmoEuxK0b7WxuaZXpDav3MBDjIj2jh_pYQXousGaN6h5at0TcNDzI$>
Looking forward to seeing you there!
Regards,
Chuck Forsyth
Associate Director of Research Computing
XSEDE Campus Champion<https://urldefense.com/v3/__https://www.xsede.org/community-engagement/campus-champions__;!!CzAuKJ42GuquVTTmVmPViYEvSg!OsTRztHqGfaR6zjiF5wuk2j5jzPiYmXPaQ6K_yVmoEuxK0b7WxuaZXpDav3MBDjIj2jh_pYQXousGaN6h5atr9FxdaE$> / ACM<https://urldefense.com/v3/__https://member.acm.org/*charles_forsyth__;fg!!CzAuKJ42GuquVTTmVmPViYEvSg!OsTRztHqGfaR6zjiF5wuk2j5jzPiYmXPaQ6K_yVmoEuxK0b7WxuaZXpDav3MBDjIj2jh_pYQXousGaN6h5atBOa0L1A$>
Research Computing | Information Technology Solutions
University of California, Riverside
Website<https://urldefense.com/v3/__https://its.ucr.edu/research-computing__;!!CzAuKJ42GuquVTTmVmPViYEvSg!OsTRztHqGfaR6zjiF5wuk2j5jzPiYmXPaQ6K_yVmoEuxK0b7WxuaZXpDav3MBDjIj2jh_pYQXousGaN6h5atoyrFIF0$> | Research Computing Slack<https://urldefense.com/v3/__https://ucr-research-compute.slack.com/__;!!CzAuKJ42GuquVTTmVmPViYEvSg!OsTRztHqGfaR6zjiF5wuk2j5jzPiYmXPaQ6K_yVmoEuxK0b7WxuaZXpDav3MBDjIj2jh_pYQXousGaN6h5at1qXTnMQ$>
https://calendly.com/forsythc<https://urldefense.com/v3/__https://calendly.com/forsythc__;!!CzAuKJ42GuquVTTmVmPViYEvSg!OsTRztHqGfaR6zjiF5wuk2j5jzPiYmXPaQ6K_yVmoEuxK0b7WxuaZXpDav3MBDjIj2jh_pYQXousGaN6h5at2Q0tXoI$>
--
To unsubscribe from this group and stop receiving emails from it, send an email to uc-research-it+unsubscribe at lists.berkeley.edu<mailto:uc-research-it+unsubscribe at lists.berkeley.edu>.
--
Philip Papadopoulos, Ph.D
Director, Research Cyber Infrastructure Center
University of California, Irvine
E-mail: ppapadop at uci.edu<mailto:ppapadop at uci.edu>
Phone: (949) 824-5343
Calendar (UCI Only): https://bit.ly/3btXoJa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ucr.edu/pipermail/ritl/attachments/20220914/17d356e4/attachment-0001.html>
More information about the RITL
mailing list