I worked at the OSL as a student years ago, and it was one of the most impactful places I've ever worked at. I learned a lot, and I wouldn't be the engineer I am today without having worked there.
Since graduating, I've also hired, and worked with multiple alumni from the OSL and they're always top notch. Anyone looking for interns or new graduates with devops/SRE or SWE experience should be looking at the OSL for talent. It's not too often you can hire a new graduate with potentially multiple years of production experience, especially in devops.
In context of HN/Y Combinator, https://www.ycombinator.com/companies/coreos was a successful container/Kubernetes focused startup founded by two OSUOSL alumni, Alex Polvi and Brandon Philips, which was eventually acquired by Red Hat.
The OSL is something special.
For a list of projects the OSL helps host, check out https://osuosl.org/communities/. You might see a project you care about in that list! As an example: they provide aarch64 and powerpc VMs for a ton of projects to do their CI/builds on.
The OSL was transformative for my career as a budding CS student in Corvallis many years ago. I can’t say enough good things about the positive impact it has on the Open Source community and the students it employs.
In my experience, there isn’t a great on-ramp for learning to be a SysAdmin (or devop, etc) in a practical sense. Learning what it takes to support systems in “Production” with actual users, and all that entails, at some point requires a hands-on approach. Finding entry-level opportunities to do that isn’t easy until you have /some/ experience. The OSL provides that, and supports countless FOSS projects in the process. It’s really a great arrangement.
Obviously I’m biased, but the Open Source Lab should be viewed as one of the Crown Jewels of OSU.
A lot of the fun parts of the computing industry have, predictably, been hollowed out by the rent seeking model of cloud and *aaS. There is some grace as it's easier than ever to build some scalable web business.. but the most fun of my career was rabbit holing on computers for the sake of computers.. working on operating systems and device drivers and network stacks. And it did and still does matter to a lot of bottom lines, but corporates have a hard time connecting the dots or doing something other than what the flock is doing.
It's a little awkward because the AI datacenter boon is a little bit of a revival for physical and systems work but it is limited to that and I am skeptical of the longevity.
Those days of having fun working on network stacks, operating systems, setting up FOSS development labs and being a good steward of things.. harder and harder to do and even harder to get started.
When I was working on GHC many years ago OSUOSL helped us by providing us access to some nice POWER7 machines (courtesy of an IBM kernel hacker who recommended and endorsed us) and we used them for years to solve weird issues. I've always thought very highly of the Open Source Lab. I hope someone can help them make it through this.
The Open Source Lab was a fundamental part of my college experience. I would not be the person I am now if not for the experience gained while employed there. It was such a great feeling to help hundreds of open source projects maintain infrastructure and services, especially some of the larger projects which have colocated hosts
I was with Mozilla when OSU's OSL was supporting the mirroring/serving of Firefox and Thunderbird globally. They were a key mirror/supporter during the early days of Mozilla and definitely contributed to the growth of Firefox.
OSUOSL and Lance specifically (the writer of this post) was extremely supportive of me during the early days of Vagrant and Packer. Lance tried many times to try to find a way for OSUOSL to help my projects but I don't think we ever formalized anything.
Regardless, they were always big users and big proponents of the OSS work I was doing. And I remember that. I think more than the OSS project support they do, the support and education they help provide for students is laudable.
I personally think corporate sponsors shouldn't blink twice at supporting OSU OSL, but I'm not surprised given the state of... things. And the individuals choosing to judge and criticize based only on a 4 bullet point budget are infuriating.
Well, I'll help. I've emailed to setup a donation.
Thanks for everything you've done Lance, OSUOSL. And thanks to anyone else who helps support them!
Lance is a great guy and that lab for a decade plus has done great work and supported so much of community. Happily supported their start and will continue to.
As someone who was a student at the OSL when Vagrant was hip, also thanks to Mitchell for creating Vagrant! We used it a ton for testing all our our configuration management.
"I have reached out to our largest corporate sponsor and they are working to increase their support as we update our contract, but that still may not be enough."
Why wouldn't they name the sponsor? Seems like a great thing some company (presumably Google?[1]) is doing, but they don't get the credit they deserve.
[1] "Google sponsors the OSL through financial grants." (Noting that none of the other sponsors seem to provide financial grants) https://osuosl.org/sponsors/
They should reach out to IBM too, since POWER and Z hardware are mostly an IBM thing. ARM should be willing to help too, since they host AARCH64 stuff too.
I find it odd that they can provide all the
infrastructure compute / storage / bandwidth
for all of these projects for $35K
Debian and Fedora on their own must be highly demanding??
Form the article:
"""
Currently provides infrastructure hosting for projects such as
Drupal,
Gentoo Linux,
Debian,
Fedora,
phpBB,
OpenID,
Buildroot/Busybox,
Inkscape,
Cinc and many more!
"""
Most of Debian's infra is elsewhere, but OSU does host a backup server, logs server, a bugs.d.o frontend, POWER/MIPS/HPPA/RISC-V build/porting servers.
They are part of gnu compile farm which donates compute to open source projects.
I used them a lot when I worked on OpenSCAD build system, there weren't a lot of places 12+ years ago you could go 'make -j 30' on a PowerPC or 'ctest' and have it run dozens of builds/tests in parallel. Really helped alot, that C++ template stuff would barely build at all on my personal machine.
Also: Lance is almost certainly working more than 40 hours a week. Also, he isn't just a systems administrator. He's a mentor, fundraiser, any literally everything else that is needed to keep the lab running. There used to be more staff, but it's hard to retain qualified individuals. He's been there for 17 years, he's not doing it for the money, he does it because the OSL is important!
$107k in 2017 and $124k in 2023. I don't know about you, but someone with 17 years experience could easily be making 2-5x that depending on the company and role.
And it's $124k on the west coast, specifically western Oregon. Coastal PNW is notoriously expensive to live in - sure, Corvallis isn't Seattle, but it's not cheap, either. Folks like to balk at numbers when it comes to publicly (or FOSS-donation-ly) funded salaries, but also balk at tying context to those numbers. It happens almost every time anyone dares try to pay their bills on open-source work: a flame war over "you don't need that number, you could move to your parent's basement in Arkansas and survive on $20k instead!".
Presumably the $150k also includes all oncosts, so the actual salary is quite a bit lower still? As a side note I don't understand the arguments about salaries for nonprofits. Sure they should not be outrageously higher than the average, but shouldn't we want to get the best people for these jobs (instead of them working on aware?), or is the argument that if you work for a nonprofit you should be doing it out of altruism and be glad you receive a salary at all?
The argument that I assume you are talking about has some nuance around it. It's mostly about politically connected or nepotistic people who are pulling large salaries for essentially little to no work. I'm sure most regular employees at a nonprofit get treated as poorly as those of us at a normal business.
But's usually not the argument being made, the complains (same in this case) are often about the salaries of the people doing the actual work. Sure I understand the complains about multi-million salaries for the CEOs of some non-for-profit (on the other hand I have the same complains about the ridiculous salaries of CEOs of for-profits), but if that's the nuance, it doesn't come out in the complains.
It is unclear from this request, but if this is the cost to the employer it is almost certainly a larger number than the actual pay which goes to the individual.
There are usually three main levels people talk about, entry, mid career, and sr. If you say SDE, I'd assume a mid-career person, which is 100% earning way more then 150k in Seattle. https://www.levels.fyi/t/software-engineer/locations/greater... has more data.
I've been in the IT field for 20+ years, but I've never made that much in a year. What's stopping someone from saying, "Hire two cheaper guys, so there's redundancy?"
They technically call it 'fringe benefits'. My university has four categories of fringe benefits:
Full
Limited
Partial
Grad Health
The only things it specifies are that partial includes social security and full includes life insurance. But given that whatever I set for a post doc/research scientist/etc. salary is the amount they are paid, I assume that everything else including payroll taxes are encompassed in that 1/3 extra for fringe.
Not if we're talking about an experienced engineer that is the only full-time staff for a year. That's the entire budget, so it feels pretty spot on. Maybe I'm out of touch or misunderstanding your point, though.
OSU OSL provides CI machines for some of the more exotic architectures like Linux on Z and POWER to some ASF projects. It would a loss to close it down.
Maybe some unicorn billionaires could spare a few millions? Especially the ones who built their wealth on top of open source libraries or databases.
A gap like cannot be closed without corporate and foundation gifts. That said, individual gifts can also contribute to the progress.
As is common with schools, parks districts, etc., the Open Source Lab partners with a 501(c)(3) organization, the Oregon State University Foundation, to accept tax-deductible donations.
For anyone who would like to directly support the Open Source Lab in staying open, please be sure to indicate "Open Source Lab Fund" on the Oregon State Foundation donation page [0]. Note that their form is *not* set up with any tracking to attribute your gift from your clickthrough, and that any general donations to the Foundation will likely *not* support the Lab in this effort to stay open.
Most gifts that fund university endowments are earmarked by the donor for specific purposes. And for the money not earmarked, you're competing against all the other priorities, including making up for various unplanned shortfalls of Federal funding.
The Foundation itself has nearly $1b in assets. The problem with these foundations is they're often streched across the entire university. Which means the foundation behind OSL is also the foundation behind intercollegiate athletics and tons of other completely unrelated programs.
> While the Oregon State College of Engineering (CoE) has generously filled this gap, recent changes in university funding makes our current funding model no longer sustainable.
I suspect this is related to the recent Trump administration actions to withhold funding from colleges and universities. OSU already voted to increase tuition recently amid concerns about future federal funding [0]:
> “New federal priorities and proposed funding cuts, especially for research, may have direct, negative consequences for OSU,” [OSU President Jayathi Murthy] said.
I worked at the OSL as a student years ago, and it was one of the most impactful places I've ever worked at. I learned a lot, and I wouldn't be the engineer I am today without having worked there.
Since graduating, I've also hired, and worked with multiple alumni from the OSL and they're always top notch. Anyone looking for interns or new graduates with devops/SRE or SWE experience should be looking at the OSL for talent. It's not too often you can hire a new graduate with potentially multiple years of production experience, especially in devops.
In context of HN/Y Combinator, https://www.ycombinator.com/companies/coreos was a successful container/Kubernetes focused startup founded by two OSUOSL alumni, Alex Polvi and Brandon Philips, which was eventually acquired by Red Hat.
The OSL is something special.
For a list of projects the OSL helps host, check out https://osuosl.org/communities/. You might see a project you care about in that list! As an example: they provide aarch64 and powerpc VMs for a ton of projects to do their CI/builds on.
The OSL was transformative for my career as a budding CS student in Corvallis many years ago. I can’t say enough good things about the positive impact it has on the Open Source community and the students it employs.
In my experience, there isn’t a great on-ramp for learning to be a SysAdmin (or devop, etc) in a practical sense. Learning what it takes to support systems in “Production” with actual users, and all that entails, at some point requires a hands-on approach. Finding entry-level opportunities to do that isn’t easy until you have /some/ experience. The OSL provides that, and supports countless FOSS projects in the process. It’s really a great arrangement.
Obviously I’m biased, but the Open Source Lab should be viewed as one of the Crown Jewels of OSU.
A lot of the fun parts of the computing industry have, predictably, been hollowed out by the rent seeking model of cloud and *aaS. There is some grace as it's easier than ever to build some scalable web business.. but the most fun of my career was rabbit holing on computers for the sake of computers.. working on operating systems and device drivers and network stacks. And it did and still does matter to a lot of bottom lines, but corporates have a hard time connecting the dots or doing something other than what the flock is doing.
It's a little awkward because the AI datacenter boon is a little bit of a revival for physical and systems work but it is limited to that and I am skeptical of the longevity.
Those days of having fun working on network stacks, operating systems, setting up FOSS development labs and being a good steward of things.. harder and harder to do and even harder to get started.
When I was working on GHC many years ago OSUOSL helped us by providing us access to some nice POWER7 machines (courtesy of an IBM kernel hacker who recommended and endorsed us) and we used them for years to solve weird issues. I've always thought very highly of the Open Source Lab. I hope someone can help them make it through this.
I was always happily surprised to find that they were hosting what I needed when I needed it.
A great lab with a long history.
The Open Source Lab was a fundamental part of my college experience. I would not be the person I am now if not for the experience gained while employed there. It was such a great feeling to help hundreds of open source projects maintain infrastructure and services, especially some of the larger projects which have colocated hosts
I was with Mozilla when OSU's OSL was supporting the mirroring/serving of Firefox and Thunderbird globally. They were a key mirror/supporter during the early days of Mozilla and definitely contributed to the growth of Firefox.
Jensen is an OSU alum--it would be nice if this reached him.
OSUOSL and Lance specifically (the writer of this post) was extremely supportive of me during the early days of Vagrant and Packer. Lance tried many times to try to find a way for OSUOSL to help my projects but I don't think we ever formalized anything.
Regardless, they were always big users and big proponents of the OSS work I was doing. And I remember that. I think more than the OSS project support they do, the support and education they help provide for students is laudable.
I personally think corporate sponsors shouldn't blink twice at supporting OSU OSL, but I'm not surprised given the state of... things. And the individuals choosing to judge and criticize based only on a 4 bullet point budget are infuriating.
Well, I'll help. I've emailed to setup a donation.
Thanks for everything you've done Lance, OSUOSL. And thanks to anyone else who helps support them!
Lance is a great guy and that lab for a decade plus has done great work and supported so much of community. Happily supported their start and will continue to.
As someone who was a student at the OSL when Vagrant was hip, also thanks to Mitchell for creating Vagrant! We used it a ton for testing all our our configuration management.
"I have reached out to our largest corporate sponsor and they are working to increase their support as we update our contract, but that still may not be enough."
Why wouldn't they name the sponsor? Seems like a great thing some company (presumably Google?[1]) is doing, but they don't get the credit they deserve.
[1] "Google sponsors the OSL through financial grants." (Noting that none of the other sponsors seem to provide financial grants) https://osuosl.org/sponsors/
"Below are groups who supported the Open Source Lab through annual contributions of $25,000 or more during fiscal years 2019 and 2020."
It seems that all of them have contributed financially (though of course, it's unclear which ones do so at present).
"annual contributions of $25,000"
I think includes hardware/service contributions (eg the IBM hardware and the bandwidth suppliers). It's a bit unclear though.
They should reach out to IBM too, since POWER and Z hardware are mostly an IBM thing. ARM should be willing to help too, since they host AARCH64 stuff too.
I feel like that shouldn’t be impossible to raise. I would be more than happy to donate $250, now we just need 999 more to do the same.
I find it odd that they can provide all the infrastructure compute / storage / bandwidth for all of these projects for $35K
Debian and Fedora on their own must be highly demanding??
Form the article: """ Currently provides infrastructure hosting for projects such as Drupal, Gentoo Linux, Debian, Fedora, phpBB, OpenID, Buildroot/Busybox, Inkscape, Cinc and many more! """
Most of Debian's infra is elsewhere, but OSU does host a backup server, logs server, a bugs.d.o frontend, POWER/MIPS/HPPA/RISC-V build/porting servers.
https://db.debian.org/machines.cgi
Similarly, I assume RedHat hosts most of Fedora's infra and OSU does POWER/IBM Z stuff for them.
I would've assumed that Red Hat/Fedora are well-catered for now in terms of POWER and Z, considering who now owns Red Hat.
They are part of gnu compile farm which donates compute to open source projects.
I used them a lot when I worked on OpenSCAD build system, there weren't a lot of places 12+ years ago you could go 'make -j 30' on a PowerPC or 'ctest' and have it run dozens of builds/tests in parallel. Really helped alot, that C++ template stuff would barely build at all on my personal machine.
Sorry to hear this
Am I reading correctly that of the $250K they need, $150K of that goes to a single staff member for 60% of their time? Does that seem...excessive?
That's 60% of the _budget_ not 60% of their time.
Also: Lance is almost certainly working more than 40 hours a week. Also, he isn't just a systems administrator. He's a mentor, fundraiser, any literally everything else that is needed to keep the lab running. There used to be more staff, but it's hard to retain qualified individuals. He's been there for 17 years, he's not doing it for the money, he does it because the OSL is important!
Oh, and since he's a public employee, you can look up the current salary and history.
https://hr.oregonstate.edu/sites/hr.oregonstate.edu/files/er...
https://www.openthebooks.com/oregon-state-employees/?F_Name_...
I'll summarize it:
$107k in 2017 and $124k in 2023. I don't know about you, but someone with 17 years experience could easily be making 2-5x that depending on the company and role.
And it's $124k on the west coast, specifically western Oregon. Coastal PNW is notoriously expensive to live in - sure, Corvallis isn't Seattle, but it's not cheap, either. Folks like to balk at numbers when it comes to publicly (or FOSS-donation-ly) funded salaries, but also balk at tying context to those numbers. It happens almost every time anyone dares try to pay their bills on open-source work: a flame war over "you don't need that number, you could move to your parent's basement in Arkansas and survive on $20k instead!".
No, you are not reading correctly.
The 60% number is the percentage of the budget, not the staff member's allocated time.
However, what do we know about the duties of this staff member? $150k isn't a very high salary for an experienced systems administrator
Presumably the $150k also includes all oncosts, so the actual salary is quite a bit lower still? As a side note I don't understand the arguments about salaries for nonprofits. Sure they should not be outrageously higher than the average, but shouldn't we want to get the best people for these jobs (instead of them working on aware?), or is the argument that if you work for a nonprofit you should be doing it out of altruism and be glad you receive a salary at all?
The argument that I assume you are talking about has some nuance around it. It's mostly about politically connected or nepotistic people who are pulling large salaries for essentially little to no work. I'm sure most regular employees at a nonprofit get treated as poorly as those of us at a normal business.
But's usually not the argument being made, the complains (same in this case) are often about the salaries of the people doing the actual work. Sure I understand the complains about multi-million salaries for the CEOs of some non-for-profit (on the other hand I have the same complains about the ridiculous salaries of CEOs of for-profits), but if that's the nuance, it doesn't come out in the complains.
Well without actual examples from any supposed position, this discussion goes nowhere.
It's higher than an SDE in Seattle, but less than a senior position at those same companies, for people who want some perspective.
Firmly "Middle ground of the area"
It is unclear from this request, but if this is the cost to the employer it is almost certainly a larger number than the actual pay which goes to the individual.
Cost of living in Corvallis <<< Cost of living in Seattle
https://www.numbeo.com/cost-of-living/compare_cities.jsp?cou...
$150k is not "higher than an SDE in Seattle", unless you meant to say "higher than the average salary for an entry-level junior SDE role in Seattle."
That does seem to pretty clearly be what they meant, given the rest of the sentence.
There are usually three main levels people talk about, entry, mid career, and sr. If you say SDE, I'd assume a mid-career person, which is 100% earning way more then 150k in Seattle. https://www.levels.fyi/t/software-engineer/locations/greater... has more data.
I've been in the IT field for 20+ years, but I've never made that much in a year. What's stopping someone from saying, "Hire two cheaper guys, so there's redundancy?"
From a university grants perspective that likely includes benefits.
Grant hiring math is
Salary + benefits = cost
Where benefits = salary *~.4
Does "benefits" also include the tax contributions the company pays? After being 1099 for so long, those definitely sound like a benefit to me!
I honestly don't know but I assume so.
They technically call it 'fringe benefits'. My university has four categories of fringe benefits:
Full
Limited
Partial
Grad Health
The only things it specifies are that partial includes social security and full includes life insurance. But given that whatever I set for a post doc/research scientist/etc. salary is the amount they are paid, I assume that everything else including payroll taxes are encompassed in that 1/3 extra for fringe.
Assuming that's the "fully loaded" cost (ie including taxes, benefits, etc), seems like it would translate to a take-home salary of $100k or less.
Not if we're talking about an experienced engineer that is the only full-time staff for a year. That's the entire budget, so it feels pretty spot on. Maybe I'm out of touch or misunderstanding your point, though.
GP is pointing out that OSU Open Source Lab is only in jeopardy because a majority of the shortfall is in labor costs. 86% to be exact.
> Am I reading correctly that of the $250K they need, $150K of that goes to a single staff member for 60% of their time? Does that seem...excessive?
When I was doing AIX and Solaris system administration in Salem Oregon, they paid me $75 an hour.
A lot of people here are comparing Corvallis to Seattle, but they’re hundreds of miles apart.
Salem is the nearest big city.
TBH, making $75 an hour in Salem was like making $150 an hour in Seattle. You can live REALLY WELL on $75 an hour in Corvallis.
OSU OSL provides CI machines for some of the more exotic architectures like Linux on Z and POWER to some ASF projects. It would a loss to close it down.
Maybe some unicorn billionaires could spare a few millions? Especially the ones who built their wealth on top of open source libraries or databases.
A gap like cannot be closed without corporate and foundation gifts. That said, individual gifts can also contribute to the progress.
As is common with schools, parks districts, etc., the Open Source Lab partners with a 501(c)(3) organization, the Oregon State University Foundation, to accept tax-deductible donations.
For anyone who would like to directly support the Open Source Lab in staying open, please be sure to indicate "Open Source Lab Fund" on the Oregon State Foundation donation page [0]. Note that their form is *not* set up with any tracking to attribute your gift from your clickthrough, and that any general donations to the Foundation will likely *not* support the Lab in this effort to stay open.
[0] https://give.fororegonstate.org/PL1Uv3Fkug, or click through from the general donation page.
Link to donation page: https://osuosl.org/donate/
Oregon State University has a $1.651 billion endowment according to Wikipedia…
https://en.m.wikipedia.org/wiki/List_of_colleges_and_univers...
Would that be an option to save it if corporate sponsorship doesn’t work out?
Most gifts that fund university endowments are earmarked by the donor for specific purposes. And for the money not earmarked, you're competing against all the other priorities, including making up for various unplanned shortfalls of Federal funding.
I ctrl-f'd Oregon State and didn't find it in the link you provided. I think you found UofO's endowment: University of Oregon - $1.651
Wikipedia states that OSU's endowment is $829.9 million (2023).[0]
[0] https://en.wikipedia.org/wiki/Oregon_State_University
The Foundation itself has nearly $1b in assets. The problem with these foundations is they're often streched across the entire university. Which means the foundation behind OSL is also the foundation behind intercollegiate athletics and tons of other completely unrelated programs.
https://projects.propublica.org/nonprofits/organizations/936...
> While the Oregon State College of Engineering (CoE) has generously filled this gap, recent changes in university funding makes our current funding model no longer sustainable.
I suspect this is related to the recent Trump administration actions to withhold funding from colleges and universities. OSU already voted to increase tuition recently amid concerns about future federal funding [0]:
> “New federal priorities and proposed funding cuts, especially for research, may have direct, negative consequences for OSU,” [OSU President Jayathi Murthy] said.
0: https://archive.is/iGX5X
I hear Microsoft loves open source, so they should be able to step up and cover this, right?
Why stop there? Facebook, Google, OpenAI, Netflix, Amazon…
open source only works when youre more than financially and location stable.
corporate fascism has artificially raised prices across the board and ensured that the next gen must work far more for less.
they work with gov to increase taxes, licensing and insurance on the individual while reducing for the corp.
higher education is yet another corrupt corp. theyre not there to help you, but are the introduction to this system.