The way to deal with this is to constantly do live events, and actually build organizational muscle. Not these massive one off events in an area the tech team has no experience in.
We should always be doing (the thing we want to do)
Somme examples that always get me in trouble (or at least big heated conversations)
1. Always be building: It does not matter if code was not changed, or there has been no PRs or whatever, build it. Something in your org or infra has likely changed. My argument is "I would rather have a build failure on software that is already released, than software I need to release".
2. Always be releasing: As before it does not matter if nothing changed, push out a release. Stress the system and make it go through the motions. I can't tell you how many times I have seen things fail to deploy simply because they have not attempted to do so in some long period of time.
There are more just don't have time to go into them. The point is if "you did it, and need to do it again ever in the future, then you need to continuously do it"
Doing dry runs regularly makes sense, but whether actually shipping it makes sense seems context-dependent. It depends on how much you can minimize the side effects of shipping a release.
Consider publishing a new version of a library: you'd be bumping the version number all the time and invalidating caches, causing downstream rebuilds, for little reason. Or if clients are lazy about updating, any two clients would be unlikely to have the same version.
Or consider the case when shipping results in a software update: millions of customer client boxes wasting bandwidth downloading new releases and restarting for no reason.
Even for a web app, you are probably invalidating caches, resulting in slow page loads.
With enough work, you could probably minimize these side effects, so that releasing a new version that doesn't actually change anything is a non-event. But if you don't invalidate the caches, you're not really doing a full rebuild.
So it seems like there's a tension between doing more end-to-end testing and performance? Implementing a bunch of cache levels and then not using it seems counterproductive.
I like all of these considerations, although I also imagine for every context there is some frequency at which it is worthwhile to invalidate the caches to ensure that all parts of the system are still functioning as expected (including the rebuilding of the caches).
It's very hard to do a representative dry run when the most likely potential points of failure are highly load-dependent.
You can try and predict everything that'll happen in production, but if you have nothing to extrapolate from, e.g. because this is your very first large live event, the chances of getting that right are almost zero.
And you can't easily import that knowledge either, because your system might have very different points of failure than the ones external experts might be used to.
> 1. Always be building: It does not matter if code was not changed...
> 2. Always be releasing...
A good argument for this is security. Whatever libraries/dependencies you have, unpin the versions, and have good unit tests. Security vulnerabilities that are getting fixed upstream must be released. You cannot fix and remove those vulnerabilities unless you are doing regular releases. This in turn also implies having good unit tests, so you can do these builds and releases with a lower probability of releasing something broken. It also implies strong monitoring and metrics, so you can be the first to know when something breaks.
Easy: Short term risk versus long term risk. If I deploy with minimal changes today, I'm taking a non-zero short-term risk for zero short-term gain.
While I too am generally a long-term sort of engineer, it's important to understand that this is a valid argument on its own terms, so you don't try to counter it with just "piffle, that's stupid". It's not stupid. It can be shortsighted, it leads to a slippery slope where every day you make that decision it is harder to release next time, and there's a lot of corpses at the bottom of that slope, but it isn't stupid. Sometimes it is even correct, for instance, if the system's getting deprecated away anyhow why take any risk?
And there is some opportunity cost, too. No matter how slick the release, it isn't ever free. Even if it's all 100% automated it's still going to barf sometimes and require attention that not making a new release would not have. You could be doing something else with that time.
In some environments, deploying to production has a massive bureaucracy tax. Paperwork, approvals, limited windows in time, can’t do them during normal business hours, etc.
Those taxes were often imposed because of past engineering errors. For example, Don't deploy during business hours because a past deployment took down production for a day.
A great engineering team will identify a tax they dislike and work to remove it. Using the same example, that means improving the success rate of deployments so you have the data (the success record) to take to leadership to change the policy and remove the tax.
The counterargument is obvious for anyone who has been on call or otherwise responsible for system stability. It's very easy to become risk-averse in any realm.
Deploying is expensive for some models. That could involve customer facing written release notes, etc. Sometimes the software has to be certified by a govt authority.
Additionally, refactor circle jerks are terrible for back-porting subsequent bug fixes that need to be cherry picked to stable branches.
A lot of of the world isn’t CD and constant releases are super expensive.
Unless Netflix eng decides to release a public postmorterm, we can only speculate. In my time organizing small-time live streams, we always had up to 3 parallel "backup" streams (Vimeo, Cloudflare, Livestream). At Netflix's scale, I doubt they could simply summon any of these providers in, but I guess Akamai / Cloudflare would have been up for it.
They've been doing live events since 2023. But it's hard to be prepared for something that's never been done by anyone before — a superbowl scale event, entirely viewed over the internet. The superbowl gets to offload to cable and over the air. Interestingly, I didn't have any problems with my stream. So it sounds like the bandwidth problems might be localized, perhaps by data center or ISP.
I had issues here and there but there was workarounds. Then, towards the end, the quality either auto negotiated or was forced down to accommodate the massive pull.
Yeah, I think people are incorrectly assuming that everyone had the same experience with the stream. I watched the whole thing and only had a few instances of buffering and quality degradation. Not more than 30 seconds total during the stream.
Even if it was only 30% of people had a problem that's still millions of unhappy users. Not great for a time sensitive event.
Also, from lurking in various threads on the topic Netflix's in app messages added to people's irritation by suggesting that they check their WiFi/internet was working. Presumably that's the default error message but perhaps that could have been adjusted in advance somehow.
It's incomprehensible to me that Netflix, one of the most highly skilled engineering teams in the world - completely sh*t the bed last night and provided a nearly unwatchable experience that was not even in the same league as pre-internet live broadcast from 30 years ago.
My bet is that a technical manager told his executive (multiple times) that he needed more resources and engineering time to make live work properly, and they just told him to make do because they didn't want to spend the money.
It could come down to something as stupid as:
Executive: "we handled [on demand show ABCD] on day one, that was XX million"
Engineering: "live is really different"
Executive: (arguing about why it shouldn't be that different and should not need a lot of new infrastructure)
Engineering: (can't really argue with his boss about this anymore after having repeated the same conversation 3 or 4 times)
-- tells the team: we are not getting new servers or time for a new project. We have to just make do with what we have. You guys are brilliant, I know you can do it!"
To me the difference is that in 2012, you had companies focusing on delivering a quality product, whether it made money or not. Today, the economic environment has shifted a lot and companies are trying to increase profits while cutting costs. The result is inevitably a decline in quality. I'm sure that Netflix could deliver a flawless live stream to millions of viewers, but the question is can they do it while making a profit that Wall Street is happy with. Apparently not.
But is there a way that Netflix might have learned from all of Youtube's past mistakes?
The only reasonable way to scale something like this up is probably to... scale it up.
Sure, there are probably some generic lessons, but I bet that the pain points in Netflix's architecture (historically grown over more than a decade and optimized towards highly cacheable content) are very different from Youtube, which has ramped up live content gradually over as many years.
The funny thing is I was just reading something on HN like three days ago about how light years ahead Netflix tech was compared to other streaming providers. This is the first thing I thought of when I saw the reports that the fight was messing up.
The average quality of talent has gone way down compared to 2012 though.
E.g. the median engineer, excluding entry level/interns, at YouTube in 2012 was a literal genius at their niche or quite close to it.
Netflix simply can’t hire literal geniuses with mid six figure compensation packages in 2024 dollars anymore… though that may change with a more severe contraction.
Everyone here talking like this something unique netflix had to deal with. Hotstar live streamed india va Pakistan cricket match with zero issues with all time high live viewership ever in the history of live telecast. Why would viewers paying $20 month want to think about their technical issues, they dropped the ball pure and simple. Tech already exists for this, it’s been done before even by espn, nothing new here.
"The festivities were streamed live from the Peacock Theater in L.A. across more than 30 platforms including YouTube, Twitch, Facebook, TikTok Live, X (Twitter), Steam, WeChat, Bilibili, Huya, DouYu, Xiaohongshu and Instagram Live"
But that's exactly the point: Netflix didn't do this in a vacuum, they did it within Netflix.
It might just have been easier to start from scratch, maybe using an external partner experienced in live streaming, but the chances of that decision happening in a tech-heavy company such as Netflix that seems to pride itself on being an industry leader are close to zero.
People just do not appreciate how many gotchas can pop up doing anything live. Sure, Netflix might have a great CDN that works great for their canned content and I could see how they might have assumed that's the hardest part.
Live has changed over the years from large satellite dishes beaming to a geosat and back down to the broadcast center($$$$$), to microwave to a more local broadcast center($$$$), to running dedicated fiber long haul back to a broadcast center($$$), to having a kit with multiple cell providers pushing a signal back to a broadcast center($$), to having a direct internet connection to a server accepting a live http stream($).
I'd be curious to know what their live plan was and what their redundant plan was.
Sorry for the off topic but what’s this thing that I only come across in Hacker News about referring to a company by their stock exchange name (APPL, MSFT, etc) outside of a stock context? It seems really weird to me.
As a counterpoint, I do this sometimes, not thinking at all about stocks but instead as a standardized abbreviation of sorts. Ms for example can mean tons of things from a title to multiple sclerosis to milliseconds. MSFT is clear and half the length.
In-group signaling for people who like playing or thinking about the stock market. Similar to how people who make travel a big part of their identity refer to cities by their airport code.
I have always assumed that a focus on stock tickers is the natural result when your primary user base is a group of people hyper focused on “total compensation” and stock grants. The name hackernews is merely a playful reference to the history of the site. Like the name “Patriot Act.”
Technically it’s one fewer keystroke (and it’s AAPL).
It’s a lot fewer keystrokes for MS (Morgan Stanley), GS (Goldman Sachs) and MSFT (Microsoft) than it is for AAPL, but it’s a force of habit for some. Once you’re used to referring to firms by their ticker symbols, you do it all the time.
E.g. an ex trader friend still says “spot” instead of “point” when referring to decimal points, even if talking in other contexts like software versions.
If commercial = public, then no - you can not use multicast for this. It is heavily used within some enterprise networks though like if you go to a gym with lots of TVs they are all likely on multicast
This is the whole point of chaos engineering that was invented at Netflix, which tests the resiliency of these systems.
I guess we now know the limits of what "at scale" is for Netflix's live-streaming solution. They shouldn't be failing at scale on a huge stage like this.
I look forward to reading the post mortem about this.
Everyone keeps mentioning at scale. I seriously doubt this was an "at scale" problem. I have strong suspicion this was a failure at the origination point being able to push a stable signal. That is not an "at scale" issue, but a hubris of we can do better/cheaper than broadcasting standard practices
I've tried to watch an old Seinfeld episode during this event. It was freezing every few minutes even at downgraded bitrate. A video that should be on my local CDN node.
I am not sure that it is an issue with the origination point. In fact I just thought it was my ISP because my daughter's boyfriend was watching and doing facetime with her and my video was dropping but his was not. I have 2gb fiber and we regularly stream five TVs without any issue, so it should not have been a bandwidth issue.
As counterpoint, I observed 2-3 drops in bitrate, but an otherwise fine experience. So the problem seems to have been in dissemination, not at the origin.
I highly doubt this. Netflix has a system of OCAs that are loaded with hard disks, are installed in ISP’s networks, and serve the majority of those ISP’s customers.
Given than many people had no problems with the stream, it is unlikely to have been an origin problem but more likely the mechanism to fanout quickly to OCAs. Normally latency to an OCA doesn’t matter when you’re replicating new catalogs in advance, but live streaming makes a bunch of code that previously “didn’t need to be fast” get promoted to the hot path.
Do you live stream the superbowl? Me and everyone I know watch it over antenna broadcast tv. I think it is easier to have millions of tvs catch airwaves vs millions of point to point https video streams.
If you watch it over cable, you're live streaming it. Let's face it, that's where the vast majority of viewers see it. Few people view OTA even if the quality is better.
Live sports do not broadcast the event directly to a streamer. They push it to their broadcast centers. It then gets distributed from there to whatever avenues it needs to go. Trying to push a live IP stream directly from the remote live venue rarely works as expected. That's precisely why the broadcasters/networks do not do it that way
> If you watch it over cable, you're live streaming it.
Those are multicast feeds.
> Trying to push a live IP stream directly from the remote live venue rarely works as expected.
In my experience it almost always works as expected. We have highly specialized codecs and equipment for this. The stream is actively managed with feedback from the receiver so parameters can be adjusted for best performance on the fly. Redundant connections and multiple backhauls are all handled automatically.
> That's precisely why the broadcasters/networks do not do it that way
We use fixed point links and satellite where possible because we own the whole pipe. It's less coordination and effort to setup and you can hit venues and remotes where fixed infrastructure is difficult or impossible to install.
Is cable video over IP now? Last time I looked (which was forever ago), even switched video was atsc with a bit of messaging for the cable box to ask what channel to tune to, and to keep the stream alive. TV over teleco systems seems to be highly multicast, so kind of similar, headend only has to send the content once, in a single bitrate.
Not really the same as an IP service live stream where the distribution point is sending out one copy per viewer and participating in bitrate adaptation.
AFAIK, Netflix hasn't publicly described how they do live events, but I think it's safe to assume they have some amount of onsite production that outputs the master feed for archiving and live transcoding for the different bitrate targets (that part may be onsite, or at a broadcast center or something cloudy), and then goes to a distribution network. I'd imagine their broadcast center/or onsite processing feeds to a limited number of highly connected nodes that feed to most of their CDN nodes; maybe more layers. And then clients stream from the CDN nodes. Nobody would stream an event like this direct from the event; you've got to have something to increase capacity.
Over the US and Canada it mostly is, though how advanced the transition is is very regional.
The plan is to drop both analog signal and digital (QAM) to reclaim the frequencies and use them for DOCSIS internet.
Newer set top boxes from Comcast (xfinity) runs over the internet connection (in a tagged VLAN on a private network, and they communicate over a hidden wifi).
> If you watch it over cable, you're live streaming it.
Which is probably done over the cableco's private network (not the public Internet) with a special VLAN used for television (as opposed to general web access). They're probably using multicast.
When Netflix started it was the first in the space and breaking ground which is how they became a "tech" company that happens to stream media however it has been 15 years and since than the cloud providers have basically build "netflix as a service". I suspect most of the big streamers are using that instead of building their own in house thing and going through all the growing pains netflix is.
> People just do not appreciate how many gotchas can pop up doing anything live.
Sure thing, but also, how much resources do you think Netflix threw on this event? If organizations like FOSSDEM and CCC can do live events (although with way smaller viewership) across the globe without major hiccups on (relatively) tiny budgets and smaller infrastructure overall, how could Netflix not?
The CCC video crew has its fair share of geeks from broadcasting corporations and studio houses. Their combined institutional knowledge about live events and streaming distribution is probably in the same ballpark as that of giant global TV networks.
They also have the benefit of having practiced their craft at the CCC events for more than a decade. Twice a year. (Their summer event is smaller but still fairly well known. Links to talks show up on HN every now and then.)
Funky anecdote: the video crew at Assembly have more broadcasting and live AV gear for their annual event than most medium-sized studios.
This is true, but scale comes after production. Once you have the video encoded on a server with a stable connection the hard part is over. What netflix failed to do is spread the files to enough servers around the globe to handle the load. I'm surprised they were unable(?) to use their network of edge servers to handle the live stream. Just run the stream with a 10 second delay and in that time push the stream segments to the edge server
This right here is where I'd expect the failure to occur. This isn't Joey Beercan running OBS using their home internet connectivity.
This is a major broadcast. I'd expect a full on broadcast truck/trailer. If they were attempting to broadcast this with the ($) option directly to a server from onsite, then I would demand my money back. Broadcasting a live IP signal just falls on its face so many times it's only the cheap bastard option. Get the video signal as a video signal away from the live location to a facility with stable redundant networking.
This is the kind of thinking someone only familiar with computers/software/networking would think of rather than someone in broadcasting. It's nice to think about disrupting, but this is the kind of failure that disruptors never think about. Broadcasters have been there done that with ensuring live broadcasts don't go down because an internet connection wasn't able to keep up.
I’ve been using vyvx since it was called global crossing/genesis, it was fairly unique when it started, but point to point ip distributon of programs has been the norm for at least 15 years. Still have backup paths on major events on a different technology, you’d be surprised how common a dual failure on two paths can be. For example output from the euro football this summer my mai paths were on a couple of leased lines with -7, but still had a backup on some local internet into a different city just incase there was a meltdown of the main providers network (it’s happened before with ipath, automation is great until it isn’t)
> how much resources do you think Netflix threw on this event?
Based on the results, I hope it was a small team working 20% time on the idea. If you tell me they threw everything they had at it to this result, then that's even more embarrassing for them.
Netflix is good only on streaming ready made content, not live streaming, but;
1. Netflix is a 300B company, this isn't a resources issue.
2. This isn't the first time they have done live streaming at this scale either. They already have prior failure experience, you expect the 2nd time to be better, if not perfect.
3. There were plenty of time between first massive live streaming to second. Meaning plenty of time to learn and iterate.
The problem is that provisioning vast capacity for peak viewership is expensive and requires long-term commitment. Some providers won't give you more connectivity to their network unless you sign a 12 month deal where you prepay that.
Peak traffic is very expensive to run, because you're building capacity that will be empty/unsused when the event ends. Who'd pay for that? That's why it's tricky and that's why Akamai charges these insane prices for live streaming.
A "public" secret in that network layer is usually not redundant in your datacenter even if it's promised. To have redundant network you'd need to double your investment and it'll seat idle of at 50% max capacity. For 2hr downtime per year when you restart the high-capacity routers it's not cost efficient for most clients.
Then sign a contract with Akamai, who has been in business for 25 years? You outsource if you aren’t planning to do something very often.
There is no middle ground where you commit a mediocre amount of resources, end up with downtime and a mediocre experience, and then go “but we saved money.”
They have the NFL next month on Christmas day. So that'll be a big streaming session but I think it'll be nothing compared to this. Even Twitter was having problems handling the live pirate streams there.
You can't solve your way out of a complex problem that you have created and which wasn't needed in the first place. The entire microservices thing was overly complex with zero benefits
I spoke to multiple Netflix senior technicians about this.
That's a ridiculous statement. PrimeVideo is the leader in terms of sports events streaming over internet and it is composed of hundreds of microservices.
Live streaming is just much harder than streaming, and it takes a years of work and a huge headcount to get something good.
This comment shows how a very random blog about a very small part of a product can dominate all conversation about it. Prime video famously did not undo anything. Out of 100+ teams one team undid one service. But somehow similar comments are common on HN. I am making no judgement or microservice or not just on this particular comment.
It was a single team for a very specific use case.
To be clear when I said that PrimeVideo is composed of hundreds of microservices, I actually meant that it's composed of hundreds of services, themselves composed, more often than not, of multiple microservices.
Depending on your definition of a microservice, my team alone owns dozens.
It wasn't even just buffering issues, the feed would just stop and never start again until I paused it and then clicked "watch live" with the remote.
It was really bad. My Dad has always been a fan of boxing so I came over to watch the whole thing with him.
He has his giant inflatable screen and a projector that we hooked up in the front lawn to watch it, But everything kept buffering. We figured it was the Wi-Fi so he packed everything up and went inside only to find the same thing happening on ethernet.
He was really looking forward to watching it on the projector and Netflix disappointed him.
Commercial boxing has always been like WWE or MMA with a thin veneer of actual sport to it, i.e. it is just entertainment[1].
To rephrase your question then what does someone think of the entertainment on display?
I don't think it was good entertainment.
None of the hallmarks of a good show was present. i.e. It wasn't close, nor was it bloody or anything unexpected like say a KO everything went pretty much as expected. It wasn't nice watch as all,no skill or talent was on disply, all Paul had to do was use his speed to backpedal from the slow weak punches of a visibly older tyson with a bum knee and land some points occasionally to win.
--
[1] There is a deeper argument here is any spectator sports just entertainment or is truly about skill talent and competition. Boxing however including the ones promoted by traditional four major associations falls clearly on the entertainment side than say another sport like NFL to me.
Reading the comments here, I think one thing that's overlooked is that Netflix, which has been on the vanguard of web-tech and has solved many complicated problems in-house, may not have had the culture to internally admit that they needed outside help to tackle this problem.
It’s insane the excuses being made here for Netflix’s apparently unique circumstances.
They failed. Full stop. There is no valid technical reason they couldn’t have had a smooth experience. There are numerous people with experience building these systems they could have hired and listened to. It isn’t a novel problem.
Here are the other companies that are peers that livestream just fine, ignoring traditional broadcasters:
- Google (YouTube live), millions of concurrent viewers
- Amazon (Thursday Night Football, Twitch), millions of concurrent viewers
- Apple (MLS)
NBC live streamed the Olympics in the US for tens of millions.
As a cofounder of a CDN company that pushed a lot of traffic, the problem with live streaming is that you need to propagate peak viewership trough a loooot of different providers. The peering/connectivity deals are usually not structured for peak capacity that is many times over the normal 95th percentile. You can provision more connectivity, but you don't know how many will want to see the event.
Also, live events can be trickier than stored files, because you can't offload to the edges beforehand to warm up the caches.
So Netflix had 2 factors outside of their control
- unknown viewership
- unknown peak capacities outside their own networks
Both are solvable, but if you serve "saved" content you optimize for different use case than live streaming.
I don't disagree that Netflix could have / should have done better. But everybody screws these things up. Even broadcast TV screws these things up.
Live events are difficult.
I'll also add on, that the other things you've listed are generally multiple simultaneous events; when 100M people are watching the same thing at the same time, they all need a lot more bitrate at the same time when there's a smoke effect as Tyson is walking into the ring; so it gets mushy for everyone. IMHO, someone on the event production staff should have an eye for what effects won't compress well and try to steer away from those, but that might not be realistic.
I did get an audio dropout at that point that didn't self correct, which is definitely a should have done better.
I also had a couple of frames of block color content here and there in the penultimate bout. I've seen this kind of stuff on lots of hockey broadcasts (streams or ota), and I wish it wouldn't happen... I didn't notice anything like that in the main event though.
Experience would likely be worse if there were significant bandwidth constraints between Netflix and your player, of course. I'd love to see a report from Netflix about what they noticed / what they did to try to avoid those, but there's a lot outside Netflix's control there.
Disney HotStar managed to stream ~60M livestreams for the Cricket world cup a year ago. The problem has been solved. Livestreaming sports just have a different QoS expectations than on demand.
I wouldn't say it's a solved problem, how many other companies are pulling off those numbers? Isn't that the current record for concurrent streams? And wasn't it mostly to mobile devices?
The size of engineering head count is not informative, it really depends on how much is in-house and how much is external for Hotstar that would be i.e parent Disney or before Fox or staffing from IT consulting organizations who will not be on payroll.
For what it is worth, all things being equal there would be lot more non engineering in Hotstar for 2000 employees versus a streaming company of similar size or scale of users. Hotstar operates in challenging and fragmented market, India has 10+ major languages(and corresponding TV, music and movie markets) Technically there is not much difference to what Netflix or Disney has to do for i18n, however operationally each market needs separate sales, distribution and operations.
---
P.S. Yes Netflix operates in more markets including India than anybody else, however if you are actually using Netflix for almost any non English content, you will know how weak their library and depth in other markets are, their usual model in most of these markets is to have few big high quality(for that market) content rather than build depth.
P.P.S. Also yes, Indian market is seeing consolidation in the sense that many releases on streaming are multiple lingual and use major stars from more than one language to draw talent ( not new, but growing in popularity as distribution becomes cheaper with streaming), however this is only seen in big banner productions as tastes are quite different in each market and can't scale for all run of the mill content.
Amazon had their fair share of livestream failures and for notably less viewers. I don't think they deserve a spot on that list. I briefly worked in streaming media for sports and while it's not a novel problem, there are so many moving parts and points of failure that it can easily all go badly.
There is no one "Amazon" here, there are at least 3:
* Twitch: Essentially invented live streaming. Fantastic.
* Amazon Interactive Video Service [0]: Essentially "Twitch As A Service", built by Twitch engineers. Fantastic.
* Prime Video. Same exact situation as Netflix: original expertise is all in static content. Lots of growing pains with live video and poor reports. But they've figured it out: now there are regular live streams (NHL and NFL), and other channel providers do live streaming on Prime Video as a distribution platform.
Doesn't twitch almost fall over (other non-massive streams impacted) when anyone gets close to 4-5m concurrent viewers? I remember last time it happened everything started falling over, even for smaller streams. Even if Netflix struggled with the event, streaming other content worked just fine for me.
It's not full stop. There are reasons why they failed, and for many it's useful and entertaining to dissect them. This is not "making excuses" and does not get in the way of you, apparently, prioritizing making a moral judgment.
On a few forum sites I'm on, people are just giving up. Looking forward to the post-mortem on how they weren't ready for this (with just a tiny bit of schadenfreude because they've interviewed and rejected me twice).
AB84 streamed it live from a box at the arena to ~5M viewers on Twitter. I was watching it on Netflix, I didn't have any problems, but I also put his live stream up for the hell of it. He didn't have any issues that I saw.
It’s not everyone. Works fine for me though I did have to reload the page when I skipped past the woman match to the Barrios Ramos fight and it was stuck buffering at 99%.
Cable TV (or even OTA antenna in the right service area) is simply a superior live product compared to anything streaming.
The Masters app is the only thing that comes close imo.
Cable TV + DVR + high speed internet for torrenting is still an unmatched entertainment setup. Streaming landscape is a mess.
It's too bad the cable companies abused their position and lost any market goodwill. Copper connection direct to every home in America is a huge advantage to have fumbled.
The interesting thing is that a lot of TV infrastructure is now running over IP networks. If I were to order a TV connection for my home I'd get an IPTV box to connect to my broadband router via Ethernet, and it'd simply tell the upstream router to send a copy of a multicast stream my way.
Reliable and redundant multicast streaming is pretty much a solved problem, but it does require everyone along the way to participate. Not a problem if you're an ISP offering TV, definitely a problem if you're Netflix trying to convince every single provider to set it up for some one-off boxing match.
When you step back and look at the situation, it's not hard to see why Netflix dropped the ball here. Here's now I see it (not affiliated with Netflix, pure speculation):
- Months ago, the "higher ups" at Netflix struck a deal to stream the fight on Netflix. The exec that signed the deal was probably over the moon because it would get Netflix into a brand new space and bring in large audience numbers. Along the way the individuals were probably told that Netflix doesn't do livestreaming but they ignored it and assumed their talented Engineers could pull it off.
- Once the deal was signed then it became the Engineer's problem. They now had to figure out how to shift their infrastructure to a whole new set of assumptions around live events that you don't really have to think about when streaming static content.
- Engineering probably did their absolute best to pull this off but they had two main disadvantages, first off they don't have any of the institutional knowledge about live streaming and they don't really know how to predict demand for something like this. In the end they probably beefed up livestreaming as much as they could but still didn't go far enough because again, no one there really knows how something like this will pan out.
- Evening started off fine but crap hit the fan later in the show as more people tuned in for the main card. Engineering probably did their best to mitigate this but again, since they don't have the institutional knowledge of live events, they were shooting in the dark hoping their fixes would stick.
Yes Netflix as a whole screwed this one up but I'm tempted to give them more grace than usual here. First off the deal that they struck was probably one they couldn't ignore and as for Engineering, I think those guys did the freaking best they could given their situation and lack of institutional knowledge. This is just a classic case of biting off more than one can chew, even if you're an SV heavyweight.
This isn't Netflix's first foray into livestreaming. They tried a livestream last year for a reunion episode of one of their reality TV shows which encountered similar issues [0]. Netflix already has a contract to livestream a football event on Christmas, so it'll be interesting to see if their engineers are able to get anything done in a little over a month.
These failures reflect very poorly on Netflix leadership. But we all know that leadership is never held accountable for their failures. Whoever is responsible for this should at least come forward and put out an apology while owning up to their mistakes.
> But we all know that leadership is never held accountable for their failures.
You've never heard of a CEO or other C-suite or VP getting fired?
It most definitely happens. On the other hand, people at every level make mistakes, and it's preferable that they learn from them rather than be fired, if at all possible.
Accountability can take many forms. I don't think they should be fired for making a mistake, I think they should release a statement recognizing their failure along with a post-mortem. Not a particularly high bar, but most leadership failures are often swept under the rug without any public accountability or evidence that they've learned anything.
We have evidence of prior failures with livestreaming from Netflix. Were the same people responsible for that failure or do we have evidence of them having learned anything between events? If anything, I'd expect the best leaders would have a track record that includes failures while showcasing their ability to overcome and learn from those mistakes. But based on what information is publicly available, this doesn't seem to be the case in this situation.
"Solved" merely means you don't need to invent something new to solve it. It doesn't mean trivial nor easy. And it definitely doesn't mean the problem is above trade-offs.
> "GPGPU compute is a solved problem if you buy Nvidia hardware" type comment
You're replacing the word hire with buy. That misconstrues the comment. If you need to do GPGPU compute and have never done it, you work with a team that has. (And if you want to build it in house, you scale to it.)
>"GPGPU compute is a solved problem if you buy Nvidia hardware"
Which is valid? If your problem can be solved by writing a check, then it's the easiest problem to have on the planet.
Netflix didn't have to put out 3 PhD dissertations on how to improve the SOTA of live streaming, they only needed to reliably broadcast a fight for a couple hours.
That is a solved problem.
Amazon and Cloudflare do that for you as a service(!). Twitch and YouTube do it literally every day. Even X started doing it recently so.
Landing on Mars is a solved problem. Nuclear bombs are a solved problem. Doesn't mean anyone can just write a check and get it done and definitely doesn't mean any business model can bear that cost.
Look. I'm a small startup employee. I have a teeny tiny perspective here. But frankly speaking the idea that Netflix could just take some off the shelf widget and stuff it in their network to solve a problem... It's an absurd statement for even me. And if there's anyone it should apply to it would be a little startup company that needs to focus on their core area.
Every off the shelf component on the market needs institutional knowledge to implement, operate, and maintain it. Even Apple's "it just works" mantra is pretty laughable in the cold light of day. Very rarely in my experience do you ever get to just benefit from someone else's hard work in production without having an idea how properly implement, operate, and maintain it.
And that's at my little tiny ant scale. To call the problem of streaming "solved" for Netflix... Given the guess of the context from the GP post?
I just don't think this perspective is realistic at all.
> the idea that Netflix could just take some off the shelf widget and stuff it in their network to solve a problem
Right. They have to hire one of the companies that does this. Each of YouTube, Twitch (Amazon), Facebook and TikTok have, I believe, handled 10+ million streams. The last two don't compete with Netflix.
I believe this is the spirit of the "solved problem" comment: not that the solution is an off-the-shelf widget, but that if it has ever been solved, then that solution could technically be used again, even if organizing the right people is exorbitantly expensive.
There are multiple companies that offer this capability today that would take a few weeks to hide behind company branding. This was a problem of netflix just not being set up for live stream but thinking they could handle it.
There are endless amounts of stories and situations in which selling something before it really exists has helped businesses. It's totally plausible that a team working on video streaming at the scale of Netflix could figure out live streaming.
Pre-optimization is definitely a thing and it can massively hurt (i.e. startups go under) businesses. Let's stop pretending any businesses would say 'no' to extra revenue even before the engineering team had full assurance there was no latency drop.
My speculation here is this was just classic SV cockiness. The team that closed this deal probably knew that they didn't have the capability but I'm sure the arguments for doing it anyways was something along the lines of: "we have the best engineers in the bay area, we can probably figure this out"
Do startups really do this? I thought the capability is built or nearly built or at least in testing already with reasonable or amazing results, THEN they go to market?
Do startups go to other startups, fortune 500 companies and public companies to make false promises with or without due diligence and sign deals with the knowledge that the team and engineers know the product doesn't have the feature in place at all?
In other words:
Company A: "We can provide web scale live streaming service around the world to 10 billion humans across the planet, even the bots will be watching."
Company B: "OK, sounds good, Yes, here is a $2B contract."
Company A: "Now team I know we don't have the capability, but how do we build, test and ship this in under 6 months???"
Startups absolutely sell things they haven't made yet and might not even be capable of doing.
Next thing you know it's 9pm on a Sunday night and your desperately trying to ship a build for a client.
Netflix isn't some scrappy company though. If I had to guess they threw money at the problem.
A much better approach would of been to slowly scale over the course of a year. Maybe stream some college basketball games first, slowly picking more popular events to get some real prod experience.
Instead this is like their 3rd or 4th live stream ever. Even a pre show a week before would of allowed for greater testing.
I'm not a CTO of a billion dollar company though. I'm just an IC who's seen a few sites go down underload.
To be fair no one knows how it's going to go before it happens. It would of been more surprising for them to pull this off without issues... It's a matter of managing those issues. I know if I had paid 30$ for a Netflix subscription to watch this specific event I'd assume I got ripped off.
You don't necessarily have to make false promises.
You can be totally honest and upfront that the functionality doesn't exist yet and needs to be built first, but that you think you understand the problem space and can handle the engineering, provided you can secure the necessary funding, where, by the way, getting a contract and some nominal revenue now could greatly help make this a reality...
And if the upside sounds convincing enough, a potential customer might happily sign up to cover part of your costs so they can be beta testers and observe and influence ongoing development.
Of course it happens all the time that the problem space turns out to be more difficult than expected, in which case they might terminate the partnership early and then the whole thing collapses from lack of funding.
If anything, startups are more transparent about it.
In the enterprise sector this is rampant. Companies sell "platforms" and those missing features are supposed to be implemented by consultants after the sale. This means the buyer is the one footing the bill for the time spent, and suffering with the delays.
Many do, as far as initial investment goes. It makes sense when you think about the capital intensive nature of most startups (including more than web startups here, e.g. lab tech commercialization). It also accurately describes a research grant.
That's for startups that can't bootstrap (most of them). For ones which can, they may still choose to do this with customers, as you describe, because it means letting their work follow the money.
They arent clowns at all.
Ita a totally different engineering problem and you cant just spin up live streaming capacity on demand. The entire system end to end isnt optimized for live streams yet.
I wonder if there will be any long term reputational repercussions for Netflix because of this. Amongst SWEs, Netflix is known for hiring the best people and their streaming service normally seems very solid. Other streaming services have definitely caught up a bit and are much more reliable then in the early days, but my impression still has always been that Netflix is a step above the rest technically.
This sure doesn't help with that impression, and it hasn't just been a momentary glitch but hours of instability. And the Netflix status page saying "Netflix is up! We are not currently experiencing an interruption to our streaming service." doesn't help either...
Not the same demographic but their last large attempt at live was through a Love is blind reunion. It was the same thing, millions of people logging in, epic failure, nothing worked.
They never tried to do a live reunion again. I suppose they should have to get the experience. Because they are hitting the same problems with a much bigger stake event.
yup wanted to say that live stream stuttering has happened before on Netflix - I don't think the reputation is deserved.
From a livestreaming standpoint, netflix is 0/x - for many large events such as love is blind, etc.
From a livestreaming standpoint, look to broadcast news, sports / Olympics broadcasters, etc and you'll see technology, equipment, bandwidth, planning, and professionalism at 1000x of netflix.
Heck, for publicly traded quarterly earnings livestream meetings, they book direct satellite time in addition to fiber to make sure they don't rely only on terrestrial networks which can fail. From a business standpoint, failure during a quarterly meeting stream can mean the destruction of a company (by making shareholders mad that they can't see and vote during the meeting making them push for internal change) - so the stakes are much higher than live entertainment streaming.
Netflix is good at many things, livestreaming is not one of those things.
for livestreams, individual events like the Olympics probably has a surge audience of 10x of netflix events.
Netflix events is small potatoes compared to other livestream stalwarts.
Imagine having to stream a cricket match internationally to UK / India / Australia with combined audience that crushes the Superbowl or a football match to all of Europe, or even something like livestreaming F1 racing that has multiple magnitudes of audience than a boxing match and also has 10x the number of cameras (at 8K+ resolution) across a large physical staging arena (the size of the track/course) in realtime, in addition to streaming directly from the cockpit of cars that are racing 200mph++.
Livestream focused outfits do this all day, everyday.
Netflix doesn't even come close to scratching the "beginner" level of these kinds of live events.
It's a matter of competencies. We wouldn't expect Netflix to be able to serve burgers like McDonald's does - Livestreaming is a completely different discipline and it's hubris on Netflix's part to assume just because they're good at sending video across the internet they can competently do livestreaming.
the point i’m making is that the netflix live streaming timeline didn’t go
chris rock -> love is blind -> mike tyson
they have had other, successful executions in between. the comment i was replying to had cherry picked failures and i’m trying to git rebase them onto main.
From what I've heard, Netflix has really diluted the culture that people know of from the Patty McCord days.
In particular, they have been revising their compensation structure to issue RSUs, add in a bunch of annoying review process, add in a bunch of leveling and titles, begin hiring down market (e.g. non-sr employees), etc.
In addition to doing this, shuffling headcount, budgets, and title quotas around has in general made the company a lot more bureaucratic.
I think, as streaming matured as a solution space, this (what is equivalent to cost-cutting) was inevitable.
If Netflix was running the same team/culture as it was 10 years ago, I'd like to say that they would have been able to pull of streaming.
Combination of 2 and 3. The business changed. Streaming was more or less a solved problem for Netflix. They needed money for content, not expensive engineers. Ted is co-ceo… you can see where the priority is.
If places like Paramount+ can figure it out, Netflix, given their 10+ year head start on streaming and strong engineering culture, should also have been able to. And if you don't like my example, literally every other streaming service has streamed live sports without issue. YT TV, Hulu, Paramount+, Amazon Prime, Peacock, even Apple TV streams live sports.
It may be "new" to them, but they should have been ready.
I won’t argue that they shouldn’t have done better, I’m only pointing out that this is fairly different from their usual product. Amazon, YouTube, and Hulu all have a ton of experience with live streaming by now. Apple has live streamed wwdc for several years.
I did expect that Netflix would have appropriately accounted for demand and scale, though, especially given the hype for this particular event.
Most people pay Netflix to watch movies and tv shows, not sports. If I hadn't checked Hacker News today, I wouldn't even know they streamed sports, let alone that they had issues with it. Even now that I do, it doesn't affect how I see their core offering, which is their library of on-demand content.
Netflix's infrastructure is clearly built for static content, not live events, so it's no shock they aren't as polished in this area. Streaming anything live over the internet is a tough technical challenge compared to traditional cable.
Has Netflix ever live streamed something before? People on reddit are reporting that if you back up the play marker by about 3 minutes the lag goes away. They've got a handle on streaming things when they have a day in advance to encode it into different formats and push it to regional CDNs. But I can't recall them ever live streaming something. Definitely nothing this hyped.
If Netflix still interviews on hacker rank puzzles I think this should be a wake up call. Interviewing on irrelevant logic puzzles is no match for systems engineering.
I did a round of netflix interviews, didn't get an offer (but passed the technical coding rounds) they absolutely had the best interview process of any company I've interviewed at my entire career.
They do make you code but the questions were
1. Not on hacker rank or leetcode
2. Pratical coding questions that didn't require anything more than basic hashmaps/lists/loops/recursion if you want. Some string parsing, etc.
They were still hard, you had to code a fast, but no tricky algorithms required. It also felt very collaborative, it felt like you were driving pair programming. Highly recommended even though didn't get an offer!
For systems design and engineering, absolutely this. I expected the very highest standards and upmost uptime from Netflix, similar to Google and Amazon.
Tells you the uselessness of their engineering blogs.
I don't spend much time streaming, but I got a glimpse of the Amazon Prime catalog yesterday, and was surprised at how many titles on the front page were movies I'd actually watch. Reminded me of Netflix a dozen years ago.
Amazon Prime isn't so great. Lots of for rent/purchase content or content with ads these days. And they end up repeating slots of content in all the rows in their UI, so I end up seeing the same suggestions everywhere rather than much that's new (other than first party productions).
To me they're basically padding their front page.
But honestly that's most of the major streaming platforms these days. I recently cancelled Disney Plus for similar reasons. The only reasons I don't cancel prime or Netflix are because I have family members I split the memberships with to share.
I recently found a lil dvd rental place in my city. It’s a non-profit, they also do archivals and stuff.
It’s pretty much a two-story townhouse packed head to toe with DVDs (lots of blu rays!)
You don’t realize how limited the streaming collection is until you’re back in a movie store, looking through thousands and thousands of movies you would never find otherwise.
Since I found it, I’ve started doing movie night every week with my friends. It’s such an absolute blast. We go to the store, each pick out a random movie that looks good (or bad, in a good way) or just different.
That's an excellent option. I think it'd be remiss not to mention local libraries. Of course, your mileage may vary, but the ones I've gone to do seem to have adequate selections. I just don't often make time to go there and browse like I would have at traditional video rental places back in the day.
Heck, mine even have some video games; though from when I've checked they're usually pretty back-reserved.
I was in high school in the early 00s, and going to the movies was such a big part of my life. Now, I never even know what's out.
I suspect life stage is a factor, but it does feel like there are many classes of entertainment (cinema and standup come to mind) that don't resonate like they used to.
Back in the day everyone was watching the same thing.
The choices for entertainment were limited to whatever was showing in movie theatres, whatever was on TV and whatever record stores were selling.
I've given Netflix a lot more money than I've gotten value out of. I've had an account for ~15y and only really use it for airplanes unless there's a specific thing I'm excited to watch.
I'm in the same boat where as soon as they make it too hard to share, I'll probably cancel it. I think the main reason their sharing crackdown hasn't been a problem so far is that I use it so seldomly, it thinks the "main" address is my parents, which makes it easy for me to pass the "are you traveling" 2FA on my own phone when I do want to watch something.
> And they end up repeating slots of content in all the rows in their UI, so I end up seeing the same suggestions everywhere rather than much that's new
All of the streaming services do this and I hate it. Netflix is the worst of the bunch, in my experience. I already scrolled past a movie, I don't want to watch it, don't show it to me six more times.
Imagine walking through a Blockbuster where every aisle was the same movies over and over again.
There's also the "FreeVee" items, which have ads regardless of whether you're a prime subscriber or not. And it feels like a lot of their catalog has been transferred over to FreeVee.
It's been pretty rough the last few years. So many great films and series, not to mention kids programming, removed to make way for mediocre NeTfLiX oRiGiNaLs and Bollywood trash.
Prime Video has to be the worst of all major streaming services. The video quality is horrible, its crippled with ads (3 not skippable ads for an episode of 45 minutes, lastly), and a lot of interesting titles are behind a "partner paywall".
I have prime and my shopping experience is crippled with ads too.
I think it got worse for sellers recently too. If I search for something, like a specific item using its description, sometimes the only result for it shows "sponsored".
It used to show up as sponsored and also unsponsored below.
If this changed, I assume it is bad for the seller. Either they pay for all search results, or their metrics are skewed because all searches were helped by "sponsorship" (and there are no longer unsponsored hits)
I was watching the rings of power and it started with a "Commercial free experience provided by so and so" with a long ad at the start of the episode, and then a third of the way into the episode, at a critical action part, it broke in the middle of the actor's sentence to a 6 minute ad block.
I exited playback and haven't gone back to finish it. I'll wait for it eventually to make it to a Blu-ray release someday.
> ut my impression still has always been that Netflix is a step above the rest technically.
I always assumed youtube was top dog for performance and stability. I can’t remember the last time I had issues with them and don’t they handle basically more traffic than any other video service?
Maybe a client issue, but i've got a low-end smart tv which handles netflix fine, but youtube is unwatchable due to buffering and failed cuts to adverts
I think Netflix will have even more sw engineers looking to work there once they notice even for average quality of work they can get paid 3 times more than their current pay.
Is it really that big a deal if you are watching a few minutes behind?
I've watched ball games on streaming networks where I can also hear a local radio broadcast, and the stream is always delayed compared to the radio, sometimes by quite a lot. But you'd never know it if you were just watching the stream.
The issue is that most people are trying to watch live which is what it's advertised as. And until they figure out that they need to watch X minutes behind, it is unwatchable. Many will not figure that out.
So for the first hour it was just total frustration until I stopped trying to go back to live mode.
Internet streams are not real-time even in the best case. There is always a few seconds of delay, often quite a bit more than that depending on number of hops and link speeds, congestion, etc.
I think why I will remember about this fight is not the (small) streaming issue I encountered as much as the poor quality of the fight itself. For me that was the reputational loss. Netflix was touting “NFL is coming to Netflix”. This fight did not really make me want to watch that.
I don't care about boxing or UFC or the grade-A douchebags that are the Paul brothers, but I tuned in just because I had the time and a Netflix subscription.
It was actually great that the fight itself was so boring because it justifies never having to spend time / money on that kind of bullshit. It was a farce. A very bright, loud, sparkly, and expensive (for some people) farce.
The value I got from it was the knowledge that missing out on that kind of thing isn't really missing out on anything at all.
Not really a joke, though? VOD has obvious methods to cheat a bit. Redundancy abounds and you can even network shape for costs. Could probably get even better compression for clear reasons.
Live, not so much. One source that you have to fanout from and absolutely no way to get cheap redundancy. Right?
Based on this I'm wondering whether it was straight up they did not expect it to be this popular?
> Some Cricket graphs of our #Netflix cache for the #PaulVsTyson fight. It has a 40 Gbps connection and it held steady almost 100% saturated the entire time.
So the issue is that Netflix gets its performance from colocating caches of movies in ISP datacenters, and a live broadcast doesn't work with that. It's not just about the sheer numbers of viewers, it's that a live model totally undermines their entire infrastructure advantage.
Correct, this is not Netflix’ regular cup of tea, and it’s a very different problem to solve. They can probably use their edge caches, but it’s challenging.
My wild assed guess is the differences in the edge nodes.
Netflix's edge nodes are optimized for streaming already encoded videos to end users. They have to transcode some number of formats from the source and send them all to the edge nodes to flow out. It's harder to manage a ton of different streams flowing out to the edge nodes cleanly.
I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients. Only one stream from source to edge to worry about and is much simpler to support and reason about.
> I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients.
Ha, no, our edge nodes don't have anywhere near enough spare CPU to do transcoding on the fly.
We have our own issues with livestreaming, but our system's developed differently over the past 15 years compared to Netflix's. While they've historically focused on intelligent pre-placement of data (which of course doesn't work for livestreaming), such an approach was never feasible for YT with the sheer size of our catalog (thanks to user-generated content).
Netflix is still new to the space, and there isn't a good substitute for real-world experience for understanding how your systems behave under wildly different traffic patterns. Give them some time.
It also helps that youtube serves shit tier quality videos more gracefully. Everyone is used to the step down to pixel-world on youtube to the point where they don’t complain much.
And decent part of these users are on free tier, so they are not paying for it. That alone gives you some level of forgiveness. At least I am not paying anything for this experience.
In my experience even YouTubeTV has problems sometimes. I'll have the 1080p (and enhanced mode also I think) quality set and still deal with a lot of compression artifacts.
Not sure how Netflix does it. But this is not very time sensitive, and I would have delayed the stream by 15 to 30 seconds to cache it and then deliver to everyone.
Yep. Having actually worked on this sort of stuff I can confirm.
Your ISP doesn't have enough bandwidth to the Internet (generally speaking) for all users to get their feed directly from a central location. And that central location doesn't have enough bandwidth to serve all users even if the ISP could. That said, the delay can be pretty small, e.g. the first user to hit the cache goes upstream, the others basically get the stream as it comes in to the cache. This doesn't make things worse, it makes them better.
I don't bet so I have no clue, but why is that? Are people able to place bets in the middle of the match or something? I would have assumed bets get locked in when the fight starts
This is kind of silly because the delay between actual event happening to showing up on OTA TV or cable TV to showing up on satellite TV can already be tens of seconds.
Or, hear me out here, it's a wild concept, just work.
You know, like every other broadcaster, streaming platform, and company that does live content has been able to do.
Acting like this is a novel, hard problem that needs to be solved and we need to "upsell" it in tiers because Netflix is incompetent and live broadcasting hasn't been around for 80+ years is so fucking stupid.
Likely these devices use different media formats and/or quality levels. And yes, it's possible one device buffers more than the other. Infinite freezes sounds like some routing issues or bugs.
When I was watching the behavior on the tv, was wondering if buffering sends some separate, non-business-as-usual requests, and that part of Netflix's delivery architecture was being overloaded.
E.g. "give me this previous chunk" vs "send me the current stream"
Buffering typically just consumes the same live stream until there's enough in the buffer. No difference other than the request rate being potentially higher. At least I can confidently say that for the standard players/video platforms. NetFlix could be doing something different. I'm not sure if they have their own protocols. But I'd be very surprised if the buffering used a completely different mechanism.
That model still works for streaming. You have a central source stream only to the distributed edge locations, then have clients only stream from their local edge location. Even if one region is overwhelmed, the rest can still work. Load on the central source is bounded.
Damn that sucks. I wonder if they could have intentionally streamed it 5 min late? I don’t have all the context around the fight though — maybe a competing service would win if Netflix intentionally induced delay?
they were introducing 5 minute delays on some of the clients. I noticed my ipad was always live and the smart tv had a 5 minute delay but you could fast forward to live.
I don't think that live doesn't work with caches. No one watching live would care about a O(s) delay, which is highly amenable to caching at ISPs and streaming changes from there to downstream clients. Offhand I'd say that would support O(ms) delay but no less.
Not sure I fully buy that. The “live” stream is rarely “live”. It’s often a highly cached buffer that’s a few mins from latest. Those in isp caches can still help here.
Streamed glitch free for me both on my phone and Xbox. The fight wasn’t so great though, but still a fun event. Jake Paul is a money machine right now.
Yea, it’s a bad look. But I switched to watching some other Netflix video and it seemed fine. Just this event had some early issues. Looks fine now though.
Most third-party internet-based streaming solutions are overlaid on top of a point-to-point network, while broadcast is one-to-many, and even cable tends to use multicast within the cable provider's network.
You have potentially different problems, e.g. limited bandwidth / spectrum. If, say, there are multiple games going on at the same time, you can only watch whichever feed the broadcaster decides to air. And, of course, regardless of the technology in use, there are matters of acquiring rights for various events. One benefit of internet-based streaming is that one service can acquire the rights and be able to reach everyone, whereas an individual cable provider might only reach its direct subscribers.
On cable(terrestrial is entirely different) even the bandwidth or spectrum is less limiting for broadcasting multiple games. Hard thing is the other parts of production, like cameras, live-directing and live commentary. Adding new channels is less challenging than actual producing content at expected level there.
Two games actually, both on Christmas Day. A day when most people are at home or the home of family or friends, and they are both pretty good late-season matchups (Chiefs-Steelers and Ravens-Texans) so I imagine viewership will be high.
If they botch the NFL games, it will surely hurt their reputation.
I don't think it'll be long-term. Most people will forget about this really quickly. It's not like there will be many people saying "Oh, you don't want to sign up for Netflix, the Tyson fight wasn't well streamed" in even 6 months nevermind 10 years.
Static files have been pretty much the standard streaming protocols for both VOD and live for the last 15 years ago. Before, it was Adobe Flash (RTMP).
With the way that they are designed, you can even use a regular CDN.
You can push these files to all the edges before you release the content which will protect your origin. Livestream all your edge servers are grabbing content from the origin unless you have another tier of regional servers to alleviate load.
Sure but that’s why your edge servers do request collapsing. And there are full blown CDN companies that will write an enterprise contract with you that can do this stuff with ease. Akamai is like 25 years old now.
Scale has increased but the techniques were figured out 20 years ago. There is not much left to invent in this space at the current moment so screwing up more than once is a bit unacceptable.
I don't think Netflix is even designed to handle very extreme multi-region live-streaming at scale as evidenced in this event with hundreds of millions simultaneously watching.
YouTube, Twitch, Amazon Prime, Hulu, etc have all demonstrated to stream simultaneously to hundreds of millions live without any issues. This was Netflix's chance to do this and they have largely failed at this.
There are no excuses or juniors to blame this time. Quite the inexperience from the 'senior' engineers at Netflix not being able to handle the scale of live-streaming which they may lose contracts for this given the downtime across the world over this high impact event.
Very embarrassing for a multi-billion dollar publicly traded company.
The assumption that it was related to insufficient investment isn’t supported by any evidence. Flawed technical decisions can be made by the most expensive engineers too.
Yeah, the funny part is that Hulu, Amazon Prime, and Peacock have all demonstrated the ability to handle an event of this caliber with no issue. Netflix now may never get another opportunity like this again.
It seemed to be some capacity issue with the CDNs. When I stopped and restarted the stream it worked again. Perhaps they do not use real time multi-cdn switching.
I know nothing about boxing and this fight was just ridiculously impressive. I kept tuning out of the earlier fights. They felt like some sort of filler. I didn’t get the allure. But Taylor v Serrano was just obvious talent that even I could appreciate it.
To be fair, a lot of people pay their ISP for a modem/router combo and connect to something like "Xfinity" at their house. So to them, there is no difference.
What do you think were the dynamics of the engineering team working on this?
I'd think this isn't too crazy to stress test. If you have 300 million users signed up then you're stress test should be 300 million simultaneous streams in HD for 4 hours. I just don't see how Netflix screws this up.
Maybe it was a management incompetence thing? Manager says something like "We only need to support 20 million simultaneous streams" and engineers implement to that spec even if the 20 million number is wildly incorrect.
There's no way 300 million people watched this, especially if that number is representing every Netflix subscriber. The largest claimed live broadcast across all platforms is last year's Super Bowl with 202 million unique viewers for at least part of it, but that includes CBS, Nickelodeon, and Univision, not just streaming audiences. Its average viewers for the whole game was 123 million, which is second all-time to the Apollo 11 moon landing.
FIFA claimed the 2022 World Cup final reached 1.5 billion people worldwide, but again that seems like it was mostly via broadcast television and cable.
As far as single stream, Disney's Hotstar claimed 59 million for last year's Cricket World Cup, and as far as the YT platform, the Chandrayaan-3 lunar landing hit 8 million.
100 million is a lot of streams, let alone 300. But also note that not every stream reaches a single individual.
And, as far as the 59 million concurrent streams in India, the bitrate was probably very low (I'd wager no more than 720p on average, possibly even 480p in many cases). It's again a very different problem across the board due to regional differences (such as spread of devices, quality of network, even behavioral differences).
The arrogant Netflix! They always brag about how technologically superior they are, and they can't handle a simple technological challenge! I didn't have a buffering issue, I had an error page - for hours! Yet, they kept advertising the boxing match to me! What a joke! If you can't stream it, don't advertise it to save face with people like me who don't care about boxing!
Every organization makes mistakes and every organization has outages. Netflix is not different. Instead, of bashing them because they are imperfect, you might want to ask what you can learn from this incident. What would you do if your service received more traffic than expected? How would you test your service so you can be confident it will stay up?
Also, I have never seen any Netflix employees who are arrogant or who think they are superior to other people. What I have seen is Netflix's engineering organization frequently describes the technical challenges they face and discusses how they solve them.
I think you’re oversimplifying it. Live event streaming is very different from movie streaming. All those edge cache servers become kinda useless and you start hitting peering bottlenecks.
Edge caches are not useless for live streaming. They're critical. The upstream from those caches has no way of handling each individual users. The stream needs to hit the edge cache and end users should be served from there.
A typical streaming architecture is multi-tiered caches, source->midtier->edge.
We don't know what happened but it's possible they ran out of capacity on their edge (or anywhere else).
Main event hasn’t even started yet. Traffic will probably 10x for that. They’re screwed. Should have picked something lower profile to get started with live streaming.
I thought Netflix’s biggest advantage was the quality/salary of its engineers.
I think that every time I wait for Paramount+ to restart after its gone black in picture on picture, and yet, I’n still on Paramount+ and not Netflix, so maybe that advantage isn’t real.
I think this was true at some point, but I’ve been disappointed in the quality of the OSS Netflix tools recently. I think before k8s and a plethora of other tools matured, they were way ahead of the curve.
I specifically found the Netflix suite for Spring very lacking, and found message oriented architectures on something like NATS a lot easier to work with.
Sigh, none of the competitors are much better. Disney, who has more than enough cash to throw at streaming, is a near constant hassle for us ( after 3 or more episodes it throws an inscrutable error on Playstation ). I would drop it, but this is the only remaining streaming service and wife is not willing to drop it ( I guess until 1 it is one error per one episode ).
Eh, the beef that I have is that I am already a paying customer. Why does it seem like I am getting subpar service in terms of delivery? I know it is a tired conversation on this forum, but corporations big and small do what they can do mess with experience to eke out few more cents from customers. It almost does not matter which industry one looks at, it is the same story; the only difference is typically how egregious things get[1].
It will never not annoy and amuse me that illegal options (presumably run by randoms in their spare time) are so much better than the offerings of big companies and their tech ‘talent’.
Illegal options also have lot less resources (revenue, service providers who are willing host/facilitate illegal activities, and so on) so it’s a fair comparison in my opinion.
> service providers who are willing host/facilitate illegal activities
At least for NFL pirate streams, it seems they tend to use "burner" tenants from Azure and AWS. Of course they get shut down, but how hard is it to spin up another one?
I have Netflix purchased legally with hard earned money. But because I had issues I looked for illegal streams, and they were bad, crashes, buffering.. you name it. So I went back to Netflix and watched it at 140p quality.
This twitter stream was the most reliable for me. Completely took Netflix out of the equation; just some dude at the event with his phone: https://x.com/i/broadcasts/1mrxmMRmXpQxy
I guarantee this is a management issue. Somebody needed to bear down at some point and put the resources into load testing. The engineers told them it probably won't be sufficient.
I assume this came down to some technical manager saying they didn't have the human and server resources for the project to work smoothly and a VP or something saying "well, just do the best you can.. surely it will be at least a little better than last time we tried something live, right?"
I think there should be a $20 million class action lawsuit, which should be settled as automatic refunds for everyone who streamed the fight. And two executives should get fired.
At least.. that's how it would be if there was any justice in the world. But we now know there isn't -- as evidenced by the fact that Jake Paul's head is still firmly attached to his body.
This is probably a naive question but very relevant to what we have here.
In a protocol where a oft-repeated request goes through multiple intermediaries, usually every intermediate will be able to cache the response for common queries (Eg: DNS).
In theory, ISPs would be able to do the same with the HTTP. Although I am not aware of anyone doing such (since it will rightfully raise concerns of privacy and tampering).
Now TLS (or other encryption) will break this abstraction. Every user, even if they request a live stream, receives a differently encrypted response.
But live stream of a popular boxing match has nothing to do with the "confidentiality" of encryption protocol, only integrity.
Do we have a protocol which allows downstream intermediates eg ISPs to cache content of the stream based on demand, while a digital signature / other attestation being still cryptographically verified by the client?
> But the real indicator of how much Sunday’s screw-up ends up hurting Netflix will be the success or failure of its next live program—and the next one, and the one after that, and so on. There’s no longer any room for error. Because, like the newly minted spouses of Love Is Blind, a streaming service can never stop working to justify its subscribers’ love. Now, Netflix has a lot of broken trust to rebuild.
From my experience, it works if your not watching it 'live'. But the moment I put my devices to 'live' it perma-breaks. 504 gateway timed out in web developer tools hitting my local CDN. probably works on some CDNs, doesnt on others. Probably works if your not 'live'
edit: literally a nginx gateway timed out screen if you view the response from the cdn... wow
I've been re-watching Silicon Valley the last few weeks and just watched the Nucleus live stream episode 2 days ago, pretty funny seeing it in real life.
They should have partnered with every major CDN and load balanced across all of them. It’s ironic how we used to be better at broadcasting live events way back in the day versus today.
The first round of the fight just finished, and the issues seem to be resolved, hopefully for good. All this to say what others have noted already, this experience does not evoke a lot of confidence in Netflix's live-streaming infrastructure.
I watched the event last night and didn't get any buffering issues, but I did notice frequent drop in video quality when watching the live feed. If I backed the video up a bit, the video quality suddenly went back up to 4k.
I had some technical experience with live video streaming over 15 years ago. It was a nightmare back then. I guess live video is still difficult in 2024. But congrats to Jake Paul and boxing fans. It was a great event. And breaking the internet just adds more hype for the next one.
I wonder how localized the issues were. I watched the Taylor/Seranno fight and the Paul/Tyson without issue and the picture quality was the best in every seen for live sports. Was blown away by how good it was. No where near what I’m getting with steaming NFL. This is what I want the future of live sports to look like. Though the commentary was so so.
I’m in the Pacific Northwest. I wonder if we got lucky on this or just some areas got unlucky.
This livestream broke the internet, no joke. youtube was barely loading and a bunch of other sites too. 130M is a conservative number given all the pirate streams.
Honestly you didn't miss much, every (real) boxing fan thought of this as a disgrace and a shame when announced. putting a 58 year old Tyson against a crackhead filled with steroids (Jake Paul) ? Either case it would have been a shame on Jake Paul for even getting in the ring with such an old boxer.
In boxing you are old by 32 or maybe 35 year old for heavy weight, and everything goes down very very fast.
Yes, it was utterly boring, but they made their money. I don't like either Paul brother, so I only watched in hopes shorter, much-older Tyson would make Jake look as foolish as he is.
Comments on forums do not provide that data. And if you want to extrapolate self-reports, it's obviously fine (to varying degrees) for the vast majority of people, but that's not the "issue."
These kind of reports are the equivalent of saying "I have power" when you're hundreds of miles away from where a hurricane landed. It's uninteresting, it's likely you have power, and it does literally nothing for people that do not have power.
It doesn't advance the topic anywhere. There are other places to report these issues directly and in aggregate with other people -- HN (and many other forums) are not that place.
For the same reason that pointing out the sun rose in the east today would be ridiculous but if it happened to rise in the west, or you perceived it to rise in the west, that would be worth sharing.
Being able to livestream a sporting event is the default now and has been for at least over a decade since HBO’s original effort to stream a Game of Thrones season opener failed because of the MSFT guy they hired, and they fixed it by handing it over to MLBAM.
Maybe that’s what Netflix should do. Buy Disney so they can get access to the MLBAM people (now Disney Streaming because they bought it from MLB).
I watched on an AppleTV and the stream was rock solid.
I don’t know if it’s still the case, but in the past some devices worked better than others during peak times because they used different bandwidth providers. This was the battle between Comcast and Cogent and Netflix.
Remember back in 2014 or so when Netflix users on Comcast were getting slow connections and buffering pauses? It didn't affect people who watched Netflix via Apple TV because Netflix served Apple TV users with a different network.
> In a little known, but public fact, anyone who is on Comcast and using Apple TV to stream Netflix wasn’t having quality problems. The reason for this is that Netflix is using Level 3 and Limelight to stream their content specifically to the Apple TV device. What this shows is that Netflix is the one that decides and controls how they get their content to each device and whether they do it via their own servers or a third party. Netflix decides which third party CDNs to use and when Netflix uses their own CDN, they decide whom to buy transit from, with what capacity, in what locations and how many connections they buy, from the transit provider. Netflix is the one in control of this, not Comcast or any ISP.
> I watched on an AppleTV and the stream was rock solid.
For me it was buffering and low resolution, on the current AppleTV model, hardwired, with a 1Gbps connection from AT&T. Some streaming devices may have handled whatever issues Netflix was having better than others, but this was clearly a bigger problem than just the streaming device.
A friend and I, in separate states, found that it wouldn’t stream from TVs, Roku, etc. but would stream from mobile. And for me, using a mobile hotspot to a laptop; though that implies checking IP address range instead of just user-agent, so that seems unlikely.
Anyway, I wouldn’t be surprised if they were prioritizing mobile traffic because it’s more forgiving of shitty bitrate.
One similar crash I remember very well was CNN on 9/11 - I tired to connect from France but is down the whole day.
Since then I am very used to it because our institutional web sites traditionally crash when there is a deadline (typically the taxes or school inscriptions).
As for that one, my son is studying in Europe (I am also in Europe), he called me desperate at 5 am or so to check if he is the only one with the problem (I am the 24/7 family support for anything plugged in). After having liberally insulted Netflix he realized he confirmed with his grandparents that he will be helping them at 10 :)
On a tangential note, the match totally looked fixed to me - Tyson was barely throwing any punches. I understand age is not on his side, but he looked plenty spry when he was ducking, weaving and dodging. It seemed to me he could have done better in terms of attacking as well.
I would argue Tyson has a shorter reach, Jake was whiffing a lot of superman punches, and all that does is waste energy. Jake might be able to throw punches, but he clearly wasn't interested in taking them. If they stood closer and slugged it out, the fight could have gone either way.
Yeah the biggest thing to me, and the commentators mentioned this as well, his legs looked REALLY wobbly.
All your attacking power comes from your legs and hips, so if his legs weren’t stable he didn’t have much attacking power.
I think he gave it everything he had in rounds 1 and 2. Unfortunately, I just don’t think it was ever going to be enough against a moderately trained 27 year old.
i thought they did DSA interviews at netflix what happened? I had to watch the fight on someone streaming to X from their phone at the event and it was better than watching on netflix..if you could watch at all. extremely embarrassing!
My theory is they've so heavily optimized for static content and distributing content on edge nodes that they were probably poorly setup for live-streaming.
This, I feel bad for their engineers who were told every bonus would be a matter of how low can they get the cost-per-GB of transferred data, leading to the glorious Netflix-in-a-box (https://openconnect.netflix.com/deploymentguide.pdf) and then management casually asks for a live show with a wildly different set of guarantee requirements.
I believe HN's algorithm tends to relatively downrank stories with a high comment-to-upvote ratio, because they are more often flamewars on divisive topics.
Another major algorithmic down-ranker is vote wars on comments.
If lots of people are upvoting and downvoting the same comments, that's treated as a signal the topic is contentious and people are likely to start behaving badly.
HN is very clear they prioritize good behavior as the long term goal, and they are as a result comfortable having contentious topics fall in the ranking even if everyone involved in the discussion feels the topic is important.
This wasn’t a “real” fight in the ring. It was clearly a hype/money fight only. The late 20 year old boxer has a massive following (or hate following) with younger age groups; and Mike Tyson brings the older age groups out. Mike has earned somewhat of a legendary status.
Leading up to the fight, there were many staged interactions meant to rile up the audience and generate hype and drive subscription revenue, and subsequently make ad spots a premium ($$$).
Unfortunately, American television/entertainment is so fake. I can’t get even be bothered to engage or invest time into it anymore.
I think they must be noticing the issues, because I've noticed they've been dropping the stream quality quite substantially... It's a clever trick, but kind of cheap to do so, because who wants to watch pixelated things?
To be brutally honest if it’s a choice between pixelated and constantly buffering, pixelated is way less bad. Constantly buffering is incredibly annoying during live sports. (but this doesn’t negate your main point which is that if people paid to watch they expect decent resolution)
Fastly says they do 6M ccv for superbowl (i'm actually surprised they let them do the entire thing and don't mix different CDNs) and I'm not sure they do encoding and manifest serving - they might just cache/deliver chunks. Do you really think tyson vs other guy was only 600k ccv? I'd be shocked if Netflix can't handle this.
You would think, but technology always finds a way to screw things up. Cox Communications has had ongoing issues with their video for weeks because of Juniper router upgrades and even the vendor can't fix it. They found this out AFTER they put it in production. Shit happens.
Yes, as I have said again and again on hacker news in different comments Netflix went overboard with their microservices and tried to position itself as a technological company when it's not. It has made everything more complex and that's why any Netflix tech blog is useless because it is not the way to build things correctly.
To understand how to do things correctly look at something like pornhub who handle more scale than Netflix without crying about it.
The other day I was having this discussion with somebody who was saying distributed counter logic is hard and I was telling them that you don't even need it if Netflix didn't go completely mental on the microservices and complexity.
What a massive blow to NFLX. They have been in the streaming game for years (survived COVID-19) and this silly exhibition match is what does them in?
I didn’t watch it live (boxing has lot its allure for me) but vicariously lived through it via social feed on Bluesky/Mastadon.
Billions of dollars at their disposal and they just can’t get it right. Probably laid off the highly paid engineers and teams that made their shit work.
It's not a "massive blow" at all. Consumers will only vaguely remember this in a month. Netflix got a lot of new signups and got to test out their streaming infrastructure to figure out what needs work.
The fight itself was lame which worked in their favor. No one really cared about not being able to see every second of the "action". It's not like it was an NBA game that came down to the last second.
Hopefully they fix it because they are hosting two Christmas NFL games this year and if you want to really piss people off you have buffering issues during NFL games lol.
It was so bad. So so bad. Like don’t use your customers as guinea pigs for live streaming. So lame. They need a new head of content delivery. You can’t charge customers like that and market a massive event and your tech is worse than what we had from live broadcast tv.
You're lucky you only had some buffering issues. This is got the case for many people, I don't know the percentage, but many people on reddit were complaining.
Does anyone have any thoughts besides "bad engineering" on what could've gone wrong? It seems like taking on a new endeavor like streaming an event that would possibly draw many hundreds of millions of viewers doesn't make sense. Is there any obvious way that this would just work, or is there obviously a huge mistake deeply rooted in the whole thing. Also, are there any educated guesses on some fine details in the codebase and patterns that could result in this?
Mine is glitchy, but if I refresh i get a good steam for a bit, then it gets low res, then freeze. If I wait for auto-reconnect it takes forever. Hard refresh and I'm good. Like, new streams to new server, then overloaded, then does as if their cluster is crashing and healing is rapid cycles. Sawtooth patterns on their charts.
And then all these sessions lag, or orphan taking up space, so many reconnections at various points in the stream.
System getting hammered. Can't wait for this writeup.
On X.com someone had a stream that was stable to at least 5 million simultaneous viewers, but then (as I expected) someone at Netflix got them to pull the plug on it. So I would expect this fight to have say, 50 million + watching? Maybe as many as 150-250 million worldwide, given this is Tyson's last fight.
Did anyone else see different behaviour with different clients? My TV failed on 25% loaded, my laptop loaded but played for a minute or two before getting stuck buffering, and my iphone played the whole fight fine. All on the same wifi network.
Woke up at 4am (EU here), to tune for the main event. Bought Netflix just for this. The women fight went good, no buffering, 4K.
As it approached the time for Paul vs Tyson, it started to first drop to 140p, and then constantly buffer. Restarted my chromecast a few times, tried from laptop, and finally caught a stream on my mobile phone via mobile network rather than my wifi.
The TV Netflix kept blaming my internet which kept coming back as “fast”.
Ended up watching the utterly disappointing, senior abuse, live stream on my mobile phone with 360p quality.
Gonna cancel Netflix and never pay for it it again, nor watch hyped up boxing matches.
I ended up turning my TV off and watching from my phone because of the buffering/freezing. The audio would continue to play and the screen would be frozen with a loading percentage that never changed.
I have Spectrum (600 Mbps) for ISP and Verizon for mobile.
There was some blog post on HN the other day where someone said they don't do chaos monkey anymore... Even then, how do you chaos test a novel event ahead of time?
From my limited understanding, the NFL heavily depends on the Netflix Open Connect platform to stream media to edge locations, which is different from live streaming. Probably, they over-pushed the HD contents.
I would have just made it simple, delay the live stream a few seconds and encode it into the same bucket where users already is playing static movies. Just have the player only allow start at the time everyone is at.
I'm watching the event as I'm writing this. I've been needing to exit the player and resume it constantly. Pretty surprising that Netflix hasn't weeded out these bugs.
I couldn’t watch a show a couple days ago. Long time customer, and first time I’ve considered cancelling. Broke the basic contract of I give $ and Netflix give show.
I did some VPN hopping and connecting to an endpoint in Dallas has allowed me to start watching again. Not live though, that throws me back into buffering hell.
Weird that an organization like Netflix is having problems with this considering their depth of both experience and pockets. I wonder if they didn't expect the number of people who were interested in finding out what the pay-per-view experience is like without spending any extra money. Still, I suppose we can all be thankful Netflix is getting to cut their live event teeth on "alleged rapist vs convicted rapist" instead of something more important.
Well he is almost 60 and the average retirement age for pro boxers is in the mid 30s. He is well past his prime and in physically demanding sport that is very hard on participants.
We all know netflix was built for static content, but its still hilarious that they have thousands of engineers making 500-1M in total comp and they couldnt live stream a basic broadcast. You probably could have just run this on AWS with a CDK configuration and quota increase from amazon
I am curious about their live streaming infrastructure.
I have done live streaming for around 100k concurrent users. I didn't setup infrastructure because it was CloudFront CDN.
Why it is hard for Netflix. They have already figured out CDN part. So it should not be a problem even if it is 1M or 100M. because their CDN infrastructure is already handling the load.
I have only work with HLS live streaming where playlist is constantly changing compared to VOD. Live video chunks work same as VOD. CloudFront also has a feature request collapsing that greatly help live streaming.
So, my question is if Netflix has already figured out CDN, why their live infrastructure failing?
Note: I am not saying my 100k is same scaling as their 100M. I am curious about which part is the bottleneck.
> Why it is hard for Netflix. They have already figured out CDN part. So it should not be a problem even if it is 1M or 100M. because their CDN infrastructure is already handling the load ... Note: I am not saying my 100k is same scaling as their 100M. I am curious about which part is the bottleneck.
100k concurrents is a completely different game compared to 10 million or 100 million. 100k concurrents might translate to 200Gbps globally for 1080p, whereas for that same quality, you might be talking 20T for 10 million streams. 100k concurrents is also a size such that you could theoretically handle it on a small single-digit number of servers, if not for latency.
> CloudFront also has a feature request collapsing that greatly help live streaming.
I don't know how much request coalescing Netflix does in practice (or how good their implementation is). They haven't needed it historically, since for SVOD, they could rely on cache preplacement off-peak. But for live, you essentially need a pull-through cache for the sake of origin offload. If you're not careful, your origin can be quickly overwhelmed. Or your backbone if you've historically relied too heavily on your caches' effectiveness, or likewise your peering for that same reason.
200Gbps is a small enough volume that you don't really need to provision for that explicitly; 20Tbps or 200Tbps may need months if not years of lead time to land the physical hardware augments, sign additional contracts for space and power, work with partners, etc.
OK, but the last time they tried a livestream (a reality show reunion), it also fell over. I suppose to their credit, my stream never outright died yesterday, it just went to potato quality.
If you're going to be having intense algorithm interviews, paying top dollar for only hiring senior engineers, building high intensity and extreme distributed systems and having SRE engineers, we best see insanely good results and a high ROI out of it.
All of the conditions was perfect for Netflix, and it seems that the platform entirely flopped.
Is this what chaos engineering is all about that Netflix was marketing heavily to engineers? Was the livestream supposed to go down as Netflix removed servers randomly?
I clicked on this thread to type that exact thing, holy smokes.
You're referring to Hooli's streaming of UFC fight that goes awry and Gavin Belson totally loses it, lol. Great scene and totally relevant to what's happening with Netflix rn.
After a few buffering timeouts during the first match, the rest of the event had no technical difficulties (in SoCal, so close to one of Netflix's HQs).
Unfortunately, except for the women's match, the fights were pretty lame...4 of the 6 male boxers were out of shape. Paul and Tyson were struggling to stay awake and if you were to tell me that Paul was just as old as Tyson I would have believed it.
Why do they want to get into the live business? It doesn't seem to synergize with their infrastructure. Sending the same stream in real time to numerous people just isn't the same task as letting people stream optimized artifacts that are prepositioned at the edge of the network.
Live is the only thing that won’t be commodified entirely. “Anyone” can pump out stream-when-you-want TV shows. Live events are generally exclusive, unpredictable, and cultural moments .
The marginal cost to add a viewer to broadcast sports is zero! That's what I am getting at. I know why someone would want this business, I just don't see what aspect of Netflix's existing business made them think they'd be good at it.
Most PPV is what, $50-$70? So subscribing to Netflix for $20 or whatever per month sounds like a bargain for anyone who is interested and not already a customer. Then assume some large percentage doesn’t cancel either because they forgot, or because they started watching a show and then decided to keep paying.
I thought it's only the best of the best of the best working at Netflix ... or maybe we can just put this myth to sleep that Netflix even knows what it's doing. The suggestions are shit, the UX is shit, apparently even the back end sucks.
I have to assume this is some snarky way of saying "violating copyright is Bad, m'kay".
Because taken at face value it's false. Any technical challenges involved in distributing a stream cannot possibly be affected by the legal status of the bits being pushed across the network.
All these engineering blog posts, distributed systems and these complex micro-services clearly didn't help with this issue.
Netflix is clearly not designed nor prepared for scalable multi-region live-streaming, no matter the amount of 'senior' engineers they throw at the problem.
> I'm an engineering manager at a Fortune 500 company. The dumbest engineer on our team left for Netflix. He got a pay raise too.
Apparently he was smart enough to get away from the Fortune 500 company he worked at, reporting to yourself, and "got a pay raise too."
> Our engineers are fucking morons. And this guy was the dumbest of the bunch.
See above.
> If you think Netflix hires top tier talent, you don't know Netflix.
Maybe you don't know the talent within your own organization. Which is entirely understandable given your proclamation:
Our engineers are fucking morons.
Then again, maybe this person who left your organization is accurately described as such, which really says more about the Fortune 500 company employing him and presumably continues to employ yourself.
IOW, either the guy left to get out from under an EM who says he is a "fucking moron" or he actually is a "fucking moron" and you failed as a manager to elevate his skills/performance to a satisfactory level.
White-Knighting for 'fucking morons' is not a good look though. You'll end up in a world where packets of peanuts have a label on saying 'may contain nuts'.
I think acting as if peanuts are actually nuts for purposes of communication is much more defensible than acting as if tomatoes are vegetables, in short you are dying on a hill that was paved over long ago.
I agree most people will conflate them, but someone who's allergic to peanuts but not tree nuts (or vice versa), i.e. the people the labels are intended for, are going to care about the difference.
Managers aren't teachers. They can spend some time mentoring and teaching but there's a limit to that. I've worked with someone who could not write good code and no manager could change that.
Most people I've worked with aren't like that of course (there's really only one that stands out), so maybe you've just been lucky enough to avoid them.
I do find it unlikely that all of his engineers are morons, but on the other hand I haven't worked for a typical fortune 500 company - maybe that's where all the mediocre programmers end up.
> or he actually is a "fucking moron" and you failed as a manager to elevate his skills/performance to a satisfactory level.
sometimes managers don't have the authority to fire somebody and are forced to keep their subordinates. Yes good managers can polish gold, but polishing poop still results in poop.
I was consulting at a place, there was a very bad programmer whose code looked sort of like this
cost arrayIneed = [];
const arrayIdontNeed = firstArray.map(item => {
if(item.hasProp) {
arrayIneed.push(item);
}
});
return arrayIneed;
the above is very much a cleaned up and elegant version of what he would actually push into the repo.
he left for a competitor in the same industry, this was at the second biggest company for the industry in Denmark and he left for the biggest company - presumably he got a pay raise.
I asked the manager after he was gone, one time when I was refactoring some code of his - which in the end just meant throwing it all out and rewriting from scratch - why he had been kept on so long, and the manager said there were some layoffs coming up and he would have been out with those but because of the way things worked it didn't make sense to let him go earlier.
> the manager said there were some layoffs coming up and he would have been out with those but because of the way things worked it didn't make sense to let him go earlier.
Incentives are fucked across the board right now.
Move on a low performer today and you'll have an uphill battle for a backfill at all. If you get one, many companies are "level-normalizing" (read: level-- for all backfills). Or perhaps your management thinks the job could be done overseas cheaper, or you get pushed to turn it into a set of tasks so you can farm it out to contractors.
So you keep at least some shitty devs to hold their positions, and as ballast to throw overboard when your bosses say "5% flat cut, give me your names". We all do it. If we get back to ZIRP I'll get rid of the actively bad devs when I won't risk losing their position entirely. Until then, it's all about squeezing what limited value they have out and keeping them away from anything important.
Hmm. Engineering managers should be setting the team culture and determining best criteria for extending an offer to a candidate. If theres a problem with the hiring process I'd look for the closest source that could or should be fixing it.
I hope to never have a manager who is mentally stack ranking me and my coworkers in terms of perceived dumbness instead of in terms of any positive trait.
Almost everyone I know manager or not is usually ranking everyone they work with on various attributes.
In fact it would be incredibly weird to ask a close friend who at their work kicks ass and who sucks and have them respond back, "I've never really thought about how good any of my coworkers were at their jobs"
dumbness is ranking intelligence, which is a positive trait, dumbness is just a metric for how often intelligence fails.
Example - the manager who started this sub-thread may be a pretty smart guy and able to accurately rate the intelligence of the engineers at his organization - but he had a minor momentary failing of intelligence to post on HN calling those engineers fucking morons.
You've got to rank how often the intelligence fails in someone to be able to figure out how reliable their intelligence is.
I'm not a manager and I don't stack rank people, but I am 100% capable of knowing when one of my co-workers or predecessors is a fucking moron.
The trick is to use my massive brain to root cause several significant outages, discover that most of them originate in code written by the same employee, and notice that said employee liked to write things like
// @ts-nocheck
// obligatory disabling of typescript: static typing is hard, so why bother with it?
async function upsertWidget() {
try {
// await api.doSomeOtherThings({ ... })
// 20 line block of commented-out useless code
// pretend this went on much longer
let result = await api.createWidget({ a, b, c })
if (!result.ok) {
result = await api.createWidget({ a, b }) // retries for days! except with different args, how fun
if (!result.ok) {
result = await api.updateWidget({ a, b, c }) // oh wait, this time we're updating
}
}
// notice that api.updateWidget() can fail silently
// also, the three function calls can each return different data, I sure am glad we disabled typescript
return result
} catch (error) {
return error // I sure do love this pattern of returning errors and then not checking whether the result was an error or the intended object
}
}
function doSomething() {
const widget = await upsertWidget()
}
...except even worse, because instead of createWidget the name was something far less descriptive, the nesting was deeper and involved loops, there were random assignments that made no goddamn sense, and the API calls just went to an unnecessary microservice that was only called from here and which literally just passed the data through to a third party with minor changes. Those minor changes resulted in an internal API that was actually worse than the unmodified third party API.
I am so tired of these people. I am not a 10x rockstar engineer and not without flaws, but they are just so awful and draining, and they never seem to get caught in time to stop them ruining perfectly fine companies. Every try>catch>return is like an icy cat hand from the grave reaching up to knock my coffee off my desk.
In this specific case, the fucking moron in question was the one who designed the code review process and hired the other engineers, and it took place a significant length of time before my involvement.
Which, yes, does raise interesting questions about how someone who can't be trusted to handle errors in an API call ended up in a senior enough position to do that.
There's a disincentive to actively block PRs if you don't want your coworkers to think you are a bad colleague / not on their side. So you often see suboptimal code making its way to production. This has a worse effect the more terrible engineers there are.
Except in this case it's clearly affecting at minimum the rest of OP's team.
At that point it's not one person being obnoxious and never approving their team members diffs and more of a team effort to do so.
But at minimum if you have a culture of trying to improve your codebase you'll inevitably set up tests, ci/cd with checks, etc. before any code can be deployed. Which should really take any weight of responsibility out of any one member of the team. Whether trying to put out code or reject said code.
I dunno, I've gone and done a "git blame" to find out who the fucking moron that wrote the code was, only to find out it was me three years ago.
Sure, there's such a thing as stupid code, but without knowing the whole context under which a bit of code was produced, unless it's utterly moronic, (which is entirely possible, dailywtf has some shiners), it's hard to really judge it. Hindsight, as applied to code, is 2020.
I agree with the general sentiment ("one instance of bad code might have been me") but not the specific sentiment ("I could easily decide to catch and ignore errors through every bit of code I worked on without knowing why that was bad, and commit other, similar crimes against good taste in the same way").
The difference for me is that this is pervasive. Yes, sometimes I might write code with a bug in error handling at 3am when I'm trying to bring a service back up, but I don't do it consistently across all the code that I touch.
I accept that the scope is hard to understand without having worked on a codebase which a genuine fucking moron has also touched. "Oh strken," you might say, "surely it can't be that bad." Yes, it can. I have never seen anything like this before. It's like the difference between a house that hasn't been cleaned in a week and a hoarder's house. If I tried to explain what hoarding is, well, maybe you'd reply that sometimes you don't do the dishes every day or mop the floor every week, and then I'd have to explain that the kitchen I'm talking about is filled from floor to roof with dirty dishes and discarded wrappers, including meat trays, and smells like a dead possum.
Hey, that possum's name was Roger and I'm really sad that it died. I've been feeding it for weeks! There are definitely bad programmers out there who's code is only suitable for public shaming via the daily wtf.
I've never seen a team that has somehow managed to hire exclusively morons. Even the shittiest of call center jobs and construction crews have a few people with enough brain cells to tie their shoelaces.
Have you considered that maybe you're being overly harsh about your co-workers? Maybe take the fact that one of them was hired by a top paying employer as a sign that you should improve your own ability to judge skill?
I've seen tons of them! The formula is to create conditions that will make even slightly competent people leave. They hire their morron nephew, he is always 30 min late then they moan when you are 5 min late because the parking lot was blocked by their car. He always leaves 2 hours early while you do overtime that they regularly forget to pay for. Your day is filled with your own work PLUS that of your retarded coworkers who only drink coffee while joking about you doing their work. You are not as fast as the last guy! haha! If something goes wrong the morrons collectively blame you, just like the last time. You get a formal warning. etc etc The other normal person they hire is let go after 2 days because they complaint which means they didnt fit the team.
And so on
If he still works there the morron who left was less of a.
Can’t speak for every place but that’s not always an option. As a teenager, I worked at Sports Direct where the management would regularly work us after our allotted hours and bar us putting the extra time onto our timesheet. If I recall correctly, the company eventually got pulled for it but the money they’d have saved over years would have outweighed the fine.
The timesheets were on paper so good luck putting your real hours on without your manager, who files it, finding out.
You’re asking children to have full understanding of their rights and how to enforce them. Also, investigations into this started in 2020: over a decade after I left. Do you think nobody had reported this in all that time? Looks like the system wasn’t working as well as you think it does.
I interviewed at Netflix a few years ago; with several of their engineers. One thing I cannot say is that they are morons.
their interview process is top notch too and while I was ultimately rejected, I used their format as the base template for how I started hiring at my company.
I don't have a dog in this fight, but you typically use your A players for hiring/interviews.
It can be both true that Netflix has God tier talent and a bunch of idiots. In fact, that's probably true of most places. I guess the ratio matters more.
This is every dev house I've worked at. For most people (mostly not the ones on HN), coding is a 9-5 job. No ambition. Just lines of code. Go home. I don't know there is anything particularly wrong with that.
You just have to accept most staff at any corporation are simply just average. There has to be an average for there to be better and worse.
It mostly makes sense to me. From their bombastic blogs to github projects full of overwrought Enterprise java design patterns. The only thing great about Netflix is it pays a lot more.
Having worked with a bunch of guys who have gone on to "top teams", I no longer believe they have top teams. My fav was the guy who said the system could scale indefinitely after it literally fell on its ass from too much traffic. He couldn't understand that just because Lambdas my themselves can scale, they are limited by the resources they use, so just ignored it and insisted that it could. The same guy also kept on saying we should change the TPEG standard because he didn't like how it worked. And these companies are seriously pretending they've got the best and brightest. If that's really true, I really need to find another profession.
I've worked for many companies that said they hired the best. And to be honest when I hire I also try to hire good people. I think I could hire better if a) I had an open cheque, b) I was running coolest project in the universe. I did hire for some interesting projects but nothing close to an open cheque. Even under these conditions it's tough to find great people. You can go after people with a proven track record but even that doesn't always guarantee their next project will be as successful.
The reality though is that large companies with thousands of people generally end up having average people. Some company may hire more PhD's. But on average those aren't better software engineers than non-PhD's. Some might hire people who are strong competitive coders, but that also on average isn't really that strong of a signal for strong engineers.
Once you have a mix of average people, on a curve, which is the norm, the question becomes do you have an environment where the better people can be successful. In many corporate environments this doesn't happen. Better engineers may have obstacles put in front of them or they can forced out of the organization. This is natural because for most organizations can be more of a political question than a technical question.
Smaller organizations, that are very successful (so can meet my two criterias) and can be highly selective or are highly desirable, can have better teams. By their nature as smaller organizations those teams can also be effective. As organizations grow the talent will spread out towards average and the politics/processes/debt/legacy will make those teams less effective.
To be fair, when you need to hire hundreds or thousands of people, you gotta hire average people. The best is a finite resource and not all of the best want to work for FAANG or any megacorp.
I used to want to work at a FAANG-like company when I was just starting out thinking they were going to be full of amazing devs. But over the years, I've seen some of the worst devs go to these companies so that just destroyed that illusion. And the more you hear about the sort of work they do, it just sounds boring compared to startups.
> Our engineers are fucking morons. And this guy was the dumbest of the bunch.
Very indicative of a toxic culture you seem to have been pulled in to and likely have contributed to by this point given your language and broad generalizations.
Describing a wide group of people you're also responsible for as "fucking morons" says more about you than them.
If your "dumbest engineer" got a job and a hefty raise going to Netflix, it means he was very capable engineer who was playing the part of moron at this Fortune 500 company because he was reporting to a manager who was calling him and the entire team morons and he didn't feel the need to go above and beyond for that manager.
Also, highly likely that it was the manager that was the moron and not everyone around him.
In terms of healthcare industry in Hungary: one worker does the same job for 700 USD a month and another for 1100 USD, the only difference is formal education and years worked in the industry. You can perform much better (by actually caring about the patients in those 12 hours you work) but you will get paid the same amount regardless. Of course if you have 3 kids (whether they are adults or not) then you do not pay taxes (or much less than someone who does not have kids or only has 2).
They obviously have some really good engineers, but many low-tier ones as well. No idea how long they stay there, though.
I'm watching the fight now and have experienced the buffering issues. Bit embarrassing for a company that fundamentally only does a single thing, which is this. Also, yeah, 900k TC and whatnot but irl you get this. Mediocre.
Or in other words: In one case the "stream" is stored on a harddrive not far away from you, only competing for bandwidth in the last section to you. In the other case the "stream" is comming over the Internet to you and everyone else at the same time.
I'm going to avoid leaving a zero-effort response like "actually you're the problem" like half of the replies and contribute:
Why do you call your engineers morons? Is it a lack of intelligence, a lack of wisdom, a lack of experience, inability to meet deadlines, reading comprehension, or something else?
I wonder if Netflix is just hiring for different criteria (e.g. you want people who will make thoughtful decisions while they want people who have memorized all the leetcode problems).
Because the idea that all the engineers that work at his large company are morons is absurd. Anyone in that situation that believes that and even more, states it, is just making their own character flaws apparent.
It’s hyperbole, like a teacher complaining to others, “my kids were all crazed animals today.”
I’ve worked with engineers where I had to wonder how they found their computer every morning. I can easily see how a few of those would make you bitter and angry.
this is why managers get a bad rap. what proportion think like this? hopefully not a large one but i do worry. ultimately if the team sucks its because of the management. theyre the ones with the greatest power to change things for the better.
What are the chances that your entire engineering team is entirely composed of low performers or people with bad attitude or whatever you designate as "fucking morons"?
It's more likely that you are bad at managing, growing and motivating your team.
Even if it was true, to refer to your team in this way makes you look like you are not ready for management.
Your duty is to get the most out of the team, and that mindset won't help you.
Don’t agree. Sometimes you can observe the world around you, and it’s not pretty. Are they not allowed to observe the truth as they see it? What if they are right?
The way to deal with this is to constantly do live events, and actually build organizational muscle. Not these massive one off events in an area the tech team has no experience in.
I have this argument a lot in tech.
We should always be doing (the thing we want to do)
Somme examples that always get me in trouble (or at least big heated conversations)
1. Always be building: It does not matter if code was not changed, or there has been no PRs or whatever, build it. Something in your org or infra has likely changed. My argument is "I would rather have a build failure on software that is already released, than software I need to release".
2. Always be releasing: As before it does not matter if nothing changed, push out a release. Stress the system and make it go through the motions. I can't tell you how many times I have seen things fail to deploy simply because they have not attempted to do so in some long period of time.
There are more just don't have time to go into them. The point is if "you did it, and need to do it again ever in the future, then you need to continuously do it"
Doing dry runs regularly makes sense, but whether actually shipping it makes sense seems context-dependent. It depends on how much you can minimize the side effects of shipping a release.
Consider publishing a new version of a library: you'd be bumping the version number all the time and invalidating caches, causing downstream rebuilds, for little reason. Or if clients are lazy about updating, any two clients would be unlikely to have the same version.
Or consider the case when shipping results in a software update: millions of customer client boxes wasting bandwidth downloading new releases and restarting for no reason.
Even for a web app, you are probably invalidating caches, resulting in slow page loads.
With enough work, you could probably minimize these side effects, so that releasing a new version that doesn't actually change anything is a non-event. But if you don't invalidate the caches, you're not really doing a full rebuild.
So it seems like there's a tension between doing more end-to-end testing and performance? Implementing a bunch of cache levels and then not using it seems counterproductive.
I like all of these considerations, although I also imagine for every context there is some frequency at which it is worthwhile to invalidate the caches to ensure that all parts of the system are still functioning as expected (including the rebuilding of the caches).
It's very hard to do a representative dry run when the most likely potential points of failure are highly load-dependent.
You can try and predict everything that'll happen in production, but if you have nothing to extrapolate from, e.g. because this is your very first large live event, the chances of getting that right are almost zero.
And you can't easily import that knowledge either, because your system might have very different points of failure than the ones external experts might be used to.
> 1. Always be building: It does not matter if code was not changed...
> 2. Always be releasing...
A good argument for this is security. Whatever libraries/dependencies you have, unpin the versions, and have good unit tests. Security vulnerabilities that are getting fixed upstream must be released. You cannot fix and remove those vulnerabilities unless you are doing regular releases. This in turn also implies having good unit tests, so you can do these builds and releases with a lower probability of releasing something broken. It also implies strong monitoring and metrics, so you can be the first to know when something breaks.
This is great, but what possible counterargument is there? We should prolong indefinitely a spooky ambiguity about whether the system works or not?
Easy: Short term risk versus long term risk. If I deploy with minimal changes today, I'm taking a non-zero short-term risk for zero short-term gain.
While I too am generally a long-term sort of engineer, it's important to understand that this is a valid argument on its own terms, so you don't try to counter it with just "piffle, that's stupid". It's not stupid. It can be shortsighted, it leads to a slippery slope where every day you make that decision it is harder to release next time, and there's a lot of corpses at the bottom of that slope, but it isn't stupid. Sometimes it is even correct, for instance, if the system's getting deprecated away anyhow why take any risk?
And there is some opportunity cost, too. No matter how slick the release, it isn't ever free. Even if it's all 100% automated it's still going to barf sometimes and require attention that not making a new release would not have. You could be doing something else with that time.
Finite compute, people, and opportunity cost.
It is just a reframing of build vs maintain.
In some environments, deploying to production has a massive bureaucracy tax. Paperwork, approvals, limited windows in time, can’t do them during normal business hours, etc.
Those taxes were often imposed because of past engineering errors. For example, Don't deploy during business hours because a past deployment took down production for a day.
A great engineering team will identify a tax they dislike and work to remove it. Using the same example, that means improving the success rate of deployments so you have the data (the success record) to take to leadership to change the policy and remove the tax.
The counterargument is obvious for anyone who has been on call or otherwise responsible for system stability. It's very easy to become risk-averse in any realm.
The common and flawed counterargument is “when we deploy, outages happen.” You’ll hear this constantly at companies with bad habits.
Deploying is expensive for some models. That could involve customer facing written release notes, etc. Sometimes the software has to be certified by a govt authority.
Additionally, refactor circle jerks are terrible for back-porting subsequent bug fixes that need to be cherry picked to stable branches.
A lot of of the world isn’t CD and constant releases are super expensive.
This is golden advice, honestly. "If you don't use it, you lose it" applied to software development.
that’s difficult to reproduce at scale; there are only so many “super bowl” events in a calendar year
> ...the tech team has no experience in
Unless Netflix eng decides to release a public postmorterm, we can only speculate. In my time organizing small-time live streams, we always had up to 3 parallel "backup" streams (Vimeo, Cloudflare, Livestream). At Netflix's scale, I doubt they could simply summon any of these providers in, but I guess Akamai / Cloudflare would have been up for it.
Agreed. This is a management failure, full stop. Unbelievable that they'd expect engineering to handle a single Livestream event of this magnitude.
They've been doing live events since 2023. But it's hard to be prepared for something that's never been done by anyone before — a superbowl scale event, entirely viewed over the internet. The superbowl gets to offload to cable and over the air. Interestingly, I didn't have any problems with my stream. So it sounds like the bandwidth problems might be localized, perhaps by data center or ISP.
Maybe they considered this event as a rehearsal for the upcoming NFL streams which I am guessing might have a wider audience
I had issues here and there but there was workarounds. Then, towards the end, the quality either auto negotiated or was forced down to accommodate the massive pull.
Yeah, I think people are incorrectly assuming that everyone had the same experience with the stream. I watched the whole thing and only had a few instances of buffering and quality degradation. Not more than 30 seconds total during the stream.
Even if it was only 30% of people had a problem that's still millions of unhappy users. Not great for a time sensitive event.
Also, from lurking in various threads on the topic Netflix's in app messages added to people's irritation by suggesting that they check their WiFi/internet was working. Presumably that's the default error message but perhaps that could have been adjusted in advance somehow.
Wow, building talent from within? I thought that went out of fashion. I think companies are too impatient to develop their employees.
The WWE is moving their programming to Netflix next year. If I were them, I'd be horrified at what I saw.
It's incomprehensible to me that Netflix, one of the most highly skilled engineering teams in the world - completely sh*t the bed last night and provided a nearly unwatchable experience that was not even in the same league as pre-internet live broadcast from 30 years ago.
My bet is that a technical manager told his executive (multiple times) that he needed more resources and engineering time to make live work properly, and they just told him to make do because they didn't want to spend the money.
It could come down to something as stupid as:
Executive: "we handled [on demand show ABCD] on day one, that was XX million"
Engineering: "live is really different"
Executive: (arguing about why it shouldn't be that different and should not need a lot of new infrastructure)
Engineering: (can't really argue with his boss about this anymore after having repeated the same conversation 3 or 4 times) -- tells the team: we are not getting new servers or time for a new project. We have to just make do with what we have. You guys are brilliant, I know you can do it!"
I thought Netflix engineers were the best and could even do mythical leetcode hards. What happened? Why are they paid half a million dollars a year?
Isn't this more of a management problem, trying to turn a not-livestream system into a livestreaming one?
In 2012 Youtube did the Red Bull stratos live stream with 8m concurrent users. We're 12 years later, Netflix fucked up.
To me the difference is that in 2012, you had companies focusing on delivering a quality product, whether it made money or not. Today, the economic environment has shifted a lot and companies are trying to increase profits while cutting costs. The result is inevitably a decline in quality. I'm sure that Netflix could deliver a flawless live stream to millions of viewers, but the question is can they do it while making a profit that Wall Street is happy with. Apparently not.
But is there a way that Netflix might have learned from all of Youtube's past mistakes?
The only reasonable way to scale something like this up is probably to... scale it up.
Sure, there are probably some generic lessons, but I bet that the pain points in Netflix's architecture (historically grown over more than a decade and optimized towards highly cacheable content) are very different from Youtube, which has ramped up live content gradually over as many years.
The funny thing is I was just reading something on HN like three days ago about how light years ahead Netflix tech was compared to other streaming providers. This is the first thing I thought of when I saw the reports that the fight was messing up.
The average quality of talent has gone way down compared to 2012 though.
E.g. the median engineer, excluding entry level/interns, at YouTube in 2012 was a literal genius at their niche or quite close to it.
Netflix simply can’t hire literal geniuses with mid six figure compensation packages in 2024 dollars anymore… though that may change with a more severe contraction.
Everyone here talking like this something unique netflix had to deal with. Hotstar live streamed india va Pakistan cricket match with zero issues with all time high live viewership ever in the history of live telecast. Why would viewers paying $20 month want to think about their technical issues, they dropped the ball pure and simple. Tech already exists for this, it’s been done before even by espn, nothing new here.
The Independent reports 35m viewers of that cricket match [0].
Rolling Stone reported 120m for Tyson and Paul on Netflix [1].
These are very different numbers. 120m is Super Bowl territory. Could Hotstar handle 3-4 of those cricket matches at the same time without issue?
[0] https://www.the-independent.com/sport/cricket/india-pakistan...
[1] https://www.rollingstone.com/culture/culture-news/jake-paul-...
The Games Awards claims to have 118 million livestreams [1], and went off without a hitch.
I watched it for the game trailers, actually shocked that it's also superbowl viewership territory.
https://variety.com/2023/digital/news/game-awards-2023-break...
"The festivities were streamed live from the Peacock Theater in L.A. across more than 30 platforms including YouTube, Twitch, Facebook, TikTok Live, X (Twitter), Steam, WeChat, Bilibili, Huya, DouYu, Xiaohongshu and Instagram Live"
India - Australia is the one of interest, scored cricket’s highest concurrent audience ever, 59 Million.
https://www.icc-cricket.com/news/biggest-cricket-world-cup-e...
Majority of superbowl viewers watch it on cable. Streaming gets fewer than 10M concurrents
Until it switches entirely to Netflix one day
Going by their failure this time around, that's not happening anytime soon.
But that's exactly the point: Netflix didn't do this in a vacuum, they did it within Netflix.
It might just have been easier to start from scratch, maybe using an external partner experienced in live streaming, but the chances of that decision happening in a tech-heavy company such as Netflix that seems to pride itself on being an industry leader are close to zero.
People just do not appreciate how many gotchas can pop up doing anything live. Sure, Netflix might have a great CDN that works great for their canned content and I could see how they might have assumed that's the hardest part.
Live has changed over the years from large satellite dishes beaming to a geosat and back down to the broadcast center($$$$$), to microwave to a more local broadcast center($$$$), to running dedicated fiber long haul back to a broadcast center($$$), to having a kit with multiple cell providers pushing a signal back to a broadcast center($$), to having a direct internet connection to a server accepting a live http stream($).
I'd be curious to know what their live plan was and what their redundant plan was.
You are making excuses for a multibillion dollar company that has been in this game for many years. Maybe the first to market in streaming.
This isn’t NFLX’s first rodeo in live streaming. Have seen a handful of events pop up in their apps.
There is no excuse. All of the resources and talent at their disposal, and they looked absolutely amateurish. Poor optics.
I would be amazed if they are able to secure another exclusive contract like this in the future.
Sorry for the off topic but what’s this thing that I only come across in Hacker News about referring to a company by their stock exchange name (APPL, MSFT, etc) outside of a stock context? It seems really weird to me.
As a counterpoint, I do this sometimes, not thinking at all about stocks but instead as a standardized abbreviation of sorts. Ms for example can mean tons of things from a title to multiple sclerosis to milliseconds. MSFT is clear and half the length.
In-group signaling for people who like playing or thinking about the stock market. Similar to how people who make travel a big part of their identity refer to cities by their airport code.
I just assume it's people who spend too much time thinking about the stock context.
I have always assumed that a focus on stock tickers is the natural result when your primary user base is a group of people hyper focused on “total compensation” and stock grants. The name hackernews is merely a playful reference to the history of the site. Like the name “Patriot Act.”
merriam-webster.com/dictionary/brevity
Writing APPL instead of Apple doesn’t get you any fewer keystrokes.
Technically it’s one fewer keystroke (and it’s AAPL).
It’s a lot fewer keystrokes for MS (Morgan Stanley), GS (Goldman Sachs) and MSFT (Microsoft) than it is for AAPL, but it’s a force of habit for some. Once you’re used to referring to firms by their ticker symbols, you do it all the time.
E.g. an ex trader friend still says “spot” instead of “point” when referring to decimal points, even if talking in other contexts like software versions.
also, the symbol for apple is AAPL.
Ugh. Similar (huge) pet peeve about people who say "n.B." instead of "note".
A company that readily admits it burns out SWRs and SREs in exchange for the big bucks.
Just what the fuck are these people doing?
If I were a major investor in them I'd be pissed.
Is multicast a thing on the commercial internet? Seems like that could help.
If commercial = public, then no - you can not use multicast for this. It is heavily used within some enterprise networks though like if you go to a gym with lots of TVs they are all likely on multicast
You’re talking about the contribution from the venue to the boardcast centre, increasingly not a full program but being mixed remotely.
That’s a very different area to transmission of live to end users.
This is the whole point of chaos engineering that was invented at Netflix, which tests the resiliency of these systems.
I guess we now know the limits of what "at scale" is for Netflix's live-streaming solution. They shouldn't be failing at scale on a huge stage like this.
I look forward to reading the post mortem about this.
Everyone keeps mentioning at scale. I seriously doubt this was an "at scale" problem. I have strong suspicion this was a failure at the origination point being able to push a stable signal. That is not an "at scale" issue, but a hubris of we can do better/cheaper than broadcasting standard practices
Perhaps it was, or perhaps it was not.
I was watching a pirated, live retransmission of the event on Twitch (in Portuguese), and there was zero buffering on my end.
I've tried to watch an old Seinfeld episode during this event. It was freezing every few minutes even at downgraded bitrate. A video that should be on my local CDN node.
I am not sure that it is an issue with the origination point. In fact I just thought it was my ISP because my daughter's boyfriend was watching and doing facetime with her and my video was dropping but his was not. I have 2gb fiber and we regularly stream five TVs without any issue, so it should not have been a bandwidth issue.
As counterpoint, I observed 2-3 drops in bitrate, but an otherwise fine experience. So the problem seems to have been in dissemination, not at the origin.
I highly doubt this. Netflix has a system of OCAs that are loaded with hard disks, are installed in ISP’s networks, and serve the majority of those ISP’s customers.
Given than many people had no problems with the stream, it is unlikely to have been an origin problem but more likely the mechanism to fanout quickly to OCAs. Normally latency to an OCA doesn’t matter when you’re replicating new catalogs in advance, but live streaming makes a bunch of code that previously “didn’t need to be fast” get promoted to the hot path.
If it was a problem at origin, why did it get better/worse as viewership fell/rose?
It is weird because this was a solved problem.
Every major network can broadcast the Super Bowl without issue.
And while Netflix claims it streamed to 280 million, that’s if every single subscriber viewed it.
Actual numbers put it in the 120 million range. Which is in line with the Super Bowl.
Maybe Netflix needs to ask CBS or ABC how to broadcast
Do you live stream the superbowl? Me and everyone I know watch it over antenna broadcast tv. I think it is easier to have millions of tvs catch airwaves vs millions of point to point https video streams.
If you watch it over cable, you're live streaming it. Let's face it, that's where the vast majority of viewers see it. Few people view OTA even if the quality is better.
Live sports do not broadcast the event directly to a streamer. They push it to their broadcast centers. It then gets distributed from there to whatever avenues it needs to go. Trying to push a live IP stream directly from the remote live venue rarely works as expected. That's precisely why the broadcasters/networks do not do it that way
> If you watch it over cable, you're live streaming it.
Those are multicast feeds.
> Trying to push a live IP stream directly from the remote live venue rarely works as expected.
In my experience it almost always works as expected. We have highly specialized codecs and equipment for this. The stream is actively managed with feedback from the receiver so parameters can be adjusted for best performance on the fly. Redundant connections and multiple backhauls are all handled automatically.
> That's precisely why the broadcasters/networks do not do it that way
We use fixed point links and satellite where possible because we own the whole pipe. It's less coordination and effort to setup and you can hit venues and remotes where fixed infrastructure is difficult or impossible to install.
Is cable video over IP now? Last time I looked (which was forever ago), even switched video was atsc with a bit of messaging for the cable box to ask what channel to tune to, and to keep the stream alive. TV over teleco systems seems to be highly multicast, so kind of similar, headend only has to send the content once, in a single bitrate.
Not really the same as an IP service live stream where the distribution point is sending out one copy per viewer and participating in bitrate adaptation.
AFAIK, Netflix hasn't publicly described how they do live events, but I think it's safe to assume they have some amount of onsite production that outputs the master feed for archiving and live transcoding for the different bitrate targets (that part may be onsite, or at a broadcast center or something cloudy), and then goes to a distribution network. I'd imagine their broadcast center/or onsite processing feeds to a limited number of highly connected nodes that feed to most of their CDN nodes; maybe more layers. And then clients stream from the CDN nodes. Nobody would stream an event like this direct from the event; you've got to have something to increase capacity.
> Is cable video over IP now?
Over the US and Canada it mostly is, though how advanced the transition is is very regional.
The plan is to drop both analog signal and digital (QAM) to reclaim the frequencies and use them for DOCSIS internet.
Newer set top boxes from Comcast (xfinity) runs over the internet connection (in a tagged VLAN on a private network, and they communicate over a hidden wifi).
> If you watch it over cable, you're live streaming it.
Which is probably done over the cableco's private network (not the public Internet) with a special VLAN used for television (as opposed to general web access). They're probably using multicast.
When Netflix started it was the first in the space and breaking ground which is how they became a "tech" company that happens to stream media however it has been 15 years and since than the cloud providers have basically build "netflix as a service". I suspect most of the big streamers are using that instead of building their own in house thing and going through all the growing pains netflix is.
You know they were commoditized when "Build Netflix" became a system-design interview question.
Solves differently though, right? Cable broadcasts are not the same as a streaming video over the internet, right?
> People just do not appreciate how many gotchas can pop up doing anything live.
Sure thing, but also, how much resources do you think Netflix threw on this event? If organizations like FOSSDEM and CCC can do live events (although with way smaller viewership) across the globe without major hiccups on (relatively) tiny budgets and smaller infrastructure overall, how could Netflix not?
The CCC video crew has its fair share of geeks from broadcasting corporations and studio houses. Their combined institutional knowledge about live events and streaming distribution is probably in the same ballpark as that of giant global TV networks.
They also have the benefit of having practiced their craft at the CCC events for more than a decade. Twice a year. (Their summer event is smaller but still fairly well known. Links to talks show up on HN every now and then.)
Funky anecdote: the video crew at Assembly have more broadcasting and live AV gear for their annual event than most medium-sized studios.
> Their combined institutional knowledge about live events and streaming distribution
Now if they could just get audio levels and compression figured out.
Scale changes everything, I don't think it's fair to shrug this off
This is true, but scale comes after production. Once you have the video encoded on a server with a stable connection the hard part is over. What netflix failed to do is spread the files to enough servers around the globe to handle the load. I'm surprised they were unable(?) to use their network of edge servers to handle the live stream. Just run the stream with a 10 second delay and in that time push the stream segments to the edge server
This right here is where I'd expect the failure to occur. This isn't Joey Beercan running OBS using their home internet connectivity.
This is a major broadcast. I'd expect a full on broadcast truck/trailer. If they were attempting to broadcast this with the ($) option directly to a server from onsite, then I would demand my money back. Broadcasting a live IP signal just falls on its face so many times it's only the cheap bastard option. Get the video signal as a video signal away from the live location to a facility with stable redundant networking.
This is the kind of thinking someone only familiar with computers/software/networking would think of rather than someone in broadcasting. It's nice to think about disrupting, but this is the kind of failure that disruptors never think about. Broadcasters have been there done that with ensuring live broadcasts don't go down because an internet connection wasn't able to keep up.
Lumen has their Vyvvyx product/service which uses fiber for broadcast television.
I’ve been using vyvx since it was called global crossing/genesis, it was fairly unique when it started, but point to point ip distributon of programs has been the norm for at least 15 years. Still have backup paths on major events on a different technology, you’d be surprised how common a dual failure on two paths can be. For example output from the euro football this summer my mai paths were on a couple of leased lines with -7, but still had a backup on some local internet into a different city just incase there was a meltdown of the main providers network (it’s happened before with ipath, automation is great until it isn’t)
> Once you have the video encoded on a server with a stable connection the hard part is over.
The hard part is over, and people new to the problem think they are almost done, but then the next part turns out to be 100x harder.
Lots of people can encode a video.
Last I checked, p2p solves a lot of the scaling issues.
Haven't Asian live sports been using p2p already two decades ago ?
(What is the biggest Peertube livestream so far ?)
Yeah, I agree with this, especially the everything part. Netflix isn't exactly a scrappy FOSS/hackers organization or similar.
> how much resources do you think Netflix threw on this event?
Based on the results, I hope it was a small team working 20% time on the idea. If you tell me they threw everything they had at it to this result, then that's even more embarrassing for them.
> If organizations like FOSSDEM and CCC can do live events (although with way smaller viewership) […]
Or, for that matter, Youtube (Live) and Twitch.
Netflix is good only on streaming ready made content, not live streaming, but;
1. Netflix is a 300B company, this isn't a resources issue.
2. This isn't the first time they have done live streaming at this scale either. They already have prior failure experience, you expect the 2nd time to be better, if not perfect.
3. There were plenty of time between first massive live streaming to second. Meaning plenty of time to learn and iterate.
The problem is that provisioning vast capacity for peak viewership is expensive and requires long-term commitment. Some providers won't give you more connectivity to their network unless you sign a 12 month deal where you prepay that.
Peak traffic is very expensive to run, because you're building capacity that will be empty/unsused when the event ends. Who'd pay for that? That's why it's tricky and that's why Akamai charges these insane prices for live streaming.
A "public" secret in that network layer is usually not redundant in your datacenter even if it's promised. To have redundant network you'd need to double your investment and it'll seat idle of at 50% max capacity. For 2hr downtime per year when you restart the high-capacity routers it's not cost efficient for most clients.
Then sign a contract with Akamai, who has been in business for 25 years? You outsource if you aren’t planning to do something very often.
There is no middle ground where you commit a mediocre amount of resources, end up with downtime and a mediocre experience, and then go “but we saved money.”
Well, they didn't want to spend the money or more likely their own technical team/boss promised that they can do it themselves.
They indeed have a great CDN network, but it's not very good for this particular type of traffic. May be they will know/fix/buy next time...
Yea, the issue here isn't just that they're having issues, it's that they're having the same issues they've had before.
They have the NFL next month on Christmas day. So that'll be a big streaming session but I think it'll be nothing compared to this. Even Twitter was having problems handling the live pirate streams there.
> Even Twitter was having problems
Is that a surprise? They're not who I would think of first as a gold standard for high viewership live streams.
Well, considering it was multiple small streams I would expect them to keep them up. No have their entire streaming service have issues.
You can't solve your way out of a complex problem that you have created and which wasn't needed in the first place. The entire microservices thing was overly complex with zero benefits
I spoke to multiple Netflix senior technicians about this.
They said that's the whole shtick.
That's a ridiculous statement. PrimeVideo is the leader in terms of sports events streaming over internet and it is composed of hundreds of microservices.
Live streaming is just much harder than streaming, and it takes a years of work and a huge headcount to get something good.
Prime famously undid some amount of their microservices recently because it couldn’t keep up, and was hideously expensive.
This comment shows how a very random blog about a very small part of a product can dominate all conversation about it. Prime video famously did not undo anything. Out of 100+ teams one team undid one service. But somehow similar comments are common on HN. I am making no judgement or microservice or not just on this particular comment.
It was a single team for a very specific use case.
To be clear when I said that PrimeVideo is composed of hundreds of microservices, I actually meant that it's composed of hundreds of services, themselves composed, more often than not, of multiple microservices.
Depending on your definition of a microservice, my team alone owns dozens.
Yeah didn't they crash on love is blind or one of their reality shows recently-ish?
It wasn't even just buffering issues, the feed would just stop and never start again until I paused it and then clicked "watch live" with the remote.
It was really bad. My Dad has always been a fan of boxing so I came over to watch the whole thing with him.
He has his giant inflatable screen and a projector that we hooked up in the front lawn to watch it, But everything kept buffering. We figured it was the Wi-Fi so he packed everything up and went inside only to find the same thing happening on ethernet.
He was really looking forward to watching it on the projector and Netflix disappointed him.
> My Dad has always been a fan of boxing
What did your Dad think about the 'boxing'?
Commercial boxing has always been like WWE or MMA with a thin veneer of actual sport to it, i.e. it is just entertainment[1].
To rephrase your question then what does someone think of the entertainment on display?
I don't think it was good entertainment.
None of the hallmarks of a good show was present. i.e. It wasn't close, nor was it bloody or anything unexpected like say a KO everything went pretty much as expected. It wasn't nice watch as all,no skill or talent was on disply, all Paul had to do was use his speed to backpedal from the slow weak punches of a visibly older tyson with a bum knee and land some points occasionally to win.
-- [1] There is a deeper argument here is any spectator sports just entertainment or is truly about skill talent and competition. Boxing however including the ones promoted by traditional four major associations falls clearly on the entertainment side than say another sport like NFL to me.
Reading the comments here, I think one thing that's overlooked is that Netflix, which has been on the vanguard of web-tech and has solved many complicated problems in-house, may not have had the culture to internally admit that they needed outside help to tackle this problem.
A combination of hubris and groupthink.
Not invented here syndrome works at first but as time progresses the internally built tools become a liability
It’s insane the excuses being made here for Netflix’s apparently unique circumstances.
They failed. Full stop. There is no valid technical reason they couldn’t have had a smooth experience. There are numerous people with experience building these systems they could have hired and listened to. It isn’t a novel problem.
Here are the other companies that are peers that livestream just fine, ignoring traditional broadcasters:
- Google (YouTube live), millions of concurrent viewers
- Amazon (Thursday Night Football, Twitch), millions of concurrent viewers
- Apple (MLS)
NBC live streamed the Olympics in the US for tens of millions.
As a cofounder of a CDN company that pushed a lot of traffic, the problem with live streaming is that you need to propagate peak viewership trough a loooot of different providers. The peering/connectivity deals are usually not structured for peak capacity that is many times over the normal 95th percentile. You can provision more connectivity, but you don't know how many will want to see the event. Also, live events can be trickier than stored files, because you can't offload to the edges beforehand to warm up the caches.
So Netflix had 2 factors outside of their control
- unknown viewership
- unknown peak capacities outside their own networks
Both are solvable, but if you serve "saved" content you optimize for different use case than live streaming.
I don't disagree that Netflix could have / should have done better. But everybody screws these things up. Even broadcast TV screws these things up.
Live events are difficult.
I'll also add on, that the other things you've listed are generally multiple simultaneous events; when 100M people are watching the same thing at the same time, they all need a lot more bitrate at the same time when there's a smoke effect as Tyson is walking into the ring; so it gets mushy for everyone. IMHO, someone on the event production staff should have an eye for what effects won't compress well and try to steer away from those, but that might not be realistic.
I did get an audio dropout at that point that didn't self correct, which is definitely a should have done better.
I also had a couple of frames of block color content here and there in the penultimate bout. I've seen this kind of stuff on lots of hockey broadcasts (streams or ota), and I wish it wouldn't happen... I didn't notice anything like that in the main event though.
Experience would likely be worse if there were significant bandwidth constraints between Netflix and your player, of course. I'd love to see a report from Netflix about what they noticed / what they did to try to avoid those, but there's a lot outside Netflix's control there.
The examples given here are not on the same scale. The numbers known so far:
- 120m viewers [1]
- Entire Netflix CDN Traffic grew 4x when the live stream started [2]
[1] https://www.rollingstone.com/culture/culture-news/jake-paul-...
[2] https://x.com/DougMadory/status/1857634875257294866
Disney HotStar managed to stream ~60M livestreams for the Cricket world cup a year ago. The problem has been solved. Livestreaming sports just have a different QoS expectations than on demand.
https://www.geeksforgeeks.org/how-did-hotstar-managed-5-9-cr...
I wouldn't say it's a solved problem, how many other companies are pulling off those numbers? Isn't that the current record for concurrent streams? And wasn't it mostly to mobile devices?
Size of Hotstar team = ~2000. Enggrs will be less than that number.
https://www.linkedin.com/company/disney-hotstar/people/
The size of engineering head count is not informative, it really depends on how much is in-house and how much is external for Hotstar that would be i.e parent Disney or before Fox or staffing from IT consulting organizations who will not be on payroll.
For what it is worth, all things being equal there would be lot more non engineering in Hotstar for 2000 employees versus a streaming company of similar size or scale of users. Hotstar operates in challenging and fragmented market, India has 10+ major languages(and corresponding TV, music and movie markets) Technically there is not much difference to what Netflix or Disney has to do for i18n, however operationally each market needs separate sales, distribution and operations.
---
P.S. Yes Netflix operates in more markets including India than anybody else, however if you are actually using Netflix for almost any non English content, you will know how weak their library and depth in other markets are, their usual model in most of these markets is to have few big high quality(for that market) content rather than build depth.
P.P.S. Also yes, Indian market is seeing consolidation in the sense that many releases on streaming are multiple lingual and use major stars from more than one language to draw talent ( not new, but growing in popularity as distribution becomes cheaper with streaming), however this is only seen in big banner productions as tastes are quite different in each market and can't scale for all run of the mill content.
Disney Streaming has 900 employees, a large majority of which are engineers.
This is the company that supplies technology to Hotstar, Hulu, MLB Live streaming, etc.
https://en.m.wikipedia.org/wiki/Disney_Streaming
Hotstar is a completely different company.
I guess one question I have is did Netflix partner with other CDNs?
Despite their already huge presence, Amazon for example has multiple CDNs involved for capacity for live events. Same for Peacock.
Netflix has 280m subscriber highly doubt half of them tuned in to watch the match, is that 130m figure official?
Amazon had their fair share of livestream failures and for notably less viewers. I don't think they deserve a spot on that list. I briefly worked in streaming media for sports and while it's not a novel problem, there are so many moving parts and points of failure that it can easily all go badly.
There is no one "Amazon" here, there are at least 3:
* Twitch: Essentially invented live streaming. Fantastic.
* Amazon Interactive Video Service [0]: Essentially "Twitch As A Service", built by Twitch engineers. Fantastic.
* Prime Video. Same exact situation as Netflix: original expertise is all in static content. Lots of growing pains with live video and poor reports. But they've figured it out: now there are regular live streams (NHL and NFL), and other channel providers do live streaming on Prime Video as a distribution platform.
[0] https://aws.amazon.com/ivs/
Doesn't twitch almost fall over (other non-massive streams impacted) when anyone gets close to 4-5m concurrent viewers? I remember last time it happened everything started falling over, even for smaller streams. Even if Netflix struggled with the event, streaming other content worked just fine for me.
IVS does not scale past 1080p: https://ivs.rocks/
> They failed. Full stop.
It's not full stop. There are reasons why they failed, and for many it's useful and entertaining to dissect them. This is not "making excuses" and does not get in the way of you, apparently, prioritizing making a moral judgment.
The big difference of all the examples you’ve mentioned is dedicated full-time crews on the ground where the events are produced.
I’m pretty confident that when the post mortem is done the issues are going to be way closer to the broadcast truck than the user.
it could be that they made use of the same advice X followed :)
On a few forum sites I'm on, people are just giving up. Looking forward to the post-mortem on how they weren't ready for this (with just a tiny bit of schadenfreude because they've interviewed and rejected me twice).
AB84 streamed it live from a box at the arena to ~5M viewers on Twitter. I was watching it on Netflix, I didn't have any problems, but I also put his live stream up for the hell of it. He didn't have any issues that I saw.
> He didn't have any issues that I saw.
He’s definitely got issues..
were calling antonio brown ab84 now? What happened to Mr. BC?
They sabotaging OP just for a reverse schadenfreude play
Can you share which forums
/r/netflix and /sp/
Chiefsplanet.com, unstuckpolitics.com, my buddies on group text :)
The post-mortem will be interesting indeed.
It’s not everyone. Works fine for me though I did have to reload the page when I skipped past the woman match to the Barrios Ramos fight and it was stuck buffering at 99%.
You skipped the best part.
Cable TV (or even OTA antenna in the right service area) is simply a superior live product compared to anything streaming.
The Masters app is the only thing that comes close imo.
Cable TV + DVR + high speed internet for torrenting is still an unmatched entertainment setup. Streaming landscape is a mess.
It's too bad the cable companies abused their position and lost any market goodwill. Copper connection direct to every home in America is a huge advantage to have fumbled.
The interesting thing is that a lot of TV infrastructure is now running over IP networks. If I were to order a TV connection for my home I'd get an IPTV box to connect to my broadband router via Ethernet, and it'd simply tell the upstream router to send a copy of a multicast stream my way.
Reliable and redundant multicast streaming is pretty much a solved problem, but it does require everyone along the way to participate. Not a problem if you're an ISP offering TV, definitely a problem if you're Netflix trying to convince every single provider to set it up for some one-off boxing match.
The Masters app is truly incredible, I don't know if it gets enough praise.
What's so great about it?
When you step back and look at the situation, it's not hard to see why Netflix dropped the ball here. Here's now I see it (not affiliated with Netflix, pure speculation):
- Months ago, the "higher ups" at Netflix struck a deal to stream the fight on Netflix. The exec that signed the deal was probably over the moon because it would get Netflix into a brand new space and bring in large audience numbers. Along the way the individuals were probably told that Netflix doesn't do livestreaming but they ignored it and assumed their talented Engineers could pull it off.
- Once the deal was signed then it became the Engineer's problem. They now had to figure out how to shift their infrastructure to a whole new set of assumptions around live events that you don't really have to think about when streaming static content.
- Engineering probably did their absolute best to pull this off but they had two main disadvantages, first off they don't have any of the institutional knowledge about live streaming and they don't really know how to predict demand for something like this. In the end they probably beefed up livestreaming as much as they could but still didn't go far enough because again, no one there really knows how something like this will pan out.
- Evening started off fine but crap hit the fan later in the show as more people tuned in for the main card. Engineering probably did their best to mitigate this but again, since they don't have the institutional knowledge of live events, they were shooting in the dark hoping their fixes would stick.
Yes Netflix as a whole screwed this one up but I'm tempted to give them more grace than usual here. First off the deal that they struck was probably one they couldn't ignore and as for Engineering, I think those guys did the freaking best they could given their situation and lack of institutional knowledge. This is just a classic case of biting off more than one can chew, even if you're an SV heavyweight.
This isn't Netflix's first foray into livestreaming. They tried a livestream last year for a reunion episode of one of their reality TV shows which encountered similar issues [0]. Netflix already has a contract to livestream a football event on Christmas, so it'll be interesting to see if their engineers are able to get anything done in a little over a month.
These failures reflect very poorly on Netflix leadership. But we all know that leadership is never held accountable for their failures. Whoever is responsible for this should at least come forward and put out an apology while owning up to their mistakes.
[0] https://time.com/6272470/love-is-blind-live-reunion-netflix/
> But we all know that leadership is never held accountable for their failures.
You've never heard of a CEO or other C-suite or VP getting fired?
It most definitely happens. On the other hand, people at every level make mistakes, and it's preferable that they learn from them rather than be fired, if at all possible.
Accountability can take many forms. I don't think they should be fired for making a mistake, I think they should release a statement recognizing their failure along with a post-mortem. Not a particularly high bar, but most leadership failures are often swept under the rug without any public accountability or evidence that they've learned anything.
We have evidence of prior failures with livestreaming from Netflix. Were the same people responsible for that failure or do we have evidence of them having learned anything between events? If anything, I'd expect the best leaders would have a track record that includes failures while showcasing their ability to overcome and learn from those mistakes. But based on what information is publicly available, this doesn't seem to be the case in this situation.
> They now had to figure out how to shift their infrastructure to a whole new set of assumptions around live events
It wasn't their first live event. A previous live event had similar issues.
Livestreaming is a solved problem. This sounds like NIH [1]. (At the very least, hire them as a back-up.)
[1] https://en.wikipedia.org/wiki/Not_invented_here
"Solved" merely means you don't need to invent something new to solve it. It doesn't mean trivial nor easy. And it definitely doesn't mean the problem is above trade-offs.
Saying live-streaming is a solved problem is like saying search is a solved problem.
> Saying live-streaming is a solved problem is like saying search is a solved problem
It is. You can hire the people who have solved it to do it for you.
> It is. You can hire the people who have solved it to do it for you.
"GPGPU compute is a solved problem if you buy Nvidia hardware" type comment
> "GPGPU compute is a solved problem if you buy Nvidia hardware" type comment
You're replacing the word hire with buy. That misconstrues the comment. If you need to do GPGPU compute and have never done it, you work with a team that has. (And if you want to build it in house, you scale to it.)
>"GPGPU compute is a solved problem if you buy Nvidia hardware"
Which is valid? If your problem can be solved by writing a check, then it's the easiest problem to have on the planet.
Netflix didn't have to put out 3 PhD dissertations on how to improve the SOTA of live streaming, they only needed to reliably broadcast a fight for a couple hours.
That is a solved problem.
Amazon and Cloudflare do that for you as a service(!). Twitch and YouTube do it literally every day. Even X started doing it recently so.
No excuses for Netflix, tbh.
Landing on Mars is a solved problem. Nuclear bombs are a solved problem. Doesn't mean anyone can just write a check and get it done and definitely doesn't mean any business model can bear that cost.
Of course it means that!
You only need a big enough check.
No it doesn't.
India has landed on Mars for a fraction of the cost it took other nations, and the ESA has never been able to pull it off.
Not every cost is fungible and money isn't always the limiting factor.
Look. I'm a small startup employee. I have a teeny tiny perspective here. But frankly speaking the idea that Netflix could just take some off the shelf widget and stuff it in their network to solve a problem... It's an absurd statement for even me. And if there's anyone it should apply to it would be a little startup company that needs to focus on their core area.
Every off the shelf component on the market needs institutional knowledge to implement, operate, and maintain it. Even Apple's "it just works" mantra is pretty laughable in the cold light of day. Very rarely in my experience do you ever get to just benefit from someone else's hard work in production without having an idea how properly implement, operate, and maintain it.
And that's at my little tiny ant scale. To call the problem of streaming "solved" for Netflix... Given the guess of the context from the GP post?
I just don't think this perspective is realistic at all.
> the idea that Netflix could just take some off the shelf widget and stuff it in their network to solve a problem
Right. They have to hire one of the companies that does this. Each of YouTube, Twitch (Amazon), Facebook and TikTok have, I believe, handled 10+ million streams. The last two don't compete with Netflix.
I believe this is the spirit of the "solved problem" comment: not that the solution is an off-the-shelf widget, but that if it has ever been solved, then that solution could technically be used again, even if organizing the right people is exorbitantly expensive.
Offering it for sale != having solved it.
There are multiple companies that offer this capability today that would take a few weeks to hide behind company branding. This was a problem of netflix just not being set up for live stream but thinking they could handle it.
>First off the deal that they struck was probably one they couldn't ignore
If you can't provide the service you shouldn't sell it?
There are endless amounts of stories and situations in which selling something before it really exists has helped businesses. It's totally plausible that a team working on video streaming at the scale of Netflix could figure out live streaming.
Pre-optimization is definitely a thing and it can massively hurt (i.e. startups go under) businesses. Let's stop pretending any businesses would say 'no' to extra revenue even before the engineering team had full assurance there was no latency drop.
My speculation here is this was just classic SV cockiness. The team that closed this deal probably knew that they didn't have the capability but I'm sure the arguments for doing it anyways was something along the lines of: "we have the best engineers in the bay area, we can probably figure this out"
Execs never listen or even ask engineers about feasibility of projects they sign up to. Hope the exec in question will be let go.
I mean, the ones that do ask don’t proceed to signing up. I think we are seeing a form of survival bias.
You've never worked in a startup have you? Or any business for that matter. You have to promise something first, then build it.
i imagine this is why a lot of products, and startups, fail.
No joke, is this actually true?
Do startups really do this? I thought the capability is built or nearly built or at least in testing already with reasonable or amazing results, THEN they go to market?
Do startups go to other startups, fortune 500 companies and public companies to make false promises with or without due diligence and sign deals with the knowledge that the team and engineers know the product doesn't have the feature in place at all?
In other words:
Company A: "We can provide web scale live streaming service around the world to 10 billion humans across the planet, even the bots will be watching."
Company B: "OK, sounds good, Yes, here is a $2B contract."
Company A: "Now team I know we don't have the capability, but how do we build, test and ship this in under 6 months???"
Startups absolutely sell things they haven't made yet and might not even be capable of doing.
Next thing you know it's 9pm on a Sunday night and your desperately trying to ship a build for a client.
Netflix isn't some scrappy company though. If I had to guess they threw money at the problem.
A much better approach would of been to slowly scale over the course of a year. Maybe stream some college basketball games first, slowly picking more popular events to get some real prod experience.
Instead this is like their 3rd or 4th live stream ever. Even a pre show a week before would of allowed for greater testing.
I'm not a CTO of a billion dollar company though. I'm just an IC who's seen a few sites go down underload.
To be fair no one knows how it's going to go before it happens. It would of been more surprising for them to pull this off without issues... It's a matter of managing those issues. I know if I had paid 30$ for a Netflix subscription to watch this specific event I'd assume I got ripped off.
You don't necessarily have to make false promises.
You can be totally honest and upfront that the functionality doesn't exist yet and needs to be built first, but that you think you understand the problem space and can handle the engineering, provided you can secure the necessary funding, where, by the way, getting a contract and some nominal revenue now could greatly help make this a reality...
And if the upside sounds convincing enough, a potential customer might happily sign up to cover part of your costs so they can be beta testers and observe and influence ongoing development.
Of course it happens all the time that the problem space turns out to be more difficult than expected, in which case they might terminate the partnership early and then the whole thing collapses from lack of funding.
If anything, startups are more transparent about it.
In the enterprise sector this is rampant. Companies sell "platforms" and those missing features are supposed to be implemented by consultants after the sale. This means the buyer is the one footing the bill for the time spent, and suffering with the delays.
“Aspirational sugar” is as common in startup culture as in Fortune 500 sales contracts, they’re just messaged and “de-risked” differently.
Many do, as far as initial investment goes. It makes sense when you think about the capital intensive nature of most startups (including more than web startups here, e.g. lab tech commercialization). It also accurately describes a research grant.
That's for startups that can't bootstrap (most of them). For ones which can, they may still choose to do this with customers, as you describe, because it means letting their work follow the money.
>> First off the deal that they struck was probably one they couldn't ignore
> If you can't provide the service you shouldn't sell it?
Then how will the folks in Sales get their commission?
Besides, not providing the service hasn't stopped Tesla from selling FSD, and their stock has been going gangbusters.
/s
Not sure why Netflix is held in high regard - this proves they're just as much clowns as the other 'big players' in the circus.
They arent clowns at all. Ita a totally different engineering problem and you cant just spin up live streaming capacity on demand. The entire system end to end isnt optimized for live streams yet.
I mean, maybe? You just made all this up.
I wonder if there will be any long term reputational repercussions for Netflix because of this. Amongst SWEs, Netflix is known for hiring the best people and their streaming service normally seems very solid. Other streaming services have definitely caught up a bit and are much more reliable then in the early days, but my impression still has always been that Netflix is a step above the rest technically.
This sure doesn't help with that impression, and it hasn't just been a momentary glitch but hours of instability. And the Netflix status page saying "Netflix is up! We are not currently experiencing an interruption to our streaming service." doesn't help either...
Not the same demographic but their last large attempt at live was through a Love is blind reunion. It was the same thing, millions of people logging in, epic failure, nothing worked.
They never tried to do a live reunion again. I suppose they should have to get the experience. Because they are hitting the same problems with a much bigger stake event.
yup wanted to say that live stream stuttering has happened before on Netflix - I don't think the reputation is deserved.
From a livestreaming standpoint, netflix is 0/x - for many large events such as love is blind, etc.
From a livestreaming standpoint, look to broadcast news, sports / Olympics broadcasters, etc and you'll see technology, equipment, bandwidth, planning, and professionalism at 1000x of netflix.
Heck, for publicly traded quarterly earnings livestream meetings, they book direct satellite time in addition to fiber to make sure they don't rely only on terrestrial networks which can fail. From a business standpoint, failure during a quarterly meeting stream can mean the destruction of a company (by making shareholders mad that they can't see and vote during the meeting making them push for internal change) - so the stakes are much higher than live entertainment streaming.
Netflix is good at many things, livestreaming is not one of those things.
Even some of the old guard can do this. The Olympics worked pretty well (despite the awkward UI), and that was Peacock/NBC.
Perhaps Netflix still needs a dozen more microservices to get this right...
All valid points though each of those examples seemingly only has a fraction of the viewers of the netflix events, right?
for livestreams, individual events like the Olympics probably has a surge audience of 10x of netflix events.
Netflix events is small potatoes compared to other livestream stalwarts.
Imagine having to stream a cricket match internationally to UK / India / Australia with combined audience that crushes the Superbowl or a football match to all of Europe, or even something like livestreaming F1 racing that has multiple magnitudes of audience than a boxing match and also has 10x the number of cameras (at 8K+ resolution) across a large physical staging arena (the size of the track/course) in realtime, in addition to streaming directly from the cockpit of cars that are racing 200mph++.
Livestream focused outfits do this all day, everyday.
Netflix doesn't even come close to scratching the "beginner" level of these kinds of live events.
It's a matter of competencies. We wouldn't expect Netflix to be able to serve burgers like McDonald's does - Livestreaming is a completely different discipline and it's hubris on Netflix's part to assume just because they're good at sending video across the internet they can competently do livestreaming.
this is false, the tom brady roast was live streamed
yes, love is blind failed, but was definitely not the most recent attempt. they did some other golf thing too, iirc
tom brady is largely a guy popular in the USA whereas Mike Tyson is globally famous. It follows that this fight would attract a larger audience.
the point i’m making is that the netflix live streaming timeline didn’t go
chris rock -> love is blind -> mike tyson
they have had other, successful executions in between. the comment i was replying to had cherry picked failures and i’m trying to git rebase them onto main.
Is anyone surprised? I don't see how their infrastructure can handle this when it was designed for non-realtime precaching of prerecorded content.
From what I've heard, Netflix has really diluted the culture that people know of from the Patty McCord days.
In particular, they have been revising their compensation structure to issue RSUs, add in a bunch of annoying review process, add in a bunch of leveling and titles, begin hiring down market (e.g. non-sr employees), etc.
In addition to doing this, shuffling headcount, budgets, and title quotas around has in general made the company a lot more bureaucratic.
I think, as streaming matured as a solution space, this (what is equivalent to cost-cutting) was inevitable.
If Netflix was running the same team/culture as it was 10 years ago, I'd like to say that they would have been able to pull of streaming.
Were they not able to hire enough top-skilled people? If not, why not?
Or did they have a lot of needs that they decided didn't require top-skilled people?
Or was this a beancounter thing, of someone deciding that the company was paying more money on staffing than they needed to, without understanding it?
Combination of 2 and 3. The business changed. Streaming was more or less a solved problem for Netflix. They needed money for content, not expensive engineers. Ted is co-ceo… you can see where the priority is.
My observation is that Netflix is one of those places that brags about how they do so much with so little employees.
Few. Little employees would be...small.
Was live streaming much of a use case for them before this?
They stream plenty of pre recorded video, often collocated. Live streaming seems like something they aren’t yet good at.
https://www.cnn.com/2023/03/04/entertainment/chris-rock-netf... (2023-03-06)
If places like Paramount+ can figure it out, Netflix, given their 10+ year head start on streaming and strong engineering culture, should also have been able to. And if you don't like my example, literally every other streaming service has streamed live sports without issue. YT TV, Hulu, Paramount+, Amazon Prime, Peacock, even Apple TV streams live sports.
It may be "new" to them, but they should have been ready.
I won’t argue that they shouldn’t have done better, I’m only pointing out that this is fairly different from their usual product. Amazon, YouTube, and Hulu all have a ton of experience with live streaming by now. Apple has live streamed wwdc for several years.
I did expect that Netflix would have appropriately accounted for demand and scale, though, especially given the hype for this particular event.
Netflix won't take a hit here.
Most people pay Netflix to watch movies and tv shows, not sports. If I hadn't checked Hacker News today, I wouldn't even know they streamed sports, let alone that they had issues with it. Even now that I do, it doesn't affect how I see their core offering, which is their library of on-demand content.
Netflix's infrastructure is clearly built for static content, not live events, so it's no shock they aren't as polished in this area. Streaming anything live over the internet is a tough technical challenge compared to traditional cable.
Netflix is trying to expand into live sports. This event wasn’t a one off thing. There is an NFL game they are streaming at the end of the year.
Has Netflix ever live streamed something before? People on reddit are reporting that if you back up the play marker by about 3 minutes the lag goes away. They've got a handle on streaming things when they have a day in advance to encode it into different formats and push it to regional CDNs. But I can't recall them ever live streaming something. Definitely nothing this hyped.
Love is blind reunion which had major problems. Chris Rock comedy which only had a few issues. The Netflix Cup which had issues.
Chris Rock comedy special, and the Tom Brady roast. Nothing on this scale, though.
If Netflix still interviews on hacker rank puzzles I think this should be a wake up call. Interviewing on irrelevant logic puzzles is no match for systems engineering.
I did a round of netflix interviews, didn't get an offer (but passed the technical coding rounds) they absolutely had the best interview process of any company I've interviewed at my entire career.
They do make you code but the questions were 1. Not on hacker rank or leetcode 2. Pratical coding questions that didn't require anything more than basic hashmaps/lists/loops/recursion if you want. Some string parsing, etc.
They were still hard, you had to code a fast, but no tricky algorithms required. It also felt very collaborative, it felt like you were driving pair programming. Highly recommended even though didn't get an offer!
For systems design and engineering, absolutely this. I expected the very highest standards and upmost uptime from Netflix, similar to Google and Amazon.
Tells you the uselessness of their engineering blogs.
I don't spend much time streaming, but I got a glimpse of the Amazon Prime catalog yesterday, and was surprised at how many titles on the front page were movies I'd actually watch. Reminded me of Netflix a dozen years ago.
Amazon Prime isn't so great. Lots of for rent/purchase content or content with ads these days. And they end up repeating slots of content in all the rows in their UI, so I end up seeing the same suggestions everywhere rather than much that's new (other than first party productions).
To me they're basically padding their front page.
But honestly that's most of the major streaming platforms these days. I recently cancelled Disney Plus for similar reasons. The only reasons I don't cancel prime or Netflix are because I have family members I split the memberships with to share.
I recently found a lil dvd rental place in my city. It’s a non-profit, they also do archivals and stuff.
It’s pretty much a two-story townhouse packed head to toe with DVDs (lots of blu rays!)
You don’t realize how limited the streaming collection is until you’re back in a movie store, looking through thousands and thousands of movies you would never find otherwise.
Since I found it, I’ve started doing movie night every week with my friends. It’s such an absolute blast. We go to the store, each pick out a random movie that looks good (or bad, in a good way) or just different.
All of a sudden, I love movies again!!
That's an excellent option. I think it'd be remiss not to mention local libraries. Of course, your mileage may vary, but the ones I've gone to do seem to have adequate selections. I just don't often make time to go there and browse like I would have at traditional video rental places back in the day.
Heck, mine even have some video games; though from when I've checked they're usually pretty back-reserved.
I was in high school in the early 00s, and going to the movies was such a big part of my life. Now, I never even know what's out.
I suspect life stage is a factor, but it does feel like there are many classes of entertainment (cinema and standup come to mind) that don't resonate like they used to.
Back in the day everyone was watching the same thing. The choices for entertainment were limited to whatever was showing in movie theatres, whatever was on TV and whatever record stores were selling.
Heh Heh Heh. Maybe there will be a resurgence of Blockbuster style retail stores... ;)
I've given Netflix a lot more money than I've gotten value out of. I've had an account for ~15y and only really use it for airplanes unless there's a specific thing I'm excited to watch.
I'm in the same boat where as soon as they make it too hard to share, I'll probably cancel it. I think the main reason their sharing crackdown hasn't been a problem so far is that I use it so seldomly, it thinks the "main" address is my parents, which makes it easy for me to pass the "are you traveling" 2FA on my own phone when I do want to watch something.
> And they end up repeating slots of content in all the rows in their UI, so I end up seeing the same suggestions everywhere rather than much that's new
All of the streaming services do this and I hate it. Netflix is the worst of the bunch, in my experience. I already scrolled past a movie, I don't want to watch it, don't show it to me six more times.
Imagine walking through a Blockbuster where every aisle was the same movies over and over again.
Amazon Prime front page includes currently a lot of ads for movies that you can rent or ”buy”. Are you sure these movies weren’t them?
This is my biggest issue with Prime Video. You never know what's included and what costs extra.
There's also the "FreeVee" items, which have ads regardless of whether you're a prime subscriber or not. And it feels like a lot of their catalog has been transferred over to FreeVee.
You can toggle to only show free content. Still get ads but theyre obvious.
> Reminded me of Netflix a dozen years ago.
It's been pretty rough the last few years. So many great films and series, not to mention kids programming, removed to make way for mediocre NeTfLiX oRiGiNaLs and Bollywood trash.
is this specifically in India? I never see bollywood stuff in the US but half the catalogue is dubbed/subbed korean dramas
Netflix pivoted to be a platform to waste as much of your time as possible vs entertain.
The Amazon originals are way better imo. They do the dark pattern crap with paid content, as one would expect from Amazon.
Every Amazon show looks the same, yellow washed or something; and they should spend more money on costumes - they get beat by low budget cosplay.
Fallout was pretty good. Very loyal to the game.
Prime Video has to be the worst of all major streaming services. The video quality is horrible, its crippled with ads (3 not skippable ads for an episode of 45 minutes, lastly), and a lot of interesting titles are behind a "partner paywall".
I have prime and my shopping experience is crippled with ads too.
I think it got worse for sellers recently too. If I search for something, like a specific item using its description, sometimes the only result for it shows "sponsored".
It used to show up as sponsored and also unsponsored below.
If this changed, I assume it is bad for the seller. Either they pay for all search results, or their metrics are skewed because all searches were helped by "sponsorship" (and there are no longer unsponsored hits)
I was watching the rings of power and it started with a "Commercial free experience provided by so and so" with a long ad at the start of the episode, and then a third of the way into the episode, at a critical action part, it broke in the middle of the actor's sentence to a 6 minute ad block.
I exited playback and haven't gone back to finish it. I'll wait for it eventually to make it to a Blu-ray release someday.
> Prime Video has to be the worst of all major streaming services
I would put Prime Video at 2nd worst. Absolute worst IME is Paramount+.
Edit: worst for streaming quality
It's also super annoying to try to watch on a computer compared to Netflix or YouTube
> ut my impression still has always been that Netflix is a step above the rest technically.
I always assumed youtube was top dog for performance and stability. I can’t remember the last time I had issues with them and don’t they handle basically more traffic than any other video service?
Maybe a client issue, but i've got a low-end smart tv which handles netflix fine, but youtube is unwatchable due to buffering and failed cuts to adverts
Maybe that’s it. I pay from premium though so don’t have the advert issue (apples to apples).
I think Netflix will have even more sw engineers looking to work there once they notice even for average quality of work they can get paid 3 times more than their current pay.
I think they have to refund the fees for a month to anyone who streamed this fight. That's the only thing that seems fair.
It has been pretty useless. At the moment seems to be working only when running in non-live mode several minutes behind.
So if there are 1 million trying to stream it, that means they would lose $15 million. So.. they might only give a partial refund.
But people should push for an automatic refund instead of a class action.
Is it really that big a deal if you are watching a few minutes behind?
I've watched ball games on streaming networks where I can also hear a local radio broadcast, and the stream is always delayed compared to the radio, sometimes by quite a lot. But you'd never know it if you were just watching the stream.
>Is it really that big a deal if you are watching a few minutes behind?
i don't bet on sports. but from friends who do: yes, it's a really really big deal.
The issue is that most people are trying to watch live which is what it's advertised as. And until they figure out that they need to watch X minutes behind, it is unwatchable. Many will not figure that out.
So for the first hour it was just total frustration until I stopped trying to go back to live mode.
Potentially more so in this brave new world of increased sports betting.
https://en.wikipedia.org/wiki/The_Sting
Offshore combined streaming and betting houses will be cleaning up the rubes.
Internet streams are not real-time even in the best case. There is always a few seconds of delay, often quite a bit more than that depending on number of hops and link speeds, congestion, etc.
well it's live sports, watching live is the big deal. Also people are gambling on the outcomes so watching a few minutes behind is a big deal
I think why I will remember about this fight is not the (small) streaming issue I encountered as much as the poor quality of the fight itself. For me that was the reputational loss. Netflix was touting “NFL is coming to Netflix”. This fight did not really make me want to watch that.
I don't care about boxing or UFC or the grade-A douchebags that are the Paul brothers, but I tuned in just because I had the time and a Netflix subscription.
It was actually great that the fight itself was so boring because it justifies never having to spend time / money on that kind of bullshit. It was a farce. A very bright, loud, sparkly, and expensive (for some people) farce.
The value I got from it was the knowledge that missing out on that kind of thing isn't really missing out on anything at all.
I used to work for a live streaming platform once. We always joked that VOD (Netflix) was "easy" compared to live.
Not really a joke, though? VOD has obvious methods to cheat a bit. Redundancy abounds and you can even network shape for costs. Could probably get even better compression for clear reasons.
Live, not so much. One source that you have to fanout from and absolutely no way to get cheap redundancy. Right?
Yes, of course but it was still a cope because we never saw more than maybe 3M concurrent viewers at a time
Based on this I'm wondering whether it was straight up they did not expect it to be this popular?
> Some Cricket graphs of our #Netflix cache for the #PaulVsTyson fight. It has a 40 Gbps connection and it held steady almost 100% saturated the entire time.
https://fosstodon.org/@atoponce/113491103342509883
There's a difference between live broadcasts and serving up content that's sitting on a server I guess?
In my country every time there's a big football match the people who try to watch it on the internet face issues.
So the issue is that Netflix gets its performance from colocating caches of movies in ISP datacenters, and a live broadcast doesn't work with that. It's not just about the sheer numbers of viewers, it's that a live model totally undermines their entire infrastructure advantage.
See: https://openconnect.netflix.com/en/
Correct, this is not Netflix’ regular cup of tea, and it’s a very different problem to solve. They can probably use their edge caches, but it’s challenging.
How YouTube does this? Netflix is like drop in the ocean when compared to.
My wild assed guess is the differences in the edge nodes.
Netflix's edge nodes are optimized for streaming already encoded videos to end users. They have to transcode some number of formats from the source and send them all to the edge nodes to flow out. It's harder to manage a ton of different streams flowing out to the edge nodes cleanly.
I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients. Only one stream from source to edge to worry about and is much simpler to support and reason about.
But that's just my wild assed guess.
> I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients.
Ha, no, our edge nodes don't have anywhere near enough spare CPU to do transcoding on the fly.
We have our own issues with livestreaming, but our system's developed differently over the past 15 years compared to Netflix's. While they've historically focused on intelligent pre-placement of data (which of course doesn't work for livestreaming), such an approach was never feasible for YT with the sheer size of our catalog (thanks to user-generated content).
Netflix is still new to the space, and there isn't a good substitute for real-world experience for understanding how your systems behave under wildly different traffic patterns. Give them some time.
It also helps that youtube serves shit tier quality videos more gracefully. Everyone is used to the step down to pixel-world on youtube to the point where they don’t complain much.
And decent part of these users are on free tier, so they are not paying for it. That alone gives you some level of forgiveness. At least I am not paying anything for this experience.
In my experience even YouTubeTV has problems sometimes. I'll have the 1080p (and enhanced mode also I think) quality set and still deal with a lot of compression artifacts.
Not sure how Netflix does it. But this is not very time sensitive, and I would have delayed the stream by 15 to 30 seconds to cache it and then deliver to everyone.
I wonder how effective it would be to cache live events with a delay. Write to the tail, read from the head.
that’s totally unacceptable for live sports which people are able to bet on
I have bad news for you. This is how it works already for “live” sports
Correct. Here are some latency numbers from the last SuperBowl: https://www.phenixrts.com/resource/super-bowl-2024
Even the best latency is dozens of seconds behind live action.
Yep. Having actually worked on this sort of stuff I can confirm.
Your ISP doesn't have enough bandwidth to the Internet (generally speaking) for all users to get their feed directly from a central location. And that central location doesn't have enough bandwidth to serve all users even if the ISP could. That said, the delay can be pretty small, e.g. the first user to hit the cache goes upstream, the others basically get the stream as it comes in to the cache. This doesn't make things worse, it makes them better.
I don't bet so I have no clue, but why is that? Are people able to place bets in the middle of the match or something? I would have assumed bets get locked in when the fight starts
Idk about traditional sports books but on Polymarket you can certainly continue betting at any time until the market resolves.
They end betting some minutes before the fight ends.
I last saw Tyson at +500 while Jake was around -800 on DraftKings somewhere in the 6th round.
a match has multiple rounds doesn't it? Seems logical to bet on individual rounds or events that can occur throughout the match.
This is kind of silly because the delay between actual event happening to showing up on OTA TV or cable TV to showing up on satellite TV can already be tens of seconds.
isn't this why people would listen via radio?
Why should they catering to such an audience in first place?
I think this could be one of upsells that Netflix could use.
Premium: get no delay
Normal users: get cache and delay
Or, hear me out here, it's a wild concept, just work.
You know, like every other broadcaster, streaming platform, and company that does live content has been able to do.
Acting like this is a novel, hard problem that needs to be solved and we need to "upsell" it in tiers because Netflix is incompetent and live broadcasting hasn't been around for 80+ years is so fucking stupid.
Every other live platform has a delay of multiple seconds
Live sports require microwave relays for high frequency sports bets
I would be surprised if they don't already do this. The question is how big a buffer to trade off for delay...
I'm curious if the root cause is more variable than usual latency.
Sample size 1, but...
I saw a ton of buffering and failure on an embedded Netflix app on a TV, including some infinite freezes.
Moved over to laptop, zero buffering.
I assume the web app runs with a lot bigger buffer than whatever is squeezed into the underpowered TV.
Likely these devices use different media formats and/or quality levels. And yes, it's possible one device buffers more than the other. Infinite freezes sounds like some routing issues or bugs.
When I was watching the behavior on the tv, was wondering if buffering sends some separate, non-business-as-usual requests, and that part of Netflix's delivery architecture was being overloaded.
E.g. "give me this previous chunk" vs "send me the current stream"
Buffering typically just consumes the same live stream until there's enough in the buffer. No difference other than the request rate being potentially higher. At least I can confidently say that for the standard players/video platforms. NetFlix could be doing something different. I'm not sure if they have their own protocols. But I'd be very surprised if the buffering used a completely different mechanism.
That model still works for streaming. You have a central source stream only to the distributed edge locations, then have clients only stream from their local edge location. Even if one region is overwhelmed, the rest can still work. Load on the central source is bounded.
Damn that sucks. I wonder if they could have intentionally streamed it 5 min late? I don’t have all the context around the fight though — maybe a competing service would win if Netflix intentionally induced delay?
they were introducing 5 minute delays on some of the clients. I noticed my ipad was always live and the smart tv had a 5 minute delay but you could fast forward to live.
I don't think that live doesn't work with caches. No one watching live would care about a O(s) delay, which is highly amenable to caching at ISPs and streaming changes from there to downstream clients. Offhand I'd say that would support O(ms) delay but no less.
Not sure I fully buy that. The “live” stream is rarely “live”. It’s often a highly cached buffer that’s a few mins from latest. Those in isp caches can still help here.
>but my impression still has always been that Netflix is a step above the rest technically.
Maybe if we're not counting Youtube as 'streaming', but in my mind no one holds a candle to YT quality in (live)streaming.
Streamed glitch free for me both on my phone and Xbox. The fight wasn’t so great though, but still a fun event. Jake Paul is a money machine right now.
Yea, it’s a bad look. But I switched to watching some other Netflix video and it seemed fine. Just this event had some early issues. Looks fine now though.
Is cable and broadcast better for live TV? No scaling issues. Doesn't matter how many people tune in.
It's fundamentally different, for sure.
Most third-party internet-based streaming solutions are overlaid on top of a point-to-point network, while broadcast is one-to-many, and even cable tends to use multicast within the cable provider's network.
You have potentially different problems, e.g. limited bandwidth / spectrum. If, say, there are multiple games going on at the same time, you can only watch whichever feed the broadcaster decides to air. And, of course, regardless of the technology in use, there are matters of acquiring rights for various events. One benefit of internet-based streaming is that one service can acquire the rights and be able to reach everyone, whereas an individual cable provider might only reach its direct subscribers.
On cable(terrestrial is entirely different) even the bandwidth or spectrum is less limiting for broadcasting multiple games. Hard thing is the other parts of production, like cameras, live-directing and live commentary. Adding new channels is less challenging than actual producing content at expected level there.
Interestingly that TV nowadays is delivered through IP.
Seems like it
There's an upcoming NFL game on Netflix next month. They need to get their shit together.
Two games actually, both on Christmas Day. A day when most people are at home or the home of family or friends, and they are both pretty good late-season matchups (Chiefs-Steelers and Ravens-Texans) so I imagine viewership will be high.
If they botch the NFL games, it will surely hurt their reputation.
For me, netflix constantly forget the last episode/spot I was in a TV show. Beyond frustrating
I don't think it'll be long-term. Most people will forget about this really quickly. It's not like there will be many people saying "Oh, you don't want to sign up for Netflix, the Tyson fight wasn't well streamed" in even 6 months nevermind 10 years.
It may vary by ISP. It’s been fine for me.
I think you are correct. Ziply Fiber said they were seeing 2.1 times their normal peak [1].
But also people were saying they weren't having any issues streaming on Ziply.
[1] https://www.reddit.com/r/ZiplyFiber/comments/1gsenik/netflix...
> their streaming service normally seems very solid
Not trying to downplay their complexity, but last I heard Netflix is splitting the shows in small data chunks and just serves them as static files.
Live streaming is a different beast
That's how streaming (usually) works. The main URL is a playlist of transport stream files and it just downloads them in the background as you go.
Static files have been pretty much the standard streaming protocols for both VOD and live for the last 15 years ago. Before, it was Adobe Flash (RTMP).
With the way that they are designed, you can even use a regular CDN.
You can push these files to all the edges before you release the content which will protect your origin. Livestream all your edge servers are grabbing content from the origin unless you have another tier of regional servers to alleviate load.
Sure but that’s why your edge servers do request collapsing. And there are full blown CDN companies that will write an enterprise contract with you that can do this stuff with ease. Akamai is like 25 years old now.
Scale has increased but the techniques were figured out 20 years ago. There is not much left to invent in this space at the current moment so screwing up more than once is a bit unacceptable.
I don't think Netflix is even designed to handle very extreme multi-region live-streaming at scale as evidenced in this event with hundreds of millions simultaneously watching.
YouTube, Twitch, Amazon Prime, Hulu, etc have all demonstrated to stream simultaneously to hundreds of millions live without any issues. This was Netflix's chance to do this and they have largely failed at this.
There are no excuses or juniors to blame this time. Quite the inexperience from the 'senior' engineers at Netflix not being able to handle the scale of live-streaming which they may lose contracts for this given the downtime across the world over this high impact event.
Very embarrassing for a multi-billion dollar publicly traded company.
They must have try to do this on the cheap, thinking they could dynamically scale on the fly for this. Big mistake.
This is a total supposition without any proof.
What more proof do you need other than the fact that streams went down worldwide on a highly anticipated event from a public company?
I wouldn't be surprised if lots of engineers at Netflix are currently now writing up a length post mortem of this.
And this is from the company that created the discipline of chaos engineering for resilience.
It is clear they under invested and took the eye of the ball with this.
This is bad, like very very bad.
The assumption that it was related to insufficient investment isn’t supported by any evidence. Flawed technical decisions can be made by the most expensive engineers too.
The evidence is that the stream went down.
We will see why it went down and to what extent they underinvested in their post mortem.
Their CDN is colo and doesn't run on AWS.
Yeah, the funny part is that Hulu, Amazon Prime, and Peacock have all demonstrated the ability to handle an event of this caliber with no issue. Netflix now may never get another opportunity like this again.
AFAIK those three farm out to CDNs with tons of edges who know what they're doing.
I have a feeling Netflix said 'how hard could this be?' and is finding out right now.
Chances are Jake might fight Connor McGregor. Sure, Connor is not as famous as Tyson but that will also invite a large number of people to stream it.
Sure they will. They’ll just set up the next event and outside of some tech folks no one will remember this.
I mean, I guarantee you every boxing fan is never going to trust Netflix again for an event like this.
0 people give a shit about boxing fans. It’s not up to them.
What are they going to do? Just not watch the next fight?
It seemed to be some capacity issue with the CDNs. When I stopped and restarted the stream it worked again. Perhaps they do not use real time multi-cdn switching.
This Serrano fight is just an insane display of excellence.
If anyone was waiting for the main card to tune in, I recommend tuning in now.
Absolutely excellent fight. 10 full rounds with full effort until the end. Fantastic.
Also, no buffering issues on my end. Have to wonder if it's a regional issue.
What was an amazing fight - that Serrano won. I have no idea how Taylor was scored the winner.
That was a savage fight!
naw, taylor head butting the whole fight was dirty and really took the wind out of it
Serrano should have won.
I know nothing about boxing and this fight was just ridiculously impressive. I kept tuning out of the earlier fights. They felt like some sort of filler. I didn’t get the allure. But Taylor v Serrano was just obvious talent that even I could appreciate it.
Serrano was robbed!
I love how I can come to HN to instantly find out if it’s Netflix or my WiFi.
Wifi wifi or wifi as in your ISP Internet connection? Sp many people now call an Internet connection "wifi".
Anyway, network cable is the only way to go!
"Many people", those who call their ISP connection WiFi, are technology potato's.
To be fair, a lot of people pay their ISP for a modem/router combo and connect to something like "Xfinity" at their house. So to them, there is no difference.
I have the wifis with the geebees.
This! I was checking my WiFi and then I instinctively checked HN and what do you know!
metoo！
Right?!
What do you think were the dynamics of the engineering team working on this?
I'd think this isn't too crazy to stress test. If you have 300 million users signed up then you're stress test should be 300 million simultaneous streams in HD for 4 hours. I just don't see how Netflix screws this up.
Maybe it was a management incompetence thing? Manager says something like "We only need to support 20 million simultaneous streams" and engineers implement to that spec even if the 20 million number is wildly incorrect.
Has there ever been a 300m concurrent live stream? I thought Disney+ had the record at something like 60m.
There's no way 300 million people watched this, especially if that number is representing every Netflix subscriber. The largest claimed live broadcast across all platforms is last year's Super Bowl with 202 million unique viewers for at least part of it, but that includes CBS, Nickelodeon, and Univision, not just streaming audiences. Its average viewers for the whole game was 123 million, which is second all-time to the Apollo 11 moon landing.
FIFA claimed the 2022 World Cup final reached 1.5 billion people worldwide, but again that seems like it was mostly via broadcast television and cable.
As far as single stream, Disney's Hotstar claimed 59 million for last year's Cricket World Cup, and as far as the YT platform, the Chandrayaan-3 lunar landing hit 8 million.
100 million is a lot of streams, let alone 300. But also note that not every stream reaches a single individual.
And, as far as the 59 million concurrent streams in India, the bitrate was probably very low (I'd wager no more than 720p on average, possibly even 480p in many cases). It's again a very different problem across the board due to regional differences (such as spread of devices, quality of network, even behavioral differences).
World Cup final, if you add up all streams worldwide?
probably an esports match hosted on bilibili
The arrogant Netflix! They always brag about how technologically superior they are, and they can't handle a simple technological challenge! I didn't have a buffering issue, I had an error page - for hours! Yet, they kept advertising the boxing match to me! What a joke! If you can't stream it, don't advertise it to save face with people like me who don't care about boxing!
Every organization makes mistakes and every organization has outages. Netflix is not different. Instead, of bashing them because they are imperfect, you might want to ask what you can learn from this incident. What would you do if your service received more traffic than expected? How would you test your service so you can be confident it will stay up?
Also, I have never seen any Netflix employees who are arrogant or who think they are superior to other people. What I have seen is Netflix's engineering organization frequently describes the technical challenges they face and discusses how they solve them.
I think you’re oversimplifying it. Live event streaming is very different from movie streaming. All those edge cache servers become kinda useless and you start hitting peering bottlenecks.
Edge caches are not useless for live streaming. They're critical. The upstream from those caches has no way of handling each individual users. The stream needs to hit the edge cache and end users should be served from there.
A typical streaming architecture is multi-tiered caches, source->midtier->edge.
We don't know what happened but it's possible they ran out of capacity on their edge (or anywhere else).
Main event hasn’t even started yet. Traffic will probably 10x for that. They’re screwed. Should have picked something lower profile to get started with live streaming.
I don’t work in tech. Is this something that engineers could respond to and reallocate resources to fix mid stream?
Not a chance. This level of infrastructure was set up days in advance - I would be unsurprised if they'd had a code freeze all week for this fight.
Had issues all stream but was perfect during the final fight.
They've done quite a bit of lower profile live streams... various events, and the Everybody's in LA chat show series.
Reminds me of Nucleus stuttering during UFC
I could hear Gavin Belson screaming during the broadcast when my stream was freezing as they were each making their entrance. Mike Judge is a prophet.
I thought Netflix’s biggest advantage was the quality/salary of its engineers.
I think that every time I wait for Paramount+ to restart after its gone black in picture on picture, and yet, I’n still on Paramount+ and not Netflix, so maybe that advantage isn’t real.
I think this was true at some point, but I’ve been disappointed in the quality of the OSS Netflix tools recently. I think before k8s and a plethora of other tools matured, they were way ahead of the curve.
I specifically found the Netflix suite for Spring very lacking, and found message oriented architectures on something like NATS a lot easier to work with.
Sigh, none of the competitors are much better. Disney, who has more than enough cash to throw at streaming, is a near constant hassle for us ( after 3 or more episodes it throws an inscrutable error on Playstation ). I would drop it, but this is the only remaining streaming service and wife is not willing to drop it ( I guess until 1 it is one error per one episode ).
Do you think YouTube couldn't handle it?
I keep hearing that a lot of people are switching to the illegal competitor for better service.
Eh, the beef that I have is that I am already a paying customer. Why does it seem like I am getting subpar service in terms of delivery? I know it is a tired conversation on this forum, but corporations big and small do what they can do mess with experience to eke out few more cents from customers. It almost does not matter which industry one looks at, it is the same story; the only difference is typically how egregious things get[1].
[1]https://www.marketwatch.com/story/i-opt-to-fly-private-no-ma...
It will never not annoy and amuse me that illegal options (presumably run by randoms in their spare time) are so much better than the offerings of big companies and their tech ‘talent’.
Illegal options would have lot less active users. So it is not a fair comparison
Illegal options also have lot less resources (revenue, service providers who are willing host/facilitate illegal activities, and so on) so it’s a fair comparison in my opinion.
> service providers who are willing host/facilitate illegal activities
At least for NFL pirate streams, it seems they tend to use "burner" tenants from Azure and AWS. Of course they get shut down, but how hard is it to spin up another one?
They still have to put it behind a privacy-friendly proxy to hide their IP address from litigators right?
Yup. Also a bit more latency since it's effectively restreaming unless it's someone at the actual event.
I have Netflix purchased legally with hard earned money. But because I had issues I looked for illegal streams, and they were bad, crashes, buffering.. you name it. So I went back to Netflix and watched it at 140p quality.
This twitter stream was the most reliable for me. Completely took Netflix out of the equation; just some dude at the event with his phone: https://x.com/i/broadcasts/1mrxmMRmXpQxy
> just some dude at the event with his phone
Antonio Brown is not “just some dude”. He’s a national treasure.
Its a good thing he's rich and famous otherwise there might actually be consequences for him illegally broadcasting this.
I guarantee this is a management issue. Somebody needed to bear down at some point and put the resources into load testing. The engineers told them it probably won't be sufficient.
I assume this came down to some technical manager saying they didn't have the human and server resources for the project to work smoothly and a VP or something saying "well, just do the best you can.. surely it will be at least a little better than last time we tried something live, right?"
I think there should be a $20 million class action lawsuit, which should be settled as automatic refunds for everyone who streamed the fight. And two executives should get fired.
At least.. that's how it would be if there was any justice in the world. But we now know there isn't -- as evidenced by the fact that Jake Paul's head is still firmly attached to his body.
This is probably a naive question but very relevant to what we have here.
In a protocol where a oft-repeated request goes through multiple intermediaries, usually every intermediate will be able to cache the response for common queries (Eg: DNS).
In theory, ISPs would be able to do the same with the HTTP. Although I am not aware of anyone doing such (since it will rightfully raise concerns of privacy and tampering).
Now TLS (or other encryption) will break this abstraction. Every user, even if they request a live stream, receives a differently encrypted response.
But live stream of a popular boxing match has nothing to do with the "confidentiality" of encryption protocol, only integrity.
Do we have a protocol which allows downstream intermediates eg ISPs to cache content of the stream based on demand, while a digital signature / other attestation being still cryptographically verified by the client?
What you describe is called a CDN and has been widely used for 20 years.
Utter incompetence from senior leadership at Netflix. They had so much time to prepare for this.
I want to index everyone sneering at this situation and never work with any of them.
Eh, punching up, while still punching, doesn’t seem that distasteful to me.
There's no up. There's just punching, and making excuses for punching.
yep, especially knowing this isn't their first rodeo... 18 months since https://time.com/6272470/love-is-blind-live-reunion-netflix/
> But the real indicator of how much Sunday’s screw-up ends up hurting Netflix will be the success or failure of its next live program—and the next one, and the one after that, and so on. There’s no longer any room for error. Because, like the newly minted spouses of Love Is Blind, a streaming service can never stop working to justify its subscribers’ love. Now, Netflix has a lot of broken trust to rebuild.
From my experience, it works if your not watching it 'live'. But the moment I put my devices to 'live' it perma-breaks. 504 gateway timed out in web developer tools hitting my local CDN. probably works on some CDNs, doesnt on others. Probably works if your not 'live'
edit: literally a nginx gateway timed out screen if you view the response from the cdn... wow
It's down permanently for me in India. We have Hotstar, which has a record of 58 million viewers during the cricket World Cup final. Way ahead.
Probably less about the level of advancement and more about their ability to stream vs play VOD. Two different kinds of infrastructure optimisation.
Wasn't that the biggest concurrent stream ever?
Live streaming and streaming prerecorded movies is a whole different ballgame.
In fact, optimizing for later can hurt the former.
Would be interesting to read any postmortems on this failure. Maybe someone will be kind enough to share the technical details for the curious crowd.
Amazon had issues last year too when they started broadcasting TNF but its fine these days.
I'm sure they will get it figured out.
Can Mike Judge please stop predicting everything?
I've been re-watching Silicon Valley the last few weeks and just watched the Nucleus live stream episode 2 days ago, pretty funny seeing it in real life.
"Puts data compression in a rear naked chokehold"
Reference for everyone else: https://m.youtube.com/watch?v=9IGvzb-KCpY
They should have partnered with every major CDN and load balanced across all of them. It’s ironic how we used to be better at broadcasting live events way back in the day versus today.
> envoy overloaded
That's the plain-text message I see when I tried to refresh the stream.
Follow-up:
My location: East SF Bay.
Now even the Netflix frontpage (post login, https://www.netflix.com/browse ) shows the same message.
The same message even in a private window when trying to visit https://www.netflix.com/browse
The first round of the fight just finished, and the issues seem to be resolved, hopefully for good. All this to say what others have noted already, this experience does not evoke a lot of confidence in Netflix's live-streaming infrastructure.
Ah, envoy. Now that is a name I have not missed.
I watched the event last night and didn't get any buffering issues, but I did notice frequent drop in video quality when watching the live feed. If I backed the video up a bit, the video quality suddenly went back up to 4k.
I had some technical experience with live video streaming over 15 years ago. It was a nightmare back then. I guess live video is still difficult in 2024. But congrats to Jake Paul and boxing fans. It was a great event. And breaking the internet just adds more hype for the next one.
I wonder how localized the issues were. I watched the Taylor/Seranno fight and the Paul/Tyson without issue and the picture quality was the best in every seen for live sports. Was blown away by how good it was. No where near what I’m getting with steaming NFL. This is what I want the future of live sports to look like. Though the commentary was so so.
I’m in the Pacific Northwest. I wonder if we got lucky on this or just some areas got unlucky.
Nucleix needs to focus on fixing middle-out compression instead of kicking cameras.
This livestream broke the internet, no joke. youtube was barely loading and a bunch of other sites too. 130M is a conservative number given all the pirate streams.
don't confuse your ISP breaking with every other provider or the rest of the Internet. It was more than fine here.
Honestly you didn't miss much, every (real) boxing fan thought of this as a disgrace and a shame when announced. putting a 58 year old Tyson against a crackhead filled with steroids (Jake Paul) ? Either case it would have been a shame on Jake Paul for even getting in the ring with such an old boxer.
In boxing you are old by 32 or maybe 35 year old for heavy weight, and everything goes down very very fast.
End of rant.
Hell, I’d complaing about Jake Paul vs Mike Tyson as well if I was a boxing fan. Even without buffering issues
Yes, it was utterly boring, but they made their money. I don't like either Paul brother, so I only watched in hopes shorter, much-older Tyson would make Jake look as foolish as he is.
They better get have some better judges and refs too. The co-headline title fight was a joke.
Netflix has some NFL games on Christmas Day. Wonder how those will go for them.
I remember when ESPN started streaming years back, it was awful. Now I almost never have problems with their live events, primarily their NHL streams.
FWIW, works fine for me.
Please don't make these types of comments, they mean nothing and they serve no purpose.
It means, it is different if the service goes down to 100, 50, 10 percent of users. I watched the show with no issues.
Comments on forums do not provide that data. And if you want to extrapolate self-reports, it's obviously fine (to varying degrees) for the vast majority of people, but that's not the "issue."
These kind of reports are the equivalent of saying "I have power" when you're hundreds of miles away from where a hurricane landed. It's uninteresting, it's likely you have power, and it does literally nothing for people that do not have power.
It doesn't advance the topic anywhere. There are other places to report these issues directly and in aggregate with other people -- HN (and many other forums) are not that place.
You’re in a fucking thread about people commenting in a forum about an outage that may or may not have been caused by Netflix
Which has many interesting facets worthy of discussion! No need to be extremely aggressive in your tone.
You are the one telling people their comments provide no value. That’s far more aggressive than using a word not even directed at you
Why is it acceptable to share that it doesn't work but not acceptable to share that it does?
For the same reason that pointing out the sun rose in the east today would be ridiculous but if it happened to rise in the west, or you perceived it to rise in the west, that would be worth sharing.
Being able to livestream a sporting event is the default now and has been for at least over a decade since HBO’s original effort to stream a Game of Thrones season opener failed because of the MSFT guy they hired, and they fixed it by handing it over to MLBAM.
Maybe that’s what Netflix should do. Buy Disney so they can get access to the MLBAM people (now Disney Streaming because they bought it from MLB).
Been working great for me as well. Starlink in Oregon.
The stream never buffered on my side but quality was for the whole duration of the stream pretty basic I doubt it was even 720p
Us too
I watched on an AppleTV and the stream was rock solid.
I don’t know if it’s still the case, but in the past some devices worked better than others during peak times because they used different bandwidth providers. This was the battle between Comcast and Cogent and Netflix.
Your device type has no influence on your provider and its bandwidth characteristics. If you're on Comcast, Apple can't magically make it not suck.
It isn't Apple, it's Netflix.
Remember back in 2014 or so when Netflix users on Comcast were getting slow connections and buffering pauses? It didn't affect people who watched Netflix via Apple TV because Netflix served Apple TV users with a different network.
> In a little known, but public fact, anyone who is on Comcast and using Apple TV to stream Netflix wasn’t having quality problems. The reason for this is that Netflix is using Level 3 and Limelight to stream their content specifically to the Apple TV device. What this shows is that Netflix is the one that decides and controls how they get their content to each device and whether they do it via their own servers or a third party. Netflix decides which third party CDNs to use and when Netflix uses their own CDN, they decide whom to buy transit from, with what capacity, in what locations and how many connections they buy, from the transit provider. Netflix is the one in control of this, not Comcast or any ISP.
https://www.streamingmediablog.com/2014/02/heres-comcast-net...
Ah, Thanks, I see what you're saying now and it makes a lot of sense. Just didn't grok it from your previous post, Sorry!
Native apps have a lot more scope for client side load balancing due to having a different security model than browsers.
If you're on Apple, Comcast can make it magically not suck. Not sure if it's relevant though.
Cogent just seems to love picking fights with everyone (see Hurricane Electric). Why are they still in business?
> I watched on an AppleTV and the stream was rock solid.
For me it was buffering and low resolution, on the current AppleTV model, hardwired, with a 1Gbps connection from AT&T. Some streaming devices may have handled whatever issues Netflix was having better than others, but this was clearly a bigger problem than just the streaming device.
A friend and I, in separate states, found that it wouldn’t stream from TVs, Roku, etc. but would stream from mobile. And for me, using a mobile hotspot to a laptop; though that implies checking IP address range instead of just user-agent, so that seems unlikely.
Anyway, I wouldn’t be surprised if they were prioritizing mobile traffic because it’s more forgiving of shitty bitrate.
I wonder if this points to network peering and edge nodes. Mobile network vs cabled network likely being routed to different places.
I just left a bar streaming it on a smart TV and back in my home it's streaming on the Roku just fine.
Guess I was looking for explanations too hard.
One similar crash I remember very well was CNN on 9/11 - I tired to connect from France but is down the whole day.
Since then I am very used to it because our institutional web sites traditionally crash when there is a deadline (typically the taxes or school inscriptions).
As for that one, my son is studying in Europe (I am also in Europe), he called me desperate at 5 am or so to check if he is the only one with the problem (I am the 24/7 family support for anything plugged in). After having liberally insulted Netflix he realized he confirmed with his grandparents that he will be helping them at 10 :)
On a tangential note, the match totally looked fixed to me - Tyson was barely throwing any punches. I understand age is not on his side, but he looked plenty spry when he was ducking, weaving and dodging. It seemed to me he could have done better in terms of attacking as well.
I would argue Tyson has a shorter reach, Jake was whiffing a lot of superman punches, and all that does is waste energy. Jake might be able to throw punches, but he clearly wasn't interested in taking them. If they stood closer and slugged it out, the fight could have gone either way.
Yeah the biggest thing to me, and the commentators mentioned this as well, his legs looked REALLY wobbly.
All your attacking power comes from your legs and hips, so if his legs weren’t stable he didn’t have much attacking power.
I think he gave it everything he had in rounds 1 and 2. Unfortunately, I just don’t think it was ever going to be enough against a moderately trained 27 year old.
i thought they did DSA interviews at netflix what happened? I had to watch the fight on someone streaming to X from their phone at the event and it was better than watching on netflix..if you could watch at all. extremely embarrassing!
My theory is they've so heavily optimized for static content and distributing content on edge nodes that they were probably poorly setup for live-streaming.
This, I feel bad for their engineers who were told every bonus would be a matter of how low can they get the cost-per-GB of transferred data, leading to the glorious Netflix-in-a-box (https://openconnect.netflix.com/deploymentguide.pdf) and then management casually asks for a live show with a wildly different set of guarantee requirements.
someone's going to have to reverse a linked list of subscriptions.
Its because they use medium-hard leetcode instead of hard. I suggest 8 rounds instead of 4
Surely more whiteboard questions are the key? Reversing a binary tree is so last year, they should make candidates reverse an octree.
How is this story not on the front page anymore? 375 comments. Seems like a big story to me.
I believe HN's algorithm tends to relatively downrank stories with a high comment-to-upvote ratio, because they are more often flamewars on divisive topics.
Another major algorithmic down-ranker is vote wars on comments.
If lots of people are upvoting and downvoting the same comments, that's treated as a signal the topic is contentious and people are likely to start behaving badly.
HN is very clear they prioritize good behavior as the long term goal, and they are as a result comfortable having contentious topics fall in the ranking even if everyone involved in the discussion feels the topic is important.
Bet they wish they'd gone with middle out compression
When they come up with that idea it's the most 18-rated and accurate way an engineer would think about it.
My kid woke me up complaining internet is not working. Turns out he is trying to watch the fight and it's not working at all here in India.
I’m not sure buffering was the biggest issue with this event. How was as 58 year old Tyson fighting a man in his 20s?
This wasn’t a “real” fight in the ring. It was clearly a hype/money fight only. The late 20 year old boxer has a massive following (or hate following) with younger age groups; and Mike Tyson brings the older age groups out. Mike has earned somewhat of a legendary status.
Leading up to the fight, there were many staged interactions meant to rile up the audience and generate hype and drive subscription revenue, and subsequently make ad spots a premium ($$$).
Unfortunately, American television/entertainment is so fake. I can’t get even be bothered to engage or invest time into it anymore.
It may as well have been a WWE special. As good as scripted. It was a business venture not a competitive fight.
I think they must be noticing the issues, because I've noticed they've been dropping the stream quality quite substantially... It's a clever trick, but kind of cheap to do so, because who wants to watch pixelated things?
To be brutally honest if it’s a choice between pixelated and constantly buffering, pixelated is way less bad. Constantly buffering is incredibly annoying during live sports. (but this doesn’t negate your main point which is that if people paid to watch they expect decent resolution)
Looks like I’m playing Tysons Punchout right now
Glass Jake?
Dumb question
Isn't live streaming at scale already solved problem by cable companies? I never seen ESPN going down during a critical event
This is not the same streaming - netflix is doing that over HTTP. Totally different tech and scaling issues
Yes and no. It's not the "same" but this is a solved problem. Fastly regularly delivers the Super Bowl to 10x as many viewers.
Netflix dropped the ball hard
Fastly says they do 6M ccv for superbowl (i'm actually surprised they let them do the entire thing and don't mix different CDNs) and I'm not sure they do encoding and manifest serving - they might just cache/deliver chunks. Do you really think tyson vs other guy was only 600k ccv? I'd be shocked if Netflix can't handle this.
You would think, but technology always finds a way to screw things up. Cox Communications has had ongoing issues with their video for weeks because of Juniper router upgrades and even the vendor can't fix it. They found this out AFTER they put it in production. Shit happens.
Yes, as I have said again and again on hacker news in different comments Netflix went overboard with their microservices and tried to position itself as a technological company when it's not. It has made everything more complex and that's why any Netflix tech blog is useless because it is not the way to build things correctly.
To understand how to do things correctly look at something like pornhub who handle more scale than Netflix without crying about it.
The other day I was having this discussion with somebody who was saying distributed counter logic is hard and I was telling them that you don't even need it if Netflix didn't go completely mental on the microservices and complexity.
What a massive blow to NFLX. They have been in the streaming game for years (survived COVID-19) and this silly exhibition match is what does them in?
I didn’t watch it live (boxing has lot its allure for me) but vicariously lived through it via social feed on Bluesky/Mastadon.
Billions of dollars at their disposal and they just can’t get it right. Probably laid off the highly paid engineers and teams that made their shit work.
It's not a "massive blow" at all. Consumers will only vaguely remember this in a month. Netflix got a lot of new signups and got to test out their streaming infrastructure to figure out what needs work.
The fight itself was lame which worked in their favor. No one really cared about not being able to see every second of the "action". It's not like it was an NBA game that came down to the last second.
> Probably laid off the highly paid engineers and teams that made their shit work.
More likely overpaying a bunch of posers without the chops, a victim of their own arrogance.
https://www.livemint.com/sports/news/mike-tyson-v-jake-paul-...
"Netflix streamed the fight to its 280 million subscribers"
Perhaps the technology was over-sold.
I’m a little amused at folks tuning in for meme / low quality personalities doing things … and getting the equivalent production values.
Hopefully they fix it because they are hosting two Christmas NFL games this year and if you want to really piss people off you have buffering issues during NFL games lol.
Maybe this was a stress test for the NFL games?
I'd expect the NFL games to have a largely American audience, but today's boxing event attracted a global audience.
Why no one mentioned the term “vaporware”? Isn’t this a classic example of one?
Chaos testing, nothing to see here.
Serves Netflix right for killing my beloved DVD rentals.
Sounds like a job for Pied Piper
I can feel the pressure on the network engineers from here XD
It was so bad. So so bad. Like don’t use your customers as guinea pigs for live streaming. So lame. They need a new head of content delivery. You can’t charge customers like that and market a massive event and your tech is worse than what we had from live broadcast tv.
So much for Netflix engineering talent aura
It’s a learning experience! I remember Conor and Floyd broke hbo and the ufc. It’s a hard problem for sure!
Some buffering issues for us, but I bet views are off the charts. Huge for Netflix, bad for espn, paramount, etc etc
You're lucky you only had some buffering issues. This is got the case for many people, I don't know the percentage, but many people on reddit were complaining.
This is bad for Netflix imo.
I guess it kinda depends on viewer counts?
Does anyone have any thoughts besides "bad engineering" on what could've gone wrong? It seems like taking on a new endeavor like streaming an event that would possibly draw many hundreds of millions of viewers doesn't make sense. Is there any obvious way that this would just work, or is there obviously a huge mistake deeply rooted in the whole thing. Also, are there any educated guesses on some fine details in the codebase and patterns that could result in this?
Mine is glitchy, but if I refresh i get a good steam for a bit, then it gets low res, then freeze. If I wait for auto-reconnect it takes forever. Hard refresh and I'm good. Like, new streams to new server, then overloaded, then does as if their cluster is crashing and healing is rapid cycles. Sawtooth patterns on their charts.
And then all these sessions lag, or orphan taking up space, so many reconnections at various points in the stream.
System getting hammered. Can't wait for this writeup.
It's far from perfect here in Canada, I keep having to pause it or go back and then load it again.
Oddly having watched PPV events via the high seas for years, it feels normal...
Just adding a data point, here in Canada on my nVidia Shield it went down to 360p a dozen times or so, but never paused at all. I guess I got lucky.
Was this the plot of a silicon valley episode?
On X.com someone had a stream that was stable to at least 5 million simultaneous viewers, but then (as I expected) someone at Netflix got them to pull the plug on it. So I would expect this fight to have say, 50 million + watching? Maybe as many as 150-250 million worldwide, given this is Tyson's last fight.
Amazon prime streams the Thursday night NFL game and they seem to have no problem.
Isn't the scale a bit different though? Surely this event was an order of magnitude more concurrent viewers than some NFL game.
Did anyone else see different behaviour with different clients? My TV failed on 25% loaded, my laptop loaded but played for a minute or two before getting stuck buffering, and my iphone played the whole fight fine. All on the same wifi network.
I hope they do a postmortem
they should also do a business postmortem, how did the exec greenlight this without livestreaming infrastructure in place?
I’m very disappointed.
Woke up at 4am (EU here), to tune for the main event. Bought Netflix just for this. The women fight went good, no buffering, 4K.
As it approached the time for Paul vs Tyson, it started to first drop to 140p, and then constantly buffer. Restarted my chromecast a few times, tried from laptop, and finally caught a stream on my mobile phone via mobile network rather than my wifi.
The TV Netflix kept blaming my internet which kept coming back as “fast”.
Ended up watching the utterly disappointing, senior abuse, live stream on my mobile phone with 360p quality.
Gonna cancel Netflix and never pay for it it again, nor watch hyped up boxing matches.
I ended up turning my TV off and watching from my phone because of the buffering/freezing. The audio would continue to play and the screen would be frozen with a loading percentage that never changed.
I have Spectrum (600 Mbps) for ISP and Verizon for mobile.
Not enough chaos monkey engineering.
There was some blog post on HN the other day where someone said they don't do chaos monkey anymore... Even then, how do you chaos test a novel event ahead of time?
Every time it buffers for me, Netflix does an internet test only for it to come back and say its fast...
I thought Hooli was Google, but may be it was Netflix after all.
I'm watching on a 'pirate' stream because my netflix stream is absolutely frozen.
From my limited understanding, the NFL heavily depends on the Netflix Open Connect platform to stream media to edge locations, which is different from live streaming. Probably, they over-pushed the HD contents.
I would have just made it simple, delay the live stream a few seconds and encode it into the same bucket where users already is playing static movies. Just have the player only allow start at the time everyone is at.
I'm watching the event as I'm writing this. I've been needing to exit the player and resume it constantly. Pretty surprising that Netflix hasn't weeded out these bugs.
I couldn’t watch a show a couple days ago. Long time customer, and first time I’ve considered cancelling. Broke the basic contract of I give $ and Netflix give show.
I switched to watching on the android app and it's been flawless. Sad, but workable
They're not used to live. I imagine that's it. All their caching infrastructure is there assuming the content isn't currently being generated.
Illegal streams are working but netflix is not. That is crazy.
This is why we need ipv6. If ipv6 was fully rolled out this livestream could have been an efficient multicast stream like what happens with ipTV.
I did some VPN hopping and connecting to an endpoint in Dallas has allowed me to start watching again. Not live though, that throws me back into buffering hell.
Guess they should have livestreamed it on X to be safe!
Or Facebook. Or YouTube. Or Vimeo. Or LiveLeak.
It’s like watching a Minecraft cosplay of the event.
It's been fine since 11:00 EST, I wonder if they started using the CDN more effectively and pushed everyone back a few minutes?
Mine just crashed and reloaded to "Envoy Overloaded"
Weird that an organization like Netflix is having problems with this considering their depth of both experience and pockets. I wonder if they didn't expect the number of people who were interested in finding out what the pay-per-view experience is like without spending any extra money. Still, I suppose we can all be thankful Netflix is getting to cut their live event teeth on "alleged rapist vs convicted rapist" instead of something more important.
> alleged rapist vs convicted rapist
And you’ll never guess which Presidential candidate they both support!
I watched the whole fight with a 2 minute delay. That was frustrating and it didn't help that Tyson lost.
Well he is almost 60 and the average retirement age for pro boxers is in the mid 30s. He is well past his prime and in physically demanding sport that is very hard on participants.
Currently trying to watch it and it's not loading at all for me. Re-subscribed specifically for the fight.
We all know netflix was built for static content, but its still hilarious that they have thousands of engineers making 500-1M in total comp and they couldnt live stream a basic broadcast. You probably could have just run this on AWS with a CDK configuration and quota increase from amazon
I am curious about their live streaming infrastructure.
I have done live streaming for around 100k concurrent users. I didn't setup infrastructure because it was CloudFront CDN.
Why it is hard for Netflix. They have already figured out CDN part. So it should not be a problem even if it is 1M or 100M. because their CDN infrastructure is already handling the load.
I have only work with HLS live streaming where playlist is constantly changing compared to VOD. Live video chunks work same as VOD. CloudFront also has a feature request collapsing that greatly help live streaming.
So, my question is if Netflix has already figured out CDN, why their live infrastructure failing?
Note: I am not saying my 100k is same scaling as their 100M. I am curious about which part is the bottleneck.
> Why it is hard for Netflix. They have already figured out CDN part. So it should not be a problem even if it is 1M or 100M. because their CDN infrastructure is already handling the load ... Note: I am not saying my 100k is same scaling as their 100M. I am curious about which part is the bottleneck.
100k concurrents is a completely different game compared to 10 million or 100 million. 100k concurrents might translate to 200Gbps globally for 1080p, whereas for that same quality, you might be talking 20T for 10 million streams. 100k concurrents is also a size such that you could theoretically handle it on a small single-digit number of servers, if not for latency.
> CloudFront also has a feature request collapsing that greatly help live streaming.
I don't know how much request coalescing Netflix does in practice (or how good their implementation is). They haven't needed it historically, since for SVOD, they could rely on cache preplacement off-peak. But for live, you essentially need a pull-through cache for the sake of origin offload. If you're not careful, your origin can be quickly overwhelmed. Or your backbone if you've historically relied too heavily on your caches' effectiveness, or likewise your peering for that same reason.
200Gbps is a small enough volume that you don't really need to provision for that explicitly; 20Tbps or 200Tbps may need months if not years of lead time to land the physical hardware augments, sign additional contracts for space and power, work with partners, etc.
Why didn’t they use Netflix AI to solve the problems?
How dare you insult the AI Gods
Was this their first time doing live content? I figured something would go wrong. I'm sure lots of people were watching.
Not their first time, but the first time at this scale.
OK, but the last time they tried a livestream (a reality show reunion), it also fell over. I suppose to their credit, my stream never outright died yesterday, it just went to potato quality.
Silicon Valley predicted this: https://youtu.be/ddTbNKWw7Zs
Sounds like a scene from: Silicon Valley - Nucleus fails
https://www.youtube.com/watch?v=9IGvzb-KCpY
Shoulda used middle out compression.
Over promised and under delivered. That’s a bad look
If you're going to be having intense algorithm interviews, paying top dollar for only hiring senior engineers, building high intensity and extreme distributed systems and having SRE engineers, we best see insanely good results and a high ROI out of it.
All of the conditions was perfect for Netflix, and it seems that the platform entirely flopped.
Is this what chaos engineering is all about that Netflix was marketing heavily to engineers? Was the livestream supposed to go down as Netflix removed servers randomly?
This reminds me of that scene in Silicon Valley
https://www.youtube.com/watch?v=9IGvzb-KCpY&t=50s
I clicked on this thread to type that exact thing, holy smokes.
You're referring to Hooli's streaming of UFC fight that goes awry and Gavin Belson totally loses it, lol. Great scene and totally relevant to what's happening with Netflix rn.
After a few buffering timeouts during the first match, the rest of the event had no technical difficulties (in SoCal, so close to one of Netflix's HQs).
Unfortunately, except for the women's match, the fights were pretty lame...4 of the 6 male boxers were out of shape. Paul and Tyson were struggling to stay awake and if you were to tell me that Paul was just as old as Tyson I would have believed it.
Wow I feel scammed. I paid for a Netflix subscription specifically for this but it's not loading so I'm watching on an illegal streaming website
Why do they want to get into the live business? It doesn't seem to synergize with their infrastructure. Sending the same stream in real time to numerous people just isn't the same task as letting people stream optimized artifacts that are prepositioned at the edge of the network.
Live is the only thing that won’t be commodified entirely. “Anyone” can pump out stream-when-you-want TV shows. Live events are generally exclusive, unpredictable, and cultural moments .
They want to break into sports because it’s such a big business and if you do sports you need to be able to stream live.
Because its the one area traditional networks have the advantage in
The marginal cost to add a viewer to broadcast sports is zero! That's what I am getting at. I know why someone would want this business, I just don't see what aspect of Netflix's existing business made them think they'd be good at it.
Most PPV is what, $50-$70? So subscribing to Netflix for $20 or whatever per month sounds like a bargain for anyone who is interested and not already a customer. Then assume some large percentage doesn’t cancel either because they forgot, or because they started watching a show and then decided to keep paying.
Not sure why this is being downvoted. I can see your point - it’s much harder to this live but a lot of their cdn infra can be reused.
Ota broadcasts are clearer
I thought it's only the best of the best of the best working at Netflix ... or maybe we can just put this myth to sleep that Netflix even knows what it's doing. The suggestions are shit, the UX is shit, apparently even the back end sucks.
Working okay for me
Everyone pointing out that their illegal streams, X streams, etc. work fine are kind of missing the point.
These secondary streams might be serving a couple thousand users at best.
Initial estimates are in the hundreds of millions for Netflix. Kind of a couple of orders of magnitude difference there.
Piracy is distributed, yes.
I think that is the point, in fact.
Everything's easy when it's someone else's content.
I have to assume this is some snarky way of saying "violating copyright is Bad, m'kay".
Because taken at face value it's false. Any technical challenges involved in distributing a stream cannot possibly be affected by the legal status of the bits being pushed across the network.
They have absolutely shit the bed here, and of course their socials are completely ignoring it.
I can't see the fight right now.
It's not lagging for me. It crashed and not coming back.
Update: Switched to the app on my phone and so far so good.
Works in Australia. Maybe their CDN is under a lot of stress?
In NZ, it was had maybe 2 low-quality moments, but never froze and was in high-definition for the rest of the time.
Don't jinx it.
All these engineering blog posts, distributed systems and these complex micro-services clearly didn't help with this issue.
Netflix is clearly not designed nor prepared for scalable multi-region live-streaming, no matter the amount of 'senior' engineers they throw at the problem.
> Netflix is clearly not designed nor prepared for scalable multi-region live-streaming
Well, yes. Who would think Netflix was designed for that? They do VOD. They're only trying to move into this now.
This. Overly complex nonsense that they have built up is crazy and the fact that we on a tech forum agree to all of this is crazier.
It's almost like this platform has been taken over by JavaScripters completely.
> taken over by JavaScripters
That is an incredible way to phrase the sentiment, thank you.
Looks like shit for me. Buffered a bit as well.
People still pay real world money to Netflix after they cancelled and how and why Warrior Nun just to see grandpa being beaten up.
I guess in the year when Trump is being reelected this is hardly a surprise.
Is this potentially an aws issue?
I would assume not because twitch runs on aws. I think netflix engineers haven't optimized as much for livestreaming like twitch has
I do not think twitch has the amount of concurrent users netflix might have had today morning for the fight.
I blame RTO and AI
yeah i'm using iptv which is just a rip of NF and its stuck buffering.
I'm an engineering manager at a Fortune 500 company. The dumbest engineer on our team left for Netflix. He got a pay raise too.
Our engineers are fucking morons. And this guy was the dumbest of the bunch. If you think Netflix hires top tier talent, you don't know Netflix.
> I'm an engineering manager at a Fortune 500 company. The dumbest engineer on our team left for Netflix. He got a pay raise too.
Apparently he was smart enough to get away from the Fortune 500 company he worked at, reporting to yourself, and "got a pay raise too."
> Our engineers are fucking morons. And this guy was the dumbest of the bunch.
See above.
> If you think Netflix hires top tier talent, you don't know Netflix.
Maybe you don't know the talent within your own organization. Which is entirely understandable given your proclamation:
Then again, maybe this person who left your organization is accurately described as such, which really says more about the Fortune 500 company employing him and presumably continues to employ yourself.IOW, either the guy left to get out from under an EM who says he is a "fucking moron" or he actually is a "fucking moron" and you failed as a manager to elevate his skills/performance to a satisfactory level.
White-Knighting for 'fucking morons' is not a good look though. You'll end up in a world where packets of peanuts have a label on saying 'may contain nuts'.
Which would be doubly silly as peanuts aren't actually nuts.
… which is why the label makes sense. They may have been contaminated with nuts during production.
I think acting as if peanuts are actually nuts for purposes of communication is much more defensible than acting as if tomatoes are vegetables, in short you are dying on a hill that was paved over long ago.
I agree most people will conflate them, but someone who's allergic to peanuts but not tree nuts (or vice versa), i.e. the people the labels are intended for, are going to care about the difference.
... or a world where grown adults pay millions of dollars to watch grown adults fighting like school children.
In fact, what am I even doing in this thread? - close-tab.
And you think white knighting for managers who call their directs all “fucking morons” is a good look?
This is the funniest thing I’ve read today
> failed as a manager to elevate...
Managers aren't teachers. They can spend some time mentoring and teaching but there's a limit to that. I've worked with someone who could not write good code and no manager could change that.
Most people I've worked with aren't like that of course (there's really only one that stands out), so maybe you've just been lucky enough to avoid them.
I do find it unlikely that all of his engineers are morons, but on the other hand I haven't worked for a typical fortune 500 company - maybe that's where all the mediocre programmers end up.
> or he actually is a "fucking moron" and you failed as a manager to elevate his skills/performance to a satisfactory level.
sometimes managers don't have the authority to fire somebody and are forced to keep their subordinates. Yes good managers can polish gold, but polishing poop still results in poop.
I was consulting at a place, there was a very bad programmer whose code looked sort of like this
cost arrayIneed = [];
const arrayIdontNeed = firstArray.map(item => {
if(item.hasProp) { arrayIneed.push(item); }
});
return arrayIneed;
the above is very much a cleaned up and elegant version of what he would actually push into the repo.
he left for a competitor in the same industry, this was at the second biggest company for the industry in Denmark and he left for the biggest company - presumably he got a pay raise.
I asked the manager after he was gone, one time when I was refactoring some code of his - which in the end just meant throwing it all out and rewriting from scratch - why he had been kept on so long, and the manager said there were some layoffs coming up and he would have been out with those but because of the way things worked it didn't make sense to let him go earlier.
> the manager said there were some layoffs coming up and he would have been out with those but because of the way things worked it didn't make sense to let him go earlier.
Incentives are fucked across the board right now.
Move on a low performer today and you'll have an uphill battle for a backfill at all. If you get one, many companies are "level-normalizing" (read: level-- for all backfills). Or perhaps your management thinks the job could be done overseas cheaper, or you get pushed to turn it into a set of tasks so you can farm it out to contractors.
So you keep at least some shitty devs to hold their positions, and as ballast to throw overboard when your bosses say "5% flat cut, give me your names". We all do it. If we get back to ZIRP I'll get rid of the actively bad devs when I won't risk losing their position entirely. Until then, it's all about squeezing what limited value they have out and keeping them away from anything important.
this however was back when incentives were not so messed up, but sure.
Hmm. Engineering managers should be setting the team culture and determining best criteria for extending an offer to a candidate. If theres a problem with the hiring process I'd look for the closest source that could or should be fixing it.
I don't think I'd want to work for you.
I hope to never have a manager who is mentally stack ranking me and my coworkers in terms of perceived dumbness instead of in terms of any positive trait.
Almost everyone I know manager or not is usually ranking everyone they work with on various attributes.
In fact it would be incredibly weird to ask a close friend who at their work kicks ass and who sucks and have them respond back, "I've never really thought about how good any of my coworkers were at their jobs"
dumbness is ranking intelligence, which is a positive trait, dumbness is just a metric for how often intelligence fails.
Example - the manager who started this sub-thread may be a pretty smart guy and able to accurately rate the intelligence of the engineers at his organization - but he had a minor momentary failing of intelligence to post on HN calling those engineers fucking morons.
You've got to rank how often the intelligence fails in someone to be able to figure out how reliable their intelligence is.
I'm not a manager and I don't stack rank people, but I am 100% capable of knowing when one of my co-workers or predecessors is a fucking moron.
The trick is to use my massive brain to root cause several significant outages, discover that most of them originate in code written by the same employee, and notice that said employee liked to write things like
...except even worse, because instead of createWidget the name was something far less descriptive, the nesting was deeper and involved loops, there were random assignments that made no goddamn sense, and the API calls just went to an unnecessary microservice that was only called from here and which literally just passed the data through to a third party with minor changes. Those minor changes resulted in an internal API that was actually worse than the unmodified third party API.I am so tired of these people. I am not a 10x rockstar engineer and not without flaws, but they are just so awful and draining, and they never seem to get caught in time to stop them ruining perfectly fine companies. Every try>catch>return is like an icy cat hand from the grave reaching up to knock my coffee off my desk.
Isn't that a problem with your code review process? Why is that person's code making it to production?
So again, maybe they're a bad employee but it seems like nothing's done to even try and minimize the risks they present.
In this specific case, the fucking moron in question was the one who designed the code review process and hired the other engineers, and it took place a significant length of time before my involvement.
Which, yes, does raise interesting questions about how someone who can't be trusted to handle errors in an API call ended up in a senior enough position to do that.
There's a disincentive to actively block PRs if you don't want your coworkers to think you are a bad colleague / not on their side. So you often see suboptimal code making its way to production. This has a worse effect the more terrible engineers there are.
Except in this case it's clearly affecting at minimum the rest of OP's team.
At that point it's not one person being obnoxious and never approving their team members diffs and more of a team effort to do so.
But at minimum if you have a culture of trying to improve your codebase you'll inevitably set up tests, ci/cd with checks, etc. before any code can be deployed. Which should really take any weight of responsibility out of any one member of the team. Whether trying to put out code or reject said code.
Turning this into an incentive that everyone values is a signal that a team has a great culture.
I dunno, I've gone and done a "git blame" to find out who the fucking moron that wrote the code was, only to find out it was me three years ago.
Sure, there's such a thing as stupid code, but without knowing the whole context under which a bit of code was produced, unless it's utterly moronic, (which is entirely possible, dailywtf has some shiners), it's hard to really judge it. Hindsight, as applied to code, is 2020.
I agree with the general sentiment ("one instance of bad code might have been me") but not the specific sentiment ("I could easily decide to catch and ignore errors through every bit of code I worked on without knowing why that was bad, and commit other, similar crimes against good taste in the same way").
The difference for me is that this is pervasive. Yes, sometimes I might write code with a bug in error handling at 3am when I'm trying to bring a service back up, but I don't do it consistently across all the code that I touch.
I accept that the scope is hard to understand without having worked on a codebase which a genuine fucking moron has also touched. "Oh strken," you might say, "surely it can't be that bad." Yes, it can. I have never seen anything like this before. It's like the difference between a house that hasn't been cleaned in a week and a hoarder's house. If I tried to explain what hoarding is, well, maybe you'd reply that sometimes you don't do the dishes every day or mop the floor every week, and then I'd have to explain that the kitchen I'm talking about is filled from floor to roof with dirty dishes and discarded wrappers, including meat trays, and smells like a dead possum.
Hey, that possum's name was Roger and I'm really sad that it died. I've been feeding it for weeks! There are definitely bad programmers out there who's code is only suitable for public shaming via the daily wtf.
Btw how do you know your current manager is not doing that.
I don't. That's why I said hope :)
You’ll know when he ends every meeting with “dummies, get back to work”
I've never seen a team that has somehow managed to hire exclusively morons. Even the shittiest of call center jobs and construction crews have a few people with enough brain cells to tie their shoelaces.
Have you considered that maybe you're being overly harsh about your co-workers? Maybe take the fact that one of them was hired by a top paying employer as a sign that you should improve your own ability to judge skill?
I've seen tons of them! The formula is to create conditions that will make even slightly competent people leave. They hire their morron nephew, he is always 30 min late then they moan when you are 5 min late because the parking lot was blocked by their car. He always leaves 2 hours early while you do overtime that they regularly forget to pay for. Your day is filled with your own work PLUS that of your retarded coworkers who only drink coffee while joking about you doing their work. You are not as fast as the last guy! haha! If something goes wrong the morrons collectively blame you, just like the last time. You get a formal warning. etc etc The other normal person they hire is let go after 2 days because they complaint which means they didnt fit the team.
And so on
If he still works there the morron who left was less of a.
At least half of that is on you. NEVER work unpaid/unlogged OT.
Can’t speak for every place but that’s not always an option. As a teenager, I worked at Sports Direct where the management would regularly work us after our allotted hours and bar us putting the extra time onto our timesheet. If I recall correctly, the company eventually got pulled for it but the money they’d have saved over years would have outweighed the fine.
The timesheets were on paper so good luck putting your real hours on without your manager, who files it, finding out.
I’d be amazed if they ever cleaned up their act.
Yes and: IIRC, the USA has at least $8b of wage theft per year.
Report that shit to your local Department of Labor equivalent. They would have gotten you, and everyone else in that store, their owed money.
You’re asking children to have full understanding of their rights and how to enforce them. Also, investigations into this started in 2020: over a decade after I left. Do you think nobody had reported this in all that time? Looks like the system wasn’t working as well as you think it does.
> Our engineers are fucking morons.
I interviewed at Netflix a few years ago; with several of their engineers. One thing I cannot say is that they are morons.
their interview process is top notch too and while I was ultimately rejected, I used their format as the base template for how I started hiring at my company.
I don't have a dog in this fight, but you typically use your A players for hiring/interviews.
It can be both true that Netflix has God tier talent and a bunch of idiots. In fact, that's probably true of most places. I guess the ratio matters more.
oR god tier talent and a bunch of other god tier talent that decided to coast and cash their fat checks
what seemed good about it that makes different then any other hiring process that seems detached from the job?
None of the Fortune 500 companies hire top talent. They have a few good people but 98% of the engineers are average at best. Over paid.
This is every dev house I've worked at. For most people (mostly not the ones on HN), coding is a 9-5 job. No ambition. Just lines of code. Go home. I don't know there is anything particularly wrong with that.
You just have to accept most staff at any corporation are simply just average. There has to be an average for there to be better and worse.
Why is your team morons? Kind of disparaging maybe? Fish rots from the head situation?
Curb your enthusiasm had a good segment of episodes that were a parody on Netflix and how they shifted hiring from merit to other criteria.
Curb did? What season/episodes?
Reputation usually lags reality by 5+ years. See: Google.
Absolutely right. Netflix was once all about the sports team mentality. Now they’re Man Utd.
Given Man Utd under Ferguson used to be THE football team, you could say they were always Man Utd and still are ;-)
Haha indeed, but I’m a gooner. You’d never see me admit that.
It mostly makes sense to me. From their bombastic blogs to github projects full of overwrought Enterprise java design patterns. The only thing great about Netflix is it pays a lot more.
Sometimes if everyone else is the problem you are the problem.
(We detached this subthread from https://news.ycombinator.com/item?id=42154036.)
You ever thought they were doing the bare minimum and studying at night to leave?
Having worked with a bunch of guys who have gone on to "top teams", I no longer believe they have top teams. My fav was the guy who said the system could scale indefinitely after it literally fell on its ass from too much traffic. He couldn't understand that just because Lambdas my themselves can scale, they are limited by the resources they use, so just ignored it and insisted that it could. The same guy also kept on saying we should change the TPEG standard because he didn't like how it worked. And these companies are seriously pretending they've got the best and brightest. If that's really true, I really need to find another profession.
I've worked for many companies that said they hired the best. And to be honest when I hire I also try to hire good people. I think I could hire better if a) I had an open cheque, b) I was running coolest project in the universe. I did hire for some interesting projects but nothing close to an open cheque. Even under these conditions it's tough to find great people. You can go after people with a proven track record but even that doesn't always guarantee their next project will be as successful.
The reality though is that large companies with thousands of people generally end up having average people. Some company may hire more PhD's. But on average those aren't better software engineers than non-PhD's. Some might hire people who are strong competitive coders, but that also on average isn't really that strong of a signal for strong engineers.
Once you have a mix of average people, on a curve, which is the norm, the question becomes do you have an environment where the better people can be successful. In many corporate environments this doesn't happen. Better engineers may have obstacles put in front of them or they can forced out of the organization. This is natural because for most organizations can be more of a political question than a technical question.
Smaller organizations, that are very successful (so can meet my two criterias) and can be highly selective or are highly desirable, can have better teams. By their nature as smaller organizations those teams can also be effective. As organizations grow the talent will spread out towards average and the politics/processes/debt/legacy will make those teams less effective.
To be fair, when you need to hire hundreds or thousands of people, you gotta hire average people. The best is a finite resource and not all of the best want to work for FAANG or any megacorp.
I used to want to work at a FAANG-like company when I was just starting out thinking they were going to be full of amazing devs. But over the years, I've seen some of the worst devs go to these companies so that just destroyed that illusion. And the more you hear about the sort of work they do, it just sounds boring compared to startups.
> I'm an engineering manager
How are you involved in the hiring process?
> Our engineers are fucking morons. And this guy was the dumbest of the bunch.
Very indicative of a toxic culture you seem to have been pulled in to and likely have contributed to by this point given your language and broad generalizations.
Describing a wide group of people you're also responsible for as "fucking morons" says more about you than them.
Its all ego when these companies think they hire the best.
Our engineers are fucking morons
If your "dumbest engineer" got a job and a hefty raise going to Netflix, it means he was very capable engineer who was playing the part of moron at this Fortune 500 company because he was reporting to a manager who was calling him and the entire team morons and he didn't feel the need to go above and beyond for that manager.
Also, highly likely that it was the manager that was the moron and not everyone around him.
> If your "dumbest engineer" got a job and a hefty raise going to Netflix, it means he was very capable engineer who was playing the part of moron
It's also possible that there's very little correlation between capability, reputation and salary.
Don't we all know someone who is overpaid? There are more than a few well known cases of particular employers who select for overpaid employees...
> well known cases of particular employers who select for overpaid employees
Not well-known enough, apparently. Where should I be applying?
There are different forms of overpayment but to give some examples:
- The recent story of AWS using serverless for video processing comes to mind [1].
- Google is renowned for rest and vest.
- Many government jobs pay more than their private counterparts.
- Military contractors
- Most of the healthcare industry
- Lobbyists
[1] https://news.ycombinator.com/item?id=35811741
In terms of healthcare industry in Hungary: one worker does the same job for 700 USD a month and another for 1100 USD, the only difference is formal education and years worked in the industry. You can perform much better (by actually caring about the patients in those 12 hours you work) but you will get paid the same amount regardless. Of course if you have 3 kids (whether they are adults or not) then you do not pay taxes (or much less than someone who does not have kids or only has 2).
> Don't we all know someone who is overpaid?
Yes, usually managers.
I can +1 with a similar anecdote.
They obviously have some really good engineers, but many low-tier ones as well. No idea how long they stay there, though.
I'm watching the fight now and have experienced the buffering issues. Bit embarrassing for a company that fundamentally only does a single thing, which is this. Also, yeah, 900k TC and whatnot but irl you get this. Mediocre.
livestream is quite different from streaming pre-processed video, so I'm not surprised by the scaling issues
i was mildly interested and managed to find a pirate livestream, it didn't have buffering issues lol
Well the pirate site was not live-streaming to 100 million users…
I know.
But given how much they spend on engineering, how much time they had and how important this event is ... mediocre performance.
All true, but this part of your GP comment:
> a company that fundamentally only does a single thing, which is this
… isn’t true. From the couch, watching Suits and watching a live sports match may seem similar but they’re very different technical problems to solve.
Or in other words: In one case the "stream" is stored on a harddrive not far away from you, only competing for bandwidth in the last section to you. In the other case the "stream" is comming over the Internet to you and everyone else at the same time.
I have to wonder if its a regional thing. I'm watching from the southern pacific in HD, and its been excellent.
I’d imagine it is fairly dependent on which cache you’re connected to.
I'm going to avoid leaving a zero-effort response like "actually you're the problem" like half of the replies and contribute:
Why do you call your engineers morons? Is it a lack of intelligence, a lack of wisdom, a lack of experience, inability to meet deadlines, reading comprehension, or something else?
I wonder if Netflix is just hiring for different criteria (e.g. you want people who will make thoughtful decisions while they want people who have memorized all the leetcode problems).
I don't understand why nobody here believes you.
There's no reason to doubt what you say, probably people identify with the mistreated one. Why?
Because the idea that all the engineers that work at his large company are morons is absurd. Anyone in that situation that believes that and even more, states it, is just making their own character flaws apparent.
It’s hyperbole, like a teacher complaining to others, “my kids were all crazed animals today.”
I’ve worked with engineers where I had to wonder how they found their computer every morning. I can easily see how a few of those would make you bitter and angry.
Let me think about it...
All the engineers in MY company are morons.
They're just bureucrats.
this is why managers get a bad rap. what proportion think like this? hopefully not a large one but i do worry. ultimately if the team sucks its because of the management. theyre the ones with the greatest power to change things for the better.
Your job must be truly awful.
Sounds like you’re a good match for their team then
What are the chances that your entire engineering team is entirely composed of low performers or people with bad attitude or whatever you designate as "fucking morons"?
It's more likely that you are bad at managing, growing and motivating your team.
Even if it was true, to refer to your team in this way makes you look like you are not ready for management.
Your duty is to get the most out of the team, and that mindset won't help you.
Don’t agree. Sometimes you can observe the world around you, and it’s not pretty. Are they not allowed to observe the truth as they see it? What if they are right?
An engineering manager who thinks his engineers are morons and dumb?
I have questions..
Top troll bro
Sounds like he got a better deal. If this is how you describe your team, I suspect they are all submitting their resumes hoping to get away from you.
Dupe: https://news.ycombinator.com/item?id=42153906