Apple loves to stealth test new tech in full public view by sneaking it into relatively mundane places, so debuting agentic AI via accessibility is very on brand.
A few other examples:
- The Touch Bar was much more than an OLED strip, it was Apple’s first move in the transition to Apple Silicon on macs. The Apple T1 chip in the 2016 Touch Bar MacBooks was the first solely Apple-designed processor to appear in a Mac and took over several responsibilities away from intel chipsets like power management, fans, sleep/wake, access to the camera & mic, and the secure enclave powering touch ID. Then the T2 added encryption of the SSD, audio management, image processing for the camera, and prevented tampering with the boot process
- The iPhone 3G shipped with a Liquidmetal SIM eject tool, which is made from a strong custom metal alloy which is "practically unbendable by hand unless you want to hurt or cut your fingers." Although Apple hasn’t released anything with the alloy since then, now nearly 20 years later Apple is rumored to be using liquid metal in their upcoming foldable iPhone.
- RealityKit had 3D scanning and a lot of other cool AR capabilities for years which didn’t make sense until the Apple Vision Pro was released.
'liquid metal' sounds cool. It's probably a metallic glass. I super dislike that it seemingly will be synonymous with the brand name by Apple even though that stuff has been around for decades.
Not that there are particularly many places where this is used - mostly because it really is just very expensive. In the awesome position that Apple is in, economic feasibility is so much easier to achieve, with like tens of millions of guaranteed parts to be preduced.
And their upcoming smart glasses are the best UX for almost everything they showed the user holding up her phone for.
I read their glasses when taking video or pics the lense will light up and or flash more prominently then Metas. Maybe that will help the whole privacy issue and also it's not Meta (do love my Meta or smart glasses as a whole will ditch Metas for Apple quickly as both pair of Metas broke & there's no store for support).
I also am looking forward to Apple smart glasses. I use Meta glasses because thre useful since I'm totally blind. I'd much rather have on device recognission though if possible. I also trust Apple more then Meta. At least I'm technically enclined enough to realize I shouldn't wear them in the bathroom or bedroom. At least when humans look at my AI prompts there maily seeing food boxes or computer bios screens.
Same for me! Both pairs of Metas are now inoperative because I can't get them back to factory reset. Once you pull that little blue tab out of them when they're new, it's GAME OVER.
Adding to the list of grievances, the functionality and the options it presented differed from app to app. I understand that the function keys also change their function app to app, but the visual noise the Touch Bar (wow - the word even gets autocorrected to have the right capitalization!) added as you switched between apps was too distracting.
I’ve always been a “remap capslock to escape” kind of guy (vim), so I didn’t mind much. Access to the brightness (screen and keyboard) and volume slider was neat but superfluous with the OG fn keys. Context-driven controls were probably the best thing about the touch bar, and I don’t think it got enough love to make that stick.
1) First generation made ESC button a touch button. Aside from ergonomics (or lack of them), at least for me, on a psychological level "abort" button needs to be something you can smash. Also, macOS already had the worst input handling under load, making it virtual button made it worse.
2) While "happy path" on macOS pretty much never requires you to use Fkeys, but my workflow does. Blindly using touch buttons is harder than real buttons.
3) I'm not huge media keys users, but I bet #2 applies here as well.
I liked the touchbar in every other sense. If it was just an addition to an existing keyboard, people wouldn't have hate it[];
[]: At that time it was hard to not be frustrated using mac (butterfly keyboard etc), so touchbar might have gotten more hate than it deserved because of overall frustration.
Also, the Touch Bar seemed to be abandoned as soon as it launched. It only ever launched on the Pro line. There were never any feature updates. They never made it flexible enough for people to customize it.
> They never made it flexible enough for people to customize it.
I feel like it was fairly customizable - the Mac system settings let you do a lot of drag and drop of controls, and
I recall iTerm having a similar interface for customizing the bar in its own settings.
I do think it should’ve been given a lot more love, but that’s Apple for ya I guess
I think a bigger issue was that so few applications used it in cool, interesting ways. It has the same appeal as the oled button boxes some people have, except it’s right there on the deck… but nobody did anything with it.
I sure did prefer the media controls on it, though. I still have a 16” here and am reminded of what could have been.
It was extremely flexible in customization options, and there were SDKs to make it do additional cool stuff in apps, but nobody really cared for the most part.
I honestly think it was mostly a "we have a custom secure coprocessor now, what can we do with it?" sort of thing, which also worked out for Touch ID and disk encryption.
One more on top of others. Many people felt it was a solution in search of a problem. As in, there was no problem i had that it solved. And it was forced on us, in place of something useful. From the start i read that as: This wont be here in a couple years. Which then made it annoying to deal with in the meantime (the hate).
Things that stick around, are generally value adding across a large or complete subset of their users. Touch bar was always niche, and thus always doomed. I think a good counter comparison is Apple VR headsets. For me, i have no use and little interest. But i can see them as a hedge at the very least, or as an enthusiast entrant into an emerging market, where future products in that segment may become interesting. And on top, it doesnt impact me - i can ignore their existence until it becomes useful.
If touch bar were launched like VR, i suspect it would have gotten similar level of dismisals, but less hate.
One thing Apple really needs to get right is speech to text transcription. They've nailed accessibility in so many ways and yet it feels like they're a decade behind on properly transcribing voices. At least half a decade.
Input on the iPhone is so dreadful nowadays. Their palm rejection is definitely worse than before, so mistyping is more frequent. Their text-correction algorithm for typing is worse than before, and it frequently makes incorrect corrections to words that I don't notice, because they change words a few words back from where I typed. And STT hasn't improved. On top of that, my fingers are tired of the phone form factor. Please make the iphone not a chore to use, apple.
Until siri can reliably handle "Navigate to <business that is a decade old>", offline and using pre-downloaded maps, I'm going to assume all the other, harder speech to text and conversational stuff is just vaporware.
I found another dreadful iPhone input "feature" yesterday. If you are browsing around in third party carplay apps, and ready to tap your selection, but instead press the accelerator first, it truncates the list to only a few items, and scrolls to the top.
Way to reduce driving distractions guys! What's next? If the car is moving, maps changes destinations?
I really wish human computer interaction research were more broadly applied, and if you do dumb stuff like all of the automotive / carplay world, then you'd be liable in court.
I once had a car that hid the backup cam behind a legal disclaimer every time you turned it on. I'm sure at least one pedestrian was hit by a car in reverse while that screen was on. The manufacturer should be 100% liable for the poor UI decision.
Wispr Flow is a masterclass in STT. Apple's solution feels like it's from the last century in comparison. Same applies with Apple's TTS when you have ElevenLabs and OpenAI running laps around it. All I need is for my iPhone to do those things natively at the same quality level (because in Apple's walled garden that's the only way to get them usable everywhere).
But Apple's uses so few system resources and runs fully on device on newer iPhone models (16+ I believe). It's so efficient. I really enjoy using Handy with Parakeet as the model, but the system resource usage is a monster compared to Apple's (although very good).
Looks like Wispr Flow uses a cloud model [0]:
> Cloud based speech processing infrastructure for 1B users
It gets to be a messy comparison because my iPhone can do STT with no latency pretty well fully on device, but Wispr Flow requires a cloud model, but to be fair, older Apple devices do as well. It's not an apples and oranges comparison, but I think those technical details make this a non direct comparison in a few ways.
For on-device with low system resource usage, Apple's is pretty damn good.
Apple's stt has been on-device for a long time now, long before iPhone 16. I haven't noticed any improvements since my first ever iphone 5S. I'm pretty sure wispr flow can use on-device models. I use Voiceink[0] which can use parakeet models on-device and can optionally use cloud models.It's like night and day comparing Apple's to Voiceink. The only advantage I find to Apple's stt is less friction. 3rd party apps just can't integrate as smoothly with the system. There's a gesture to activate Appledictation when Voiceover is on.
That doesn't work if you have limited or no connectivity (e.g. on a mountain range). There are also privacy concerns, e.g. a doctor using it to transcribe medical information.
FWIW - I also really like Wispr Flow, but I moved to running the 'Whisper Large' model locally using Handy (https://github.com/cjpais/Handy), which has been essentially as good, while also having lower latency.
Handy is great. It exposes a bunch of open models beyond Whipser too, and though I haven’t tried too many of them, I’ll throw in a rec for the Parakeet model which feels pretty much on par with Whisper for accuracy and is way way faster.
Would be great if they could at least fix two major bugs:
* input simply fails (seemingly) randomly where it is supported and many apps from major vendors don't support dictation input at all (e.g. OneNote) (there should at least be a fallback (a la Dragon Dictate from decades ago) for those cases
* capitalization is still random leaving you with many errors to correct
but Apple mostly seems to see accessibility as something to use to enable performative press releases not actual functionality...
It’d be amazing if speech-to-text could take into account context as well: Greek if I’m speaking Greek, Korean if I’m speaking Korean, or for (int i = 0; i < count; ++i) if I’m dictating code.
There’s so much complaining about their keyboard issues, and it’s really an infuriating part of the iOS experience. The phone being hard to grip/slippery doesn't help, no…
A while ago I signed up as a sighted person on Be My Eyes. I didn't get as many calls as I had hoped, but I was glad to help out on the few that I could. One call was to read envelopes of incoming mail, another was to read pill bottles, and then there was the two funny guys on big cozy chairs with shopping bags of cereal boxes and wanted to know what was what. I remember one guy really didn't like one type. The app had a unique feature for the sighted person to turn on the camera of the vision impaired person.
I still have the Be My Eyes app installed but haven’t had a call in over a year - I think it’s a testament to how powerful AI vision models have become. I find it cool that AI works well enough for vision impaired people to solve all their problems.
However there was something very human and nice about helping out a stranger with a small random task from time to time. I fondly remember one older lady who spilled a box of blueberries on the kitchen floor and I helped her hunt them all down by guiding her around. It was 10 minutes of connection with a random person doing something fun and which is till remember fondly 4 years later
Ever since Be My Eyes introduced there AI features it's my understanding that there's been a lot less need for volunteers. I'm totally blind, and started using the app once they added in AI. It works great for for things like reading food labels after my kids have movved things around, determining if the tv has been left on, etc. I think I would use the volunteer feature if I still lived alone, but I don't.
They always had significantly more people willing to help than in need of help. Which is good, not going to knock that. I signed up for it many years ago but didn’t get a single call, so eventually I just uninstalled it.
There would have to be a person pretending to be blind on the other side of the phone call, recording your answers. At that point is it not just easier to have that person do the labeling themselves?
Fun fact: This video was made accessible to sighted people because no blind person would ever listen to voice at that speed. Honestly if you ever observe a blind person using computers you'd impressed how they can listen to audio at unimaginable speeds.
I think it takes quite a lot of practice to reach this speed. It's not rare among blind developers, but I think it still takes a lot of work to get there. Pretty impressive!
I wish more people would watch videos like this just because having a realistic idea of how blind people do certain tasks can help you move from pity or even compassion to a more productive kind of understanding. I think sometimes when you haven't seen it, you can't really even imagine how it can be done.
I listen to a lot of podcasts and listen at 1.5-2.0 speed and it’s to the point that I literally cannot stand listening to 1.0 speed anymore as they go too slowly (depending on the content of course).
Same. Returning to 1x speed makes people sound (to my 2x-abused ears) drunk and slurring their works. If I want to listen to something slowly and carefully, I will just about tolerate 1.25x.
What really frustrates me is watching/listening to discussion of music, because I am forced to listen to the talking at 1x because the music sounds wrong (and is wrong) at anything other than 1x.
I'm so glad YouTube and other podcast players have moved to support 3.0 speed. As I get comfortable with one, I move it up some. For things like sports and "did you know" content, I can go 2.5 if I'm not multitasking. For technical content, sometimes I'm stuck at 1.0.
You can get browser extensions to do it for all media controls on any site. YouTube's "Premium" for 3x is laughable when it's an internal browser function.
Same haha. But for me 1.5x is the sweet spot. Anything more and I find myself rewinding a lot. I want to feel comfortable absorbing info and not on constant alert.
I am jealous. I can't listen and retain most podcasts at more than 1.0x. I even disable the podcast player functionality that eliminates pauses and silent sections.
I do the opposite in a few. There's some I follow weekly and it's only an hour or so each. I drop it to .7 or .8 because I want to get a bit more time with the hosts. Possibly stupid but I've sort of got used to some of these folks at that speed, and the normal speed is 'weird'. One is a political podcast, and when they play clips of Trump, he does always sound very drunk, but the hosts themselves (to me) don't sound drunk, just... measured. Some of it may be audio quality - I'm getting their microphone directly, often the audio clips are from field recordings.
Unless you're completely technologically illiterate, the kind of person who has no idea how to install an app or sign up for an online account, you're probably doing something of the sort.
If you are dedicated, few weeks to few months of usage with regular ramp up. You should be careful with adjusting which symbols are read though and sometimes the programing languages matters because different symbols have different significance for understanding the code.
I did IT for a community Center way back in the day and the director was blind. I was blown away by how fast his screen reader read things out to him - completely incomprehensible to me - and his efficiency with keyboard shortcuts would put even vim/emacs elitists to shame.
The way (Windows) screen readers handle web navigation is basically Vim in disguise.
You have two modes: "focus mode", where you can edit text in text fields and keys are passed straight to the browser, and "browse mode", where keys move a virtual cursor around the page.
In browse mode, navigating with just arrow keys all the time would be just as slow as you might imagine, so you use single-key keyboard shortcuts to move by role, E.G. to the next heading, button, table or unvisited link.
The keyboard layout is optimized for memorizability and not efficiency, you use the actual arrow keys instead of hjkl for example, but the concepts are eerily similar.
There are a couple of other approaches to solve this problem, Mac OS's Voice Over is much more Emacs-like for example, and each approach has its own pros and cons, but that's definitely one way to do it.
Probably because it's an advertisement, and super fast robot voices can feel extremely harsh and annoying. Even blind people who rely on them find them overstimulating sometimes, lol.
Indeed, and not just fast, but often heavily robotic (which many sighted people struggle to understand even at 1.5x). I remember reading about a blind person who learned how to do echo-location using sound, and it seemed like such a cool superpower, that one of these days I'm going to take the plunge and unplug my monitor and start learning how to really use the tools. I worked with a blind person a few years back who got almost double the battery life from his laptop as the rest of us by having the screen off all the time, so that alone would be a nice feature. I may never get to the epic level of echo-location, but if I get even half-way there it would be awesome. With a bonus of being able to actually QA a11y changes.
Twenty years ago I took a level 1 tech support call from a visually impairment guy and it took about 3.2 seconds to realise his condition was no impediment for using a computer because of the screen reader tech he was using.
I listen to a lot of podcasts and YouTube videos at 3x or 4x speed now, having slowly built up the skill over a few years. It's pretty nice now and saves time, and it's remarkable how well the human brain can adapt to such input.
You recall nothing and you know it. You're just wasting time you could use for something useful or meaningful in your life. Kids call it "Anxiety cope" but I don't agree.
> Honestly if you ever observe a blind person using computers you'd impressed how they can listen to audio at unimaginable speeds.
Even better, fire up Orca (or whatever screenreader application your OS comes with) yourself and try to use your computer while shutting your eyes, kind of eye-opening (no pun intended) what kind of experience these sort of users typically get. And also, you quickly start to understand why they set the speech rate for their voice synthesizer to be so fast, it's almost unbearable navigating applications (and particularly lists) otherwise.
When I was at Google, I'd periodically test our (internal-only) app with Chromevox with the display off. It's not that it sounded like it would be easy, but it really is a challenge, and I can only imagine the muscle memory built up over time of trying to work around accessibility bugs and strange behaviors.
Unfortunately it seems impossible to get all that much funding for accessibility work :/ I wonder what ever happened to the Newton accessibility bus intended to supplement Wayland...
I’ve worked at Apple Facebook and Google. Apple was the only one that made a11y bugs and a face to face consultation with a blind developer to show you how your app sucked, mandatory before you could launch.
> I wonder what ever happened to the Newton accessibility bus intended to supplement Wayland...
Hm, never heard about it, but now I'm wondering too. I just finished implementing proper accessibility support for my native app toolkit for Linux, macOS and Windows, but only done it for X11 so far, I was just gonna get started with Wayland. What is the accessibility story on Wayland, couldn't people rely on the same protocols as with X11? That was my impression, but haven't really dig into yet.
There are apps I use semi-regularly that less-experienced screen reader users thought were inaccessible, and I couldn't even explain what they were doing wrong from memory. The ways of working around accessibility issues are just so ingrained in me that all I can usually remember is "yeah I did this somehow, but it was six months ago and I have absolutely no idea which specific tricks I needed for this one."
> you quickly start to understand why they set the speech rate for their voice synthesizer to be so fast, it's almost unbearable navigating applications (and particularly lists) otherwise.
I imagine that for coding it also helps deal with the fundamental problem of an ephemeral stream rather than a persistent document that you can navigate visually in multiple dimensions. Working memory is limited, and getting more text in in a short period of time probably helps you work within that better. I also imagine that working with text via audio all the time gradually stretches and improves memory.
It's not the ephemeral stream that's the problem, it's the limited bandwidth.
You can show a lot more info on a screen than you can transmit through speech in a short period of time. That doesn't mean you read faster than you listen, just that sighted people essentially use their eyeballs as an "input device" to decide what information to look at.
If there's an object on the screen that you want to examine but that you don't need to click, you can just "navigate to it" with your eyeballs, without ever touching a mouse or keyboard. We don't have that luxury.
This means we need a much more efficient system for navigating what's on the screen, but that only gets you so far. Eventually, the easiest way to deal with this problem is just to increase the bandwidth of your channel, and you do that by increasing the speech rate.
The difference is that the voice in the video is a natural, human voice. It's the robotic sounding voices that always pronounce the same letters the exact same way (mostly the Eloquence family of voices) that enable blind people to listen at superhuman speeds. You can't listen to a real voice that fast.
I know plenty of blind people who have their voice speed unbearably slow and barely scratch the surface of what technology can do for them. I think an interface where you can tell your phone what to do in natural language will really help a lot of less technical people.
I'm not getting my hopes up though given apple's history with Siri, which is truly awful.
Apple's history with accessibility is, on the whole, pretty good. I strongly suspect that the "coming soon" part of this means "after we integrate Google Gemini models into the system," so I don't think you should use the current state of Siri as a yardstick. (I actually have decent luck with the current Siri, but I don't push it very much and have sort of adapted myself to its limitations; on the flip side, I have a lot of skepticism around LLMs, but they're really a quantum leap in natural language processing capability over what came before, and the use cases they're showing here seem to be right in the LLM wheelhouse -- with the asterisk of "you're still always going to have to check its work.")
> I strongly suspect that the "coming soon" part of this means "after we integrate Google Gemini models into the system…"
I don’t think the Google's tech has anything to do with these features.
This would had to have been in the works long before the Google announcement. Also, these are enhancements of existing iOS and macOS features. They don’t require an LLM anyway; these features use Apple’s Machine Learning models.
For example, creating subtitles for videos? iOS 16 introduced Live Captions for FaceTime calls in 2022 [1].
This has been the typical pattern for Apple for the last few years. The flashy features are announced at WWDC, accessibility has a dedicated, earlier press release. Before this practice, accessibility announcements would usually be tucked in some WWDC slide that most people wouldn't even notice.
The thing that disappointed me about this amazing announcement was “coming later this year“. They should probably give us dates for a little while at least until we get the (<)$95 checks.
I just would not wanna promise anything. Except “available for download this Friday“ once the gold master is passing tests.
The "coming later this year" language is disappointing to some people, but that's just Apple propriety. Allow me to explain.
"Coming later this year" means it's part of a publicly committed release — iOS 27, macOS 27, etc. — not vaporware.
The annual pre-WWDC accessibility announcement is a tradition, and with the conference less than a month away, expect more detail then. New a11y features have a good chance of appearing in the 10am PT keynote or the Platforms State of the Union, the developer-focused follow-up at 1pm PT.
That said, things are still fluid with three weeks to go — features can be added or pulled at any time. If something gets bumped from the main presentations, there will almost certainly be a dedicated video session covering it.
As for availability: some of these features will land in the iOS 27 and macOS 27 betas, which drop during WWDC for Apple Developer Program members. The public beta follows in July, and there's a free tier of the developer program if you want early access.
Don't expect everything at once, though. Some features won't arrive until the September release candidates — and even then, a few may ship labeled "beta" or "experimental," or hold for a future dot release.
Being able-bodied and sighted is probably the biggest disadvantage for using iOS.
Twenty years and text input & manipulation on iPhone sucks a big fat hair pair of dogs balls still.
The last time I daily drove Android was over two years ago and it was immeasurably less God-damn-I-wanna-dig-Jobs-corpse-up-n-give-the-guy-a-piece-of-my-mind, only problem is his grave is unmarked. Arsehole!
Whenever my sister (blind) and I (visually impaired) visit my mom (blind) we secretly turn up the reading speed on her TV just a little because we can't stand how unbearably slow she keeps it, but if we turn it up quickly, she'll freak out.
After a few more years of Thanksgivings and Christmases and Mothers' Days, we'll finally train her up to a reasonable speed lmao.
This is heartwarming. The audio equivalent to the practice of sighted people fixing the bad default settings on boomers’ televisions each Thanksgiving.
Whether that control you see visually is actually accessible to a blind user is a different matter entirely. Further, it maxes out at 2x, but a blind person would typically screen read at the equivalent of 3-6x.
> Another way in which it's not cheap to lose sight, I guess.
True.
We can frame it even more strongly: "default societal practices actively discriminate against people with disabilities; they intentionally, consciously choose to make life harder for people who're disadvantaged".
I don't get it. The speed of TTS can be adjusted, right?
Pretty sure there's enough blind people who don't listen to voice at insane speeds, because they listen in their non-native second language or for whatever other reason. What's wrong in using lowest common denominator that's 100% accessible to those people as well as people who want faster speeds? Unlike "too fast", "too slow" doesn't get entirely inaccessible, it's just boring.
I don’t think it’s meant to be criticism. It’s an interesting piece of information that gives a peek into how those with vision impairment consume content. There’s nothing wrong with it; but it was enlightening to consider the experience for those of us who have not been forced to.
Some blind people listen to things at superhuman speeds, but not all blind people. Using a normal reading speed is a sensible choice for an ad trying to appeal to blind people since you don't want to intimidate those who don't use superhuman speeds.
Going from that to "heh a sighted person made this because it's normal speed" is simply incorrect.
It was the sort of statement an HNer might make to showcase some trivia they have about some other group, but they oversold it.
> Pretty sure there's enough blind people who don't listen to voice at insane speeds, because they listen in their non-native second language or for whatever other reason.
Yes, for lots of reasons. It takes practice to get up to a high speed with a given TTS. People who go blind later in life are just beginning, and it can take a long time for them to get up to really high speeds. You may also need to reset somewhat when you change from one TTS to another. And blind people's ears are subject to problems just like anyone else's; if your hearing isn't great you may need slower speeds or higher volumes or both. That's why even though most people use screenreaders at much higher speeds, the defaults when you turn on a new device are painfully slow. You have to set a conservative default so people with less experience/worse ears/whatever can get by.
Anyway I don't think it's a criticism. It's just noting that it doesn't depict how most people will use end up using it, and if you're curious about what typical usage sounds like, you should look for another example.
No. It's not criticism. What they're saying is that the video was shot with a default that a sighted person could understand, because any blind person would naturally have their speed set to much higher than that.
It's like how in videos that teach people a foreign language, everyone speaks slowly and uses simple words, even though native speakers don't talk like that at all. The GP is simply saying that an actual blind person would be way more efficient at it, but they made the video with inefficient settings so sighted people could understand what was going on.
I've heard textual description tracks on television programs before. They come fast, but not screen-reader fast. To the untrained ear a blind person's screen reader sounds like when you somehow get the TI-99/4A's speech synthesizer to read from invalid memory.
The audio description tracks are a different genre than screenreadera perform. They're acting, by actors, carefully written and performed to fit into the gaps in the dialogue while preserving the mood and flow of the show. I think speeding them up or making them robotic would ruin them, while both of those traits are actually desirable for screenreaders.
I think we should reserve judgment until this lands in the hands of the people it helps.
My experience is limited to my elderly parents who have trouble seeing. With the text size Apple allows them to set it to, their phones are unreadable. Text runs off the screen in every app, 1st and 3rd party.
In their bill example, the user is told to confirm with the provider. Why not offer to call the number on the bill? Instead of telling them to use text detection, do it for them? Presumably Apple Intelligence would already have that capability. I’m afraid this will be a gimmick at best.
EDIT: Forgot to mention, the grip is good to see. Hopefully they don’t charge the apple tax on it.
Yeah, I used to use iOS with text one step above the default size, and text was often cut off.
I have a problem with astigmatic halation that makes ‘dark mode’ difficult to read. Since iOS 26, multiple aspects of the system have been made dark only, contrary to the system setting. Writing text correctly should be the lowest of low-hanging fruit.
I suspect this is more of a flashy ‘AI’ promotion rather than reflective of any real commitment.
I had to set macOS on high contrast to be able to differentiate ui elements at glance. But most electron-based apps do not get the hint or even provide a high contrast theme.
I think the above person was making a commentary about the things Apple chooses not to do. Apple strategy is often to be intentionally last to market, after the dust settles.
I agree. There seems to be a lot of potential in this space (from my outsider view). I really hope that this issue from an earlier article (https://news.ycombinator.com/item?id=48178378) doesn't become common enough to make useful functionality like this a danger. Seems unlikely in the short term but as use cases grow, so might the bad actors.
Let's be honest, compare the amount of money a corporation can make helping visually impaired people to the amount of money they can make replacing software developers and financial analysts.
Don't get me wrong, Apple using these technologies to help humans who are in need of help is laudable. But let's not pretend we don't know why most corporations don't look into this kind of thing. I think if we're being honest, we all very much know why they leave this sort of thing to the always nebulous "others".
> “When we work on making our devices accessible by the blind,” he said, “I don’t consider the bloody ROI.” It was the same thing for environmental issues, worker safety, and other areas that don’t have an immediate profit. The company does “a lot of things for reasons besides profit motive. We want to leave the world better than we found it.”
Apple's competitors have had these features for years (Android for 7, Windows for 1), so it's really an indictment of Apple. They give lip service to helping the visually impaired, and this press release is good marketing for the non-visually impaired people who don't know this.
Aren't the LLM-based features of this announcement catch-up features? Describing the contents of the screen is something Gemini has been doing on Pixel phones for a while. It's a fairly obvious use case for a multimodal AI.
My one hope is that this eventually becomes widespread enough to stop alt text scolds.
Seems like everyone skipped over this part, but optical controls for motorized wheelchairs is a cool idea (at least to me, maybe that's an old idea).
Full VR hasn't done well, but it does continue to make me wonder if there's a market for a stripped and slimmed device. I'd maybe be interested in a device that does optical controls if it fit in regular-sized glasses. I'd be super interested if it had a HUD system (even a super basic one that can only show a handful of symbols). Better still if it had some basic audio, but maintaining the "regular glasses" form factor is more important to me than the HUD or audio.
It's been done for a while - follow the links to who they reference. ie https://www.tolt.tech but it's their integration they've done into the OS is interesting.
Seems like a pretty strong indicator that AR glasses are still being worked on, this definitely feels like one of those features Apple ships to refine before the proper hardware is ready.
I lack that confidence here, because it doesn't appear to really involve AR/VR. Eye tracking is the only feature this really uses afaict. The AR seems like a net negative here, it's just the only device Apple has that has a consistent and remotely convenient view of your eyes.
The device is large and makes the user look weird and non-present, which are net negatives.
The only benefit of the AR is showing the directional arrows, but they could get the same thing with much less weird looking non-prescription glasses with arrows sharpied on them. More realistically, anyone really using for mobility probably develops muscle memory for which direction to look to go where and then they don't even need that. At that point it's just a really expensive, really clunky camera.
It’s still useful to get the information instantly and verify it later. Arguably asking someone you trust to read the number for you might be a better idea than calling the company. Not everybody has that option though.
Aaaand the logistics of making that call to the company to confirm the amount on the bill can get awkward. IVR and hold-time hell just to get a human to have to explain your predicament as to why you're asking for such a mundane piece of information that was in fifty other touchpoints that you couldn't access as quickly or easily.
(I'm also picturing the poor CSR at the other end of the phone wading through hundreds and hundreds of call logs over the years for simple requests and managers up above screaming 'why is this guy calling us all the damn time costing us money'...)
Once you paid the same bill for a few months you'll know how much your phone bill will roughly be and you'll not have to do. They obviously have to put that line in there, just like ChatGPT saying "Please verify everything we tell you" in the footer.
It might be useful if it remembered a bill for, say, 60 days, and could also comment on percent difference since the last one. "The total amount due on the bill is $83.89 which is 4% higher than last month's bill from the same company."
They won't, its literally part of their sales funnel. They've specifically engineered a bad experience for anyone outside the ecosystem by making it all of their friend's problem too. Its very important for their stock price that text messages sent by non apple products are just slightly more difficult to read.
"Vehicle Motion Cues come to visionOS, which can help reduce motion sickness for people who use Apple Vision Pro as a passenger in a moving vehicle. Vision Pro will also support face gestures for performing taps and system actions, plus a new way to select elements with one’s eyes while using Dwell Control."
I use them as well. I'm usually the driver so I don't typically look at my phone while the car is moving, but I recently rode along with a family member to an event. They handed me their iPhone to look at something and I felt totally disoriented trying to look at a moving screen in a moving car. I had to resist the urge to turn on the motion cues.
It's really useful for having a decent screen up in front of me when I'm a passenger trying to do something on the laptop. Saves staring down at my lap, and removes any motion on my screen from the peripheral view of the driver.
Still somewhat odd when a bus drives out from behind your Terminal mind.
From the article: "new features for controlling power wheelchairs with Apple Vision Pro"
Someone using this feature will want motion cues as well.
And in your quote: Dwell Control is a feature set to interact with an Apple Vision Pro using only your eyes. Lingering your gaze on a button will press it. An AVP is now more comfortable to use in more situations because of motion cues.
Planes? Trains? If you haven't used these motion dots, they actually do work wonders. My wife gets motion sickness and could barely ever look at her phone when riding as a passenger in the car, even just to type in directions. With the motion dots she does just fine.
A lot of us forget it, but things like text to speech, subtitles etc are there for the differently abled
Without that, there wouldn’t really be great vlm and conversational models.
The AI companies might have paid for the dictation of some videos on their own but voice assistants etc wouldn’t have existed and our ability to have AI that eventually understands the world would be much much harder.
lol I’m saying working on accessibility features has helped more than those of us that are sighted. Often times for a lot of us, it’s a drag and comes lower on the priority list, but without it AI, llms etc wouldn’t have the ability to programmatically understand the world.
> A lot of us forget it, but things like text to speech, subtitles etc are there for the differently abled
They are there for everyone. You don’t need to have a permanent disability to benefit from accessibility features. A device designed to work one handed is useful to someone without an arm or a person with two arms who is holding a baby. Subtitles are useful to someone who can’t hear or someone lying to a sleeping spouse or in a noisy place.
“Accessibility needs can be permanent, temporary or situational.”
These are great improvements, it's good to see Apple investing in improvements like this (especially with the Vision Pro) but I can't help but feel that they utility will remain very low until they make the Vision Pro look significantly less distopian than it does.
The form-factor is a significant issue for real-world usage, and it's kind of unclear if there is a plan for a future product line given its (pretty abysmal) initial receiption.
I don’t think abysmal is the right word. The hardware was widely praised except for being dorky looking and a few other complaints.
The price and lack of content and developer interest have been the main problems.
And ultimately, people just don’t seem that interested in this product category. Meta ran into the same issue, though at least they targeted gaming where there is a decent niche.
VR/AR tech seems cool and futuristic, but hasn’t quite found its killer app yet.
Meta did sell over 20 million headsets. The Quest is definitely lower-margin hardware than Vision Pro, but in terms of install base that's an order of magnitude larger audience.
Apple really screwed themselves by only supporting WebXR for cross-platform VR experiences. Soon Valve will ship the Steam Frame, which will likely cost a fraction of the Vision Pro and support bog-standard PC games like H3VR, flight simulators and flatscreen PC titles. Meanwhile, AVP owners will have paid $3,500 for a more powerful chip/headset with a fraction of the content library and featureset that Valve and Meta offer. Vision Pro's lack of audience is entirely a self-imposed failure, it seems.
On-device video subtitles generation is exciting, should help with watching videos on mute. This seems like a low hanging fruit that should've already been grabbed by an app but I can't find any.
Honestly as a blind person and blind developer myself, most of these features get a shrug at best.
For one, there's already a bunch of third-party apps that do most if not all of this (Seeing AI, Envision AI, BeMyEyes, Aira, etc.). So at best, this does what all those apps are doing but faster and on-device, which may or may not mean it is also more inaccurate, we'll have to see.
In the meantime, Mac OS's screen reader, VoiceOver, has been left to essentially exist in maintenance mode for years, where users have had to build, arguably impressive, third-party solutions to add features to the thing that comparable screen readers on Windows have had for a really long time.
Through that lens, this all looks a bit performative to me, but again, maybe I'll be pleasantly surprised.
The one thing I'm mildly excited to see is the improvement to Voice Control, as guessing what the programmatic name of a button is or having to constantly use a numbers grid to target elements doesn't sound fun.
To respond to what I see in some of the comments:
- On speech rate: It does take quite a bit of practice to crank up the speech rate and there's a degree of retraining you need to do when you switch voices. A lot of more "human" sounding voices are harder to follow at super high speeds which is why a lot of people prefer more robotic but consistent speech and generally aren't convinced by AI-powered TTS yet; they often fall apart if you raise the speech rate past a certain point.
- Re: actually waiting for the target audience's verdict: This is so important. I see more and more companies, individuals etc. talk about accessibility, build accessibility solutions and evangelize AI for accessibility without EVER talking to the people they claim to help.
This will almost certainly mean mistakes will be made, up to and including doing more harm than good.
If you want to do accessibility right, that includes AI products of any kind, hire people with lived experience or you'll get the equivalent of machine-translated text, hackerproof security in one click or an AI-powered coffee bar that orders thousands of rubber gloves.
Coincidental note: I have time for new projects right now :P
Are these already available? I regularly read these announcements and years later I still don't know where to find them, or are not actually functional.
Accessibility features are such a great way to keep technology focused on real-world problems and real-world experiences.
I think the trap in creating anything is doing it for a crowd. Art, software, anything... it turns out better when it is made with a specific, named individual in-mind.
Accessibility features are almost always championed and field-tested with one specific loved one in mind and I think that's what keeps the technical solutions personable and grounded.
These are pretty helpful features for differently abled people. I think it would be really cool if Apple made AI glasses that could communicate with the iPhone thus eliminating the need to point your phone at everything (especially, if you are moving outdoors or in a crowd).
I'm not blind but I sometimes I can't process where things are, even if in front of me. Would be cool to just point to a messy table and see where the keys are. If they offer this as some Vision/Core ML feature, I'd implement the messy table app as soon as these features land. Probably already possible, but simpler if they release this.
> A new power wheelchair control feature leverages the precision eye-tracking system on Apple Vision Pro to offer a responsive input method for compatible alternative drive systems. [0]
The above caption for Apple Vision Pro is for a video that to me, as an Apple Vision Pro user, is discomforting.
More questions are raised than are answered by the short video: Is the user able to fit the Apple Vision Pro by him/herself? What happens when dwelling on a directional control misregisters? Can the user recalibrate the "Eyes and Hands" setting? Dwelling on a control displaces focus and there may be impeding objects in the path of the power wheelchair. Is this really a good idea?
To my sensibility, the video is unsettling (at best), especially given how cumbersome Apple Vision Pro is.
Your concerns are completely nonsensical. It's clearly being marketed as a healthcare tool for people with debilitating injuries that preclude the use of hand-powered wheelchair controls, severe situations where there's no neck-down control and users would be limited to controls like head-tilt or mouth actuated systems. These people obviously require daily care to simply get them out of bed and into the chair and back again every single day - their nurse could just put on their Vision Pro for them! This seems like an incredible leap forward for people in this situation, if they iterate on this and it gets better then this could be a very viable wheelchair control system in the future.
Putting aside the fact that no company should have direct access to anyone's brain, how cool would it be to be building toward VISOR (from TNG) instead of this. If we could translate sensor signals to the neural circuitry of the brain directly, we wouldn't even need an LLM in the mix. But to have it as an overlay, as supplementary data! With the ability to turn it off of course. (Would a person even be able to turn it off? In the same sense as whether someone can "turn off" social media?) If only we had meaningful human rights and institutions that really protected them... I still can't fully give up the techno-optimism that made me love tech in the first place (and TNG for that matter).
As Apple shifts towards services and fancy software features, I wonder how do they expect to stay competitive by only releasing them for a subset of languages.
I have difficulty trusting this. There are plenty of videos online of LLMs making up stuff like "I just ate a hot dog, is there mustard around my mouth?" "No, everything is clean" while there is a big yellow stain om the guy's face
The problem is using a language model to assess images.
Probably 80% of "LLM's are below expectation" complaints (from the general population) involves some form of image analyses.
Image tokenization is hard because unlike language tokenization, where every token is extremely dense with meaning, image tokens tends to be meaningless or irrelevant but are processed all the same.
Give an SOTA LLM a picture of toothpicks and ask it to move one to make a square, and it will probably struggle and fumble it. But give a mid-size LLM from 2 years ago the same problem in verbal form, and it will nail it almost every time.
That takeaway is, do everything you can to avoid having the LLM need to rely on images for the answer.
Like coding, creating images or text, maybe the alternative of doing it yourself is too easy or enjoyable for you. Don't expect that will be true for everyone.
Maybe on a smartphone, but usually not on a computer. Keyboards are pretty good.
The other thing is that if you're around others, voice input means you have no privacy. Even if you're not doing anything particularly private, it's a bit awkward and potentially embarrassing. If you use touch input in conjunction with a screen reader, you can be more like a "normal" user in that what you're doing is just between you and your phone.
I feel that if we build the UI for blind users first, we would get much more powerful systems, rather than building UI for the seeing users and then slapping a CV to text model of what's shown on the screens.
Did not test it yet, but blind users may be more prone to dominate Command Line Interfaces, which are becoming increasingly popular due to its easy integration with LLM
Kudos to apple for providing some of the best accessibility features across their devices. I’ve always appreciated the consistency of reduce transparency, increased contrast, reduced motion, reduced white point, touch areas, color blindness support. And they work well across third party apps. That demands a lot of effort on the API and UI framework to have broad support for something that is mostly a non-sellable feature.
Most apps have terrible accessibility labels because developer don't bother, which breaks every screen reader pipeline downstream. The Voice Control "say what you see" feature routes around that by letting users describe a button in plain language. That's a real fix for a problem caused by humans being lazy about ally.
And this is why androidlets will never win. They’re too busy selling your data to ever think of disabled people or usability.
iOS is just painfully good. I can pause a video, put my finger on text inside the video, and copy it. Until they added it, I didn’t even know how much I needed that.
I'm super glad that they're doing this, but once again unexcited for another decade of Apple self-privileging on this stuff so they're the only ones allowed to touch or improve any of this surface, or UX outside an app's tiny box.
People talk a lot about how MacOS has gone downhill but I feel like it would have been a good start if developers could continue to patch over Apple's shortcomings like they used to be able to.
I imagine that we would be a few years into a spectrum of tools like this if they didn't lock it down like they do.
Totally aware that plenty of HN commenters are very glad that Apple keeps this locked down. I'm just the other opinion, that's all.
I don’t want to discredit more advancements in accessibility, but this feels like accessibility porn.
I have fond memories of an old coworker 10 years ago who is blind. He would use his phone no problem, texting, going about his day, he was even on Tinder (credit to Tinder for making their app so accessible long ago). He would commute on his own, walk to the train station, even transfer to another train during peak rush hour. I’m not saying it was all easy for him, but nothing in this video really stood out to me more than what shirt was on the bed. I know other services/apps have long existed to be the “eyes” for people who need support, but this video feels….uneventful?
I may be cynical about this though, as I often hate how Apple’s marketing makes these emotional bids about how life-critical they are to society - which is fair to a degree..but it just feels cheap to be glamorising “look! we saved this person from pending doom, cool right??”
for every person like your coworker, there are probably several who have a much harder time with technology and who would benefit from a simpler interface.
If this includes improvements to the screen recognition feature in Voice Over, it could provide accessibility for apps where the developer doesn't care about accessibility, which is extremely common.
The vision capabilities could be useful if they are done well, but I suspect that will always be covered better by 3rd party apps.
I mean even if it is marketing for them they still did the work and developed these features. I had some vision issues recently and was glad there were options to make text more legible to me.
Additionally I don't believe this is just marketing. This is adaption to a changing market. Apple's customer base is aging and having these kinds of features will allow them to keep using Apple products for a longer.
They have done the work, but I don’t see much work that’s beyond what’s been previously capable without Apple Intelligence. The marketing of Apple Intelligence is weak here, not the foundational abilities.
Apple loves to stealth test new tech in full public view by sneaking it into relatively mundane places, so debuting agentic AI via accessibility is very on brand.
A few other examples:
- The Touch Bar was much more than an OLED strip, it was Apple’s first move in the transition to Apple Silicon on macs. The Apple T1 chip in the 2016 Touch Bar MacBooks was the first solely Apple-designed processor to appear in a Mac and took over several responsibilities away from intel chipsets like power management, fans, sleep/wake, access to the camera & mic, and the secure enclave powering touch ID. Then the T2 added encryption of the SSD, audio management, image processing for the camera, and prevented tampering with the boot process
- The iPhone 3G shipped with a Liquidmetal SIM eject tool, which is made from a strong custom metal alloy which is "practically unbendable by hand unless you want to hurt or cut your fingers." Although Apple hasn’t released anything with the alloy since then, now nearly 20 years later Apple is rumored to be using liquid metal in their upcoming foldable iPhone.
- RealityKit had 3D scanning and a lot of other cool AR capabilities for years which didn’t make sense until the Apple Vision Pro was released.
'liquid metal' sounds cool. It's probably a metallic glass. I super dislike that it seemingly will be synonymous with the brand name by Apple even though that stuff has been around for decades.
Not that there are particularly many places where this is used - mostly because it really is just very expensive. In the awesome position that Apple is in, economic feasibility is so much easier to achieve, with like tens of millions of guaranteed parts to be preduced.
Also their first own modem, shipped in their cheapest tier starting with the iPhone 16e.
And their upcoming smart glasses are the best UX for almost everything they showed the user holding up her phone for.
I read their glasses when taking video or pics the lense will light up and or flash more prominently then Metas. Maybe that will help the whole privacy issue and also it's not Meta (do love my Meta or smart glasses as a whole will ditch Metas for Apple quickly as both pair of Metas broke & there's no store for support).
I also am looking forward to Apple smart glasses. I use Meta glasses because thre useful since I'm totally blind. I'd much rather have on device recognission though if possible. I also trust Apple more then Meta. At least I'm technically enclined enough to realize I shouldn't wear them in the bathroom or bedroom. At least when humans look at my AI prompts there maily seeing food boxes or computer bios screens.
Same for me! Both pairs of Metas are now inoperative because I can't get them back to factory reset. Once you pull that little blue tab out of them when they're new, it's GAME OVER.
I’ve had excellent luck with their warranty process.
There is a factory reset procedure, I think you hold the shutter button while switching them off and on again.
Dang, really? Why would that be? I mean, I believe you, I'm just shocked. I'm glad I read this, as I was thinking of getting some but not now.
Man I miss the touch bar. Never got why people hated it so much
The hate was because it replaced function keys many people use by tactile touch, without looking. Doing the same on a touch screen is very difficult.
If the bar had been added on top of those, I don't think there would've been the same kind of hate for it.
I didn't really mind the fn keys being there. I rarely use function keys unless I'm RDP'd to a Windows machine.
What drove me crazy though was the escape key. They later added the physical escape key back but I think at that point it was a bit too late.
Adding to the list of grievances, the functionality and the options it presented differed from app to app. I understand that the function keys also change their function app to app, but the visual noise the Touch Bar (wow - the word even gets autocorrected to have the right capitalization!) added as you switched between apps was too distracting.
I’ve always been a “remap capslock to escape” kind of guy (vim), so I didn’t mind much. Access to the brightness (screen and keyboard) and volume slider was neat but superfluous with the OG fn keys. Context-driven controls were probably the best thing about the touch bar, and I don’t think it got enough love to make that stick.
Ah yea, I've only owned one with the physical escape key. That would be annoying.
1) First generation made ESC button a touch button. Aside from ergonomics (or lack of them), at least for me, on a psychological level "abort" button needs to be something you can smash. Also, macOS already had the worst input handling under load, making it virtual button made it worse.
2) While "happy path" on macOS pretty much never requires you to use Fkeys, but my workflow does. Blindly using touch buttons is harder than real buttons.
3) I'm not huge media keys users, but I bet #2 applies here as well.
I liked the touchbar in every other sense. If it was just an addition to an existing keyboard, people wouldn't have hate it[];
[]: At that time it was hard to not be frustrated using mac (butterfly keyboard etc), so touchbar might have gotten more hate than it deserved because of overall frustration.
Also, the Touch Bar seemed to be abandoned as soon as it launched. It only ever launched on the Pro line. There were never any feature updates. They never made it flexible enough for people to customize it.
> They never made it flexible enough for people to customize it.
I feel like it was fairly customizable - the Mac system settings let you do a lot of drag and drop of controls, and I recall iTerm having a similar interface for customizing the bar in its own settings.
I do think it should’ve been given a lot more love, but that’s Apple for ya I guess
I think a bigger issue was that so few applications used it in cool, interesting ways. It has the same appeal as the oled button boxes some people have, except it’s right there on the deck… but nobody did anything with it.
I sure did prefer the media controls on it, though. I still have a 16” here and am reminded of what could have been.
It was extremely flexible in customization options, and there were SDKs to make it do additional cool stuff in apps, but nobody really cared for the most part.
I honestly think it was mostly a "we have a custom secure coprocessor now, what can we do with it?" sort of thing, which also worked out for Touch ID and disk encryption.
I didn't like it, and was happy when they got rid of it. But I didn't hate it.
I did hate the butterfly keyboard that was introduced at the same time. Probably Apple's biggest hardware mistake of the past 15 years or so.
One more on top of others. Many people felt it was a solution in search of a problem. As in, there was no problem i had that it solved. And it was forced on us, in place of something useful. From the start i read that as: This wont be here in a couple years. Which then made it annoying to deal with in the meantime (the hate).
Things that stick around, are generally value adding across a large or complete subset of their users. Touch bar was always niche, and thus always doomed. I think a good counter comparison is Apple VR headsets. For me, i have no use and little interest. But i can see them as a hedge at the very least, or as an enthusiast entrant into an emerging market, where future products in that segment may become interesting. And on top, it doesnt impact me - i can ignore their existence until it becomes useful.
If touch bar were launched like VR, i suspect it would have gotten similar level of dismisals, but less hate.
One thing Apple really needs to get right is speech to text transcription. They've nailed accessibility in so many ways and yet it feels like they're a decade behind on properly transcribing voices. At least half a decade.
Input on the iPhone is so dreadful nowadays. Their palm rejection is definitely worse than before, so mistyping is more frequent. Their text-correction algorithm for typing is worse than before, and it frequently makes incorrect corrections to words that I don't notice, because they change words a few words back from where I typed. And STT hasn't improved. On top of that, my fingers are tired of the phone form factor. Please make the iphone not a chore to use, apple.
Until siri can reliably handle "Navigate to <business that is a decade old>", offline and using pre-downloaded maps, I'm going to assume all the other, harder speech to text and conversational stuff is just vaporware.
I found another dreadful iPhone input "feature" yesterday. If you are browsing around in third party carplay apps, and ready to tap your selection, but instead press the accelerator first, it truncates the list to only a few items, and scrolls to the top.
Way to reduce driving distractions guys! What's next? If the car is moving, maps changes destinations?
I really wish human computer interaction research were more broadly applied, and if you do dumb stuff like all of the automotive / carplay world, then you'd be liable in court.
I once had a car that hid the backup cam behind a legal disclaimer every time you turned it on. I'm sure at least one pedestrian was hit by a car in reverse while that screen was on. The manufacturer should be 100% liable for the poor UI decision.
Wispr Flow is a masterclass in STT. Apple's solution feels like it's from the last century in comparison. Same applies with Apple's TTS when you have ElevenLabs and OpenAI running laps around it. All I need is for my iPhone to do those things natively at the same quality level (because in Apple's walled garden that's the only way to get them usable everywhere).
But Apple's uses so few system resources and runs fully on device on newer iPhone models (16+ I believe). It's so efficient. I really enjoy using Handy with Parakeet as the model, but the system resource usage is a monster compared to Apple's (although very good).
Looks like Wispr Flow uses a cloud model [0]:
> Cloud based speech processing infrastructure for 1B users
It gets to be a messy comparison because my iPhone can do STT with no latency pretty well fully on device, but Wispr Flow requires a cloud model, but to be fair, older Apple devices do as well. It's not an apples and oranges comparison, but I think those technical details make this a non direct comparison in a few ways.
For on-device with low system resource usage, Apple's is pretty damn good.
[0] https://wisprflow.ai/post/technical-challenges
Apple's stt has been on-device for a long time now, long before iPhone 16. I haven't noticed any improvements since my first ever iphone 5S. I'm pretty sure wispr flow can use on-device models. I use Voiceink[0] which can use parakeet models on-device and can optionally use cloud models.It's like night and day comparing Apple's to Voiceink. The only advantage I find to Apple's stt is less friction. 3rd party apps just can't integrate as smoothly with the system. There's a gesture to activate Appledictation when Voiceover is on.
Apple runs on-device on older models, too, just wimpier models.
human resources (my voice and time) are far more valuable than the system resources. going to the cloud is absolutely worth it to prevent a typo
That doesn't work if you have limited or no connectivity (e.g. on a mountain range). There are also privacy concerns, e.g. a doctor using it to transcribe medical information.
FWIW - I also really like Wispr Flow, but I moved to running the 'Whisper Large' model locally using Handy (https://github.com/cjpais/Handy), which has been essentially as good, while also having lower latency.
Handy is great. It exposes a bunch of open models beyond Whipser too, and though I haven’t tried too many of them, I’ll throw in a rec for the Parakeet model which feels pretty much on par with Whisper for accuracy and is way way faster.
I don't think things have improved much on that front since Colin Hughes gave a run down on Voice Control's problems several years ago
https://www.theregister.com/on-prem/2023/08/16/those-who-rel...
Would be great if they could at least fix two major bugs:
* input simply fails (seemingly) randomly where it is supported and many apps from major vendors don't support dictation input at all (e.g. OneNote) (there should at least be a fallback (a la Dragon Dictate from decades ago) for those cases * capitalization is still random leaving you with many errors to correct
but Apple mostly seems to see accessibility as something to use to enable performative press releases not actual functionality...
It’d be amazing if speech-to-text could take into account context as well: Greek if I’m speaking Greek, Korean if I’m speaking Korean, or for (int i = 0; i < count; ++i) if I’m dictating code.
Apple dictation on MacOS is actually pretty dang good. I've got it bound to a double-tap on fn and I use it pretty regularly.
try wisprflow and then tell us it's good
I just installed this and already despise its pricing model. I trust this product approximately zero.
there are plenty of free alternatives using the same models
I use Aqua Voice because Apple STT is so frustrating.
There’s so much complaining about their keyboard issues, and it’s really an infuriating part of the iOS experience. The phone being hard to grip/slippery doesn't help, no…
A while ago I signed up as a sighted person on Be My Eyes. I didn't get as many calls as I had hoped, but I was glad to help out on the few that I could. One call was to read envelopes of incoming mail, another was to read pill bottles, and then there was the two funny guys on big cozy chairs with shopping bags of cereal boxes and wanted to know what was what. I remember one guy really didn't like one type. The app had a unique feature for the sighted person to turn on the camera of the vision impaired person.
https://www.bemyeyes.com
I still have the Be My Eyes app installed but haven’t had a call in over a year - I think it’s a testament to how powerful AI vision models have become. I find it cool that AI works well enough for vision impaired people to solve all their problems.
However there was something very human and nice about helping out a stranger with a small random task from time to time. I fondly remember one older lady who spilled a box of blueberries on the kitchen floor and I helped her hunt them all down by guiding her around. It was 10 minutes of connection with a random person doing something fun and which is till remember fondly 4 years later
Ever since Be My Eyes introduced there AI features it's my understanding that there's been a lot less need for volunteers. I'm totally blind, and started using the app once they added in AI. It works great for for things like reading food labels after my kids have movved things around, determining if the tv has been left on, etc. I think I would use the volunteer feature if I still lived alone, but I don't.
I haven’t had a single alert from it since the AI stuff rolled out
> I didn't get as many calls as I had hoped
They always had significantly more people willing to help than in need of help. Which is good, not going to knock that. I signed up for it many years ago but didn’t get a single call, so eventually I just uninstalled it.
Who called you? Blind people?
That’s the app. Or vision-impaired, at least.
But in practice it could just be anyone whose outsourcing vision right? It's basically a free manual vision service.
There would have to be a person pretending to be blind on the other side of the phone call, recording your answers. At that point is it not just easier to have that person do the labeling themselves?
In practice, it's also to help vision impaired people
Are you implying that it might be exploited by unethical tech bros looking to build data sets?
It is a very manual process and would probably be an inefficient way to collect that data.
Fun fact: This video was made accessible to sighted people because no blind person would ever listen to voice at that speed. Honestly if you ever observe a blind person using computers you'd impressed how they can listen to audio at unimaginable speeds.
https://youtu.be/wKISPePFrIs?si=ahGfFp0U7-pTU9w6&t=43
my go to example of this is this talk by Saqib Shaikh (a blind software engineer at Microsoft) giving a talk about Visual Studio. Link is timestamped
I think it takes quite a lot of practice to reach this speed. It's not rare among blind developers, but I think it still takes a lot of work to get there. Pretty impressive!
I wish more people would watch videos like this just because having a realistic idea of how blind people do certain tasks can help you move from pity or even compassion to a more productive kind of understanding. I think sometimes when you haven't seen it, you can't really even imagine how it can be done.
I listen to a lot of podcasts and listen at 1.5-2.0 speed and it’s to the point that I literally cannot stand listening to 1.0 speed anymore as they go too slowly (depending on the content of course).
Same. Returning to 1x speed makes people sound (to my 2x-abused ears) drunk and slurring their works. If I want to listen to something slowly and carefully, I will just about tolerate 1.25x.
What really frustrates me is watching/listening to discussion of music, because I am forced to listen to the talking at 1x because the music sounds wrong (and is wrong) at anything other than 1x.
The funny thing is that slow talkers sound normal at 2x speed. It's jarring when you hear their actual speech.
I listen to podcasts at 1x. But there are a few people I've done podcasts with that I do various audio tricks to speed up.
Playing music at 1x should be a pretty simple feature to add to those apps.
Ideally it should be done while encoding.
I'm so glad YouTube and other podcast players have moved to support 3.0 speed. As I get comfortable with one, I move it up some. For things like sports and "did you know" content, I can go 2.5 if I'm not multitasking. For technical content, sometimes I'm stuck at 1.0.
You can get browser extensions to do it for all media controls on any site. YouTube's "Premium" for 3x is laughable when it's an internal browser function.
Another fun thing is if you use an extension you can fast-forward through the advertisements too. For some channels I use around 3.5x playback speed.
Ublock origin blocks the ads entirely on Firefox.
Premium is for up to 4x, not just 3x
That’s an amusing observation.
Likewise, YouTube’s “premium” feature of not displaying ads is laughable when displaying content is literally an internal browser function.
I pay anyway, because I was going to pay for an on-demand streaming music service anyway.
Something that the Overcast podcast player does (and probably others) is silence removal, which in some ways is even better than the raw speedup.
Same haha. But for me 1.5x is the sweet spot. Anything more and I find myself rewinding a lot. I want to feel comfortable absorbing info and not on constant alert.
I am jealous. I can't listen and retain most podcasts at more than 1.0x. I even disable the podcast player functionality that eliminates pauses and silent sections.
I do the opposite in a few. There's some I follow weekly and it's only an hour or so each. I drop it to .7 or .8 because I want to get a bit more time with the hosts. Possibly stupid but I've sort of got used to some of these folks at that speed, and the normal speed is 'weird'. One is a political podcast, and when they play clips of Trump, he does always sound very drunk, but the hosts themselves (to me) don't sound drunk, just... measured. Some of it may be audio quality - I'm getting their microphone directly, often the audio clips are from field recordings.
Except Marc Andreessen, I can’t decode his speech at 2x
Maybe it’s just a matter of practice.
> It's not rare among blind developers
It's not rare among the blind in general.
Unless you're completely technologically illiterate, the kind of person who has no idea how to install an app or sign up for an online account, you're probably doing something of the sort.
If you are dedicated, few weeks to few months of usage with regular ramp up. You should be careful with adjusting which symbols are read though and sometimes the programing languages matters because different symbols have different significance for understanding the code.
Ho-ly cow. That is very impressive.
I'm not even sure what to say, but discoveries like this are why I use hackernews, I'd never have known this otherwise.
To be fair, the acoustics of the room that talk was given in are... not too great, to put it mildly.
I can easily understand Eloquence (the speech synthesizer he's using) at that speed, but I struggled a bit with this one.
Woah, this is really cool to see
I did IT for a community Center way back in the day and the director was blind. I was blown away by how fast his screen reader read things out to him - completely incomprehensible to me - and his efficiency with keyboard shortcuts would put even vim/emacs elitists to shame.
The way (Windows) screen readers handle web navigation is basically Vim in disguise.
You have two modes: "focus mode", where you can edit text in text fields and keys are passed straight to the browser, and "browse mode", where keys move a virtual cursor around the page.
In browse mode, navigating with just arrow keys all the time would be just as slow as you might imagine, so you use single-key keyboard shortcuts to move by role, E.G. to the next heading, button, table or unvisited link.
The keyboard layout is optimized for memorizability and not efficiency, you use the actual arrow keys instead of hjkl for example, but the concepts are eerily similar.
There are a couple of other approaches to solve this problem, Mac OS's Voice Over is much more Emacs-like for example, and each approach has its own pros and cons, but that's definitely one way to do it.
Probably because it's an advertisement, and super fast robot voices can feel extremely harsh and annoying. Even blind people who rely on them find them overstimulating sometimes, lol.
Indeed, and not just fast, but often heavily robotic (which many sighted people struggle to understand even at 1.5x). I remember reading about a blind person who learned how to do echo-location using sound, and it seemed like such a cool superpower, that one of these days I'm going to take the plunge and unplug my monitor and start learning how to really use the tools. I worked with a blind person a few years back who got almost double the battery life from his laptop as the rest of us by having the screen off all the time, so that alone would be a nice feature. I may never get to the epic level of echo-location, but if I get even half-way there it would be awesome. With a bonus of being able to actually QA a11y changes.
> echo-location
We all do that, I mean unless you’re hearing impaired.
Everyone’s familiar with dropping a coin or such and knowing exactly where it landed without looking.
That’s more passive sonar though.
Do I recall seeing videos of guys mountain biking and making a hissing sound for an active sonar style echo location?
Or am I making that memory up.
> blind person who learned how to do echo-location using sound
RIP kid https://youtu.be/fnH7AIwhpik
I’m not gonna watch that as I’d rather stick to my head-canon that he had an altercation with a dolphin.
:) IIRC that video would have been fully produced/published during his lifetime (but 100% would have to avoid the comments!)
If he’d like your humor I like it too :dolphin:
Twenty years ago I took a level 1 tech support call from a visually impairment guy and it took about 3.2 seconds to realise his condition was no impediment for using a computer because of the screen reader tech he was using.
I listen to a lot of podcasts and YouTube videos at 3x or 4x speed now, having slowly built up the skill over a few years. It's pretty nice now and saves time, and it's remarkable how well the human brain can adapt to such input.
I’m the opposite, I can’t stand the fast speaking videos. But I also speed up 1.2x to 1.5x if the videos were too slow.
I’m struggling to understand your definition of opposite here.
Wouldn’t opposite mean you listen at sub 1x speed.
Whereas as your definition seems to be ”I’m the same, but less so.”
You recall nothing and you know it. You're just wasting time you could use for something useful or meaningful in your life. Kids call it "Anxiety cope" but I don't agree.
Maybe you can't but I can recall whatever I need to.
Can you recall 3 lines of dialogue from the latest movie you watched?
If you're listening to a podcast at 3x you're trying to learn something. No one is trying to learn watching a movie.
> Honestly if you ever observe a blind person using computers you'd impressed how they can listen to audio at unimaginable speeds.
Even better, fire up Orca (or whatever screenreader application your OS comes with) yourself and try to use your computer while shutting your eyes, kind of eye-opening (no pun intended) what kind of experience these sort of users typically get. And also, you quickly start to understand why they set the speech rate for their voice synthesizer to be so fast, it's almost unbearable navigating applications (and particularly lists) otherwise.
When I was at Google, I'd periodically test our (internal-only) app with Chromevox with the display off. It's not that it sounded like it would be easy, but it really is a challenge, and I can only imagine the muscle memory built up over time of trying to work around accessibility bugs and strange behaviors.
Unfortunately it seems impossible to get all that much funding for accessibility work :/ I wonder what ever happened to the Newton accessibility bus intended to supplement Wayland...
I’ve worked at Apple Facebook and Google. Apple was the only one that made a11y bugs and a face to face consultation with a blind developer to show you how your app sucked, mandatory before you could launch.
> I wonder what ever happened to the Newton accessibility bus intended to supplement Wayland...
Hm, never heard about it, but now I'm wondering too. I just finished implementing proper accessibility support for my native app toolkit for Linux, macOS and Windows, but only done it for X11 so far, I was just gonna get started with Wayland. What is the accessibility story on Wayland, couldn't people rely on the same protocols as with X11? That was my impression, but haven't really dig into yet.
The muscle memory build-up is definitely real.
There are apps I use semi-regularly that less-experienced screen reader users thought were inaccessible, and I couldn't even explain what they were doing wrong from memory. The ways of working around accessibility issues are just so ingrained in me that all I can usually remember is "yeah I did this somehow, but it was six months ago and I have absolutely no idea which specific tricks I needed for this one."
That time my Mac display broke and I had to log in taught me much about how important learning accessibility is even for non blind people.
> you quickly start to understand why they set the speech rate for their voice synthesizer to be so fast, it's almost unbearable navigating applications (and particularly lists) otherwise.
I imagine that for coding it also helps deal with the fundamental problem of an ephemeral stream rather than a persistent document that you can navigate visually in multiple dimensions. Working memory is limited, and getting more text in in a short period of time probably helps you work within that better. I also imagine that working with text via audio all the time gradually stretches and improves memory.
It's not the ephemeral stream that's the problem, it's the limited bandwidth.
You can show a lot more info on a screen than you can transmit through speech in a short period of time. That doesn't mean you read faster than you listen, just that sighted people essentially use their eyeballs as an "input device" to decide what information to look at.
If there's an object on the screen that you want to examine but that you don't need to click, you can just "navigate to it" with your eyeballs, without ever touching a mouse or keyboard. We don't have that luxury.
This means we need a much more efficient system for navigating what's on the screen, but that only gets you so far. Eventually, the easiest way to deal with this problem is just to increase the bandwidth of your channel, and you do that by increasing the speech rate.
The difference is that the voice in the video is a natural, human voice. It's the robotic sounding voices that always pronounce the same letters the exact same way (mostly the Eloquence family of voices) that enable blind people to listen at superhuman speeds. You can't listen to a real voice that fast.
I know plenty of blind people who have their voice speed unbearably slow and barely scratch the surface of what technology can do for them. I think an interface where you can tell your phone what to do in natural language will really help a lot of less technical people.
I'm not getting my hopes up though given apple's history with Siri, which is truly awful.
Apple's history with accessibility is, on the whole, pretty good. I strongly suspect that the "coming soon" part of this means "after we integrate Google Gemini models into the system," so I don't think you should use the current state of Siri as a yardstick. (I actually have decent luck with the current Siri, but I don't push it very much and have sort of adapted myself to its limitations; on the flip side, I have a lot of skepticism around LLMs, but they're really a quantum leap in natural language processing capability over what came before, and the use cases they're showing here seem to be right in the LLM wheelhouse -- with the asterisk of "you're still always going to have to check its work.")
> I strongly suspect that the "coming soon" part of this means "after we integrate Google Gemini models into the system…"
I don’t think the Google's tech has anything to do with these features.
This would had to have been in the works long before the Google announcement. Also, these are enhancements of existing iOS and macOS features. They don’t require an LLM anyway; these features use Apple’s Machine Learning models.
For example, creating subtitles for videos? iOS 16 introduced Live Captions for FaceTime calls in 2022 [1].
[1]: https://www.apple.com/newsroom/2022/05/apple-previews-innova...
Coming soon very likely means iOS 27.
This has been the typical pattern for Apple for the last few years. The flashy features are announced at WWDC, accessibility has a dedicated, earlier press release. Before this practice, accessibility announcements would usually be tucked in some WWDC slide that most people wouldn't even notice.
> accessibility has a dedicated, earlier press release
IIRC, it's timed to land around Global Accessibility Awareness Day (May 21).
https://accessibility.day/
The thing that disappointed me about this amazing announcement was “coming later this year“. They should probably give us dates for a little while at least until we get the (<)$95 checks.
I just would not wanna promise anything. Except “available for download this Friday“ once the gold master is passing tests.
The "coming later this year" language is disappointing to some people, but that's just Apple propriety. Allow me to explain.
"Coming later this year" means it's part of a publicly committed release — iOS 27, macOS 27, etc. — not vaporware.
The annual pre-WWDC accessibility announcement is a tradition, and with the conference less than a month away, expect more detail then. New a11y features have a good chance of appearing in the 10am PT keynote or the Platforms State of the Union, the developer-focused follow-up at 1pm PT.
That said, things are still fluid with three weeks to go — features can be added or pulled at any time. If something gets bumped from the main presentations, there will almost certainly be a dedicated video session covering it.
As for availability: some of these features will land in the iOS 27 and macOS 27 betas, which drop during WWDC for Apple Developer Program members. The public beta follows in July, and there's a free tier of the developer program if you want early access.
Don't expect everything at once, though. Some features won't arrive until the September release candidates — and even then, a few may ship labeled "beta" or "experimental," or hold for a future dot release.
Being able-bodied and sighted is probably the biggest disadvantage for using iOS.
Twenty years and text input & manipulation on iPhone sucks a big fat hair pair of dogs balls still.
The last time I daily drove Android was over two years ago and it was immeasurably less God-damn-I-wanna-dig-Jobs-corpse-up-n-give-the-guy-a-piece-of-my-mind, only problem is his grave is unmarked. Arsehole!
Whenever my sister (blind) and I (visually impaired) visit my mom (blind) we secretly turn up the reading speed on her TV just a little because we can't stand how unbearably slow she keeps it, but if we turn it up quickly, she'll freak out.
After a few more years of Thanksgivings and Christmases and Mothers' Days, we'll finally train her up to a reasonable speed lmao.
This is heartwarming. The audio equivalent to the practice of sighted people fixing the bad default settings on boomers’ televisions each Thanksgiving.
Blind people can't change video speed? The control is available right there.
Yes, the audio speed can be adjusted.
Whether that control you see visually is actually accessible to a blind user is a different matter entirely. Further, it maxes out at 2x, but a blind person would typically screen read at the equivalent of 3-6x.
Huh, 2x is low even sometimes for sighted people.
Related, it seems like YouTube recently paywalled speed increase beyond 2x. Another way in which it's not cheap to lose sight, I guess.
> Another way in which it's not cheap to lose sight, I guess.
True.
We can frame it even more strongly: "default societal practices actively discriminate against people with disabilities; they intentionally, consciously choose to make life harder for people who're disadvantaged".
> Another way in which it's not cheap to lose sight, I guess.
Seems like it would be a win-win to have a user setting to opt out of video in exchange for ungating that feature.
No they are saying that the audio playing for tts would be at like 2.4x what's in the commercial.
I don't get it. The speed of TTS can be adjusted, right?
Pretty sure there's enough blind people who don't listen to voice at insane speeds, because they listen in their non-native second language or for whatever other reason. What's wrong in using lowest common denominator that's 100% accessible to those people as well as people who want faster speeds? Unlike "too fast", "too slow" doesn't get entirely inaccessible, it's just boring.
Such a random reason to criticize for.
I don’t think it’s meant to be criticism. It’s an interesting piece of information that gives a peek into how those with vision impairment consume content. There’s nothing wrong with it; but it was enlightening to consider the experience for those of us who have not been forced to.
Seems like I brought my own negativity into this...
I don't think you did.
Some blind people listen to things at superhuman speeds, but not all blind people. Using a normal reading speed is a sensible choice for an ad trying to appeal to blind people since you don't want to intimidate those who don't use superhuman speeds.
Going from that to "heh a sighted person made this because it's normal speed" is simply incorrect.
It was the sort of statement an HNer might make to showcase some trivia they have about some other group, but they oversold it.
> Pretty sure there's enough blind people who don't listen to voice at insane speeds, because they listen in their non-native second language or for whatever other reason.
Yes, for lots of reasons. It takes practice to get up to a high speed with a given TTS. People who go blind later in life are just beginning, and it can take a long time for them to get up to really high speeds. You may also need to reset somewhat when you change from one TTS to another. And blind people's ears are subject to problems just like anyone else's; if your hearing isn't great you may need slower speeds or higher volumes or both. That's why even though most people use screenreaders at much higher speeds, the defaults when you turn on a new device are painfully slow. You have to set a conservative default so people with less experience/worse ears/whatever can get by.
Anyway I don't think it's a criticism. It's just noting that it doesn't depict how most people will use end up using it, and if you're curious about what typical usage sounds like, you should look for another example.
No. It's not criticism. What they're saying is that the video was shot with a default that a sighted person could understand, because any blind person would naturally have their speed set to much higher than that.
It's like how in videos that teach people a foreign language, everyone speaks slowly and uses simple words, even though native speakers don't talk like that at all. The GP is simply saying that an actual blind person would be way more efficient at it, but they made the video with inefficient settings so sighted people could understand what was going on.
I briefly worked at a call center and I would hear supervisors listening to recorded calls at warp speed.
> boiler call center
What does this mean?
I've heard textual description tracks on television programs before. They come fast, but not screen-reader fast. To the untrained ear a blind person's screen reader sounds like when you somehow get the TI-99/4A's speech synthesizer to read from invalid memory.
The audio description tracks are a different genre than screenreadera perform. They're acting, by actors, carefully written and performed to fit into the gaps in the dialogue while preserving the mood and flow of the show. I think speeding them up or making them robotic would ruin them, while both of those traits are actually desirable for screenreaders.
How did you come across those tracks? Never have myself.
dont you worry, as a sighted person I am also infuriated by apples slooow reading speed, e.g. for "Announce Notifications".
Also as a sighted person, this is why I hate the modern trend of using the video format to show 3-4 bullet points. Just give me the text.
This looks like a genuinely useful application of LLMs.
I wish more companies focused on how they can help humans instead of replacing us or squeezing us as hard as possible in the name of productivity.
I think we should reserve judgment until this lands in the hands of the people it helps.
My experience is limited to my elderly parents who have trouble seeing. With the text size Apple allows them to set it to, their phones are unreadable. Text runs off the screen in every app, 1st and 3rd party.
In their bill example, the user is told to confirm with the provider. Why not offer to call the number on the bill? Instead of telling them to use text detection, do it for them? Presumably Apple Intelligence would already have that capability. I’m afraid this will be a gimmick at best.
EDIT: Forgot to mention, the grip is good to see. Hopefully they don’t charge the apple tax on it.
Yeah, I used to use iOS with text one step above the default size, and text was often cut off.
I have a problem with astigmatic halation that makes ‘dark mode’ difficult to read. Since iOS 26, multiple aspects of the system have been made dark only, contrary to the system setting. Writing text correctly should be the lowest of low-hanging fruit.
I suspect this is more of a flashy ‘AI’ promotion rather than reflective of any real commitment.
I had to set macOS on high contrast to be able to differentiate ui elements at glance. But most electron-based apps do not get the hint or even provide a high contrast theme.
This is what Apple does best.
They treat new industry advancements as technology, not products itself.
AI will be a feature to improve the customer experience, not the product itself.
These features have existed on Android devices for years. What Apple does best is marketing.
https://blog.google/products-and-platforms/platforms/android...
https://android-developers.googleblog.com/2024/09/talkback-u...
I think the above person was making a commentary about the things Apple chooses not to do. Apple strategy is often to be intentionally last to market, after the dust settles.
I agree. There seems to be a lot of potential in this space (from my outsider view). I really hope that this issue from an earlier article (https://news.ycombinator.com/item?id=48178378) doesn't become common enough to make useful functionality like this a danger. Seems unlikely in the short term but as use cases grow, so might the bad actors.
Its with their servers right? Do they trust a iPhone with their life? Or they are trusting their data center?
Looks like some of the features might use on-device models. They mention subtitle generation works on-device.
> help humans instead of replacing us or squeezing us as hard as possible in the name of productivity
Increasing their productivity is helping humans.
Let's be honest, compare the amount of money a corporation can make helping visually impaired people to the amount of money they can make replacing software developers and financial analysts.
Don't get me wrong, Apple using these technologies to help humans who are in need of help is laudable. But let's not pretend we don't know why most corporations don't look into this kind of thing. I think if we're being honest, we all very much know why they leave this sort of thing to the always nebulous "others".
Tim Cook has been pretty clear where he stands:
> “When we work on making our devices accessible by the blind,” he said, “I don’t consider the bloody ROI.” It was the same thing for environmental issues, worker safety, and other areas that don’t have an immediate profit. The company does “a lot of things for reasons besides profit motive. We want to leave the world better than we found it.”
— https://www.forbes.com/sites/stevedenning/2014/03/07/why-tim...
Again, it's absolutely great that Apple does these things!
I was just answering the question of why other corporations don't.
Money.
There's relatively little money in helping the visually impaired. You have to do it because you want to do it. Not because you're going to get rich.
Apple's competitors have had these features for years (Android for 7, Windows for 1), so it's really an indictment of Apple. They give lip service to helping the visually impaired, and this press release is good marketing for the non-visually impaired people who don't know this.
>But let's not pretend we don't know why most corporations don't look into this kind of thing.
I assume almost everyone looks into spending less money than more money for equivalent goods and services.
Aren't the LLM-based features of this announcement catch-up features? Describing the contents of the screen is something Gemini has been doing on Pixel phones for a while. It's a fairly obvious use case for a multimodal AI.
My one hope is that this eventually becomes widespread enough to stop alt text scolds.
"looks like" there are a lot of automated accessibility systems that fall woefully short in practical use
this sort of thing really needs input from someone that uses it before we can judge it
Seems like everyone skipped over this part, but optical controls for motorized wheelchairs is a cool idea (at least to me, maybe that's an old idea).
Full VR hasn't done well, but it does continue to make me wonder if there's a market for a stripped and slimmed device. I'd maybe be interested in a device that does optical controls if it fit in regular-sized glasses. I'd be super interested if it had a HUD system (even a super basic one that can only show a handful of symbols). Better still if it had some basic audio, but maintaining the "regular glasses" form factor is more important to me than the HUD or audio.
It's been done for a while - follow the links to who they reference. ie https://www.tolt.tech but it's their integration they've done into the OS is interesting.
Seems like a pretty strong indicator that AR glasses are still being worked on, this definitely feels like one of those features Apple ships to refine before the proper hardware is ready.
I lack that confidence here, because it doesn't appear to really involve AR/VR. Eye tracking is the only feature this really uses afaict. The AR seems like a net negative here, it's just the only device Apple has that has a consistent and remotely convenient view of your eyes.
The device is large and makes the user look weird and non-present, which are net negatives.
The only benefit of the AR is showing the directional arrows, but they could get the same thing with much less weird looking non-prescription glasses with arrows sharpied on them. More realistically, anyone really using for mobility probably develops muscle memory for which direction to look to go where and then they don't even need that. At that point it's just a really expensive, really clunky camera.
> The total amount due on the bill is $83.89. Please verify this amount with your utility provider or by using Text Detection before making a payment.
1. Use AI to determine how much a bill is for
2. Call up the people who billed you and ask them how much they billed you
3. Pay billed amount
It’s still useful to get the information instantly and verify it later. Arguably asking someone you trust to read the number for you might be a better idea than calling the company. Not everybody has that option though.
And not everybody wants to use that option all the time. Asking a human makes you feel dependent more than using a tool does.
Aaaand the logistics of making that call to the company to confirm the amount on the bill can get awkward. IVR and hold-time hell just to get a human to have to explain your predicament as to why you're asking for such a mundane piece of information that was in fifty other touchpoints that you couldn't access as quickly or easily.
(I'm also picturing the poor CSR at the other end of the phone wading through hundreds and hundreds of call logs over the years for simple requests and managers up above screaming 'why is this guy calling us all the damn time costing us money'...)
Once you paid the same bill for a few months you'll know how much your phone bill will roughly be and you'll not have to do. They obviously have to put that line in there, just like ChatGPT saying "Please verify everything we tell you" in the footer.
It might be useful if it remembered a bill for, say, 60 days, and could also comment on percent difference since the last one. "The total amount due on the bill is $83.89 which is 4% higher than last month's bill from the same company."
I presume that calling customer support is at least as frustrating for people with disabilities as it is for anyone.
It's a shame Apple removed the screen reader announcements ("the Apple logo") from the youtube version of the commercial.
https://www.youtube.com/watch?v=B3SmsSCvoss
Those made the ad stand out in my opinion.
Change the audio language to "English descriptive".
Generated subtitles for video does sound useful. Sometimes actors mumble so horribly, I don’t understand a word they’re saying.
Nice. It would be good if Apple could find the time to improve the readability of the white text on green bubbles too.
edit: it seems that asking Apple to follow their own accessibility guidelines isn't popular on HN :-(
They won't, its literally part of their sales funnel. They've specifically engineered a bad experience for anyone outside the ecosystem by making it all of their friend's problem too. Its very important for their stock price that text messages sent by non apple products are just slightly more difficult to read.
That wouldn't let them say “AI”.
"Vehicle Motion Cues come to visionOS, which can help reduce motion sickness for people who use Apple Vision Pro as a passenger in a moving vehicle. Vision Pro will also support face gestures for performing taps and system actions, plus a new way to select elements with one’s eyes while using Dwell Control."
Maybe just don't wear them in a car?
Wearing a headset in the back of an Uber doesn't sound that crazy,
I use those motion cues on my iPhone even though I don't struggle with motion sickness https://www.youtube.com/shorts/OxbjggMcKrk
I use them as well. I'm usually the driver so I don't typically look at my phone while the car is moving, but I recently rode along with a family member to an event. They handed me their iPhone to look at something and I felt totally disoriented trying to look at a moving screen in a moving car. I had to resist the urge to turn on the motion cues.
It's really useful for having a decent screen up in front of me when I'm a passenger trying to do something on the laptop. Saves staring down at my lap, and removes any motion on my screen from the peripheral view of the driver.
Still somewhat odd when a bus drives out from behind your Terminal mind.
From the article: "new features for controlling power wheelchairs with Apple Vision Pro"
Someone using this feature will want motion cues as well.
And in your quote: Dwell Control is a feature set to interact with an Apple Vision Pro using only your eyes. Lingering your gaze on a button will press it. An AVP is now more comfortable to use in more situations because of motion cues.
Maybe just rethink your "maybe just" comment...?
Trains. Airplanes. TFA said vehicle, not car.
Planes? Trains? If you haven't used these motion dots, they actually do work wonders. My wife gets motion sickness and could barely ever look at her phone when riding as a passenger in the car, even just to type in directions. With the motion dots she does just fine.
Trains are a thing.
>Maybe just don't wear them in a car?
Why not?
Because the more we reject our shared reality and substitute it with each our own, the less humane we become.
The AVP is primarily an AR device, not VR.
God forbid a person rejects the shared reality of a boring 12 hour flight and substitutes it with their own. Some real deep thoughts here
I’ve met some very interesting people on flights. I’ve done some great work. I’ve had some great ideas.
Don’t be so scared of variety. You just keep subjecting yourself to more of the same. The unending familiarity makes you dull.
A lot of us forget it, but things like text to speech, subtitles etc are there for the differently abled
Without that, there wouldn’t really be great vlm and conversational models.
The AI companies might have paid for the dictation of some videos on their own but voice assistants etc wouldn’t have existed and our ability to have AI that eventually understands the world would be much much harder.
So we're blaming disabled people now.
lol I’m saying working on accessibility features has helped more than those of us that are sighted. Often times for a lot of us, it’s a drag and comes lower on the priority list, but without it AI, llms etc wouldn’t have the ability to programmatically understand the world.
You however…. Maybe need to switch to decaf?
> A lot of us forget it, but things like text to speech, subtitles etc are there for the differently abled
They are there for everyone. You don’t need to have a permanent disability to benefit from accessibility features. A device designed to work one handed is useful to someone without an arm or a person with two arms who is holding a baby. Subtitles are useful to someone who can’t hear or someone lying to a sleeping spouse or in a noisy place.
“Accessibility needs can be permanent, temporary or situational.”
https://www.coursearc.com/accessibility-content-fundamentals...
Can Apple “unveil” Touch ID as a “new” accessibility feature because Face ID is an accessibility nightmare?
Luckily the European Accessibility Act has pretty much made PDF/UA a requirement.
This should really be the last resort.
Accessible PDFs are quite rare in reality. Especially if there are tables, graphics, maths, forms or anything more than plane text.
Agreed, it is a problem, but it is being legislated in as required for many businesses in the EU going forward.
These are great improvements, it's good to see Apple investing in improvements like this (especially with the Vision Pro) but I can't help but feel that they utility will remain very low until they make the Vision Pro look significantly less distopian than it does.
The form-factor is a significant issue for real-world usage, and it's kind of unclear if there is a plan for a future product line given its (pretty abysmal) initial receiption.
I don’t think abysmal is the right word. The hardware was widely praised except for being dorky looking and a few other complaints.
The price and lack of content and developer interest have been the main problems.
And ultimately, people just don’t seem that interested in this product category. Meta ran into the same issue, though at least they targeted gaming where there is a decent niche.
VR/AR tech seems cool and futuristic, but hasn’t quite found its killer app yet.
Meta did sell over 20 million headsets. The Quest is definitely lower-margin hardware than Vision Pro, but in terms of install base that's an order of magnitude larger audience.
Apple really screwed themselves by only supporting WebXR for cross-platform VR experiences. Soon Valve will ship the Steam Frame, which will likely cost a fraction of the Vision Pro and support bog-standard PC games like H3VR, flight simulators and flatscreen PC titles. Meanwhile, AVP owners will have paid $3,500 for a more powerful chip/headset with a fraction of the content library and featureset that Valve and Meta offer. Vision Pro's lack of audience is entirely a self-imposed failure, it seems.
On-device video subtitles generation is exciting, should help with watching videos on mute. This seems like a low hanging fruit that should've already been grabbed by an app but I can't find any.
Honestly as a blind person and blind developer myself, most of these features get a shrug at best. For one, there's already a bunch of third-party apps that do most if not all of this (Seeing AI, Envision AI, BeMyEyes, Aira, etc.). So at best, this does what all those apps are doing but faster and on-device, which may or may not mean it is also more inaccurate, we'll have to see. In the meantime, Mac OS's screen reader, VoiceOver, has been left to essentially exist in maintenance mode for years, where users have had to build, arguably impressive, third-party solutions to add features to the thing that comparable screen readers on Windows have had for a really long time.
Through that lens, this all looks a bit performative to me, but again, maybe I'll be pleasantly surprised.
The one thing I'm mildly excited to see is the improvement to Voice Control, as guessing what the programmatic name of a button is or having to constantly use a numbers grid to target elements doesn't sound fun.
To respond to what I see in some of the comments:
- On speech rate: It does take quite a bit of practice to crank up the speech rate and there's a degree of retraining you need to do when you switch voices. A lot of more "human" sounding voices are harder to follow at super high speeds which is why a lot of people prefer more robotic but consistent speech and generally aren't convinced by AI-powered TTS yet; they often fall apart if you raise the speech rate past a certain point. - Re: actually waiting for the target audience's verdict: This is so important. I see more and more companies, individuals etc. talk about accessibility, build accessibility solutions and evangelize AI for accessibility without EVER talking to the people they claim to help. This will almost certainly mean mistakes will be made, up to and including doing more harm than good. If you want to do accessibility right, that includes AI products of any kind, hire people with lived experience or you'll get the equivalent of machine-translated text, hackerproof security in one click or an AI-powered coffee bar that orders thousands of rubber gloves. Coincidental note: I have time for new projects right now :P
Unfortunate thumbnail on that embedded video
Are these already available? I regularly read these announcements and years later I still don't know where to find them, or are not actually functional.
Accessibility features are such a great way to keep technology focused on real-world problems and real-world experiences.
I think the trap in creating anything is doing it for a crowd. Art, software, anything... it turns out better when it is made with a specific, named individual in-mind.
Accessibility features are almost always championed and field-tested with one specific loved one in mind and I think that's what keeps the technical solutions personable and grounded.
These are pretty helpful features for differently abled people. I think it would be really cool if Apple made AI glasses that could communicate with the iPhone thus eliminating the need to point your phone at everything (especially, if you are moving outdoors or in a crowd).
Didn’t they already have subtitle generation for uncaptioned video?
Edit: was thinking about this feature https://support.apple.com/guide/iphone/get-live-captions-of-...
I'm not blind but I sometimes I can't process where things are, even if in front of me. Would be cool to just point to a messy table and see where the keys are. If they offer this as some Vision/Core ML feature, I'd implement the messy table app as soon as these features land. Probably already possible, but simpler if they release this.
> A new power wheelchair control feature leverages the precision eye-tracking system on Apple Vision Pro to offer a responsive input method for compatible alternative drive systems. [0]
The above caption for Apple Vision Pro is for a video that to me, as an Apple Vision Pro user, is discomforting.
More questions are raised than are answered by the short video: Is the user able to fit the Apple Vision Pro by him/herself? What happens when dwelling on a directional control misregisters? Can the user recalibrate the "Eyes and Hands" setting? Dwelling on a control displaces focus and there may be impeding objects in the path of the power wheelchair. Is this really a good idea?
To my sensibility, the video is unsettling (at best), especially given how cumbersome Apple Vision Pro is.
[0] https://www.apple.com/newsroom/2026/05/apple-unveils-new-acc...
Your concerns are completely nonsensical. It's clearly being marketed as a healthcare tool for people with debilitating injuries that preclude the use of hand-powered wheelchair controls, severe situations where there's no neck-down control and users would be limited to controls like head-tilt or mouth actuated systems. These people obviously require daily care to simply get them out of bed and into the chair and back again every single day - their nurse could just put on their Vision Pro for them! This seems like an incredible leap forward for people in this situation, if they iterate on this and it gets better then this could be a very viable wheelchair control system in the future.
actual useful & impactful A.I features not the snake oil being sold daily.
Putting aside the fact that no company should have direct access to anyone's brain, how cool would it be to be building toward VISOR (from TNG) instead of this. If we could translate sensor signals to the neural circuitry of the brain directly, we wouldn't even need an LLM in the mix. But to have it as an overlay, as supplementary data! With the ability to turn it off of course. (Would a person even be able to turn it off? In the same sense as whether someone can "turn off" social media?) If only we had meaningful human rights and institutions that really protected them... I still can't fully give up the techno-optimism that made me love tech in the first place (and TNG for that matter).
Brain Control Interface support was already announced last year and afaik is part of iOS already.
https://developer.apple.com/documentation/accessibility/brai...
As Apple shifts towards services and fancy software features, I wonder how do they expect to stay competitive by only releasing them for a subset of languages.
They roughly know how many of their users use a particular language.
I have difficulty trusting this. There are plenty of videos online of LLMs making up stuff like "I just ate a hot dog, is there mustard around my mouth?" "No, everything is clean" while there is a big yellow stain om the guy's face
The problem is using a language model to assess images.
Probably 80% of "LLM's are below expectation" complaints (from the general population) involves some form of image analyses.
Image tokenization is hard because unlike language tokenization, where every token is extremely dense with meaning, image tokens tends to be meaningless or irrelevant but are processed all the same.
Give an SOTA LLM a picture of toothpicks and ask it to move one to make a square, and it will probably struggle and fumble it. But give a mid-size LLM from 2 years ago the same problem in verbal form, and it will nail it almost every time.
That takeaway is, do everything you can to avoid having the LLM need to rely on images for the answer.
I thought all the recent models are "multimodal"? Is the image part just sticking an image recognizer in front of the text model?
Most of those videos are chatGPT voice mode, which still used gpt 4o last time I checked. it is far from SOTA.
Like coding, creating images or text, maybe the alternative of doing it yourself is too easy or enjoyable for you. Don't expect that will be true for everyone.
Did you reply to the wrong person? What are you even trying to say here?
Surely a blind person relies a lot on audio input?
Maybe on a smartphone, but usually not on a computer. Keyboards are pretty good.
The other thing is that if you're around others, voice input means you have no privacy. Even if you're not doing anything particularly private, it's a bit awkward and potentially embarrassing. If you use touch input in conjunction with a screen reader, you can be more like a "normal" user in that what you're doing is just between you and your phone.
There's my dopamine hit for the year.
I feel that if we build the UI for blind users first, we would get much more powerful systems, rather than building UI for the seeing users and then slapping a CV to text model of what's shown on the screens.
Did not test it yet, but blind users may be more prone to dominate Command Line Interfaces, which are becoming increasingly popular due to its easy integration with LLM
Her phone in the thumbnail has oval camera bumps. It is also extra long. Mine has round camera bumps. Is that a new iphone?
Kudos to apple for providing some of the best accessibility features across their devices. I’ve always appreciated the consistency of reduce transparency, increased contrast, reduced motion, reduced white point, touch areas, color blindness support. And they work well across third party apps. That demands a lot of effort on the API and UI framework to have broad support for something that is mostly a non-sellable feature.
Most apps have terrible accessibility labels because developer don't bother, which breaks every screen reader pipeline downstream. The Voice Control "say what you see" feature routes around that by letting users describe a button in plain language. That's a real fix for a problem caused by humans being lazy about ally.
Since Apple uses Gemini to power its AI, are those features actually powered by Google Gemini?
They don't get, but they will be using Gemini derived models with iOS 27. For now it's all their own models.
this is such a great use case for the technology
And this is why androidlets will never win. They’re too busy selling your data to ever think of disabled people or usability.
iOS is just painfully good. I can pause a video, put my finger on text inside the video, and copy it. Until they added it, I didn’t even know how much I needed that.
Until they added it, you didn't need it, then suddenly a phone was unusable without it.
I can do that on my Pixel 6.
Features don't exist until Apple "invents" them /sarcasm
Now we know why the new AirPods will have cameras!
I'm super glad that they're doing this, but once again unexcited for another decade of Apple self-privileging on this stuff so they're the only ones allowed to touch or improve any of this surface, or UX outside an app's tiny box.
People talk a lot about how MacOS has gone downhill but I feel like it would have been a good start if developers could continue to patch over Apple's shortcomings like they used to be able to.
I imagine that we would be a few years into a spectrum of tools like this if they didn't lock it down like they do.
Totally aware that plenty of HN commenters are very glad that Apple keeps this locked down. I'm just the other opinion, that's all.
I don’t want to discredit more advancements in accessibility, but this feels like accessibility porn.
I have fond memories of an old coworker 10 years ago who is blind. He would use his phone no problem, texting, going about his day, he was even on Tinder (credit to Tinder for making their app so accessible long ago). He would commute on his own, walk to the train station, even transfer to another train during peak rush hour. I’m not saying it was all easy for him, but nothing in this video really stood out to me more than what shirt was on the bed. I know other services/apps have long existed to be the “eyes” for people who need support, but this video feels….uneventful?
I may be cynical about this though, as I often hate how Apple’s marketing makes these emotional bids about how life-critical they are to society - which is fair to a degree..but it just feels cheap to be glamorising “look! we saved this person from pending doom, cool right??”
for every person like your coworker, there are probably several who have a much harder time with technology and who would benefit from a simpler interface.
If this includes improvements to the screen recognition feature in Voice Over, it could provide accessibility for apps where the developer doesn't care about accessibility, which is extremely common.
The vision capabilities could be useful if they are done well, but I suspect that will always be covered better by 3rd party apps.
I mean even if it is marketing for them they still did the work and developed these features. I had some vision issues recently and was glad there were options to make text more legible to me.
Additionally I don't believe this is just marketing. This is adaption to a changing market. Apple's customer base is aging and having these kinds of features will allow them to keep using Apple products for a longer.
They have done the work, but I don’t see much work that’s beyond what’s been previously capable without Apple Intelligence. The marketing of Apple Intelligence is weak here, not the foundational abilities.