My Notes on Apple Math Notes

(mlajtos.mu)

176 points | by mlajtos 4 days ago ago

70 comments

  • alsetmusic 4 days ago

    Placing text representations of the symbols above them (as interpreted by the handwriting algo) seems so obvious that it’s the sort of thing that I’ll be frustrated if Apple doesn’t adopt it. It generates confidence in the output.

    Some things aren’t obvious until we see them. This is one.

    • mlajtos 4 days ago

      I thought the same few years ago – putting recognized symbol above is simple and effective. But you can get the same effect (confidence in recognized input) with the Smart Script technique when done right.

      I briefly experimented with that in 2022 (nothing published though) and it worked pretty well. However, the implementation has to be good – you can't just beautify every symbol because it becomes distracting really fast. The less confidence system has about some symbol, the more it should fix it. High-confidence misclassifications will still be a theoretical problem, but I didn't experienced them that often.

    • MBCook 4 days ago

      I don’t know, it seems unnecessary to me.

      Why do I need a second copy on top of the one I just wrote?

      I don’t think misinterpretation would happy nearly enough for it to be more useful than intrusive.

      • wonger_ 4 days ago

        I like having confidence that the intermediate representation is correct before moving forward. Especially since OCR usually makes mistakes on handwriting. It's not a redundant copy, it's reassurance.

        • mlajtos 4 days ago

          Yes, the handwriting recognition system needs to communicate results as soon as possible to instill trust in the user. I need to be able to spot and fix misinterpretations as quickly as possible, otherwise errors accumulate real fast.

      • IOT_Apprentice 3 days ago

        If it were an option you could contextually disable it via Apple Pencil it would be cool.

  • wpm 4 days ago

    Maybe I'm a freak but almost all of the proposed "improvements" in this article I find distracting, annoying, and would turn me off of ever using the feature. I do NOT want my notes or content dancing around on the screen or having motion displayed because its evaluating equations or statements while I'm writing them. It's also why I disable most autocomplete and warnings in code editors. I don't complaints about syntax errors for lines I haven't finished writing.

    Like, why would I care about the value of `a` before I've finished writing the equation? Sometimes a tool just shutting the hell up is a good thing.

    • fastasucan 3 days ago

      What makes me crazy about my developer setup in vscode right now is that it is complaining about unused imports (even auto removing them before I turned that off) when writing an import statement. However if I try to avoid that by referencing that module before import it is (naturally) complaining about unknown module referenced.

      • jazzyjackson 3 days ago

        I was really shocked at vscodes default behavior of adding import statements to my code without asking and have strayed away from any kind of automation since

        • strogonoff 3 days ago

          VS Code does not just add imports to your code. I switched away from VSC when I found out the official Python extension (which you basically have to use to get things done) automatically installs things like Python 3 typings—ordinary modules, completely unvetted (e.g., typings for Django are maintained by a random guy who last time I used it had nothing to do with Django core team) and installed from PyPI (which totally did not have an avalanche of packages with embedded malware just recently)—and has no option to disable this.

          • daelon 2 days ago

            Is there somewhere I can read more about this?

        • dr_kiszonka 2 days ago

          PyCharm does that too, which can be very confusing.

    • treyfitty 4 days ago

      > I don’t complaints about syntax error for lines I haven’t finished writing

      How about “complaints about” syntax errors after you’ve finished writing?

      Jokes aside, to answer why knowing the value of ‘a’ might be useful before writing the full equation: ever work in excel and you’re building up a calculation, and when finished, your answer isn’t what you expected? Tracing that missing parentheses or brain-fart typo can be frustrating. I imagine that’s the same reason why some people would care about the value of ‘a’ before they’ve finished.

    • mlajtos 4 days ago

      Fair point. Interactivity is a spectrum that should match the user preference. Interactivity helps when you are learning the tool, because you can do & fix mistakes faster. But when you want to think deeply, it should stand away and be there only when you explicitly request it.

    • paulcole 4 days ago

      Yeah it’s almost like people design things without asking you specifically for feedback.

      Lots of people like autocomplete and animations.

      • wpm 4 hours ago

        My bad for attempting to start a discussion on a discussion thread.

        I hope you’re not this miserable and rude in real life.

  • dhosek 4 days ago

    For quite a while, I’ve kept track of how many pages are left in the books I’m reading by having a note with entries along the lines of

    Vargas Llosa 727-516=211

    I’d forgotten about math notes / assumed it only applied to handwritten notes so the first time I updated one of these notes after updating my iOS it was a little shocking to retype the = and have the difference generated automatically instead of having to figure it out on my own. I’ve been holding off updating my Macs, but I can see this feature being really handy as a sort of lite spreadsheet replacement.

    • rgovostes 2 days ago

      It can be annoying when unexpected, though. I was writing out a substitution for a recipe and typed “1/4 oz =“ and it completed to “0.25 oz”. Useful.

    • mlajtos 4 days ago

      Yes, having an ability to do simple math across the whole ecosystem is extremely useful.

  • happyopossum 4 days ago

    The 18.2 beta shows you the equation Apple thinks you wrote as soon as you draw the = .

  • keyle 4 days ago

    Related, has anyone noticed with the new Sequoia, Apple Notes, which I use extensively, if you start searching or clicking as soon as the window open, you get a frozen application that won't recover - it must be killed? Is that being reported and if not where to?

    I don't really care for the suggestion that 2 + 2 = 4, I'd rather the app be responsive on open.

    • dagmx 4 days ago

      I haven’t experienced it myself but you should file issues by going to the feedback page on the Apple site or going to applefeedback:// in a browser. The latter is better because it a shortcut to the embedded feedback app which collects diagnostic data.

    • saagarjha 3 days ago

      If you take a sample of the application you might be able to figure out what it's doing.

    • neom 4 days ago

      My regular spotlight started doing that, if I do command space the box shows up and then spotlight hangs and then crashes, 2 month old MBa with sequoia update.

      • wpm 4 days ago

        Nuke your spotlight index. You can add your entire "Macintosh HD" to the Spotlight Privacy settings to exclude it from indexing, and then just remove it, and it'll force it to rebuild.

        You can also run `mdutil -E`.

        • neom 4 days ago

          worked - tnx!!

      • shmoogy 4 days ago

        I only get this when I paste and it's usually only edge or chromium apps (slack). I can't replicate it on demand to bug report, but it's very annoying when it happens.

  • peppertree 4 days ago

    I wish there's a way to restrict math notes within a bounding box. I use notes with pencil to help my kids with their math homework and it started randomly trying to solve what I was writing. I like it but wish there's more control.

    • MBCook 4 days ago

      You can turn it off on a per note basis in the three dot menu.

      there is no way to default it to off, which I have seen some people request.

    • epistasis 4 days ago

      AI demos have gone so wild that managers are apparently mistaking demos for products.

      I miss Steve Jobs, he would get the team back on track and drive the field forward more quickly by demanding reasonable UIs.

      • MBCook 4 days ago

        What’s wrong with this UI? It seems basically perfect. You write math equations and it gives you the answer.

        • epistasis 3 days ago

          I was replying to a comment with exactly what's wrong. A sibling comment to mine has a way to turn it off, but that's the wrong way to approach the feature. Instead, the user should be given more control, as the grandparent requested.

  • jerojero 3 days ago

    Live graphs, function definitions and support for more calculus would be something good for the future, as it is right now this is good for basic middle school math and quick calculations.

    Linear algebra, calculus notation, etc. these are necessary improvements. I think they might come in the future though, I know this will become on par with a graphic calculator.

    • mlajtos 3 days ago

      Yes, the first version is already usable for everyday simple stuff. In the future, it will definitely gain more features. But you are wrong in one thing – it will become much-much more than graphing calculator.

  • odysseus 4 days ago

    In Apple Math Notes, why can't you type a simple list of numbers, either in a table or in columnar format, like:

    239.12

    +242.33

    +673.34

    -------

    =

    And get an answer after the equals sign? (a la Soulver) Isn't this an obvious use case? Was Apple trying really hard not to copy Soulver's implementation?

    You can type them one after another on the same line and get an answer, but that's bad for readability. I want to add up numbers in a column like Soulver and a spreadsheet lets you do, and do that inside Notes.

    • mlajtos 4 days ago

      You can do this in Math Notes – scribble numbers in a column, underline it and boom, there is your result. Doesn't work in typed text though.

      • odysseus 3 days ago

        Thanks, at least you can do it with an Apple Pencil or your finger. Seems like more of a demo than a feature if you can’t do it with typed text too.

  • spwa4 3 days ago

    But the problem here is: you need to pay for software development. This requires a few years of effort by a capable machine learning engineer that's also a pretty good software developer. I would love to try this, mostly to become a capable machine learning engineer. But ... what's missing is the $, like in so many software demands.

    I looked up the paper on this, but there are no good models for it that generate proper syntax. So for one thing, whoever does this would need to grind out creating at minimum a few 10,000 math exercises, to get the software to work.

    So this costs, say, ~400.000 USD, and 2 years of time, counting taxes, equipment, office, ... Perhaps "only" 100.000 if you're a professor and willing to use a PhD student (and deal with the issues that will give).

    Now I don't mean to disparage this proposal. But this seems to be how everything works except "improve our existing boring software 1%" ... And it's frustrating. At best, I can find machine learning jobs that boil down to "make LLMs solve tickets" or, worse "make LLMs write spam" (usually customize/personalize existing spam mails). Beats solving tickets myself, that it definitely does, but something like this ...

    I don't have trouble finding software development jobs. However, I cannot ever seem to do something like this. I would so love to make something like this, to really take a hard problem, work it and make something beautiful. But this is not a hobby effort. This requires spending a few years of effort making something beautiful.

    If this is done as a hobby project I can make a nice tech demo, no doubt, but it will be only what I make of it, and it'll be a limited cool tech demo, just like what Apple did (making a gradio that does a few examples correctly isn't very hard), what I'd use it for. Not because I don't want to make the next generation of advanced calculators, but because I'd never have the time to do that.

    • mlajtos 3 days ago

      You are not wrong. In 2021, I did bootstrap everything on my own – dataset, segmentation algorithm, neural net, UI, math engine, etc. And in the end, it worked properly only for me. But that was my aim – own complete stack to be able to try out everything that might be interesting without being burdened by 3rd party solutions. It was a prototype that I could show to people – spread the idea. Doing product of that magnitude on my own would never fly.

      I don't think Apple did a "cool tech demo". In my view, they published first version of a very ambitious project. They worked on it for years with top talent. And they felt this is the minimum lovable product on which they will iterate.

      And if you would like to build product-ready solution similar to Math Notes, you can leverage existing stuff. MyScript.com have really good handwriting recognition system with bells and whistles. You can take a lot of shortcuts.

      • spwa4 3 days ago

        Well, I hope you'll open source it. But I will 100% understand you not doing this.

        And it's just a general observation. Being well paid to make a technically impressive GOOD product for people, I would so love to be in that situation. But barring a FANG job, I don't know what to do.

  • eviks 2 days ago

    > First, the handwriting recognition worked flawlessly, despite my awful handwriting.

    This handwriting is not awful, so no surprise it's recognized well

    But the bad feedback criticism is spot on, though the better solution is to replace symbols in-place, why would you prefer your "awful handwriting" to beautiful math symbols with excellent legibility?

    (and btw, pemdas forgot implied multiplication)

    • mlajtos 2 days ago

      > But the bad feedback criticism is spot on, though the better solution is to replace symbols in-place, why would you prefer your "awful handwriting" to beautiful math symbols with excellent legibility?

      You mean it should replace your handwriting with typed symbols?

      • eviks 2 days ago

        Yes

        • mlajtos 2 days ago

          You can try out MyScript Calculator, which is doing exactly that – https://www.myscript.com/calculator

          I prefer to keep my handwriting over the printed symbols. This might sound crazy, but when I see my handwriting transform into printed symbols, I feel as if the system stole it from me. :) Another crazy take – it is more difficult to read handwritten script, so you slow down your thinking, which might help you to think more deeply. I can't back these claims with any evidence other than "I like it that way"..

  • amai 3 days ago

    I think this might be relevant: https://mathpix.com/handwriting-recognition

  • tomashubelbauer 3 days ago

    Math Notes is pretty cool, I used it recently to figure out what spec AC I needed to buy to climate control my non-insulated barn workshop while I'm working there. However, I struggled to get it to convert units.

    An example with non-representative numbers:

        A = 10000 W
        B = A / 1 kW =
    
    This gives 10 000 000 W². I was hoping for 10 kW.
    • setopt 3 days ago

      Did you try “A / ( 1 kW )” just to see if that works? Sounds like it doesn’t understand that 1 and kW are parts of the same “thing”.

      • tomashubelbauer 3 days ago

        Good call! Seems as though that just gives a unitless "10". Which is cool, I also struggled to make it give me unitless values I could use as factors without it starting to make up sci-fi units. But it could use some more of that Apple polish and just get this right.

    • mlajtos 3 days ago

      I want to see more threads like this one – people using Math Notes for all kinds of needs. :)

    • 3 days ago
      [deleted]
  • nullhole 4 days ago

    Suggestion for the website: ~double the height of the first video element; as it is, if the mouse is over the video, the play button / scrubber overlay will be displayed & obscure the video itself.

    (the quote at the bottom of the article is excellent and apt, too :))

    • mlajtos 4 days ago

      Fixed, thank you for your suggestion!

      (I love that quote.)

  • donatj 3 days ago

    I really want to like Apples handwriting recognition, it's half the reason I bought an Apple Pencil with my new iPad. It's just kind of difficult to use in practice.

    It's just not that good at recognizing my handwriting, and best I can tell has zero mechanism for teaching it to better understand.

    When enabling the "autocorrect" where it cleans up your writing, it so often replaces my handwriting with the entirely wrong word as to be infuriating. On top of that, it often does so long after I wrote it so I might not even notice that it happened until I try to read the note back later.

    As the author mentioned, having to switching to eraser and back is painful and annoying. I really wish the options for reflowing a document were a lot more intuitive. If I need to go back and add a sentence or two mid note I can try to draw a line to make space but it rarely works like one would expect.

    I think all in all the feature is neat but half baked, and they really need to do a bunch more user testing to get it into something that you would want to use rather than a neat tech demo.

  • pazimzadeh 4 days ago

    living graphs would be great.

    • MBCook 4 days ago

      They are. Once you make a graph, you can do something like tap on one of the values in the equation and do a gesture to change the value and the graph will update live.

      I don’t know if I’m explaining it well. It was shown in the original demo of math notes during WWDC.

      • mlajtos 4 days ago

        It is true that in the WWDC24 keynote, Jenny Chen demoed a graph that could be changed by dragging a slider of one of the variables [0], but as you can see in demo in the article, graphs are dead – they do not respond to changing variables. Or maybe I am just doing something wrong...

        [0]: https://youtu.be/RXeOiIDNNek?t=2900

        • Jtsummers 3 days ago

          That works in 18.1. Tap the number you want to change in the equation and you get a slider. Change the number and the graph changes.

          • mlajtos a day ago

            You are right, graphs are alive in 18.1. I fixed the article. Thank you.

        • pederb72 3 days ago

          Looks like the graph is rendering a different expression than the one you are modifying. If you tap to select the graph element view there’s a graph config button you can tap to select which expressions are actually active for that view.

          • mlajtos a day ago

            Problem was that iPadOS 18.0 did not support alive graphs.

        • Deadsunrise 3 days ago

          I just tested your example and is working fine on two different ipads on iPadOS 18.2 Beta 1 and 2.

          • mlajtos a day ago

            You are right, graphs are alive since 18.1. I fixed the article. Thank you.

          • 3 days ago
            [deleted]
  • rgreekguy 4 days ago

    I do not do many calculations, so my only issues where when I tried it out.

    If you open the calculator app and go to handwritten calculations (whatever they're called), you have all that cool stuff. But if you open Notes, you have to click "Insert result" every single time, for better or for worse.

    Also about UX improvements: lmao. There is still no way to add a new line with Scribble, or sanely write in a small text box (I guess, it's only in third-party applications I have had this issue).

    • mlajtos 4 days ago

      Yes, math engine works differently when you are in Notes and Math Notes. There is a setting ("Insert results" vs. "Suggest results") that controls this behavior.

  • mlajtos 4 days ago

    [flagged]

  • amelius 4 days ago

    Aren't we supposed to discuss technology here before it hits the stores?

    • mlajtos 4 days ago

      In the article I discuss tech that doesn't exist yet.

      • wiredfool 3 days ago

        Not seeing it on the App Store, though it’s one of the auto completes now. Is this really a thing, and if so, where is it?

        • Jtsummers 3 days ago

          Notes, calculator, and freeform, though not exactly the same features in each. Freeform (which really ought to) doesn't suggest inserting graphs (on iOS, maybe on iPad?) for instance.