Author here. This is hr-diffuse-1-nano: bidirectional Mamba-2 + LLaDA-style masked diffusion at 288M params, cross- arch distilled from SmolLM-135M, trained on 1xh100 for ~$500.
The honest headline results: 14% infill recovery where autoregressive models score ~0 (they can't condition on text after the blank), 7.5% repetition-loop rate vs 37.5% for the teacher, and a genuinely negative result I think is the most useful part: six different self-correction methods all failed at this scale, while a 300k-param external critic head detects errors far above chance. Small models don't doubt; they rationalize.
I'm pretty busy, so I only skimmed the article, but it's actually really interesting, and also informative as I'm not familiar with diffusion models. Maybe I'll some ask questions/write later. I do want to encourage you, but, honestly the websites are a bit over the top and there's no way to know how much human input actually went in to them.
Experimental science is very messy, as you've learnt. Agreed with the other commenter, there's value (for others and especially yourself) in writing down what went wrong, and the things in the "Small models cannot judge themselves" is so reminiscent of failure modes I've experienced myself. There are usually awful or subtle bugs, training just doesn't work, and even if the results are "interesting" rather than "bad", it can still be incredibly difficult to decide what to conclude from them. To distill knowledge from observations/experiments is the problem of science. You read papers about experiments seem neat and the results profound, but the truth is they're probably a mess too and the evidence for the conclusions is probably a lot weaker than it looks; ML experiments can be unreproducible too.
I suggest that you were running experiments at too large a scale given your resources: you should try to sort out these critical issues on a smaller scale. Yes, the painful problem with ML is that things change qualitatively with scale, you just don't know if a larger scale will fix your issues. But most of these bugs didn't need scale to discover. Think about how you could have more easily discovered them.
Sorry to tell you that your comment was dead (silently blocked, invisible to most users) until I vouched for it. Don't be discouraged from posting on HN. Clearly both you're a real person, and you wrote this with an LLM (quite understandably), but people are really put off by text that smells LLM generated, and it's really easy to tell. HN is flooded with LLM comments lately, they go dead. You can use an LLM to help write, but don't let it determine the content, be genuine, and make sure it doesn't read like one. They can write in any style.
totally agree with your points on benchmarking and research
also while the website ui was supported by an LLM, all the actual writing was by me(a human) whether it looks somewhat AI generated(well I guess I'm a little bias considering I wrote it) I think I've been speding too much time with the models these days
in all genuinely appreciate the thoughtful insights on my researcH!
I have to ask: the middle paragraph of this comment reads exactly like something that Codex wrote. Exactly. Is that what happened, or have you spent so much time with these models that you started writing like them?
Really impressive for a 13 year old, and refreshingly honest writeup. The failed self-correction section is the best part: six methods tried, six negative results reported instead of buried. That's rarer than the architecture itself. Curious whether the shared+LoRA bidirectionality idea holds up once you run it past 2000 steps.
I would like this a lot more if you wrote it yourself, and if it wasn’t an ask for money.
Playing with agents can get expensive quickly, please be careful.
Author here. This is hr-diffuse-1-nano: bidirectional Mamba-2 + LLaDA-style masked diffusion at 288M params, cross- arch distilled from SmolLM-135M, trained on 1xh100 for ~$500.
The honest headline results: 14% infill recovery where autoregressive models score ~0 (they can't condition on text after the blank), 7.5% repetition-loop rate vs 37.5% for the teacher, and a genuinely negative result I think is the most useful part: six different self-correction methods all failed at this scale, while a 300k-param external critic head detects errors far above chance. Small models don't doubt; they rationalize.
Weights are open: https://huggingface.co/devnull37/hr-diffuse-1-nano. Happy to answer anything about the architecture, the failed runs.
Hi!
I'm pretty busy, so I only skimmed the article, but it's actually really interesting, and also informative as I'm not familiar with diffusion models. Maybe I'll some ask questions/write later. I do want to encourage you, but, honestly the websites are a bit over the top and there's no way to know how much human input actually went in to them.
Experimental science is very messy, as you've learnt. Agreed with the other commenter, there's value (for others and especially yourself) in writing down what went wrong, and the things in the "Small models cannot judge themselves" is so reminiscent of failure modes I've experienced myself. There are usually awful or subtle bugs, training just doesn't work, and even if the results are "interesting" rather than "bad", it can still be incredibly difficult to decide what to conclude from them. To distill knowledge from observations/experiments is the problem of science. You read papers about experiments seem neat and the results profound, but the truth is they're probably a mess too and the evidence for the conclusions is probably a lot weaker than it looks; ML experiments can be unreproducible too.
I suggest that you were running experiments at too large a scale given your resources: you should try to sort out these critical issues on a smaller scale. Yes, the painful problem with ML is that things change qualitatively with scale, you just don't know if a larger scale will fix your issues. But most of these bugs didn't need scale to discover. Think about how you could have more easily discovered them.
Sorry to tell you that your comment was dead (silently blocked, invisible to most users) until I vouched for it. Don't be discouraged from posting on HN. Clearly both you're a real person, and you wrote this with an LLM (quite understandably), but people are really put off by text that smells LLM generated, and it's really easy to tell. HN is flooded with LLM comments lately, they go dead. You can use an LLM to help write, but don't let it determine the content, be genuine, and make sure it doesn't read like one. They can write in any style.
Hey! Thanks so much for the thoughtful reply...
totally agree with your points on benchmarking and research
also while the website ui was supported by an LLM, all the actual writing was by me(a human) whether it looks somewhat AI generated(well I guess I'm a little bias considering I wrote it) I think I've been speding too much time with the models these days
in all genuinely appreciate the thoughtful insights on my researcH!
I have to ask: the middle paragraph of this comment reads exactly like something that Codex wrote. Exactly. Is that what happened, or have you spent so much time with these models that you started writing like them?
haha yeah I wrote this fully by hand
the only LLM's input into this was as a peer reviewer
I guess I have spent a bit too much time with the models these days
(for the record I mostly use Fable 5)
Really impressive for a 13 year old, and refreshingly honest writeup. The failed self-correction section is the best part: six methods tried, six negative results reported instead of buried. That's rarer than the architecture itself. Curious whether the shared+LoRA bidirectionality idea holds up once you run it past 2000 steps.
thanks a lot for the support!!
yeah for the next training run im planning on testing that out extensively before putting into training