Related shameless plug: I have a small tool that prepends each output line with a timestamp (can be absolute, relative, or elapsed since last line), might be useful or pair well with this tool in similar scenarios (not needed when dealing with already timestamped logs, of course).
Looks like the `ets` readme has a direct comparison:
> The purpose of ets is similar to that of moreutils ts(1), but ets differentiates itself from similar offerings by running commands directly within ptys, hence solving thorny issues like pipe buffering and commands disabling color and interactive features when detecting a pipe as output. (ets does provide a reading-from-stdin mode if you insist.) ets also recognizes carriage return as a line seperator, so it doesn't choke if your command prints a progress bar. A more detailed comparison of ets and ts can be found below.
This seems great. Unfortunately, I will only recall it exists once I am into minute-unknown-but-way-too-long-for-comfort while waiting for a command to complete.
Edit: Maybe you should highlight the trailing pipe example (eg `ping localhost | grep icmp | ets`)? That is way more convenient to use than the introductory samples indicate. Being able to slap it onto the end of a pipeline I am developing feels less invasive than potentially have to quote my command for the sake of a debugging tool.
I wrote a similar tool[0] a few days back because I wanted a bit more accuracy with timestamp (to measure CPU/idle time) and `ts` occasionally had a deviation of several 10ms. If I knew this, and is accurate enough for me, I might not have written one.
I love it when someone has a problem I have very often, but they have the insight to actually realize that it's a problem, and then do something about it.
Haha, thanks! Yes, sometimes I do it too, but for this spacing thing I kept hitting enter and never realized this was an actual problem that I could do something about!
Also, I have to say, getting a 3D printer has definitely done this for things around the house. Every small niggle that used to fly under the radar and tolerated now has a tiny plastic thing fixing it once and for all.
I am disappointed I couldn't find any of the OPUS games by SIGONO on your website. Though I think some (or at least OPUS: The Day We Found Earth) were temporarily removed from the play store.
Realizing is not the hardest part, imo. I avoid publishing my little subprojects/libs/funcs simply because of the burden of publishing maintenance, and that is before connecting with people about issues, frs and so on. Our packaging tradition feels more like a marketplace (with all quirks and procedures) rather than a place where you can just share code and let anyone use it.
It really never ceases to amaze me how many basic affordances terminals lack, that we hack around with bash configuration or helper tools.
Of course, I know it's a genuine hard area, and not one I've put any time into fixing, but it's also so high leverage--so many developers spend a ton of time in this environment.
I get the exact opposite impression. The fact that you can simply press enter to mark the output is yet another sign of how malleable terminal interfaces are. You simply can't do that with GUI applications unless the developer created a button implementing that specific functionality.
I would love a feature to scroll back to previous input lines, I don't think I ever found something that supports it (I haven't spent a lot of time searching for it admittedly)
iTerm on macOS does this. It calls them "marks", and you can set them anywhere, but by default it creates a mark for each prompt. You can then jump between them with cmd-arrow.
iTerm also suppers showing the timestamp of each line, and it visualizes this in a nice way where lines emitted around the same time have their timestamps replaced with a vertical line.
Yes, say you start a command that outputs a long buffer of lines, and you want to go back to the early part of that output, even better if you could jump back multiple scroll buffer points
That is generally the responsibility of the terminal, not the shell. If your terminal doesn't have a scrollbar try hitting "scroll lock" and then moving up and down with page up/page down. Some really dumb terminals have no scrollback, but most modern ones do.
I may be misunderstanding, I'm not talking about just the scrollback buffer (I don't think?), I want to not hunt for the line in the buffer and instead jump to the line of the last input command in the buffer. The "jumping between prompts" mentioned in the other comment is more or less that feature - but it seems for that implementation you need to customize the prompt with a special sequence and have an emulator that supports jumping to that seq. I wonder if something like tmux would do it w/o customization...
I find this bar annoying as hell, tbh, especially in interactive modes.
Also Konsole messes up its ansi states all the time. It’s the only terminal in which I can hit <up>, sometimes see “~[[A;” or similar after my prompt, hit <enter> and it actually executes the previous command.
Maybe you want this: https://github.com/lambdaspace/ShellNotebook Or that prompt jumping thing. This isn't something I've looked for either but I guess I just got used to the way things are. If I'm running a command with long output I usually use a pager, and when I run something that takes a long time I usually save the output to a file. If it doesn't take long then you can run it again with a pager. Scrolling back is not super useful sometimes because output can change throughout a session. But it is useful enough to think about this problem, I suppose.
I often use this technique in my dev debug logs. If the last print() was more than a few seconds ago, it also prints “(2.4 seconds passed)” in gray text before printing another line. This helps with visually separating debug requests, button clicks, etc.
Mine is not a tool, I just define my log function that tracks last call time.
const now = Date.now()
if (now - last_time >= 2000) {
console.log(…)
}
last_time = now
console.log(…args)
Things like that should be in all dev toolboxes by default, I think. It’s amazing how we create animated experiences etc for users but when it comes to development it’s just println(). The cobbler’s children go barefoot.
Yeah, if I have a stuck output (that doesn’t include a date), I’ll sometimes hit return, and add `# 12/23/24 10:37` and hit return again. That way I know when I last looked at it.
This tool might be nicer, but I’m not sure if know when I’d need to use it a priori.
I have a similar but different need, wonder if anyone has the same frustrations and knows a solution:
I often press enter key a few times to produce a noticeable gap between different runs of the same command. Yes, in theory, the shell prompt is enough to separate the runs, and I could customize the color, font etc, but that's still not as good as a few (almost) blank lines.
The workflow is like this: often I need to run a command that produces a decent amount of output and repeat that a few times. Not 3 lines, not 1k lines, but something like 40-100 lines -- not short enough that I can easily see both the beginning and the end, not long enough that I have to log it to a file. I want to be able to scroll to the top of the output and read it.
The "multiple enter key" approach works well enough, but is a bit repetitive and sometimes I forget to do that (then can't find where the output begins). I could also append printf "\n\n\n" but apparently it's annoying. I wonder if there is something simpler and works well. The tool in this article doesn't exactly match my need, as it prints a spacer immediately after output pauses, not after a command finishes.
macOS’s built-in Terminal.app has a feature I love for this, which I haven’t seen anything else replicate, which is that pressing Cmd-Up just scrolls up to the last command prompt in your scroll buffer. Just press it as many times as you want to find the beginning of previous commands you typed.
I’m not sure how it’s implemented… maybe they take note of what your prompt looks like and generate a regex for it? Or maybe they take note of when the last time it was that your shell process had no child processes and read from stdin… but it works quite reliably.
As a windows user, the fact that hitting a button immediately would reflect inside a running output stream, feels strange. But its benefit is clearly shown here.
I’m writing a terminal emulator that solves this (and many other) common problems with the CLI. For example, one of its features is it highlights command output so you can easily see where each commands output is.
iTerm on macOS does this. It calls them "marks", and you can set them anywhere, but by default it creates a mark for each prompt. You can then jump between them with cmd-arrow. iTerm also suppers showing the timestamp of each line, and it visualizes this in a nice way where lines emitted around the same time have their timestamps replaced with a vertical line.
As an aside I have foolishly never once thought that you could pipe both stdout and stderr to another program at the same time. You learn something new each day.
Fair warning - this is a bash-ism, so if you're writing a script that needs to run on multiple shells and/or a more primitive shell, you should avoid it.
Next level: Make it a wrapper `spacers ... cmd ...`. This way you can allocate PTY, and make the app believe it is still talking to the terminal and the use colors and alter its output in more fancy ways.
BTW. Somewhat similiar (better sometimes) result can be achieved simply with:
When used with Python, it's important to add "PYTHONUNBUFFERED=1" env variable for this to work not just in the terminal but also when piping to files etc.
The environment variable you're thinking of is `PYTHONUNBUFFERED`, not `UNBUFFERED` [0]. If you want a generic solution to this problem, try the `unbuffer` command [1].
For anyone who's interested, I recommend reading Julia Evan's article [2] about this problem and its solutions.
FWIW, I'd accept either one. Some names don't take an extra s by convention, like Jesus, but otherwise it seems in practice that names that feel like plurals don't get it. I'd write "Jess's" but "Jenkins'".
(I consider myself a follower of Lynne Truss's, and have deliberately not considered a grammar reference for this answer, as I'm sure this is within the interested reader's ability and shan't spoil these readers' fun.)
I find it fascinating what new programmers prioritize. I’ve been developing on Unix and Unix like systems for 40 years and I’ve never needed a solution like this. But for some reason this appears to be a banger to some of you. Baffling but fascinating.
What is a "need"? I'm sure you can imagine why this could be useful. Is this life-changing? Does anyone need it? Probably not.
Your comment just makes you come off as some narrow-minded, graybeard elitist, and possibly even a little bit dumb, given how obvious it is why this could be useful. "New programmers", etc. Why not just .. not comment instead? Literally the easiest thing in the world for you to do since it requires you to do nothing.
If he thinks it's silly then he's welcome to say that. If someone with a lot of seniority wonders why you think you need a tool, maybe that's a hint that it isn't as obviously useful as you think. In this case, I get it, but a simple awk script can more or less do this work as well IF you think you need it.
iTerm has this if you install the shell integration[1], which basically puts an escape code in your PS1 that tells iTerm where each command begins and ends.
Related shameless plug: I have a small tool that prepends each output line with a timestamp (can be absolute, relative, or elapsed since last line), might be useful or pair well with this tool in similar scenarios (not needed when dealing with already timestamped logs, of course).
https://github.com/zmwangx/ets
There is also `ts` from moreutils[0]. One of a few gems there. And moreutils is (probably) already in "your" distro.
[0] https://joeyh.name/code/moreutils/
Looks like the `ets` readme has a direct comparison:
> The purpose of ets is similar to that of moreutils ts(1), but ets differentiates itself from similar offerings by running commands directly within ptys, hence solving thorny issues like pipe buffering and commands disabling color and interactive features when detecting a pipe as output. (ets does provide a reading-from-stdin mode if you insist.) ets also recognizes carriage return as a line seperator, so it doesn't choke if your command prints a progress bar. A more detailed comparison of ets and ts can be found below.
[delayed]
This seems great. Unfortunately, I will only recall it exists once I am into minute-unknown-but-way-too-long-for-comfort while waiting for a command to complete.
Edit: Maybe you should highlight the trailing pipe example (eg `ping localhost | grep icmp | ets`)? That is way more convenient to use than the introductory samples indicate. Being able to slap it onto the end of a pipeline I am developing feels less invasive than potentially have to quote my command for the sake of a debugging tool.
I wrote a similar tool[0] a few days back because I wanted a bit more accuracy with timestamp (to measure CPU/idle time) and `ts` occasionally had a deviation of several 10ms. If I knew this, and is accurate enough for me, I might not have written one.
[0] https://www.sadiqpk.org/projects/tis
Very useful, thanks for sharing.
I love it when someone has a problem I have very often, but they have the insight to actually realize that it's a problem, and then do something about it.
Says the person who built a website listing games with fair pricing[1] and more. You’re guilty of that yourself. Much appreciated too!
[1]: https://nobsgames.stavros.io/
Haha, thanks! Yes, sometimes I do it too, but for this spacing thing I kept hitting enter and never realized this was an actual problem that I could do something about!
Also, I have to say, getting a 3D printer has definitely done this for things around the house. Every small niggle that used to fly under the radar and tolerated now has a tiny plastic thing fixing it once and for all.
I am disappointed I couldn't find any of the OPUS games by SIGONO on your website. Though I think some (or at least OPUS: The Day We Found Earth) were temporarily removed from the play store.
I am disappointed you didn't submit it!
Realizing is not the hardest part, imo. I avoid publishing my little subprojects/libs/funcs simply because of the burden of publishing maintenance, and that is before connecting with people about issues, frs and so on. Our packaging tradition feels more like a marketplace (with all quirks and procedures) rather than a place where you can just share code and let anyone use it.
I think that's you. I just dump all my code in the open, without feeling any obligation.
A bit fancier than my awk version
Or if you want to clear the screen tooIt really never ceases to amaze me how many basic affordances terminals lack, that we hack around with bash configuration or helper tools.
Of course, I know it's a genuine hard area, and not one I've put any time into fixing, but it's also so high leverage--so many developers spend a ton of time in this environment.
I get the exact opposite impression. The fact that you can simply press enter to mark the output is yet another sign of how malleable terminal interfaces are. You simply can't do that with GUI applications unless the developer created a button implementing that specific functionality.
I would love a feature to scroll back to previous input lines, I don't think I ever found something that supports it (I haven't spent a lot of time searching for it admittedly)
If I understood you right, some terminal emulators support this:
https://codeberg.org/dnkl/foot#jumping-between-prompts
iTerm on macOS does this. It calls them "marks", and you can set them anywhere, but by default it creates a mark for each prompt. You can then jump between them with cmd-arrow.
iTerm also suppers showing the timestamp of each line, and it visualizes this in a nice way where lines emitted around the same time have their timestamps replaced with a vertical line.
Are you asking about something different than up-arrow or the 'history' command?
Yes, say you start a command that outputs a long buffer of lines, and you want to go back to the early part of that output, even better if you could jump back multiple scroll buffer points
That is generally the responsibility of the terminal, not the shell. If your terminal doesn't have a scrollbar try hitting "scroll lock" and then moving up and down with page up/page down. Some really dumb terminals have no scrollback, but most modern ones do.
I may be misunderstanding, I'm not talking about just the scrollback buffer (I don't think?), I want to not hunt for the line in the buffer and instead jump to the line of the last input command in the buffer. The "jumping between prompts" mentioned in the other comment is more or less that feature - but it seems for that implementation you need to customize the prompt with a special sequence and have an emulator that supports jumping to that seq. I wonder if something like tmux would do it w/o customization...
oh maybe https://unix.stackexchange.com/questions/226731/jump-to-last... hmm - seems like the more searchable term is "prompt jumping"
You mean like tmux or screen? EDIT: apparently not, sorry.
VSCode terminal also does this I believe.
KDE's Konsole has a feature to highlight "newly printed lines", but no timestamps.
https://askubuntu.com/questions/1467849/what-is-the-bar-on-t...
I find this bar annoying as hell, tbh, especially in interactive modes.
Also Konsole messes up its ansi states all the time. It’s the only terminal in which I can hit <up>, sometimes see “~[[A;” or similar after my prompt, hit <enter> and it actually executes the previous command.
Since it's KDE, there's naturally a setting to disable it.
I think iTerm does this
Maybe you want this: https://github.com/lambdaspace/ShellNotebook Or that prompt jumping thing. This isn't something I've looked for either but I guess I just got used to the way things are. If I'm running a command with long output I usually use a pager, and when I run something that takes a long time I usually save the output to a file. If it doesn't take long then you can run it again with a pager. Scrolling back is not super useful sometimes because output can change throughout a session. But it is useful enough to think about this problem, I suppose.
I often use this technique in my dev debug logs. If the last print() was more than a few seconds ago, it also prints “(2.4 seconds passed)” in gray text before printing another line. This helps with visually separating debug requests, button clicks, etc.
Mine is not a tool, I just define my log function that tracks last call time.
Things like that should be in all dev toolboxes by default, I think. It’s amazing how we create animated experiences etc for users but when it comes to development it’s just println(). The cobbler’s children go barefoot.Very nice! I’m one of those habitual return-pressers, and this tool accomplishes the exact same thing but better.
Yeah, if I have a stuck output (that doesn’t include a date), I’ll sometimes hit return, and add `# 12/23/24 10:37` and hit return again. That way I know when I last looked at it.
This tool might be nicer, but I’m not sure if know when I’d need to use it a priori.
At least in zsh, you can do `exec > >(spacer)` but it seems to break the tty by stripping control characters from input.
I like to pressing Enter - like an animal -, too! :) It puts spaces exactly wheere I want them.
I have a similar but different need, wonder if anyone has the same frustrations and knows a solution:
I often press enter key a few times to produce a noticeable gap between different runs of the same command. Yes, in theory, the shell prompt is enough to separate the runs, and I could customize the color, font etc, but that's still not as good as a few (almost) blank lines.
The workflow is like this: often I need to run a command that produces a decent amount of output and repeat that a few times. Not 3 lines, not 1k lines, but something like 40-100 lines -- not short enough that I can easily see both the beginning and the end, not long enough that I have to log it to a file. I want to be able to scroll to the top of the output and read it.
The "multiple enter key" approach works well enough, but is a bit repetitive and sometimes I forget to do that (then can't find where the output begins). I could also append printf "\n\n\n" but apparently it's annoying. I wonder if there is something simpler and works well. The tool in this article doesn't exactly match my need, as it prints a spacer immediately after output pauses, not after a command finishes.
macOS’s built-in Terminal.app has a feature I love for this, which I haven’t seen anything else replicate, which is that pressing Cmd-Up just scrolls up to the last command prompt in your scroll buffer. Just press it as many times as you want to find the beginning of previous commands you typed.
I’m not sure how it’s implemented… maybe they take note of what your prompt looks like and generate a regex for it? Or maybe they take note of when the last time it was that your shell process had no child processes and read from stdin… but it works quite reliably.
Personally, I have a command called `--` that prints a terminal-wide horizontal red line using ‘━’ (U+2501 if HN eats it). So I run things like
Then I can see the start easily when scrolling back.You could temporarily change your prompt:
Or if the repeated prompt is part of your visual pattern matching:Or
May also add 100 '-'s before or after newlines for better separation.Hmm... this looks like it works, to save in bashrc and activate it once:
As a windows user, the fact that hitting a button immediately would reflect inside a running output stream, feels strange. But its benefit is clearly shown here.
I’m writing a terminal emulator that solves this (and many other) common problems with the CLI. For example, one of its features is it highlights command output so you can easily see where each commands output is.
You can also collapse output too
https://github.com/lmorg/mxtty
Sometimes its nice to use something like tmux for this. Open a new pane for such a run and all output will be in the scroll buffer of that one pane.
iTerm on macOS does this. It calls them "marks", and you can set them anywhere, but by default it creates a mark for each prompt. You can then jump between them with cmd-arrow. iTerm also suppers showing the timestamp of each line, and it visualizes this in a nice way where lines emitted around the same time have their timestamps replaced with a vertical line.
As an aside I have foolishly never once thought that you could pipe both stdout and stderr to another program at the same time. You learn something new each day.
I knew you could do that, but never knew you could do it with "|&" as a shorthand. TIL!
I found this comment in a bash documentation example to verify:
# To send stderr through a pipe, "|&" was added to Bash 4 as an abbreviation for "2>&1 |".
Fair warning - this is a bash-ism, so if you're writing a script that needs to run on multiple shells and/or a more primitive shell, you should avoid it.
I wasn't aware of |& -- pretty sweet. I always redirected via 2>&1 and then piped.
Wouldn’t putting a \n at the start of PS1 do a similar thing?
Next level: Make it a wrapper `spacers ... cmd ...`. This way you can allocate PTY, and make the app believe it is still talking to the terminal and the use colors and alter its output in more fancy ways.
BTW. Somewhat similiar (better sometimes) result can be achieved simply with:
(echo "first"; sleep 1; echo "second"; sleep 0.5; echo "third") | ts -i "%.S"
00.000004 first
00.987787 second
00.501363 third
When used with Python, it's important to add "PYTHONUNBUFFERED=1" env variable for this to work not just in the terminal but also when piping to files etc.
The environment variable you're thinking of is `PYTHONUNBUFFERED`, not `UNBUFFERED` [0]. If you want a generic solution to this problem, try the `unbuffer` command [1].
For anyone who's interested, I recommend reading Julia Evan's article [2] about this problem and its solutions.
[0]: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUN...
[1]: https://manpages.ubuntu.com/manpages/noble/man1/unbuffer.1.h...
[2]: https://jvns.ca/blog/2024/11/29/why-pipes-get-stuck-bufferin...
Thanks, I forgot about the PYTHON prefix, fixed it.
On MacOS, can install via homebrew's `expect`: https://apple.stackexchange.com/a/193152/330523
Apparently I knew about this 2.5 years ago as I commented on the accepted answer back then.
The Julia article was a good read. Probably something I run into once a year, but glad to have a better understanding of it.
Julia Evan's article is a great write-up indeed, I'll submit it as a main link.
It's "Julia's article", not "The Julia article".
Update: it is was on here just a month ago actually: https://news.ycombinator.com/item?id=42275033
It's Julia Evans' article, not Julia Evan's.
I don't know if Julia Evan even has an article.
I was taught that the s is omitted after plural nouns, not after nouns ending in s.
Julia Evans's house, but the Evans' house.
FWIW, I'd accept either one. Some names don't take an extra s by convention, like Jesus, but otherwise it seems in practice that names that feel like plurals don't get it. I'd write "Jess's" but "Jenkins'".
(I consider myself a follower of Lynne Truss's, and have deliberately not considered a grammar reference for this answer, as I'm sure this is within the interested reader's ability and shan't spoil these readers' fun.)
I find it fascinating what new programmers prioritize. I’ve been developing on Unix and Unix like systems for 40 years and I’ve never needed a solution like this. But for some reason this appears to be a banger to some of you. Baffling but fascinating.
You've never ran tail -f? Curious, what exactly do you do?
What is a "need"? I'm sure you can imagine why this could be useful. Is this life-changing? Does anyone need it? Probably not.
Your comment just makes you come off as some narrow-minded, graybeard elitist, and possibly even a little bit dumb, given how obvious it is why this could be useful. "New programmers", etc. Why not just .. not comment instead? Literally the easiest thing in the world for you to do since it requires you to do nothing.
If he thinks it's silly then he's welcome to say that. If someone with a lot of seniority wonders why you think you need a tool, maybe that's a hint that it isn't as obviously useful as you think. In this case, I get it, but a simple awk script can more or less do this work as well IF you think you need it.
I like it! but it would be even better as a shell's plugin maybe?
It drives me batty that terminals don’t have a trivial way to “scroll to last command”.
iTerm has this if you install the shell integration[1], which basically puts an escape code in your PS1 that tells iTerm where each command begins and ends.
1. https://iterm2.com/documentation-shell-integration.html
Some do! For example, I used Wezterm, and it offers that capability [1].
[1]: https://wezfurlong.org/wezterm/config/lua/keyassignment/Scro...
Woah! Thanks for pointing this feature out. This will be super handy to skip large amounts of output.
Terminal.app on macOS can do this with Command + up-arrow.
(IIRC this is technically just a hotkey for navigating marks or bookmarks or something, and it (book)marks each command.
- In Python, Sarge has code to poll stdout and stderr
- Years ago for timesheets, I created a "gap report" that finds when the gap between the last dated events exceeds a threshold.
- The vscode terminal indicates which stdout, stderr are from which process with a circle .
This says it's "Integrated > Shell Integration: Decorations Enabled" to configure the feature; https://stackoverflow.com/questions/73242939/vscode-remove-c...
- Are there other shells that indicate which stdout and stderr are from which process? What should it do about e.g. reset ANSI shell escape sequences?
echo "printf '\n'" >> ~/.config/fish/functions/fish_right_prompt.fish....
pretty cool idea, wish I had thought of it.
I also pipe to `ts` and successors since that gives me a running timer of when the last thing happened. It’s in moreutils.