- Choose which transformations you want (break on pipe, args, redirect, whatever)
- "Peeks" into stringified commands (think xargs, parallel) and formats those, too
- Auto-breaks at a given width (e.g., 80 characters)
- Shows you non-standard aliases, functions, files, etc. that you might not have in your shell environment
- Breaks up long jq lines with jqfmt because—let's be honest—they're getting out of hand
As a security researcher and tool developer, I often encounter (or create) long pipelined Bash commands. While quick and powerful, they can be a nightmare to read or debug. I created sol to make it easier to understand and share these commands with others.
Feature request, which I would love to have in all my automation scripts:
Replace short flags with the long switches. Short flags are great when typing in a terminal but I don't want to figure out 2 years from now what the
obscurecommand -v -f -n
I try to use long options in my script, therefore (especially in a team, where not everyone is familiar with every single command)
Line 6:
curl -fsSL "${url}"
^-- SC8537 (warning): use long options instead (`--fail`, `--silent`, `--show-error`, `--location`). foo -a -b \
| bar -c -d -e \
| baz -e -f
foo -a -b | \
bar -c -d -e | \
baz -e -f
https://unix.stackexchange.com/questions/253518/where-are-ba...
When writing bash, if I have a command with many switches, I use an array to avoid backslashes and make the intent clearer:
curl_cmd=(
curl
--silent
--fail
--output "$output_file"
"$url"
)
"${curl_cmd[@]}"
These are usually a step between "overely complicated one liner" and structured script, and often get refactored to functions etc if the script evolves that far. But lots don't, and if I just want something readable, that also lends itself to comments etc, this works for me.
{ foo -a -b }|{ bar -c -d -e }|{ baz -e -f }
But I suspect it's not everyone's cup of tea!
foo -a -b |
bar -c -d -e |
baz -e -f
foo -a -b \
| bar -c -d -e \
| baz -e -f \
&& echo "DONE." # && /bin/true
foo && \
bar && \
baz && \
:
https://unix.stackexchange.com/questions/253518/where-are-ba...
RUN foo && \
bar && \
:
RUN foo &&
bar &&
:
RUN <<EOF
foo
bar
EOF
That's a great technique, but the `:` as no-op is tough to expect bash-normies to understand (and a tough "operator" to search for). Thanks for sharing, it'll definitely stay in my back pocket!
foo --long-something \
| bar --long-other
foo --long-flag \
| bar \
--long-other-flag \
--long-option a \
--long-option b \
| baz \
--long-another-flag \
--long-flag-again \
> /path/to/some/file
foo \
| bar \
| baz
and echo done
test -n "$SOMETHING"
or set -x SOMETHING some-default
command $SOMETHING
One of my favorite patterns for readability is to use indented, long-form pipelines like this to set variables. They work fine inside subshells, but for uniformity and clarity I prefer to do
shopt -s lastpipe
foo \
| bar \
| baz \
| read SOMEVAR
If you do this, you end up with a script is mostly 'flat' (having very little nested control flow, largely avoiding loops), has very few variable assignments, and predictably locates the variable assignments it does have at the ends of pipelines. Each pipeline is a singular train of thought requiring you to consider context and state only at the very beginning and very end, and you can typically likewise think of all the intermediate steps/commands in functional terms.
I tend to write all of my shell scripts this way, including the ones I write interactively at my prompt. One really cool thing about shell languages is that unlike in 'real' programming languages, loops are actually composable! So you can freely mix ad-hoc/imperative and pipeline-centric styles like this (example is Fish):
find -name whatever -exec basename '{}' \;
| while read -l data
set -l temp (some-series $data)
set -l another (some-command $temp)
blahdiblah --something $temp --other $another
end \
| bar \
| baz \
> some-list.txt
I tend to follow my guideline suggested earlier and do this only close to the very beginning or very end of a pipeline if that loop actually modifies the filesystem or non-local variables, but it's really nice to have that flexibility imo. (It's especially handy if you want to embed some testing or conditional logic into the pipeline to filter results in a complex way.)
One stanza I have at the beginning of every script:
[[ -n “${TRACE:-}” ]] && set -o xtrace
This was typed from memory on mobile so if the above is bugged, my bad :)
Shell script is really good for some things, so we're gonna end up writing it. And if we write it, it might as well be legible, right? Shell scripts deserve the same care that other programs do.
> This lets you trace any script just by setting the environment variable. And it’s nounset-safe.
Nice! I think I'll start adding the same.
> This was typed from memory on mobile so if the above is bugged, my bad :)
I think it's fine aside from the smart quote characters your phone inserted instead of plain vertical double quotes around the parameter expansion you use to make the line nounset-friendly!
# MAGIC DEBUGGING LINE
#trap '(read -p "[$BASH_SOURCE:$LINENO] $BASH_COMMAND? ")' DEBUG
I need to write up some thoughts on bash being effectively a lisp if you stare at it the right way.
Yes, you need a recent Bash for this (and lots of other nice things like 'associative arrays' (maps)). To free myself up to write more concise and legible scripts, I lean into bashisms and make use of external programs without restriction, regardless of what's shipped by default on any system. To recover portability where I need it, I use Nix to fix my Bash interpreter versions as well as the versions of external tools.
When my scripts are 'packaged' by Nix, their bodies are written as multiline strings in Nix. In that case, Nix takes care of setting the shebang to a fixed version of Bash, and I interpolate external commands in like this:
"${pkgs.coreutils}/bin/realpath" . # these do
"${pkgs.coreutils}/bin/pwd" --physical # the same thing
In other contexts (e.g., cloud-init), I use bash a bit more restrictively depending on what versions I'm targeting. But I still use Nix to provide dependencies so that my scripts use the same versions of external tools regardless of distro or platform:
nix shell nixpkgs#toybox --command which which # these do
nix run nixpkgs#which -- which # the same thing
There are is a really cool tool[1] by the Nixer abathur for automagically converting naively-written scripts to ones that pin their deps via Nix, as in the first example. I'm not using it yet but I likely will if the scripts I use for our development environments at work get much bigger-- that way I can store them as normal scripts without any special escaping/templating and linters will know how to read them and all that.
Anyhow, it's totally safe to install a newer Bash on macOS, and I recommend doing it for personal interactive use and private scripts. Pkgsrc and MacPorts both have Bash 5.x, if you don't have or want Nix.
> I need to write up some thoughts on bash being effectively a lisp if you stare at it the right way.
You should! It's totally true, since external commands and shell builtins are in a prefix syntax just like Lisp function calls. Some shells accentuate this a bit, like Fish, where subshells are just parens with no dollar signs so nested subcommands look very Lisp-y, and Elvish, whose author (xiaq) has tried to lean into that syntactic convergence designing a new shell (drawing inspiration from Scheme in various places).
> I found this gizmo somewhere and it's worth sharing as well...
# MAGIC DEBUGGING LINE
#trap '(read -p "[$BASH_SOURCE:$LINENO] $BASH_COMMAND? ")' DEBUG
--
But the opposite might work, does anyone have a good minifier they could recommend (preferably one that does more than just whitespace mangling, eg also renames variables, chains executions, etc) that doesn’t introduce bash-isms into the resulting script?