Users should be computer-friendly, instead of computers being user-friendly!

How the One-Liner For-Loop in Bash Goes

Date/Time Permalink: 12/01/06 02:51:17 am
Category: HOWTOs and Guides

Every now and then when I'm pecking around on the Linux command line, I have a need to perform the same operation on a whole group of files. The syntax for doing so is something I always have to look up. It's one of those jobs that aren't important enough to write a shell script for, and not common enough that I'll remember it next time. So here's one of the best-kept secrets to Linux command-line file management. The syntax:

you%comp# for FILE in $(ls); do [COMMAND]; done

One element at a time:

  • for: begins the loop syntax in Bash.
  • FILE: can be any variable name you create. Remember that a new variable in Bash goes without it's dollar-sign friend on the first usage.
  • $(ls): this is a 'pseudo-variable'; I'm telling it to take the output of the command enclosed in () and use that as the value, in this case it's the output of the ls command. You can use any command in here - even another for loop!
  • ; that's a semicolon! We have completed the first of three "lines" of the command.
  • do: this is the action you're going to tell Bash to perform on each pass of the loop.
  • [COMMAND]: any command you would normally be running one at a time - apply it to the variable in place of a file name.
  • ; another semicolon! This completes the second "line" of the command. We're almost...
  • done: DO NOT put a semicolon after 'done'! Just hit 'Enter' and reap the rewards of your magic spell.

A real-world example: Say you have downloaded a batch of files that are now archived and compressed (with bzip2, yet!), and sitting in a directory by themselves. To unpack all of the little .tar.bz2 archives in one step just cd to that directory and type:

for FILE in $(ls); do tar -jxf $FILE; done

and all the tar commands get executed on each file at once. To remove the no-longer-needed zipped files in the same step, alter it to:

for FILE in $(ls); do tar -jxf $FILE && rm $FILE; done

Here's another example: Say you've downloaded a batch of files which some egomaniac prefixed their name to at the beginning of each file name. You want to rename the files without the prefix characters. We can do this with Bash's variable indexing syntax, and even make sure we don't touch any other kind of file:

you%comp# ls
NAME_file1 NAME_file2 NAME_file3 NAME_file4 NAME_file5

you%comp# for FILE in $(ls NAME_*); do mv $FILE ${FILE:5}; done

you%comp# ls
file1 file2 file3 file4 file5

This last example is especially the kind of problem I run into when receiving files from somebody who doesn't know better than to put spaces into file names. Replacing bogus characters in the middle of file names with something like underscores would require more sophisticated methods, probably involving one of tr, sed, or awk.

One more: Ever get a Python program from a case-indifferent system and the filenames are all uppercase but the code expects lowercase? For smashing all uppercase to lowercase in the current directory, try:

for FILE in $(ls); do mv $FILE $(echo $FILE | tr [A-Z] [a-z]); done

Happy hacking!

UPDATE: Thanks to sharp-eyed visitor Stephen P, who reminds us that filenames with spaces in them require quotes around the $FILE variable (like this: "$FILE") or else Bash will treat the command like you mean two files! That is a 'gotcha' I neglect often!

EVEN BETTER UPDATE: Here's a Bash script I doodled out to change spaces to underscores in filenames, kissing that problem goodbye for good!

Follow me on Twitter for an update every time this blog gets a post.
Stumble it Reddit this share on Facebook

suddenly the moon