Every now and then when I'm pecking around on the Linux command line, I have a need to perform the same operation on a whole group of files. The syntax for doing so is something I always have to look up. It's one of those jobs that aren't important enough to write a shell script for, and not common enough that I'll remember it next time. So here's one of the best-kept secrets to Linux command-line file management. The syntax:
you%comp# for FILE in $(ls); do [COMMAND]; done
One element at a time:
- for: begins the loop syntax in Bash.
- FILE: can be any variable name you create. Remember that a new variable in Bash goes without it's dollar-sign friend on the first usage.
- $(ls): this is a 'pseudo-variable'; I'm telling it to take the output of the command enclosed in () and use that as the value, in this case it's the output of the ls command. You can use any command in here - even another for loop!
- ; that's a semicolon! We have completed the first of three "lines" of the command.
- do: this is the action you're going to tell Bash to perform on each pass of the loop.
- [COMMAND]: any command you would normally be running one at a time - apply it to the variable in place of a file name.
- ; another semicolon! This completes the second "line" of the command. We're almost...
- done: DO NOT put a semicolon after 'done'! Just hit 'Enter' and reap the rewards of your magic spell.
A real-world example: Say you have downloaded a batch of files that are now archived and compressed (with bzip2, yet!), and sitting in a directory by themselves. To unpack all of the little .tar.bz2 archives in one step just cd to that directory and type:
for FILE in $(ls); do tar -jxf $FILE; done
and all the tar commands get executed on each file at once. To remove the no-longer-needed zipped files in the same step, alter it to:
for FILE in $(ls); do tar -jxf $FILE && rm $FILE; done
Here's another example: Say you've downloaded a batch of files which some egomaniac prefixed their name to at the beginning of each file name. You want to rename the files without the prefix characters. We can do this with Bash's variable indexing syntax, and even make sure we don't touch any other kind of file:
you%comp# ls
NAME_file1 NAME_file2 NAME_file3 NAME_file4 NAME_file5you%comp# for FILE in $(ls NAME_*); do mv $FILE ${FILE:5}; done
you%comp# ls
file1 file2 file3 file4 file5
This last example is especially the kind of problem I run into when receiving files from somebody who doesn't know better than to put spaces into file names. Replacing bogus characters in the middle of file names with something like underscores would require more sophisticated methods, probably involving one of tr, sed, or awk.
One more: Ever get a Python program from a case-indifferent system and the filenames are all uppercase but the code expects lowercase? For smashing all uppercase to lowercase in the current directory, try:
for FILE in $(ls); do mv $FILE $(echo $FILE | tr [A-Z] [a-z]); done
Happy hacking!
UPDATE: Thanks to sharp-eyed visitor Stephen P, who reminds us that filenames with spaces in them require quotes around the $FILE variable (like this: "$FILE") or else Bash will treat the command like you mean two files! That is a 'gotcha' I neglect often!
EVEN BETTER UPDATE: Here's a Bash script I doodled out to change spaces to underscores in filenames, kissing that problem goodbye for good!
5 feedbacks
Comments:
I have an email server that stores (spam) in folders for each user, buried a few directories down. For example:
/home/e-smith/files/users/moronuser/Maildir/.junkmail/cur/
I would like to essentially run the following command on each users home
directory:
rm -f /home/e-smith/files/users/%username%/Maildir/.junkmail/cur/*
Can you please tell me how to write a one liner to do this? I have over 700 users on the system. This email server is E-Smith (linux box) with bash and perl.
Thank you for your help
Chris
cmwithrow@wcsk12.org
rm -f /home/e-smith/files/users/*/Maildir/.junkmail/cur/*
For instance, to empty everybody's trash, I know I can type
rm /home/*/.Trash/*
leaving out the risky -f option to rm unless I know it's going to have folders.
Thanks, it worked great.
FILE="important documents"
rm $FILE # Deletes 2 files, important AND documents
rm "$FILE" # Deletes 1 file, "important documents"