Loop over list of files

Ubuntu - Command Line Tools

Ubuntu / Linux bash shell

Shell loop through files - Examples

Run a command on each file

do something (echo) with all .txt files

for f in *.txt; do echo ${f}; done;


Example: Search string ‘ABC’ in all .txt files

for f in *.txt ; do echo --${f}-- ; grep "ABC" ${f} ; done

same loop over files, but using a pipe (reading from standard input), and a while-loop

ls *.txt | while read f; do echo ${f}; done;


do something with a small set of files

for f in file1.txt file2.txt file3.txt; do echo ${f}; done;

file1.txt

file2.txt

file3.txt

same, but separately defined list of files

filelist="file1.txt file2.txt file3.txt"

for f in ${filelist}; do echo ${f}; done;


reading list of files from file 'filelist.txt' (for larger number of files)

ls *.csv > filelist.txt # define your list of files

for f in `cat filelist.txt`; do echo ${f}; done;

if a line may include spaces better use a while loop:

cat filelist.txt | while read LINE; do echo "${LINE}"; done


loop over filenames, but remove file extension, see → basename string split

for f in Data/*.txt; do FILENAME=${f%%.*}; echo ${FILENAME}; done;

Data/fileA .txt

Data/fileB .txt

loop over filenames, but remove file extension and path prefix

for f in Data/*.txt ; do FILENAME=`basename ${f%%.*}`; echo ${FILENAME}; done

Data/ fileA .txt

Data/ fileB .txt


Exclude samples that are already processed

process input files Data/*.fastq only if result-files Result/*.txt does not exist

for f in Data/*.fastq; do \

SAMPLE=`basename ${f%%.*}`; \

if [ ! -f Results/${SAMPLE}.txt ]; then \

echo "processing sample ${SAMPLE}"; \

# do something \

fi; \

done


# do something 10 times

for N in {1..10}; do \

echo ${N}; \

done

# double loop over a series of numbers and letters

for N in {1..5}; do

for S in {A..C}; do

echo ${N} ${S};

done;

done;

1 A

1 B

1 C

2 A

2 B

2 C

°°°

general use of: if-then-else (-f check if normal file)

if [ -f /path/to/${SAMPLE} ];

then

# do something if file exist

else

# do something if file does not exist

fi

Copy selected subset of files

# copy all files in filelist.txt to newDir/

ls *.csv > filelist.txt # define your list of files (edit text file)

mkdir newDir # create a new directory

for f in `cat filelist.txt`; do echo copying ${f}; cp ${f} newDir/; done;

# add path if files are not located in working directory

for f in `cat filelist.txt`; do echo copying ${f}; cp path/to/files/${f} newDir/; done;



Read more

→ rename multiple files ( add or remove prefix and file extension)

→ recursive loop over all files including files in all sub-directories ( see command → find )

http://mywiki.wooledge.org/BashPitfalls/