START OF WEEK 5 of Linux <<< FILE TIMESTAMP >>> -The up to date file is: /usr/courses/cps393/dwoit/courseNotes/ -If you are viewing your own copy, check it is up to date -If you are viewing from a link in the Course Outline, be aware it may be outdated. <<< TESTING & DEBUGGING SHELL PGMS >>> --------------------------------- use shell parameters -x or -v to debug interactively bash -v pgm displays every line of pgm just before executing it bash -x pgm like -v but substitutes values of variables in each line eg. #!/bin/bash #Source: findtrunc truncates $1 to 8 characters and #searches for it in file gfile # a msg is printed to indicate if found/not found trunc=`echo "$1" | cut -c1-8` if [ "`grep $trunc gfile`" ] then echo Found else echo not found fi try to run: findtrunc abcdefghijkl prints Found #if abcdefgh in gfile try to run: findtrunc no output, just hangs there. To stop it: ^C or ^D it prints "not found" What is the problem with findtrunc? Why does it hang? Use -x to find out why: bash -x findtrunc ++ echo '' ++ cut -c1-8 + trunc= ++ grep gfile <---- hangs, wanting to grep string gfile from stdin. If type ^D signals end of stdin, giving... + '[' '' ']' <---- since grep printed nothing on stdout + echo not found <---- since [ ] is considered false not found <---- terminates can tell where hanging In this case, we forgot the argument to findtrunc, so it was null Can fix this by adding an enclosing if [ "$1" ] (if $1 not null) or something similar, such as if [ $# -ge 1 ] , if [ -n "$1" ] ----------------------------------------------------- | Aside | would be nice if could do this: | echo "$1" | cut -c1-8 | read trunc | but can not work due to a bash "feature" (expl. in CPS590) | However, could instead use a here-string, as in: | read trunc <<< $(echo "$1" | cut -c1-8) | Or, could use ( ) or { } to extend scope as in: | echo "$1" | cut -c1-8 | ( read trunc | if [ "`grep $trunc gfile`" ] | then echo Found | else echo not found | fi | ) | Or, could use a while, as in: | echo "$1" | cut -c1-8 | while read trunc | The loop would only run once, but would work OK. | Or, could use process substitution, as in | read trunc < <(echo "$1" | cut -c1-8) | Essentially takes stdin from a temporary named pipe | (expl in CPS590) ----------------------------------------------------- <<< SHIFT >>> ------------- #!/bin/bash #Source: shiftex #pgm to print its args. while [ "$1" ] #<--- stops when $1 is null do echo "$1" #<--- so spaces preserved shift #<--- moves $2 to $1, $3 to $2 etc. done exit 0 <<< XARGS >>> --------- -to perform a command on a group of things from stdin -helps you take output of one command, and pass it as an argument to another command e.g., list lines containing 'bash' from all files whose NAME starts with 'f', where files are located anywhere in the filetree under current directory > find . -type f -name "f*" | xargs grep bash #prints LINES in files f* that contain the string bash #assuming only such files were ./f1, ./f2, and ./f3, would #end up running: grep bash ./f1 ./f2 ./f3 #Note that without xargs, it does something much different > find . -type f -name "f*" | grep bash #prints NAMES of files f* whose NAME also contains string bash > find . -type d -empty | xargs rmdir #remove empty directories from filetree under current dir #assuming the only empty directories were ./a, ./b/c, ./d #ends up running: rmdir ./a ./b/c ./d #The -p option PROMPTS before doing the command. #Assuming same as above, except adding -p (enter y or n) > find . -type d -empty | xargs -p rmdir rmdir ./a ./b/c ./d ?...y #now those 3 dirs are deleted (if typed n rmdir not run) #If no command is given, xargs uses echo by default. e.g., > ls | xargs #assuming current dir contains only files a, b, c, d #ends up running: echo a b c d #The -ni option says instead of sending stdin as one string, #split the string into chunks of i words > ls | xargs -n2 #assuming current dir contains only files a, b, c, d #ends up running: echo a b ; echo c d #both -n and -p together > echo abc de fghi jkl mn | xargs -n2 -p echo abc de ?...y abc de echo fghi jkl ?...n echo mn ?...y mn > #option -0 (zero) processes whitespace in items properly. #often used with -print0 (zero) option of find (which produces #output with whitespace properly preserved #e.g., suppose files included these 3 files which all contained string bash: # f0 f1 'f s' #To find files starting with f which contain string bash: > find . -type f -name "f*" -print0 | xargs -0 grep bash #If don't use -print0 and -0 the whitespace causes errors #To delimit input, xargs uses blanks and newlines #-d changes input delimiter (a single character only) > echo "abc.d.efghi jkl.m.nop" | xargs -d. -n2 abc d efghi jkl m nop The -I option runs the command once for each item in stdin (like -n1) except each time, {} in command is replaced with item: ls | xargs -I{} cp {} {}.bak # make backup copy of #all files in current dir. #assuming current dir contains only files a, b, c #ends up running: cp a a.bak ; cp b b.bak ; cp c c.bak Note: if b is a dir get cp b b.bak -- error, but xargs just prints msg on stderr and continues on with next item e.g., what does this print on stdout? (echo a b c ; echo d e ; echo f) | xargs -I{} echo :{}: HMWK: 1. write a shell program called clean that removes all files called "core" from your directory structure, and then removes all files that have size 0 (0 bytes) from your dir structure. That is your WHOLE dir structure, not just the current dir. (note: "find" can find files that are zero bytes long) HMWK: 1.write a shell program called avgs which will read lines from stdin such as the following (where the dots indicate more lines of the same kind: SID LNAME I T1/20 T2/30 92876035 WANG S 15 26 95908659 CHIANG R 10 29 . . . . . . . . . . . . . . . 91234987 MYRTH R 15 16 Your shell program will print out a line such as: Average of T1/20 is 17 and of T2/30 is 24 where 17 and 24 are the averages of the last 2 columns respectively. Note that your program must keep a total and count for each of the last 2 columns, and must not include data from the first line in the totals and counts. 2. Modify avgs so that the "title" line, e.g., SID LNAME I T1/20 T2/30 above, could be located at ANY LINE of stdin (ie., not necessarily the FIRST line, as above.) 3. Modify avgs so that the highest and lowest marks are printed for each of T1 and T2, as well as averages. Your program should produce output such as the following: T1/20: Average: 17 Highest: 19 Lowest: 1 T2/30: Average: 24 Highest: 29 Lowest: 9 4. Modify avgs so that the output above is in PERCENT, e.g., T1/20: Average: 85% Highest: 95% Lowest: 5% T2/30: Average: 80% Highest: 96% Lowest: 3% If variable x contains "T1/20" you could use an echo piped to a cut, and assign the result to another variable, or you could use ${x#*/} which will match pattern "*/" against the beginning of the contents of x, delete the shortest part that matches, and return the rest. Thus declare -i y=${x#*/} would assign 20 to y. Note that pattern */ means anything followed by a "/" 5. Write a shell program called findext which will list directories under a given path that contain files/dirs having the given extensions. Program findext takes at least 2 command line arguments. Its usage is: findext path ext1 ext2 ... where "path" is the path to the start of the directory structure you wish to search, and ext1, ext2, etc are extensions. Program findext will print out the directories under path which contain files/dirs that end in .ext1 .ext2 etc Your program does not need to work properly if the user passes glob constructs or special chars as arguments. For example: findext /home/dwoit/courses c scm will list all directories under /home/dwoit/courses that contain files/dirs named *.c and/or *.scm If findext is sent less than 2 command line args, it should print on stderr Usage: findext path arg1 arg2 and exit with exitcode 1 (the word findext above should be printed using $0) If it is passed 2 or more arguments, it should print a list of dirs in which files/dirs of the form *.ext1 *.ext2 etc reside. Then it should exit with exitcode 0. Note that only the DIRECTORY names are printed. Therefore you will have to chop off the trailing file/dir name (the part after the last "/") You can do this with ${var%/*} which says to match pattern "/*" (slash followed by anything) at the END of variable var; and delete the shortest part that matches and return the rest. You will also find utilities sort and uniq useful. <<< BUILT-IN COMMANDS >>> ------------------------- Saw some already: echo, exit, pwd, shopt, break, cd, test, read, typeset, etc (man bash and search for "^SHELL BUILTIN COMMANDS") << EVAL >> eg) v1="cat fn | grep 5" #suppose want to run the command in v1. Try: $v1 #but get file fn catted to screen, followed by: cat: '|': No such file or directory cat: grep: No such file or directory cat: 5: No such file or directory #why? Because does not recognize the "|" as a pipe. #solution: eval $v1 eg) v1="cat fn | grep 5" v2="grep 3 | more " #we wish to execute the command: cat fn | grep 5 | grep 3 | more #so we try this: $v1 | $v2 #but it does not work. Get errors because shell does not #recognize 1st and 3rd "|" as pipes (since part of variables). #get these errors (assuming fn exists): grep: |: No such file or directory grep: more: No such file or directory eval "$v1 | $v2" #sends lines of fn with both "5" and "3" in #them to more eval: 2 passes: 1) makes substitutions eval "cat fn | grep 5 | grep 3 | more" 2) result evaluated i.e, meta chars etc. interpreted properly #!/bin/bash #Source: printXth #Expects a bunch of command line arguments. #Prompts user and reads an integer (call it X) #then it prints the Xth command line argument echo -n "enter an integer: " read X the_arg='$'${X} #note the difference in output: echo "command line arg number ${X} is: ${the_arg}" eval echo "command line arg number ${X} is: ${the_arg}" #note: if wanted to preserve whitespace in argument, need to quote it # and need to protect those quotes: #eval echo "command line arg number ${X} is: \"${the_arg}\"" HMWK: Use eval to write a shell program called revargs which will print out its command line args in reverse order. White-space preservation not necessary. Write a shell program the does not use eval to print its command line args in reverse order. <<< SHELL CMDS >>> ------------------- most shell cmds (not builtin ones) in "bins" /bin, /usr/bin , /usr/local/bin , even ~/bin shell searches directories to find cmds. searches those in "PATH" to see path echo $PATH /usr/bin:/bin:/etc:/usr/local/bin:: ^ | current dir To add (or change) default path, reset PATH in .bash_profile (or, .profile if that is what you use instead) Also add to .bashrc for non-interactive, non-login shells, which you only get by opening a new terminal window from GUI once you are already logged into your machine (your linux laptop, a lab machine booted to linux). Putty windows are login shells, so .bashrc not relevant. Note Mac OS X’s Terminal.app runs .profile for both types of windows. PATH=${PATH}:${HOME}/bin export PATH - Adds /home/dwoit/bin to end of current path (for me) (I keep some personal shell pgms in /home/dwoit/bin) -export makes PATH global (so all subsequent processes can use it.) If 2 different cmds with same name, shell executes 1st one it finds in path i.e. if you write your own "rm" in ~dwoit/bin/rm to use instead of /bin/rm you must put YOUR ~dwoit/bin before /bin in PATH. However, the best way to accomplish getting your own rm run is to make it a function or alias, which are always done first EOH 2 <<< SHELL VARIABLES >>> ------------------------ problem: referencing a variable before it is assigned a value (i.e. programmer error) it will be null & shell pgm may behave unexpectedly e.g., grep "$var" fname If $var unset, get grep "" fname, which prints all lines of fname which is probably not what you expect set -u # if any variable unset, referencing it causes error message set +u # no error message for unset var fix above: set -u grep "$var" fname -bash: var: unbound variable can set DEFAULT values: ${ var:=value } eg) DO NOT RUN THIS!! #!/bin/bash #Source: ertmp # assumes variable temp is set to some valid temp # dir in your environment and exported cd $temp rm -r * # rm contents of $temp cd .. rmdir $temp # rm dir $temp exit 0 usage: ertmp #will erase dir $temp and contents if forgot to ever set temp to be some directory what happens? Erases all files in your home directory! Why? Fix: test -d ~/tmp || mkdir ~/tmp cd ${temp:=~/tmp} rm * cd .. rmdir $temp At least this way if accidentally unset $temp, it erases some directory called ~/tmp (/home/dwoit/tmp for me) Just for example. If writing such a shell program, would write something better such as: if [ -z "$temp" ] ... for more default values see man bash (search for := ) Can use set to set value of $1 $2, $3, etc. in shell program or on command line set "val1" "val2"... #sets $1 to "val1", $2 to "val2" ... set -- #unsets them all set - * #sets $1, $2, $3 ... to all files in current dir <<< HERE DOCUMENTS >>> #!/bin/bash #Source: menupgm #prints a menu & executes selected choice #reprints menu for another selection #continues until user selects the exit option # #illustrates use of "here document" # -any string could replace string "here" in pgm # -uses following lines as stdin until # the string "here" appears in col. 1 # while [ true ] do clear #print menu display cat << here MAIN MENU 1) Print working dir. 2) List all entries in current dir. 3) Print date & time 4) Display contents of file 5) Create backup files X) Exit here #prompt for user input using continuation char echo -e "Please enter selection ${LOGNAME}: \c" read selection case $selection in 1) pwd ;; 2) ls -1 ;; 3) date ;; 4) echo -e "Enter a file name: \c" read fname if [ -f $fname ] then echo "Contents of $fname are: " more $fname else echo "file $fname does not exist" fi ;; 5) echo -e "Enter filenames: \c" read fnames for fn in $fnames do cp $fn $fn.bak done ;; X) exit 0 ;; *) echo -e "Invalid choice. Try again \a" ;; esac #pause before redisplaying menu echo -e "\n\n Press return to continue \c" read hold done exit 0 vvvvvvvvvvvvvvv optional vvvvvvvvvvvv Woit shows cleanFiles.sh and cleanFilesFixed.sh ^^^^^^^^^^^^^^^ optional ^^^^^^^^^^^^ END OF WEEK 5 (UNIX) May do u5Lab now