START OF WEEK 2 of Linux <<< FILE TIMESTAMP >>> -The up to date file is: /usr/courses/cps393/dwoit/courseNotes/ -If you are viewing your own copy, check it is up to date -If you are viewing from a link in the Course Outline, be aware it may be outdated. <<< PIPES >>> a pipe | connects stdout of one command to stdin of another e.g., cd /usr/courses/cps393/dwoit/courseNotes/Programs/linux ls | more # lists all files in current dir, one # screen at a time ls -t | head # lists the 10 most recently modified files in this dir ls -t | head >outfile # puts those filenames in file outfile last | head #show first 10 logins on this machine since wtmp #last output has oldest at end. #print line 54 of AlanTuringBio80 cat AlanTuringBio80 | head -54 | tail -1 #print line 54 in reverse character order cat AlanTuringBio80 | head -54 | tail -1 | rev #print the 18-20th-last lines of AlanTuringBio80 tac AlanTuringBio80 | head -20 | tail -3 | tac cat AlanTuringBio80 | tail -20 | head -3 pipes contribute to shell`s usefulness and flexibility -no need to create new commands to perform a task--can *combine* existing commands to perform more complex tasks -no need for "temp" files << TEE COMMAND >> saves the data passing through the pipe writes stdin to stdout *and* to a given file e.g., changing: cat AlanTuringBio80 | head -54 | tail -1 to: cat AlanTuringBio80 | head -54 | tee first54 | tail -1 the same, but ALSO saves first 54 lines in file first54 <<< GLOB CONSTRUCTS, METACHARACTERS AND OTHER SPECIAL CHARACTERS >>> ? * [ ] #some glob constructs #man bash search for: Pathname Expansion saves typing, time encourages good naming conventions when shell encounters a glob construct: -first expands construct -then does resulting command * string of characters (incl. null string) ? single character [...] single char from set of chars within the brackets e.g., assume /home/dwoit has 7 files: lab1.jav lb2.jav new.c pie.c prog1.c prog2.c top.c /home/dwoit> ls *.c new.c pie.c prog1.c prog2.c top.c /home/dwoit> ls * lab1.jav lb2.jav new.c pie.c prog1.c prog2.c top.c /home/dwoit> ls prog?.c prog1.c prog2.c /home/dwoit> ls p*.c prog1.c prog2.c pie.c /home/dwoit> ls ???.* lb2.jav new.c pie.c top.c /home/dwoit> ls prog[123].c prog1.c prog2.c /home/dwoit> ls prog[abc].c #since no expansion, uses literal string ls: cannot access 'prog[abc].c': No such file or directory /home/dwoit> ls [m-s]* # all files starting with # m or n or ... s new.c pie.c prog1.c prog2.c [1-58] matches any single char in 1,2,3,4,5,8 [A-Za-z] matches any single char in A,B,...Z,a,b,...z systems may differ depending on $LC_COLLATE or globalasciiranges, etc. see man bash search for Pattern Matching [!ABC] matches any single char *except* A, B or C [!a-z0-4] matches any single char *except* a,b,...z,0,1,2,3,4 (in some bash, ^ can be used instead of !) These Character Classes can be used within [ ] [:alnum:] [:alpha:] [:blank:] [:cntrl:] [:digit:] [:graph:] [:lower:] [:print:] [:punct:] [:space:] [:upper:] [:xdigit:] /home/dwoit> ls [[:upper:]]* matches files whose name starts with an upper-case letter A-Z What is the difference between these?: ls [[:upper:][:upper:]]* ls [[:upper:]][[:upper:]]* First one matches files whose name starts with an upper case letter Second one matches files whose name starts with two upper case letters -------------------------------------------------------------------------- Optional. If you want to change the collate order in the shell to ASCII you can execute shopt -s globasciiranges Once you do that, then the ordering is A-Za-z (see man ASCII for order), and a command such as ls [A-Z]* will list only those files starting with an upper-case letter, and ls [a-z] only those starting with lower-case letters. You MIGHT already have globalasciiranges set to on. You can see what bash options are set by executing: echo $BASHOPTS If you see globalasciiranges in the list, then it`s on for you. If it`s not on, then do the shopt command above. globasciiranges only pertains to USING globs (as in ls commands), but not to the order that ls displays files. If you want to change the order of displayed files, you still have to change LC_COLLATE, as follows... What gets matched in a range [], depends on the value of LC_COLLATE (use the locale command to display your LC values). Some years, students have LC_COLLATE set to "en_CA.UTF-8". This collates as AaBbCc...Zz. That is why files will be listed in that order, e.g.: Art bart Mart zart If you want to collate according to ASCII order (man ASCII for order), which has, among other things, A-Z before a-z, you can set LC_COLLATE="C". Then the above files would be listed as: Art Mart bart zart You can change it by putting export LC_COLLATE="C" in your .bashrc and .bash_profile. Or, you can change it temporarily, just for one command by setting it just before the command; for example: LC_COLLATE="C" ls -la Or you can change it in the current shell by typing export LC_COLLATE="C" It will stay in effect in that shell until you exit. -------------------------------------------------------------------------- Other special characters: ~ your home directory name (/home/dwoit for prof) ~usr the home dir of user with userid "usr" ~- your previous working dir (note in bash cd - same as cd ~-) ~+ your current working dir \ protects the character following it: echo * vs. echo \* \ at end of line, means continued on next line (see program trump_born) ; separates commands: mkdir newDir ; cd newDir ; touch newFile from any dir, ls ~ lists your home dir (/home/dwoit for prof) cd ~/bin changes into dir /home/dwoit/bin for prof toggle between 2 dirs with "cd -" in bash (or with "cd ~-" ) /home/dwoit> cd bin /home/dwoit/bin> cd - /home/dwoit> cd - /home/dwoit/bin> execute a pgm called myprog in dmason`s dir cps209 /home/dwoit> ~dmason/cps209/myprog # assuming x perms on dirs, and file # exists and has r and x perms /home/dwoit/bin> echo my working dir is ~+ my working dir is /home/dwoit/bin /home/dwoit/bin> echo all the files in this dir are * all the files in this dir are file1 file2 file3 # assuming bin has only those 3 files special characters not expanded if within single or double quotes. escape character \ before any special character works like quotes. /home/dwoit/bin> echo "all the files in this dir are *" all the files in this dir are * /home/dwoit/bin> echo all the files in this dir are "*" all the files in this dir are * /home/dwoit/bin> echo all the files in this dir are \* all the files in this dir are * << PATTERNS AND REGULAR EXPRESSIONS IN SHELL >> -------------------------------------------------------------------------- Optional. The extended globs below require bash option extglob to be set. To see which shell options are set: echo $BASHOPTS If extglob is not set, set it by: shopt -s extglob -------------------------------------------------------------------------- *(exp) 0 or more occurrences of exp +(exp) 1 or more occurrences of exp ?(exp) 0 or 1 occurrences of exp @(exp1|exp2|...) exp1 or exp2 or ... !(exp) anything that does not match exp e.g., if you had files in the current dir: A Ax Axxx Axxxx Ay X X.bak x xx xxxx then shopt -s extglob ls A*(x) prints A Ax Axxx Axxxx ls A*(xx) prints A Axxxx ls A+(x) prints Ax Axxx Axxxx ls A?(x) prints A Ax ls X?(.bak) prints X X.bak ls @(*xx|*ak) prints Axxx Axxxx X.bak xx xxxx ls !(@(*xx|*ak)) prints A Ax Ay X x Note: See "man bash" and search for "Pattern Matching" for help on globs, patterns, etc. HMWK: How would you list all file/dir names that: 1. contained the letter "e"; 2. had a total of 7 characters and no extension; 2a. are 7 characters long; 3. had 3 character extensions; 3a. had 3 character alpha-numeric extensions; 4. contained the word "tst" *somewhere* in the name; 5. started with the letter "A" and contained a dash (-); 6. are at least 2 characters long and do not have the following letters anywhere in the first 2 characters: a,b,c,f,h,x,y,z 7. end in .ex with an optional c at the end 8. are not java or text files (do not end in .jav or .txt) 9. end in a : followed by a digit from 2-8 followed zero or more digits. EOH 1 <<< INTERACTIVE SHELL USE >>> -typing on command line -one-time task -prototyping <<< SHELL PROGRAMS (SHELL SCRIPTS, SHELLS) >>> -executable file containing shell commands -can run like a program (type its name on command line) Why? -task is complex -repeat it later e.g., #!/bin/bash #source: showSource #shell program to list all source java and c files in current directory echo -n "Current directory is: " pwd echo "C files are:" ls *.c echo "Java files are:" ls *.jav To run: Make sure it is executable (x permissions for user running it), then > cd /usr/courses/cps393/dwoit/courseNotes/Programs/linux > ./showSource Current directory is: /usr/courses/cps393/dwoit/courseNotes/Programs/linux C files are: blah.c Java files are: blah.jav color.jav pgm1.jav TextStackViewer.jav ttt.jav VisibleStack.jav xyz.jav > > cd #for prof > cp /usr/courses/cps393/dwoit/courseNotes/Programs/linux/showSource . > ./showSource Current directory is: /home/dwoit C files are: ls: cannot access '*.c': No such file or directory Java files are: DAC.jav skipList.jav ---------------------------------------------------------------------------------- Optional: To turn on syntax highlighting in vi(m) (if not already on): -when in vi(m) editing a shell program go into command mode (hit esc a few times) type :syntax on behold the nice colors -if highlighting seems ugly (black/white?) then make your terminal an xterm first. exit vi(m) so you are in the shell type export TERM=xterm-color now try going back into vi(m) and turning highlighting on as above If vim can tell the type of file, it uses that highlighting, e.g., vim myfile.c vim understands it a C program file and hilights accordingly. If vim cannot figure it out (e.g., C file named myfile), then force it by telling vim to set the filetype, as in :se ft=c To highlight a bash program use :se ft=sh ---------------------------------------------------------------------------------- Shell Script Arguments: $1 $2 $3 etc. $0 name of shell pgm $# the number of args passed $@ lists args as one string: "arg1 arg2 ..." $* lists args as separate: "arg1" "arg2" ... e.g., #!/bin/bash #source: prnargs #shell pgm to print out its first 2 args echo The first argument is: $1 echo The second argument is: $2 echo dollar zero is: $0 exit 0 To run: Make sure it is executable, then /home/dwoit> cp /usr/courses/cps393/dwoit/courseNotes/Programs/linux/prnargs . /home/dwoit> ./prnargs "Hello there" world The first argument is: Hello there The second argument is: world dollar zero is: ./prnargs Note: spacing in program differs from spacing in output Why? echo squeezes multiple whitespace to single space Could line it up with quotes: echo "The first argument is: $1" echo "The second argument is: $2" Could line it up with a horizontal tab (see man echo) echo -e "The first argument is:\t $1" echo -e "The second argument is:\t $2" NOTE: Could leave out the ./ if current dir in your path. e.g., /home/dwoit> prnargs (instead of ./prnargs) If it says prnargs "not found" you must put the current dir in your path e.g., in your .profile or .bash_profile or .bashrc at the end put: PATH=$PATH:: export PATH Must login again for it to take effect. Now, whenever you login, current dir always in path. <<< FILTERS >>> filter: refines or transforms its input to produce (usually different) output most shell commands are filters. stdin -------> filter --------> stdout e.g., more > (globally search for a regular expression and print it) grep string filename(s) string: the string to search for filename(s): one or more files to search in result: displays lines of file(s) that that contain string grep string searches stdin instead of file(s) sends to stdout lines of stdin that contain string ctrl-d "ends" stdin when you are typing on keyboard e.g. grep 'applet' lab1.jav lab2.jav sends to stdout (screen) those lines of files lab1.jav and lab2.jav that contain the string applet /home/jchan> grep 'applet' lab1.jav lab2.jav lab1.jav: applet.init(); lab1.jav: applet.start(); lab2.jav: applet.num=6; e.g. You forgot the exact syntax for 'for' construct in java. You know one of your .jav files uses 'for'. Do grep 'for' *.jav to get a list of all files/lines containing for Then edit or more one file to see complete for construct Note: single quotes are good for now. Once we study variables, you may have reason to use double quotes around the search string. Options: -i ignore case of search string -v print lines *not* matching search string -x search string must match ENTIRE line -w search string must match a whole word (man grep gives all the options) The grep search string above can be specified as a *Regular Expression* i.e., can contain *metacharacters* that work something like glob constructs . any character, except newline (like ? in glob) * 0 or more repetitions of previous character (unlike glob) ^ beginning of line $ end of line [...] any character inside the brackets (like glob) [^...] any character not inside the brackets (like ! in glob) \{m\} exactly m repetitions of previous character \{m,\} at least m repetitions of previous character \{m,n\} any number of repetitions of prev char between m and n inclusive \< beginning of word \> end of word remember to quote these as in: grep '\' a_file \(str\) group string str To match a special char, such as $ you need \$ e.g., searching in file fname (and later stdin): grep '^Assignment' fname # lines starting with Assignment grep -v '^Assignment' fname # lines not starting with Assignment grep 'Assignment$' fname # lines ending with Assignment grep 'd.g' fname # lines containing dag, dbg, dcg, d0g, d1g, etc grep 'su*m' fname # lines containing sm, sum, suum, suuum, etc grep 'suu*m' fname # lines containing sum, suum, suuum, etc grep '\' fname # the word so (vs. social, absolute) grep '[A-Za-z][A-Za-z]*' fname # lines containing ANY alpha string (no blank # lines, no lines with only digits, etc) grep '^[A-Za-z][A-Za-z]*$' fname # lines containing ONLY alpha characters grep 'xyz\.[^ ]* ' # lines containing string xyz followed by a dot followed by 0 or more non-spaces, then one space e.g., "xyz. " "xyz.w " "xyz.ta " etc grep 'xyz\.w\{2,3\}X' # lines containing xyz.wwX or xyz.wwwX grep 'A\(bc\)\{2,3\}D' # lines containing AbcbcD or AbcbcbcD grep -x "abc" # lines that contain exactly and only abc Extended Regular Expressions: use grep -E (or egrep on older versions) | OR + 1 or more repetitions of previous character ( and ) do not need escaping (substring) { and } do not need escaping (repetition) \n backreference: \1 replaced by substring 1, \2 by substring 2, etc e.g., grep -E 'dog|cat|bird' fname # lines with at least 1 of dog, cat, or bird grep -E '[A-Za-z]+' fname # lines with at least one alpha char grep '[A-Za-z][A-Za-z]*' fname # same as above without -E grep -E 'xyz\.w{2,3}X' # lines containing xyz.wwX or xyz.wwwX grep -E 'A(ab){2,3}B' # lines containing AababB or AabababB grep -E 'A([aei]+)B(.*)C\2\1' # lines containing strings such as: # AieiBxyzCxyziei or AiBCi etc Note1: See "man grep" and search for "REGULAR EXPRESSIONS" for help on them Note2: All the examples above could use stdin instead of a file, as in: grep '\' then it will search whatever you enter on stdin. ctrl-d to end input Note3: All the examples above could search multiple files, as in: grep '\' file1 xyz.? rt* HMWK: In the shell, is grep '[A-Za-z][A-Za-z]*' fname the same as grep -i '[A-Z]*' fname ? Why or why not? Why do you think "\<" and "\>" are used instead of simply "<" and ">" for beginning/end of line? For questions 1-6 below, write a different shell program for each. 1. list all lines in file fname containing the string "dog"; 2. list all lines in fname containing the word "dog"; 3. list all lines in fname containing the string "dog" at the beginning of a line; 4. list all lines in fname containing the word "dog" at the beginning of a line. 5. list all lines in fname containing exactly 6 "a"s in a row. 6. list all lines in fname containing one or more words that start with 93 and end in a sequence of any number of W (not 0) 7. Give 2 different grep commands that will list lines of fname that end in a lower-case vowel. 8. Use the man page to find the option of grep that shows the matched string in a different color. Try it. 9. How would you change your program for #1 above so that instead of doing grep directly from the file, it cat`s the file, then pipes that to grep which greps from stdin? 10. Print the first 3 lines of AlanTuringBio80 that contain the string 'machine' in any case. Use grep, pipe(s), and head. EOH 2 << FIND (LIST ALL FILES AND DIRS MEETING CERTAIN CRITERIA) >> e.g., this lists all java files in portion of the filesystem rooted at /home/dwoit: find /home/dwoit -name "*.jav" ^ ^ | | | | | match any entry with name *.jav (glob) dir to search (this dir and all its subdirectories, recursively). find . -type f # lists all *files* in filesystem rooted at the current dir (.) (i.e., no directories listed) find . -type d # same, but *directories* find /usr -name tali 2>/dev/null #if takes too long ctrl-c and... find /usr/[!c]* -name tali 2>/dev/null #eliminate /usr/courses #or use -prune (see below) find . -type d -perm -g+r,u+r # all dirs rooted in current dir that are readable by BOTH owner and group find . -type d -perm /g+r,u+r # are readable by EITHER or BOTH owner and group find . -type d -perm /a+w # have at least one write bit set find . -type d ! -perm /a+x # are not executable for anybody LOTS of ways to specify perms--see man find cd /usr/courses/cps393/dwoit/courseNotes/Programs/c find . -name "m*.c" #try it. note results from ./c1 ./c2 ./c3 and ./c4 find . -name c2 -prune -o -name "m*.c" -print #does not recurse down dir ./c2 #note -o is "or". #need -print to print the one that recurses #otherwise dir name ./c2 itself is still printed #-print is normally implicit. However, if explicitly # include it, then only commands WITH -print get printed # and not any other commnds (just right-side of -o and # not left side, since left has no -print) #eliminate /usr/courses using -prune: find /usr -name courses -prune -o -name tali -print 2>/dev/null LOTS of options to find. Some useful ones are: -newer xyz #list entries that have been modified more recently than file xyz -maxdepth 1 #descend only 1 level down directories (can use any integer) Why quote arguments to -name?: If argument has unquoted glob constructs, bash EXPANDS it first and then sends result to find. Sometimes this is a problem. e.g.: cd /usr/courses/cps393/dwoit/courseNotes/Programs/c find . -name "m*.c" #versus find . -name m*.c Give same result. However... If /usr/courses/cps393/dwoit/courseNotes/Programs/c has a file named mX.c, the two commands above are NOT the same because bash turns second one into: find . -name mX.c If current dir has files named mX.c and mY.c, unquoted find gives error because too many arguments to -name. Quoting stops bash expansion. The -follow option: Use -follow to tell find to recurse down symbolic links. Not relevant unless there ARE symbolic links in your search Some years, student home directories are symbolic links, so you need -follow to search it. E.g., /home/dwoit IS a symbolic link, so I need: find /home/dwoit -type d -follow The -> (arrow) in the result below says /home/dwoit is a symbolic link: ls -ld /home/dwoit lrwxrwxrwx 1 root root 12 Feb 23 2018 /home/dwoit -> ../fac/dwoit/ HMWK: 1. Use the "find" command to list all entries in directory /bin that start with the letter "m". 2. What would happen if we did not put a name such as "*.jav" in quotes in the find command? Make sure you have some .jav files in your directory and try it out. 3a. Look at the man pages for "find" to determine what the option "-mtime" does in the find command. Then... Figure out a sequence of commands (involving find, temporary files, and grep) to display those FILES in the current directory whose contents were changed within the last 24 hours and whose names contain the string "tst". 3b. How would you do 3a using a pipe and no temporary files? 4a. Make a shell program to do question 3a above. Call your program tstRecent Your program should delete any temporary files it creates. Test it to make sure your program works. 4b. Rewrite your program from 4a to use pipes instead of temp files. 5. Use the man pages to find out how find`s -maxdepth option works. Use -maxdepth 1 to list all entries in the given directory only (not the whole filetree under it) << MORE ADVANCED HEAD, TAIL OPTIONS >> Saw previously: head sends to stdout the first 10 lines of stdin tail sends to stdout the last 10 lines of stdin They can take filenames as arguments: head -2 f1 f2 f3 sends first 2 lines of f1, f2, f3 to stdout They use stdin if no filename(s) given: head -2 > cut extract specified input columns paste combine input columns both take input from stdin or files e.g., cut -c8 myfile # outputs 8th column of each line of myfile cut -c5-7,25- # outputs columns 5,6,7,25... of each line of stdin cut -f2 myfile # outputs second field of each line of myfile # (assumes fields separated by a tab) cut -d' ' -f3 # outputs field 3, fields delimited by one space paste f1 f2 # linei to stdout contains linei of f1 followed by # a tab followed by linei of f2 paste -d' ' f1 f2 # uses single space instead of tab Aside: If you have things separated by tabs in a file, and want the tabs converted to spaces, use the command: expand cat -A will show tabs, newlines, etc. (see man page) HMWK: (1) Create a file called stdnt-file with last-names in column 1, first-names in column 2, id numbers in column 3, userids in column 4. Populate your file by adding some fake students. What combination of commands (and possibly temp files) can you use to create a file, stdnt-ids, that contains only the id numbers and userids from stdnt-file? (2) How can you combine commands (and possibly temp files) to create another file, stdnt1, that contains only the first and last students in stdnt-file? (3) Display the time userid dwoit (or whoever you want) logged in on some connection. You will find the "who" command useful. It tells you who is logged in, and what they are doing. For an explanation do: man who Note that "dwoit" (or whoever) will likely have multiple entries (one for each "terminal" connection), so your output may have multiple lines. Some shell programs: e.g., #!/bin/bash #source profs #prints names of profs logged in who | grep dwoit | cut -c1-8 | uniq who | grep eharley | cut -c1-8 | uniq who | grep aabhari | cut -c1-8 | uniq who | grep -w mes | cut -c1-8 | uniq #string in other userids exit 0 #or who | grep -Ew "dwoit|eharley|mes" | cut -c1-8 | sort -u Try this: use any of ls, cut, sort, uniq, head, tail and pipes to get a listing of all the different GROUPS files/dirs belong to in the current directory. e.g., #!/bin/bash #source prof1 #prints names of a prof logged in #name supplied as command line argument who | grep -w $1 | cut -c1-8 | uniq See what happens if you use prof1 without an argument. How do you fix it? (hint: use quotes) Note 2 pipes and line continuation character \ #!/bin/bash #source: trump_born echo "Donald Trump's Birthday: " lynx -dump "https://en.wikipedia.org/wiki/Donald_Trump" 2>/dev/null | \ grep "Trump.*born" | head -1 echo "" exit 0 Program grande_born replaces Donald_Trump with Ariana_Grande Program when_born takes arguments Firstname and Lastname instead of hardcoding HMWK: Use various commands and pipes to do the following (do not use any temp files): 1. List, in long form, all those files in your home directory that have read, write and execute perms for you, and whose name contains the string "txt". 2. Do the above question again, but list in short form (just the file names.) 3. List any lines in the first 10 lines of a file, myf, that contain the word "dog". (you should create different versions of myf to test your commands.) 4. Find all lines of file, myf, that contain all the words "cat", "dog" and "mouse", in any order, and start with the letter "A". 5. List all the files in your home dir that contain the string "so" somewhere within them. Next, just list the *number* of files in your home dir that contain the string "so" (you should use, among other commands, the command "wc -l", word-count (lines option), which you should look up in the man pages.) 6. Use only ls, grep, cut, pipes to answer this question. What sequence of pipes and commands could you issue from your home dir to list, in alphabetical order, the absolute path names of all the directories in your file system (if you NEED to, use the sort command.) 7. List the names of any subdirectories of your home directory that have rwx permissions for user and group, and no permissions for others--note: list just the directory names in short form (the names only.) 8. List just names of all files/directories that have no extension; 9. List just names of all files/directories that contain no vowels at all. 10. Write a shell program called nw that takes either zero or one command line arguments. nw with no arguments should print out the 10 newest (most recently modified) files/dirs in the current directory. nw -n (nw with one argument, -n) prints out the n newest files/dirs in the current directory. e.g., nw -3 prints out the 3 most recently modified files/dirs If the argument is incorrect, then your program is allowed to do unpredictable things! 11. List names in reverse alphabetical order, of directories that have "r" permissions for "other". (the "reverse" option of the sort command may be useful.) 12. List in alphabetical order all files in the current directory that have been edited exactly 3 days ago. Do not list the same file more than once. (the uniq command will eliminate duplicate lines). END OF WEEK 2 (UNIX) May do u2Lab now