------------------------------------------------------------------------------- Filename Path, Basename, and Suffix file="/path/to/the/file.sfx" Extract Path (NOTE: may be empty) path="`expr "$file" : '\(.*\)/'`" : ${path:=.} Remove Path (basename alternative) basename="`expr "//$file" : '.*/\([^/]*\)'`" Get Suffix (path kept if present) suffix="`expr "$file" : '.*\.\([^./]*\)$'`" Remove Suffix (path kept if present) name="`expr "$file" : '\(.*\)\.[^./]*$' \| "$file"`" Example file processing loop of mine (path is often omitted as it is not used) for i in "$@" ; do [ ! -r "$i" ] && { echo "No such file \"$i\""; continue; } path="`expr "$i" : '\(.*\)/'`" # get file path (if any) name="`expr "//$i" : '.*/\([^/]*\)'`" # remove path to file suffix="`expr "$name" : '.*\.\([^./]*\)$'`" # extract last suffix name="`expr "$name" : '\(.*\)\.[^.]*$'`" # remove last suffix : ${path:=.} ... done Bash (zsh, ksh93) methods -- note the pairings > f=file.tar.bz2 > echo "${f##*.}" bz2 > echo "${f%.*}" file.tar > echo "${f#*.}" tar.bz2 > echo "${f%%.*}" file Check input file fully. FILE="${1:-/dev/stdin}" [ ! -f $FILE ] && Usage "Error \"$FILE\": does not exists" [ ! -r $FILE ] && Usage "Error \"$FILE\": is not readable" What file descriptors are open (including the 'ls' to the /dev/fd directory) ls -l /dev/fd The permissions define if the descriptor is actually read or write ------------------------------------------------------------------------------- Copy-N-Paste Problem... When you copy and paste text from any Xwindow application (For example: XTerm, "vim" in XTerm, GVim, Gedit, parcellite (clipboard), xselection, ....) The lines pasted contain RETURN characters as the end-of-line markers. For Example try Copy-N-Paste this to the command line... cat -v - This is a multi-line file for Copy-N-Paste Testing. (Press the 'return' key see results. Followed by ^D for EOF) The output from copy will only show RETURN chars, without expected NEWLINE characters. Many things do not handle this properly (including "cat") This can be a real problem when you create Copy-N-Paste HERE files. Bash automatically handles such newlines, but many other applications do not. This can make Copy-N-Paste into anything that is not a shell or a editor a real problem, without some 'fixer' program. ------------------------------------------------------------------------------- HERE, Here, here files can be fun! Important: See previous Copy-N-Paste problem. echo <<'EOF' This is documentation that would be printed as it appear here, until it sees the defined label on a line by itself. The quotes around the label define the quoting used within the herefile itself. As such "" can be used to allow variable substitution within the here file. The problem is that you can NOT indent the text with ANY whitespace EOF echo <<-'EOF' This used to have tabs on the front that you can use to indent the documentation. The tabs will be removed... However it relies of TABs and loosing them can cause the shell to miss the EOF marker. Tabs are not something that you can gurantee will not be lost! EOF sed 's/^ *| //' <<-'EOF' | This is a space indented documention up to the 'bar'+'space'. | This looks neater than a normal tab indented 'herefile'. | | It also preserves indenting in the document! | Like these last two lines. | | However the final 'EOF' label still needs to be either hard to | the left, or tab indented. I hate relying on tab indents, you | can't see them and they are often lost during code editing! EOF THE IDEAL SOLUTION... Use single quotes for the DATA block. This solves both problems (for bourn shells) And can be used in 'copy and paste' documentation. Only problem is you need to escape single quotes. echo ' | Single '\''Quotes'\'' | Double "Quotes" | Indented Lines | # Hash Comments are preserved | (or can be removed by sed if desired) | | Variable Inclusion such as HOME='"$HOME"' | | Basically this works extrememly well, without problems | # This is a comment and is ignored! ' | sed -n '/^ *|/!d; s/^ *|//; s/^ //; p' > FILE NOTE the 'echo' not only removes the need for a tab indented EOF, but also removes the need for end of data control characters. The 'sed' script also stops the problem of differences with the EOL caused by Copy-N-Paste problem. Perl version (much simplier, and handles tabs and spaces) echo ' | Line One | Line Two | Line Three # This is a comment and is ignored! ' | perl -ne 's/^\s*\| ?// && print' > FILE NOTE that any line not starting with a 'bar' is ignored, so you can include installation comment in the here file. This is a version that also removes comments from inside the 'here' file Useful for those files that normally can't accept comments, such as the firewall configuration file. echo ' # append these ports to firewall | --port=111:tcp # rpcd - the portmapper, to find these ports | --port=111:udp # --port=662:tcp # statd ??? # --port=662:udp | --service=nfs # port tcp:2049 ' | perl -ne 's/\s*#.*$//; s/^\s*\| ?// && print' \ >> /etc/sysconfig/system-config-firewall This one flushes and pauses after each line of input. It also outputs a copy to stderr... echo ' | HELO xyzzy | MAIL FROM: root@nowhere.land | RCPT TO: anthony | DATA | From: A.Thyssen@test | To: no-one-really@foobar.gu.edu.au | Subject: Testing | | This is a test | . | QUIT ' | perl -ne '$|=1; s/^\s*\| ?//||next; print; print stderr; sleep 1' | socat - tcp:localhost:smtp ------------------------------------------------------------------------------- Safe Temporary Files. Single File... umask 77 tmp=`mktemp "${TMPDIR:-/tmp}/$PROGNAME.XXXXXXXXXX"` || { echo >&2 "$PROGNAME: Unable to create temporary file"; exit 10;} trap 'rm -f "$tmp"' 0 trap 'exit 2' 1 2 3 15 command > "$tmp" Note the temporary file is already created with appropriate restricted permissions. The $TMPDIR is a typical environment variable users often set to specify a different 'personal' directory for temporary files (generally for disk space and security). Multiple Files (directory)... If Multiple temporary files are needed, a better idea is to create your own temporary directory (using a -d flag) tmpdir=`mktemp -d "${TMPDIR:-/tmp}/$PROGNAME.XXXXXXXXXX"` || { echo >&2 "$PROGNAME: Unable to create temporary directory"; exit 1;} trap 'rm -rf "$tmpdir"' 0 # remove when finished (on end or exit) trap 'exit 2' 1 2 3 15 # terminate script on signal (don't just die) command > "$tmpdir/A" Now you can use any filename in the temporary directory, and all will be automatically cleaned up when finished. The alternative to using "mktemp" is to just use "$PROGNAME.$$" though that may still clash. OR in BASH use "$PROGNAME-$$-$RANDOM" in sh you can use "$PROGNAME-$$-`awk 'BEGIN { srand (); print rand() }'`" so as to ensure there is less likely hood of a clash ------------------------------------------------------------------------------- Creating a Large File This generates a large file that does NOT leave any 'holes' in disk usage. Create a 8Mbyte byte file dd if=/dev/zero of=/export/swap/XXX bs=8k count=1024 Fill a whole disk partition with 4 Mbyte files (a poor mans "shread") dd if=/dev/zero of=00000 bs=4k count=1024 i=0; while i=$(($i+1)); do cp 00000 `printf '%05d' $i` || break; done ------------------------------------------------------------------------------- Generating file sequences... See "generating a list of numbers" in "script.hints" http://www.ict.griffith.edu.au/anthony/info/shell/script.hints Saving data into numbered files... Just save files in sequence... for num in `seq -f %03d 999`; do #num=`printf %03d $num` # do whatever and save to next sequence file touch file_$num.suffix done For restartable processes... Start with the next file number that is not present (ignoring gaps). This will continue to generate files in sequence even if gaps are present in the previously generated files. # Determine the checkpoint number for the last file generated num=`ls file_*.suffix 2>/dev/null | tail -n1 | sed 's/[^0-9]//g'` [ ! "$num" ] && num=0 # set to zero if no previous file found while :; do num=`expr $num + 1` # next file number to be created num=`printf %03d $num` # do whatever and save to next sequence file touch file_$num.suffix # break when finished! done The above will handle multiple processes, and files being randomly deleted. But it may need a locking mechnisim on the determination of the next numbered file to be generated, if mutliple processes could be creating files. ------------------------------------------------------------------------------- File Renaming "mved" is a shell script basied on sed. http://www.ict.griffith.edu.au/anthony/software/#mved mved =.01 = "rename" under ubuntu or "mv_perl" from http://www.ict.griffith.edu.au/anthony/software/#mv_perl rename 's/\.01$//' *.01 A bash solution for i in *.01; do mv "$i" "${i%%.01}"; done ------------------------------------------------------------------------------- Reading Files. Shell built-in (bash) -- is faster time `IFS=$'\n'; for I in $(seq 1 1000); do B=($(<".plan")); done` real 0m0.913s Cat time for I in $(seq 1 1000); do B="`cat .plan`"; done real 0m1.403s Sed time for I in $(seq 1 1000); do B="`sed 's/^/|/' .plan`"; done real 0m1.805s Awk time for I in $(seq 1 1000); do B="`awk '{print $0}' .plan`"; done real 0m1.833s ------------------------------------------------------------------------------- Shell File Descriptors... open file permanentaly, read lines one at a time, close exec 5<&0 0/dev/null` Good to read single characetrs from stdin in "raw" mode.. See below... This can also be used to extract say 3 bytes (count=3) at position 5 (skip=4) echo '1234567890' | dd bs=1 skip=4 count=3 2>/dev/null; echo '' 567 The 2>/dev/null junks the dd report it outputs to STDERR the final 'echo' is to add a extra return character to the end of the 3 bytes 'dd' extracts. WARNING: null characters will not be hanlded by shells correctly! Also see next sections. ------------------------------------------------------------------------------- Turn off input echo /bin/stty cbreak -echo /dev/tty 2>&1 read password /bin/stty -cbreak echo /dev/tty 2>&1 Also See http://www.ict.griffith.edu.au/anthony/info/crypto/password_input.txt ------------------------------------------------------------------------------- Read any character from file stream (BASH) I uploaded this to LinuxQuestions Tutorial http://www.linuxquestions.org/linux/answers/Programming Example Solution... printf 'a\tb\rc\nd\0e' | while IFS= read -r -s -d '' -n1 char do if [[ "$char" == '' ]]; then echo -n "-NULL-" elif [[ "$char" == $'\r' ]]; then echo -n "-RETURN-" elif [[ "$char" == $'\n' ]]; then echo -n "-NEWLINE-" elif [[ "$char" == $'\t' ]]; then echo -n "-TAB-" else echo -n "$char" fi done echo "" result a-TAB-b-RETURN-c-NEWLINE-d-NULL-e perfect! The loop exits (read returns false) on EOF only. NOTE that read will only return true if it actually reads a character! If read returns true BUT the variable "$char" is empty then either a NULL character was recieved, or a delimiter was recieved (turned off) The IFS is need to prevent the shell ignoring leading whitespace or separators. Same goes for -d (or return will be regarded as EOL). The -r is needed to disable backslash escaping. And finally the -s turns of echoing. If the -d was not given then a NEWLINE is a delimeter, and the result will become.. a-TAB-b-RETURN-c-NULL-d-NULL-e Note "printf" built-in was used so we can output a real NULL character. printf 'a\tb\rc\nd\0e' | od -xc ------------------------------------------------------------------------------- Read any character with timeout (BASH only) Note this does handle signal processing (^C interupt or ^D vs EOF) handling For that see the non-BASH method (using "stty" and "dd") below. Also see http://www.ict.griffith.edu.au/anthony/info/crypto/password_input.txt IFS= read -r -s -d '' -n 1 -t 2 "${1:-KEY}" If a -t (timeout) option was also used, then you need to test if the return status from read more throughly. There are four conditions to the read... Normal character ("$char" is not the empty string) printf 'a' | IFS= read -r -s -d '' -n 1 -t 2 char; echo $? 0 echo "$char" a NULL character recieved ("$char" is assigned the empty string) printf '\0' | IFS= read -r -s -d '' -n 1 -t 2 char; echo $? 0 [[ -n $char ]] && echo NULL NULL EOF recieved ("$char" not modified - ignore it) : | IFS= read -r -s -d '' -n 1 -t 2 char; exit $? 1 Timeout (142 = 128 + 14 or SIGALRM) sleep 3 | IFS= read -r -s -d '' -n 1 -t 2 char; echo $? 142 Probably similar for broken pipe! If you use a zero timeout " -t 0 " so as to 'poll' to see if there is any input waiting, then you will not get a timeout exit status. Instead (at least on my machine) I get a exit status of 1 the same as for an EOF condition, so you can not determine between a EOF or 'No Character' conditions. The only method I have found is to use a small perl script to make a IO library 'select()' system call to see if input is available before doing the "read". But that is not pure-BASH. See http://www.ict.griffith.edu.au/anthony/software/#shell_select ------------------------------------------------------------------------------- Read a single character (non-BASH) Including NULL characters. readkey() { old_tty_settings=`stty -g` #stty -icanon && /bin/stty eol ^A # for SYSV stty stty cbreak -echo /dev/tty 2>&1 key=`dd if=/dev/tty bs=1 count=1 2>/dev/null` # read a key press stty "$old_tty_settings" echo "$key" } echo -n "Enter a character: " key=`readkey` echo "Thank you for typing \"$key\"" =======8<-------- For say a 4 second timed key input add this to the stty... min 0 time 40 OR add a -t 4 for the BASH The stty setting defaults for this is: min 1 time 0 NOTE: that stty modes are tricky, particulaly when giving it control characters as arguments. For example linux stty command will accept the two characters "^A" to represent the single character ctrl-A, while other UNIX machines must actually recieve a single "ctrl-A" (character code 1) character, to work as expected. An alternative "readkey()" function presented from a eZine "Shell Corner" This also handles users inputing signal characters (like ^C) =======8<-------- readkey() { oldstty=`stty -g` stty -icanon -isig -echo min 1 time 0 dd bs=1 count=1 <&0 2>/dev/null stty $oldstty } key=`readkey` # read in a key from user and convert to octal character code echo "Enter a key" charcode=`readkey | od -b | awk '{ print $2 }'` # If user pressed Ctrl-C (character code 3) fake an interupt if [ $charcode = 003 ] ; then echo "Interupt!" exit 1 fi # output the octal character code of key pressed echo "$charcode" =======8<-------- This uses the output of the stty itself to reset it to normal, turns off signals (you will have to look for interupt characters yourself). To avoid problem about just what character was pressed by the user, the character was immediately piped into a "od" command to convert to a octal character code. You can put the script into that special stty mode at the start, and only exit that mode using a trap trap 'stty $oldstty' EXIT ------------------------------------------------------------------------------- Read Line with default # USAGE: readline var prompt default bashversion=${BASH_VERSION%%.*} if [ ${bashversion:-0} -ge 4 ]; then ## bash4.0 has an -i option for editing a supplied value readline() { read -ep "${2:-"$prompt"}" -i "$3" "$1" } elif [ ${BASHVERSION:-0} -ge 2 ]; then # bash 2 and 3 readline() { history -s "$3" printf "Press up arrow to edit default value: '%s'\n" "${3:-none}" read -ep "${2:-"$prompt"}" "$1" } else # Other older bourne shells readline() { printf "Press enter for default of '%s'\n" "$3" printf "%s " "${2:-"$prompt"}" read eval "$1=\${REPLY:-"$3"}" } fi ------------------------------------------------------------------------------- Read a Line from Stdin with a Timeout (non-BASH) WARNING: this really only works from a script! Do not use cut and paste to try this. Using Stty (fails) # read with 4 second timeout old_tty_settings=`stty -g` stty min 0 time 40 read input stty "$old_tty_settings" echo $input Note the stty timer is reset after every keystoke! It is not a total timeout. Timed version (fails) : #Dan Mercer: damercer@mmm.com wake_up() { echo "Morning already" } trap wake_up USR1 (sleep 20;kill -usr1 $$) & echo "Waiting for answer..." read ans echo "Here we are" The read however does not abort on modern systems. It used to be that the system call is cancaled on signal input. Ideally we would background the read, and kill it when the sleep completes, but we can't get the result of the read from a background session! ------------------------------------------------------------------------------- How do I get a particular line or range of lines from a file? Using absolute Line numbers use first line only head -1 sed q first N lines head - sed q awk '{print} NR== {exit}' awk 'NR== {exit} {print}' single line sed -n 'p;q' sed 'q;d' range of lines sed -n ',$p;q' sed ',$!d;q' last line sed '$!d' last two lines sed '$!N;$!D' second last line sed '$!{h;d;}; x' last lines tail + sed -e :a -e '$q;N;,$D;ba' NOTE: The `q' in the sed commands above is to avoid un-needed computer cycles, similarly for the exit in the nawk script below. Sed could also replace the head and tail commands, in a similar fashion, but is slower. The problem with the above is if you want the lines relative to both the start and the end of the file at the same time. "Sed" is a stream editor but if you are dealing with a REAL file and not a pipe you can also use "ed" All lines but last 15 lines echo '1,$-15 p' | ed -s file or sed -e :a -e '$d; N; 2,15ba' -e 'P;D' file The 10th to the 5th last line echo '$-10,$-5 p' | ed -s file A random line from a file however has fewer simple solutions, without having to calculate to first find the number of lines in the file before hand. For example, this does not work though logically it should... nawk ' BEGIN {srand(); RNUM = int(LNUM * rand())+1} NR == RNUM {print $0; exit 0} ' LNUM=`wc -l < file` file problems RNUM = 0 = no output, RNUM = 1 = first line, RNUM = LNUM-1 = 2rd last line RNUM = LNUM = NO OUTPUT <--- The problem A perl solution was given in the perl cookbook, which only requires a single pass though a file to extract a single random line. It works on incrementally adjusting the probabilities as each line is read, so any line gets a equal chance of being picked, in one pass. ------------------------------------------------------------------------------- Print lines between markers AAAA to BBBB sed -n '/AAAA/,$p; /BBBB/q' file Excluding the end marker sed -n '/BBBB/q; /AAAA/,$p' file Excluding both marker sed -n '/BBBB/q; 1,/AAAA/d; p' file Print Paragraphs that contain AAA BBB or CCC sed -e '/./{H;$!d;}; x;/AAA/!d;/BBB/!d;/CCC/!d' ------------------------------------------------------------------------------- uniq without sorting! perl -ne 'print unless $a{$_}++' ------------------------------------------------------------------------------- Delete lines in one file from another file Situation: you have a master file "master" and you want to remove all entries given in the file "list" from that master list... If the sorting of the master file does not matter then ou can use... cat master list list | sort | uniq -u > new_master An alturnative I usally use is to use comm on sorted versions of the lists. sort -o master master sort -o list list comm -23 master list > new_master Of course you can also get the items which are not present in the master file, and those which are in BOTH files (the union) with this method to (just different 'comm' options). fgrep??? This is untested fgrep -v -f list master > new_master WARNING; it does not limit test to whole lines! small list elements could sub-string match a larger longer line!! ------------------------------------------------------------------------------- Delete ALL blank lines sed '/^$/d' file Compress multiple blank lines to one line (paragraph separators) sed '/^$/{ N; /^\n$/D; }' file sed '/./,/^$/!d' file cat -s perl -00 -pe '' # WOW! perl -ne 'print if /\S/../^\s*$/' perl -ne 'if (/\\S/){print;$i=0}else{print unless $i;$i=1}' vi file :v/./,/./-j :wq ------------------------------------------------------------------------------- Reverse the line order in a file 1/ My first idea was to put some line numbers in front of infile (e.g. using 'pr' oder 'nl' or the like), than sorting using `| sort -r -n +0 -1 |' and removing the line numbers afterwards (e.g. with `sed'). Seems to be a little bit awkward... --- Peter Funk pf@artcom0.north.de 2/ Awk it awk '{x[NR]=$0}END{for(i=NR;i>0;i--){print x[i]}}' infile >outfile NOTE: this will barf of extreamly large files. However the sort solution works on large files --- dsilvia@blunt.net.com Dave S. 3/ A GNU text utility "tac" the reverse of "cat"! --- Noah Friedman friedman@gnu.ai.mit.edu 4/ a ed 'in-place' solution (twice as fast as the above awk solution) ed - infile <<-EOF g/^/m0 w EOF NOTE: this will gag on long lines instead of long files NOTE that :g/^/m0 will also work directly in vi too 5/ the perl solution is very simple! perl -e "print reverse <>" 6/ The utility "rev" does this (seeks and line start?) 7/ sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//' ------------------------------------------------------------------------------- reverse characters in a line sed -e ' # skip lines with lest than two characters /../! b # Embed newlines at both ends # then slowly meove markers toward middle :x s/\(\n.\)\(.*\)\(.\n\)/\3\2\1/ tx # remove newline markers s/\n//g ' ------------------------------------------------------------------------------- Lock File (file creation) Methods for lockfiles File Permission ( file mode = 000 ) Create a lockfile with zero permission. The example below uses this method in shell. It creates the lockfile and places the current pid of the process in it. NOTE a trap should be provided to remove the lock file on any abnormal exit by the program running this. WARNING: this technique does not work for ROOT which will always succeed to create the file even if it exists. Lockfile() { # create a lockfile with the process ID in it masksave=`umask`; umask 777 ( echo $$ > $1 ) 2>/dev/null; success=$? while [ $success -ne 0 ]; do sleep 2 ( echo $$ > $1 ) 2>/dev/null; success=$? done umask $masksave } Exclusive Open This requires the open(2) command to use the O_CREAT & O_EXCL flags to ensure that a file is opens if it had to be created. The csh `noclobber' flag should do this (Look at source). NOTE: This does NOT work over NFS (unless version 3 release) Hard Links to a file `ln' This works for root but on System V the `ln' command removes any file that the link is created for (ala `mv' ) as such on System V this fails for both users and root. This is the method normally used for passwd file locking. NOTE: This method is known to work over NFS Symbolic Links `ln -s' It is not known if this has the same problem on System V or if this is atomic over NFS. File rename using rename() Create a unique file with the process PID /tmp/data.lock.$PID then rename it to the lock filename /tmp/data.lock lock directory instead of a file `mkdir' This should work properly in all cases but is unknown if this is atomic over NFS. NOTE: The lockfile often contains the process-ID of the process locking the file. When the program notices that a file has a lock, it can then check to see if the other process still exists (using a kill 0 and looking at the return code and errno). This implementation fails miserably with NFS-mounted files, because they could easily be lock by a process on a remote machine. This is a fairly brain-dead locking mechanism which was fine when everyone worked on a single vax without networked file systems, but is now obviously inadequate. Still it works for many situations. Alternitivly a scheme in checking the lockfile creation date is possible. For more locking info please see C/locking.hints ------------------------------------------------------------------------------- To match all files in a directory (required 3 pattern matches) .[^.] .??* * ^This may be a ! n some shells, or in real ancheit shells (sh on ultrix) no `not' function may not be provided for shell regex. ------------------------------------------------------------------------------- Is a directory empty? # Anthony Thyssen -- my own solution - simple and obvious to script reader # Does not seem to need protection against bad filenames! if [ -z "`ls -A $dir`" ]; # then empty # Jim Rogers -- 28790008@hplsdv7.hp.com if [ `ls -A $dir | wc -l` -eq 0 ]; # then empty # David W. Tamkin -- dattier@gagme.chi.il.us if [ ". .. * ?" = "`echo .* * ?`" ] # then empty # Look for 'total 0' -- fails if a specifically named file is presen # # S.Kondakci -- jpc.694747676@avdms8.msfc.nasa.gov if ls -a $dir | grep 'total 0'; # then empty # modified by Stan Ryckman -- sgr@alden.UUCP if ls -l $dir | fgrep -x 'total 0'; # then empty # Maarten Litmaath -- maart@paramount.nikhefk.nikhef.nl (built-in cmds only) DirEmpty() { # is directory empty -- uses built in cmds only used cd "${1-.}" set .* ? * case $#$3$4 in 4\?\*) exit 0 esac exit 1 } Find all empty directories find . -depth -type d -print0 | \ xargs -0n1 sh -c '[ `ls -A "$0" | wc -l` -eq 0 ] && echo "$0";' The above can also be extended to find directories with less or more than a certian number of files, or even directories containing a specific file. ------------------------------------------------------------------------------- Read/Delete file (protect the source) for i in file1 file2 file3 ... fileN do (rm -f - "$i"; cat) < $i done | ...{pipeline}... Can also be used to write to the same filename (different Inode). Just replace cat with your modify data function (rm -f file; cat - > file) < file WARNING: the above could go really wrong, particularly on user interupt. I do not recomend it in anything but temporary files. ------------------------------------------------------------------------------- Is one file newer than another ls: newer() { # is file 1 newer than all others given [ `ls -1rtd "$@" | tail -1` = "$1" ] } older() { # is file 1 older than all others given [ `ls -1td "$@" | tail -1` = "$1" ] } # NOTE: the use of -r ensudes that ls is forced to reorder the files # to produce a true result. This ensures that if the files are the # same age (in which case ls don't bother re-ording) the result is always # false. find: newer () { # is file 1 newer than file 2 [ "`find . -name $1 -newer $2 ! -type d -print`" ] } make: # Rich Salz --- rsalz@bbn.com newer () { # is file 1 newer than file 2 echo "$1 : $2 ; @/bin/false" >/tmp/x$$ make -qf /tmp/x$$; status=$? rm -f /tmp/x$$ exit $status } bash: if [ "$file1" -nt "$file2" ] perl: if (-M $file1 < -M $file2) ksh: if [[ "$file1" -nt "$file2" ]] multiple find: * # multiple file test # Alex P. Ugolini, Jr. --- ugolinia@mr.med.ge.com newlist=`find file1 file2 file3 file4 -newer filen -print` multiple ls: This is the best solution. ls can list a large collection of files in the right order. You can then if you want sed for newer or older files than a known filename (present or not) # Tom Christiansen --- tchrist@convex.COM set `ls -td file1 file2` echo $1 is newer NOTE: the find, and ls solutions could result in all the files in a directory being stat'ed, so caution is required if dealing with VERY large directories. ------------------------------------------------------------------------------- Read File Permissions (and other attributes) easilly Perl perl -e 'printf "%o\n", (stat(shift))[2];' ~ 40755 NOTE: 40000 = directory See: man 2 stat stat (Linux coreutils) Basically this is a formatable "ls" command stat ~ --printf=%A"\n" drwxr-xr-x stat ~ --printf=%a"\n" 755 ------------------------------------------------------------------------------- find -older option to make a find . -older testfile -print do the following find . \( ! -newer testfile -a ! -name testfile \) -print BUG: on the rare instance of two files being the same age this will fail ------------------------------------------------------------------------------- File modification times (preferable to the second) EG: the "ls" command fails for files older than 6 months :::::> ls -l oldfile -rwxr-xr-x 1 root 106496 Oct 11 1990 oldfile TAR (to the minute) :::> tar cf - oldfile | tar tvf - rwxr-xr-x 0/10 106496 Oct 11 12:51 1990 oldfile CPIO :::> echo oldfile | cpio -oac | cpio -ictv 209 blocks 100755 root 106496 Oct 11 12:51:48 1990 oldfile 209 blocks PERL *** :::> perl -e 'require "ctime.pl"; print &ctime((stat(shift))[9]),"\n";' \ oldfile Fri Nov 7 6:05:02 2003 NOTE: both cpio and tar methods will read the whole file, even though it isn't nessary. As such these commands can be very slow on large files. The best method is the perl one if you have it which only does the stat() system call, and does not open or access the the file at all. ------------------------------------------------------------------------------- Complex Find Commands... For details of the order in which UNIX find traverses a directory tree and processes files, see the document "info/perl/dir_traversal_notes" Adding a suffix to a filename found find /path -type f -exec command {}.suffix \; This doesn't work as find only expands the "{}" token and doesn't recognise "{}.suffix". Solutions :- shell argument to command find /path -type f -exec sh -c 'command $1.suffix' -- {} \; WARNING: find -exec will pause while command executes, rather than continue to search for the next match. construct a command with sed and feed to shell find /path -type d -print | sed 's:.*:command &.suffix:' | sh The Sed solution is probably the most versatile. BUT it is dangerious when malicious or uncontrolled filenames are posible. Use xargs.... find /path -type f -print0 | xargs -0r -I{} command {}.suffix If the command itself can handle the suffix addition better still For example using a file renaming script (like "mv_perl") find /path -type f -print0 | xargs -0r mv_perl 's/$.old/.new/" Find - Grep note... Grep will NOT output a filename if only one argument is provided. as such it is a good idea in a find-xargs-grep as that a extra /dev/null be added to the grep command line to ensure two files are always given for grep to process... find /path -name "*.txt" | xargs grep "string" /dev/null Gnu-grep can use a -H option to force filename output find /path -name "*.txt" | xargs grep -H "string" parallel This is a drop in "xargs" replacement, but can do its own simple find or run a feed of shell commands. However it makes use of multi-processors to run the multiple commands "xargs" would normally generate, and run them in parallel. It is also a perl script using only standard perl libraries, so no archeture specific binary is needed. Poor mans "xargs" (to collect groups of filenames) using "fmt" ls | fmt |\ while read args; do grep "some words" $args done fmt collects the arguments into 'lines' and then you can run one command per 'line' or collected filenames! ------------------------------------------------------------------------------- Compare log files. New log file matches the old log file but may have extra lines at the end of the file only. status = 0 if this is the case. NOTE: This test does not check if the files are reversed! IE: the files match but it is the old one that is longer instead of the new log file. if comm -3 $new $old | cat $old - | cmp -s - $new; then mv $new $old # replace old log with new log elif comm -3 $old $new | cat $new - | cmp -s - $old; then : # ignore new file as it matches but is shorter else echo error # old and new log files do not match at all fi ------------------------------------------------------------------------------- Read from a pipeline using a filename! Some commands can only read information from a actual filename and not a from standard input. IE: the data can't be read from another command directly. On linux your can force a command to read standard input, by reading from the /dev/fd0 device... command | read_from /dev/fd/0 BUT this is a pipeline, so the "read_from" gets run in a sub-shell! But a more generic solution (works on more machines) also exists... Use a named pipe! mknod pipe p command > pipe & read_from pipe I use these techniques to pre-process files for commands which does not accept standard input (or doing so has disadvantages). For Example. * Sun pkgadd will not take standard input so I use a named pipe to allow me to de-compress a gziped pkg package into the pkgadd command without needing to decompress the stored package itself. * SGI xfsrestore must read from a file (or device), if you want to still control it interactivally (EG; stdin is still needed!). However that command can't properly read from a remote sun tape drive (rmt command incompatiblity). Solution was read the tape though a network pipe manually using `dd' into a named pipe (file could be too big to save to a temporary file). Then get xfsretore read from that named pipe. mknod /tmp/pipe p ssh -n -x TAPE_HOST dd ibs=10k if=/dev/rmt/0hn > /tmp/pipe & xfsrestore -i -f /tmp/pipe . On Bash named pipes or /dev/fd/? usage is built into the shell. read_from <(command) or for writing to the file write_to >(command) The `pipe' argument is substituted with the appropriate named pipe or /dev/fd? device name for the system being used. ------------------------------------------------------------------------------- Program output Buffered output can be: line by line, as printed, or in large blocks If a program is "expecting" a particular output (like a prompt) from a command before sending the next command to that same program, the output string may never be recieved as it is buffered forever (as the buffer is never full, or a newline seen!). This results in a Deadlock for interactive program control. The buffering is due to the interaction of different programs and the stdio library. If a program uses low-level writes, the output will be blocked in the system writes, as such partical lines or whole paragraphs can be the case. Network Packets are preformed in this way. The same thing will happen if a program turns off or removes the standard IO libraries buffering mechnisim (using setbuf, or setvbuf). However if a program uses the high level stdio library routines for writing (putchar, printf, etc..) the stdio library will buffer the output until it is flushed (written with a low-level write). This will be done when : 1/ The program does a forced flush. This includes the cases of a program: turning off buffering, or opened file in append mode. 2/ If output is to a tty (or pty), when end of line is reached 3/ Otherwise when the buffer is full. This buffering causes many of the problems with deadlocks in a programs interactive (both input and output) control of another program. See "interactive.hints" in this directory. Solution.. * A program such as "pty" (ask archie) can force programs to think it is talking to a tty when it in fact isn't. What this program does is run the command in a psuedo-tty but this you don't have to worry about. In particular the pty package provides a script called (don't ask) "condom" which accepts a command and runs that command as if it was talking to a tty, regardless of the piping arrangements around it. * The "expect" package also launches the command in its own pty for this same reason. For more information on this problem see co-processes.hints http://www.ict.griffith.edu.au/anthony/info/shell/co-processes.hints -------------------------------------------------------------------------------