------------------------------------------------------------------------------- Sed - An Introduction and Tutorial https://www.grymoire.com/Unix/Sed.html Original Sed manual and tutorial (good starting point) http://sed.sourceforge.net/grabbag/tutorials/sed_mcmahon.txt Eric Pement's One-liners for Sed (Dec 2005) http://sed.sourceforge.net/sed1line.txt http://www.pement.org/sed/sed1line.txt Eric Pement - If-then-else in Sed http://www.pement.org/sed/ifelse.txt Also see regex.hints for hints and tips of matching regular expressions And sed 1 liners http://sed.sourceforge.net/sed1line.txt Also examples from the gnu "info sed" ------------------------------------------------------------------------------- Character Escapes https://www.gnu.org/software/sed/manual/html_node/Escapes.html So with GNU-sed to replace an ESC character with 'ESC' s/\033/ESC/ However escapes are translated BEFORE being passed to the RE parser as such /[a]/ and \x5ba\x5d/ are exactly the same thing! That is only match 'a' and not '[a]' as the second probably intended. ------------------------------------------------------------------------------ Word Boundarys (sed and grep) Differences between \< \> \b Posix defines a word as consecutive characters: a-z A-Z 0-9 _ Which is also defined by the regular expression \w echo '...abc,,,' | sed 's/\/|/g' ...abc|,,, echo '...abc,,,' | sed 's/\b/|/g' ...|abc|,,, The RE matches the zero width 'transition' and not any character Useful to limit matched to the specific boundary echo 'acegi z' | grep -o '[acegi ]*\>' | cat -A acegi$ echo 'acegi z' | grep -o '[acegi ]*\b' | cat -A acegi $ ------------------------------------------------------------------------------ Print specific lines print first line of file (emulates "head -1") sed q print first 10 lines of file (emulates behavior of "head") sed 10q print the last line of a file (emulates "tail -1") sed '$!d' # method 1 sed -n '$p' # method 2 print the next-to-the-last line of a file sed -e '$!{h;d;}' -e x # for 1-line files, print blank line sed -e '1{$q;}' -e '$!{h;d;}' -e x # for 1-line files, print the line sed -e '1{$d;}' -e '$!{h;d;}' -e x # for 1-line files, print nothing print the last 2 lines of a file (emulates "tail -2") sed '$!N;$!D' print the last 10 lines of a file (emulates "tail") sed -e :a -e '$q;N;11,$D;ba' ------------------------------------------------------------------------------ Delete between Line Markers AAAA and BBBB Inclusive delete (delete start and end) sed '/AAAA/,/BBBB/d' Exclusive delete (output start and end) sed '1,/AAAA/p; /BBBB/,$p; d' or sed '/AAAA/,/BBBB/{ /AAAA/p; /BBBB/p; d }' delete end but not the start sed '/AAAA/p; /AAAA/,/BBBB/d' or sed '/AAAA/,/BBBB/{ /AAAA/p; d }' delete start but output end sed '/AAAA/,/BBBB/{ /BBBB/p; d }' or sed '/BBBB/p; /AAAA/,/BBBB/d }' See next... ------------------------------------------------------------------------------ Print lines between markers AAAA to BBBB Including markers sed -n '/AAAA/,$p; /BBBB/q' file Excluding the end marker sed -n '/BBBB/q; /AAAA/,$p' file or sed -n '/AAAA/,$p; /BBBB/Q' file Excluding both markers sed -n '/BBBB/q; 1,/AAAA/d; p' file or sed '1,/AAAA/d; /BBBB/Q' file ------------------------------------------------------------------------------- Line numbers left aligned sed = file | sed 'N;s/\n/\t/' right aligned sed = filename | sed 'N; s/^/ /; s/ *\(.\{6,\}\)\n/\1 /' counting lines (using "sed", not "wc") sed -n '$=' file ------------------------------------------------------------------------------ Delete ALL blank lines sed '/^$/d' file Delete multiple blank lines (paragraph separators) sed '/^$/{ N; /^\n$/D; }' file or sed '/./,/^$/!d' file ------------------------------------------------------------------------------- Center all lines center all text in the middle of 79-column width. Method 1 spaces at the beginning of the line are significant, trailing spaces are appended at the end of the line. sed -e :a -e 's/^.\{1,77\}$/ & /;ta' Method 2 spaces at the beginning of the line are discarded in centering the line, and no trailing spaces appear at the end of lines. sed -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/' ------------------------------------------------------------------------------- Join lines Append next line if a line ends in '\' (shell line continuation) sed -e :a -e '/\\$/N; s/\\\n//; ta' Append lines starting with space to the previous line (single space separated) (GNU-sed) sed -e :a -e '$!N;s/\n\s\+/ /;ta' -e 'P;D' ------------------------------------------------------------------------------- Overlapping Regex sed will not do overlapping regex echo "a b c d" | sed 's/\(\S\) \(\S\)/\1_\2/g' # => a_b c_d perl can however use a loop to solve the problem... echo "a b c d" | perl -pe 'while( s/(\S) (\S)/${1}_$2/g ) {}' # => a_b_c_d --- Using zero-width lookahead and look-behind https://www.regular-expressions.info/lookaround.html Using positive look-behind and positive lookahead echo "a b c d" | perl -pe 's/(?<=\S) (?=\S)/_/g' # => a_b_c_d using negative look-behind and negative lookahead echo "a b c d" | perl -pe 's/(? a_b_c_d ------------------------------------------------------------------------------- Replace second occurance of pattern sed '/OLD/{:1 n;/OLD/{s/OLD/NEW/;:2 n;$!b2};b1}' Looks for a line with "OLD" ....When found, it loops (#1), getting the next line, till it finds another instance of "OLD" ........If found, it performs the substitution, then enters the "do nothing" loop (#2) to end of file ........If not, it falls thru to the end ....If not, it falls thru to the end does however does not cover when the 2 patterns on the same line. Solution sed '/OLD/{s/OLD/NEW/2;t2;:1 n;/OLD/{s/OLD/NEW/;:2 n;$!b2};b1}' Using GNU sed... Load the whole file into pattern space. Replace the 2rd occurance. sed ':a N;$!ba; s/pattern/replacement/2' file > fileout =============================================================================== Programming with sed.... ------------------------------------------------------------------------------ Delimiting "sed" Delimitors (also see next) Say you want to search for a path name that contains slashes `/' unfortunatally these are the sed regular expresion delimiters so you need to escape the slashes before you can search for it. This is not as easy as it sounds. old_path="/some/path/to/look/for" new_path="/some/path/to/look/for" old_path=`echo "$old_path" | sed 's/\//\\\\\//g'` new_path=`echo "$new_path" | sed 's/\//\\\\\//g'` sed 's'"$old_path"'/'"$new_path"'/g' $old_file > $new_file ------------------------------------------------------------------------------- Sed with arbatitry patterns (using any litterial string within SED) NOTE trying to use SED to do a literial replacement of ANY string is a bad idea. There are just too may things that needs to be escaped. Escaping a string literial regex in sed search='abc\n\t[a-z]\+\([^ ]\)\{2,3\}\3' # EXAMPLE search_esc=$(sed 's/[^^]/[&]/g; s/\^/\\^/g' <<<"$search") sed -n "s/$search_esc/foo/p" <<<"$search" # good if echo "foo" Note: Every character except ^ is placed in its own character set [...] expression to treat it as a literal. You cannot do this with ^ so ^ chars. Are then escaped as \^. Escaping a string literal as replacement/substition in sed replace='Laurel & Hardy; PS\2' # EXAMPLE replace_esc=$(sed -e 's/[]\/$*.^[]/\\&/g' <<<"$replace") sed -n "s/\(.*\) \(.*\)/$replace_esc/p" <<<"foo bar" BASH alternative outputvar=${inputvar//"$string"/"$replace"}. PERL Alternative -- simple from=$'Cost\(*):\n$3.' to='You owe me $1/$& for'$'\n''eating A\1 sauce.' # This should convert 'from' to 'to' # Note that the replacement value needs NO escaping. perl -s -0777 -pe 's/\Q$from\E/$to/' -- -from="$from" -to="$to" <<<"$from" Referances https://stackoverflow.com/questions/29613304/ # includes multi-line solutions https://stackoverflow.com/questions/407523/ ------------------------------------------------------------------------------- Substitutions Rules for template files Sed substitions comamnd.... Thsi uses '|' as as to avoid conflict with '/' echo "SED SUBSTITUTIONS \"$@.sed\" ==> \"$@\"" sed -e 's|===BROWSERDIR===|'"$BROWSERDIR"'|' \ -e 's|===LIBRARY_RC===|'"$LIBRARY_RC"'|' \ -e '/===COLOR_SETTINGS_DIVIDER===/d' \ < $@.sed >$@ Some of the special character around the substition strings are as follows so take your pick. ===LABEL=== @@LABEL@@ @#LABEL#@ An alternative templating method is "envsubst", but has little documentation ------------------------------------------------------------------------------- Sed Programming Methology Example: This sed script grabs a 'face' file (which always starts with a space or tab) and repositions it correctly at the end of the mail header (denoted by a absolutely blank line). NOTE the use of lables. cat $FACE - $SIGNED | sed -n ' # first collect the new face at the start of the input 1 { s/^/X-Face:/; h; d; } /^[ \t]/{H;d;} # read the header; if X-face found just copy all : head /^X-Face:/ b body /^$/ b face p; n; b head # output a face at end of header :face x; p; g; p; n # just copy the rest of the file : body p; n; b body ' | /lib/sendmail "$@" ------------------------------------------------------------------------------- Removing a new line (joining lines) -- does not appear posible! ------------------------------------------------------------------------------- Adding a newline to sed (just add return -- messy) bourne shells: sed 's/;$/; }\ /' filename Csh family: sed 's/;$/; }\\ /' filename Of course you can always use a tr which is slower but neater sed 's/;$/; }@/' | tr '@' '\012' GNU sed allows you to use \r (for return) But uses \n on the input side. Do not use \n on output as that is 'null' sed 's/;$/; }\r/' In a similar way it allows the use of \t for tab. ------------------------------------------------------------------------------- Moving a line to the start of the file EG something1 this something2 ====> something1 this something2 something3 something3 sed -n ' /this/!H # if no match append to hold space /this/{x; H; } # found so exhange and append (prepend) ${g; s/\n//; p; }'