This link has been bookmarked by 972 people and liked by 1 people. It was first bookmarked on 02 Mar 2006, by Joel Liu.
-
16 Nov 19
Flashman"Sed is the ultimate stream editor. If that sounds strange, picture a stream flowing through a pipe. Okay, you can't see a stream if it's inside a pipe. That's what I get for attempting a flowing analogy. You want literature, read James Joyce.
Anyhow, sed is a marvelous utility. Unfortunately, most people never learn its real power. The language is very simple, but the documentation is terrible. The Solaris on-line manual pages for sed are five pages long, and two of those pages describe the 34 different errors you can get. A program that spends as much space documenting the errors as it does documenting the language has a serious learning curve.
Do not fret! It is not your fault you don't understand sed. I will cover sed completely. But I will describe the features in the order that I learned them. I didn't learn everything at once. You don't need to either. " -
29 Apr 19
-
13 Feb 19
-
23 Nov 18
Thomas JamesSed: An Introduction and Tutorial by Bruce Barnett https://t.co/RmJsW8v1S9
-
05 May 18
-
05 Dec 17
-
05 Oct 17
-
24 Jul 17
-
18 Jun 17
kevinoempty
-
24 May 17
-
28 Apr 17
-
19 Apr 17
Andy Breeding"#!/bin/sh
sed -n '
# by default - do not print anything
/ONE/ {
# append a line
N
# if TWO found, print the first line
/\n.*TWO/ P
}' file" -
30 Mar 17
-
16 Feb 17
-
09 Feb 17
-
03 Feb 17
-
09 Dec 16
-
07 Dec 16
-
04 Dec 16
-
most people never learn its real power
-
the documentation is terrible
-
has a serious learning curve
-
I will cover sed completely.
-
I didn't learn everything at once. You don't need to either.
-
most people only learn the substitute command: s
-
changes all occurrences of the regular expression into a new value.
-
in the "old" file to "night" in the "new" file:
-
sed s/day/night/ old >new
-
I will henceforth quote future examples to emphasize the "best practice.
-
's/day/night/'
-
line oriented.
-
but only the first occurrence was changed. That is the default behavior
-
Delimiter
-
Search
-
Replacement
-
few fine points that any future sed expert should know about.
-
only 63 more sections to cover. :-)
-
-
24 Oct 16
-
-
Last modified: Thu Apr 23 16:37:48 EDT 2015
-
When I first wrote this (in 1994), most versions of sed did not allow you to place comments inside the script. Lines starting with the '#' characters are comments. Newer versions of sed may support comments at the end of the line as well.
-
Sed is the ultimate stream editor.
-
Sed has several commands, but most people only learn the substitute command: s.
-
<old
-
>new
-
A simple example is changing "day" in the "old" file to "night" in the "new" file:
-
Or another way (for UNIX beginners),
-
and for those who want to test this:
-
This will output "night".
-
I didn't put quotes around the argument because this example didn't need them. If you read my earlier tutorial on quotes, you would understand why it doesn't need quotes. However, I recommend you do use quotes. If you have meta-characters in the command, quotes are necessary. And if you aren't sure, it's a good habit, and I will henceforth quote future examples to emphasize the "best practice."
-
Using the strong (single quote) character, that would be:
-
Another important concept is that sed is line oriented. Suppose you have the input file:
-
The output would be
-
one
-
Note that this changed "one" to "ONE" once on each line. The first line had "one" twice, but only the first occurrence was changed.
-
\)
-
\(
-
If you want to change a pathname that contains a slash - say /usr/local/bin to /common/bin - you could use the backslash to quote the slash:
-
Some call this a 'Picket Fence' and it's ugly. It is easier to read if you use an underline instead of a slash as a delimiter:
-
Some people use colons:
-
Others use the "|" character.
-
And remember that you need three delimiters. If you get a "Unterminated `s' command" it's because you are missing one of them.
-
Using & as the matched string
-
The solution requires the special character "&." It corresponds to the pattern found.
-
You can have any number of "&" in the replacement string. You could also double a pattern, e.g. the first number of a line:
-
Let me slightly amend this example. Sed will match the first string, and make it as greedy as possible. I'll cover that later. If you don't want it to be so greedy (i.e. limit the matching), you need to put restrictions on the match.
-
" Extended regular expressions have more power, but sed scripts that treated "+" as a normal character would break. Therefore you must explicitly enable this extension with a command line option.
GNU sed turns this feature on if you use the "-r" command line option. So the above could also be written using
-
I have already described the use of "(" ")" and "1" in my tutorial on regular expressions. To review, the escaped parentheses (that is, parentheses with backslashes before them) remember a substring of the characters matched by the regular expression.
-
You can use this to exclude part of the characters matched by the regular expression. The "\1" is the first remembered pattern, and the "\2" is the second remembered pattern. Sed has up to nine remembered patterns.
-
Therefore if you type
-
This will output "abcd" and delete the numbers.
-
by using extended regular expressions (note that '(' and ')' no longer need to have a backslash):
-
\1
-
\1
-
If you want to detect duplicated words, you can use
-
If you want to eliminate duplicated words, you can try:
-
\1
-
or with extended regular expressions
-
This, when used as a filter, will print lines with duplicated words.
-
Sed only operates on patterns found in the in-coming data. That is, the input line is read, and when a pattern is matched, the modified output is generated, and the rest of the input line is scanned. The "s" command will not scan the newly created output.
-
You can combine a number with the g (global) flag. For instance, if you want to leave the first word alone, but change the second, third, etc. to be DELETED instead, use /2g:
-
When the "-n" option is used, the "p" flag will cause the modified line to be printed. Here is one way to duplicate the function of grep with sed:
-
If you want to duplicate the functionality of grep, combine the -n (noprint) option with the /p print flag:
-
If you have a large number of sed commands, you can put them into a file and use
-
where sedscript could look like this:
-
When there are several commands in one file, each command must be on a separate line.
-
-
11 Oct 16
-
around the argument because this example didn't
-
ng, and commands to execute after the pattern is matched. Therefore I recommend this style:
-
-
05 Oct 16
-
16 Sep 16
-
29 Aug 16
-
19 Jul 16
-
Sed is the ultimate stream editor.
-
-
31 May 16
-
25 May 16
-
If you want to write a program to make changes in a file
-
ultimate stream editor
-
anguage is very simple, but the documentation is terrible
-
most people only learn the substitute command: s
-
changes all occurrences of the regular expression into a new value
-
If you have meta-characters in the command, quotes are necessary
-
sed is line oriented
-
only the first occurrence was changed
-
elimiter. It is conventionally a slash, because this is what ed, more, and vi use
-
can be anything you want
-
s_/usr/local/bin_/common/bin_
-
:/usr/local/bin:/common/bin:
-
As long as it's not in the string you are looking for
-
need three delimiters
-
special character "&."
-
match the first string, and make it as greedy as possible
-
sed turns this feature on if you use the "-r" command line option
-
Mac OS X and FreeBSD uses -E
-
Sed has up to nine remembered patterns.
-
using extended regular expressions (note that '(' and ')' no longer need to have a backslash):
-
"\1" doesn't have to be in the replacement string
-
can be in the pattern you are searching for
-
If you want to eliminate duplicated words
-
detect duplicated words,
-
sed -n '/\([a-z][a-z]*\) \1/p'
-
extended regular expressions
-
sed -En '/([a-z]+) \1/p'
-
will print lines with duplicated words
-
additional flags after the last delimiter
-
will only change the first occurrence of the word on a line
-
"g" after the last delimiter and use the work-around:
-
the input line is read, and when a pattern is matched, the modified output is generated, and the rest of the input line is scanned
-
s" command will not scan the newly created output.
-
an add a number after the substitution command to indicate you only want to match that particular pattern. Example:
sed 's/[a-zA-Z]* //2' <old >new
-
I've heard that combining the number with the g command does not work on The MacOS
-
new text is printed instead of the old one
-
By default, sed prints every line
-
If you use an optional argument to sed, "sed -n," it will not, by default, print any new lines. I'll cover this and other options later. When the "-n" option is used, the "p" flag will cause the modified line to be printed.
-
pecify a file that will receive the modified data
-
must have exactly one space between the w and the filename
-
could also use this method to log error or debugging information to a special file
-
sed 's/^#.*//' f1 f2 f3 | grep -v '^$' | wc -l
-
will not print anything unless an explicit request to print is found
-
-
24 May 16
-
16 May 16
siri sarased complete tutorial
-
07 May 16
-
06 May 16
-
03 May 16
-
29 Apr 16
-
05 Apr 16
-
20 Mar 16
-
I didn't put quotes around the argument because this example didn't need them.
-
Another important concept is that sed is line oriented.
-
only the first occurrence
-
The slash as a delimiter
The character after the s is the delimiter. It is conventionally a slash, because this is what ed, more, and vi use. It can be anything you want, however. If you want to change a pathname that contains a slash - say /usr/local/bin to /common/bin - you could use the backslash to quote the slash:
sed 's/\/usr\/local\/bin/\/common\/bin/' <old >new
Gulp. Some call this a 'Picket Fence' and it's ugly. It is easier to read if you use an underline instead of a slash as a delimiter:
sed 's_/usr/local/bin_/common/bin_' <old >new
Some people use colons:
sed 's:/usr/local/bin:/common/bin:' <old >new
Others use the "|" character.
sed 's|/usr/local/bin|/common/bin|' <old >new
Pick one you like. As long as it's not in the string you are looking for, anything goes. And remember that you need three delimiters. If you get a "Unterminated `s' command" it's because you are missing one of them.
-
Sed will match the first string, and make it as greedy as possible.
-
"[0-9]*" matches zero or more numbers. "[0-9][0-9]*" matches one or more numbers. Another way to do this is to use the "+" meta-character and use the pattern "[0-9]+" as the "+" is a special character when using "extended regular expressions." Extended regular expressions have more power, but sed scripts that treated "+" as a normal character would break. Therefore you must explicitly enable this extension with a command line option.
-
Sed has up to nine remembered patterns.
-
after the last delimiter.
-
These flags can specify what happens when a match is found.
-
You can add a number after the substitution command to indicate you only want to match that particular pattern.
-
You can combine a number with the g (global) flag. For instance, if you want to leave the first word alone, but change the second, third, etc. to be DELETED instead, use /2g:
-
you can specify a file that will receive the modified data.
-
You must have exactly one space between the w and the filename.
-
the "w" has to be the last flag.
-
The long argument version is
sed --expression='s/a/A/' --expression='s/b/B/' <old >new
-
Sed has more commands that make grep unnecessary. And grep -c can replace wc -l.
-
sed 's/PATTERN/&/p' file
acts like the cat program if PATTERN is not in the file: e.g. nothing is changed. If PATTERN is in the file, then each line that has this is printed twice. Add the "-n" option and the example acts like grep:
sed -n 's/PATTERN/&/p' file
Nothing is printed, except those lines with PATTERN included.
The long argument of the -n command is either
sed --quiet 's/PATTERN/&/p' file
or
sed --silent 's/PATTERN/&/p' file
-
-
03 Mar 16
-
19 Feb 16
-
Gulp. Some call this a 'Picket Fence' and it's ugly. It is easier to read if you use an underline instead of a slash as a delimiter
-
:
sed 's_/usr/local/bin_/common/bin_' <old >new
Some people use colons:
sed 's:/usr/local/bin:/common/bin:' <old >new
Others use the "|" character.
sed 's|/usr/local/bin|/common/bin|' <old >new
-
you need three delimiters. If you get a "Unterminated `s' command" it's because you are missing one of them.
-
the "+" is a special character when using "extended regular expressions." Extended regular expressions have more power, but sed scripts that treated "+" as a normal character would break. Therefore you must explicitly enable this extension with a command line option.
GNU sed turns this feature on if you use the "-r" command line option.
-
-
17 Feb 16
-
22 Jan 16
-
18 Jan 16
-
Sed - An Introduction and Tutorial by Bruce Barnett
-
-
28 Dec 15
-
20 Nov 15
-
20 Oct 15
-
18 Sep 15
-
print and delete are opposite functions
-
"!p" is similar to "d,"
-
"!d" is similar to "p."
-
-
12 Sep 15
-
once on each line
-
It is conventionally a slash, because this is what ed, more, and vi use
-
It can be anything you want, however
-
make it as greedy as possible
-
use the "-r" command line option
-
-
27 Aug 15
-
23 Aug 15
-
11 Aug 15
-
06 Aug 15
-
31 Jul 15
-
11 Jul 15
-
s Substitute command /../../ Delimiter one Regular Expression Pattern Search Pattern ONE Replacement string
The search pattern is on th
-
The solution requires the special character "&.
-
"\1" to put the first pattern back unchanged
-
's/\([a-zA-Z]*\) \([a-zA-Z]*\) /\1 /'
-
s/[a-zA-Z]* //2'
-
's/[a-zA-Z]* /DELETED /2g'
-
Don't get /2 and \2 confused
-
number flag i
-
add a colon after the 80th character in each line
-
"p" flag will cause the modified line to be printed
-
Please note that the "w" has to be the last flag.
-
A sed guru never uses two processes when one can do.
Multiple commands with -e command
-
sed -e 's/a/A/' -e 's/b/B/' <old >new
-
the /PATTERN/ address
-
placing a backslash at the end of each line:
-
You have only learned one command,
-
- Specifying a line by its number.
- Specifying a range of lines by number.
- All lines containing a pattern.
- All lines from the beginning of a file to a regular expression
- All lines from a regular expression to the end of the file.
- All lines between two regular expressions.
- Specifying a line by its number.
-
sed '3 s/[0-9][0-9]*//' <file >new
-
delete the first number on all lines that start with a "#," use:
-
'/^#/
-
s/[0-9][0-9]*//
-
I placed a space after the "/expression/" so it is easier to read. It isn't necessary, but without it the command is harder to fathom
-
sed '\_/usr/local/bin_ s_/usr/local_/common/all_'
-
This illustrates why sed scripts get the reputation for obscurity.
-
search for all lines that start with a "g,"
-
s/g/s/g'
-
Use comments liberally in a sed script
-
Er, I take that back. It's hopeless
-
sed '/start/,/stop/ s/#.*//'
-
As far as I am concerned, the only time the semicolon is useful is when you want to type the sed script on the command line.
-
02 Jul 15
-
17 Jun 15
-
05 Jun 15
-
22 May 15
-
31 Mar 15
-
25 Mar 15
-
12 Mar 15
-
04 Mar 15
phronima axxon"greedy"
-
16 Feb 15
-
09 Feb 15
-
06 Feb 15
-
04 Feb 15
-
02 Feb 15
-
16 Jan 15
-
02 Jan 15
-
- Specifying a line by its number.
- Specifying a range of lines by number.
- All lines containing a pattern.
- All lines from the beginning of a file to a regular expression
- All lines from a regular expression to the end of the file.
- All lines between two regular expressions.
- Specifying a line by its number.
useful is the ability to restrict the operation to certain lines. Some useful restrictions might be:
-
- Specifying a line by its number.
- Specifying a range of lines by number.
- All lines containing a pattern.
- All lines from the beginning of a file to a regular expression
- All lines from a regular expression to the end of the file.
- All lines between two regular expressions.
- Specifying a line by its number.
useful is the ability to restrict the operation to certain lines. Some useful restrictions might be:
-
- Specifying a line by its number.
- Specifying a range of lines by number.
- All lines containing a pattern.
- All lines from the beginning of a file to a regular expression
- All lines from a regular expression to the end of the file.
- All lines between two regular expressions.
- Specifying a line by its number.
What would be useful is the ability to restrict the operation to certain lines. Some useful restrictions might be:
-
- Specifying a line by its number.
- Specifying a range of lines by number.
- All lines containing a pattern.
- All lines from the beginning of a file to a regular expression
- All lines from a regular expression to the end of the file.
- All lines between two regular expressions.
- Specifying a line by its number.
useful is the ability to restrict the operation to certain lines. Some useful restrictions might be:
-
If you want to write a program to make changes in a file, sed is the tool to use.
-
Sed is the ultimate stream editor.
-
A simple example is changing "day" in the "old" file to "night" in the "new" file:
-
sed s/day/night/ <old >new
-
sed s/day/night/ old >new
-
echo day | sed s/day/night/
-
f you have meta-characters in the command, quotes are necessary.
-
Using the strong (single quote) character, that would be:
sed 's/day/night/' <old >new
-
echo Sunday | sed 's/day/night/'
This would output the word "Sunnight" because sed found the string "day" in the input.
-
Another important concept is that sed is line oriented
-
Suppose you have the input file:
one two three, one two three four three two one one hundred
and you used the command
sed 's/one/ONE/' <file
The output would be
ONE two three, one two three four three two ONE ONE hundred
-
Note that this changed "one" to "ONE" once on each line
-
That is the default behavior
-
There are four parts to this substitute command:
s Substitute command /../../ Delimiter one Regular Expression Pattern Search Pattern ONE Replacement string
-
The search pattern is on the left hand side and the replacement string is on the right hand side.
-
The character after the s is the delimiter. It is conventionally a slash, because this is what ed, more, and vi use.
-
you could use the backslash to quote the slash:
sed 's/\/usr\/local\/bin/\/common\/bin/' <old >new
-
easier to read if you use an underline instead of a slash as a delimiter:
sed 's_/usr/local/bin_/common/bin_' <old >new
Some people use colons:
sed 's:/usr/local/bin:/common/bin:' <old >new
Others use the "|" character.
sed 's|/usr/local/bin|/common/bin|' <old >new
-
you want to search for a pattern and add some characters, like parenthesis, around or near the pattern you found
-
How can you put the string you found in the replacement string if you don't know what it is?
-
The solution requires the special character "&." It corresponds to the pattern found.
sed 's/[a-z]*/(&)/' <old >new
-
You can have any number of "&" in the replacement string.
-
% echo "123 abc" | sed 's/[0-9]*/& &/' 123 123 abc
-
there is another way to write the above script.
-
"[0-9]*" matches zero or more numbers. "[0-9][0-9]*" matches one or more numbers. Another way to do this is to use the "+" meta-character and use the pattern "[0-9]+" as the "+" is a special character when using "extended regular expressions."
-
GNU sed turns this feature on if you use the "-r" command line option.
-
could also be written using
% echo "123 abc" | sed -r 's/[0-9]+/& &/' 123 123 abc
-
If you want it to make changes for every word, add a "g" after the last delimiter and use the work-around:
sed 's/[^ ][^ ]*/(&)/g' <old >new
-
Sed only operates on patterns found in the in-coming data. That is, the input line is read, and when a pattern is matched, the modified output is generated, and the rest of the input line is scanned.
-
's/loop/loop the loop/g' <old >new
This will not cause an infinite loop.
-
his flag makes the pattern match case insensitive. This will match abc, aBc, ABC, AbC, etc.:
sed -n '/abc/I p' <old >new
-
One method of combining multiple commands is to use a -e before each command:
sed -e 's/a/A/' -e 's/b/B/' <old >new
-
If you want to duplicate the functionality of grep, combine the -n (noprint) option with the /p print flag:
sed -n '/PATTERN/p' file
-
If you have a large number of sed commands, you can put them into a file and use
sed -f sedscript <old >new
where sedscript could look like this:
# sed comment - This script changes lower case vowels to upper case s/a/A/g s/e/E/g s/i/I/g s/o/O/g s/u/U/g
-
-
05 Dec 14
-
03 Dec 14
-
22 Nov 14
-
19 Nov 14
-
04 Nov 14
-
It is easier to read if you use an underline instead of a slash as a delimiter:
-
The solution requires the special character "&." It corresponds to the pattern found.
-
GNU sed turns this feature on if you use the "-r" command line option
-
The "\1" is the first remembered pattern, and the "\2" is the second remembered pattern. Sed has up to nine remembered patterns.
-
-
23 Oct 14
-
30 Aug 14
-
30 Jul 14
-
Hyung-Joo LimSed: An Introduction and Tutorial by Bruce Barnett http://t.co/BeWGYaX6qZ
-
22 Jul 14
-
07 Jul 14
-
12 Mar 14
-
08 Mar 14
-
26 Feb 14
-
21 Feb 14
-
10 Feb 14
stuza1 ."Sed - An Introduction and Tutorial by Bruce Barnett
!!!!!!!!!!!!!!!!!!!!!!!!!
Quick Links - NEW
Sed Commands
: label # comment {....} Block
= - print line number a \ - Append b label - Branch
c \ - change d and D - Delete g and G - Get
h and H - Hold i \ - Insert l - Look
n and N - Next p and P - Print q - Quit
r filename - Read File s/..../..../ - Substitute t label - Test
w filename - Write Filename x - eXchange y/..../..../ - Transform
Sed Pattern Flags
/g - Global
/I - Ignore Case
/p - Print
/w filename - Write Filename
Sed Command Line options
-e script (--expression=script)
-f scriptfile (--file=scriptfile)
-h (--help)
-n (--quiet --silent)
-V (--version)
Table of Contents
The Awful Truth about sed
The essential command: s for substitution
The slash as a delimiter
Using & as the matched string
Using \1 to keep part of the pattern
Sed Pattern Flags
/g - Global replacement
Is sed recursive?
/1, /2, etc. Specifying which occurrence
/p - print
Write to a file with /w filename
/I - Ignore Case
Combining substitution flags
Arguments and invocation of sed
Multiple commands with -e command
Filenames on the command line
sed -n: no printing
Using 'sed /pattern/'
Using 'sed -n /pattern/p' to duplicate the function of grep
sed -f scriptname
sed in shell scripts
Quoting multiple sed lines in the C shell
Quoting multiple sed lines in the Bourne shell
sed -v
sed -h
A sed interpreter script
Sed Comments
Passing arguments into a sed script
Using sed in a shell here-is document
Multiple commands and order of execution
Addresses and Ranges of Text
Restricting to a line number
Patterns
Ranges by line number
Ranges by patterns
Delete with d
Printing with p
Reversing the restriction with !
Relationships between d, p, and !
The q or quit command
Grouping with { and }
Operating in a pattern range except for the patterns
Writing a file with the 'w' command
Reading in a file with the 'r' command
SunOS and the # Comment Command
Adding, Changing, Inserting new lines
Append a line with 'a'
Insert a line with 'i'
Change a line with 'c'
Leading tabs and spaces in a sed script
Adding more than one line
Adding lines and the pattern space
Address ranges and the above commands
Multi-Line Patterns
Print line number with =
Transform with y
Displaying control characters with a l
Working with Multiple Lines
Matching three lines with sed
Matching patterns that span multiple lines
Using newlines in sed scripts
The Hold Buffer
Exchange with x
Example of Context Grep
Hold with h or H
Keeping more than one line in the hold buffer
Get with g or G
Flow Control
Testing with t
Debugging with l
An alternate way of adding comments
The poorly documented ;
Passing regular expressions as arguments
Inserting binary characters
Command Summary
In Conclusion
More References
Copyright 1994, 1995 Bruce Barnett and General Electric Company
Copyright 2001,2005,2007,2011,2013 Bruce Barnett
All rights reserved
You are allowed to print copies of this tutorial for your personal use, and link to this page, but you are not allowed to make electronic copies, or redistribute this tutorial in any form without permission.
Original version written in 1994 and published in the Sun Observer
Introduction to Sed
How to use sed, a special editor for modifying files automatically. If you want to write a program to make changes in a file, sed is the tool to use.
There are a few programs that are the real workhorse in the UNIX toolbox. These programs are simple to use for simple applications, yet have a rich set of commands for performing complex actions. Don't let the complex potential of a program keep you from making use of the simpler aspects. I'll start with the simple concepts and introduce the advanced topics later on.
When I first wrote this, most versions of sed did not allow you to place comments inside the script. Lines starting with the '#' characters are comments. Newer versions of sed may support comments at the end of the line as well.
The Awful Truth about sed
Sed is the ultimate stream editor. If that sounds strange, picture a stream flowing through a pipe. Okay, you can't see a stream if it's inside a pipe. That's what I get for attempting a flowing analogy. You want literature, read James Joyce.
Anyhow, sed is a marvelous utility. Unfortunately, most people never learn its real power. The language is very simple, but the documentation is terrible. The Solaris on-line manual pages for sed are five pages long, and two of those pages describe the 34 different errors you can get. A program that spends as much space documenting the errors than it does documenting the language has a serious learning curve.
Do not fret! It is not your fault you don't understand sed. I will cover sed completely. But I will describe the features in the order that I learned them. I didn't learn everything at once. You don't need to either.
The essential command: s for substitution
Sed has several commands, but most people only learn the substitute command: s. The substitute command changes all occurrences of the regular expression into a new value. A simple example is changing "day" in the "old" file to "night" in the "new" file:
sed s/day/night/ <old >new
Or another way (for UNIX beginners),
sed s/day/night/ old >new
and for those who want to test this:
echo day | sed s/day/night/
This will output "night".
I didn't put quotes around the argument because this example didn't need them. If you read my earlier tutorial on quotes, you would understand why it doesn't need quotes. However, I recommend you do use quotes. If you have meta-characters in the command, quotes are necessary. And if you aren't sure, it's a good habit, and I will henceforth quote future examples to emphasize the "best practice." Using the strong (single quote) character, that would be:
sed 's/day/night/' <old >new
I must emphasize that the sed editor changes exactly what you tell it to. So if you executed
echo Sunday | sed 's/day/night/'
This would output the word "Sunnight" because sed found the string "day" in the input.
Another important concept is that sed is line oriented. Suppose you have the input file:
one two three, one two three
four three two one
one hundred
and you used the command
sed 's/one/ONE/' <file
The output would be
ONE two three, one two three
four three two ONE
ONE hundred
Note that this changed "one" to "ONE" once on each line. The first line had "one" twice, but only the first occurrence was changed. That is the default behavior. If you want something different, you will have to use some of the options that are available. I'll explain them later.
So let's continue.
There are four parts to this substitute command:
s Substitute command
/../../ Delimiter
one Regular Expression Pattern Search Pattern
ONE Replacement string
The search pattern is on the left hand side and the replacement string is on the right hand side.
We've covered quoting and regular expressions.. That's 90% of the effort needed to learn the substitute command. To put it another way, you already know how to handle 90% of the most frequent uses of sed. There are a ... few fine points that any future sed expert should know about. (You just finished section 1. There are only 63 more sections to cover. :-) Oh. And you may want to bookmark this page, .... just in case you don't finish.
The slash as a delimiter
The character after the s is the delimiter. It is conventionally a slash, because this is what ed, more, and vi use. It can be anything you want, however. If you want to change a pathname that contains a slash - say /usr/local/bin to /common/bin - you could use the backslash to quote the slash:
sed 's/\/usr\/local\/bin/\/common\/bin/' <old >new
Gulp. Some call this a 'Picket Fence' and it's ugly. It is easier to read if you use an underline instead of a slash as a delimiter:
sed 's_/usr/local/bin_/common/bin_' <old >new
Some people use colons:
sed 's:/usr/local/bin:/common/bin:' <old >new
Others use the "|" character.
sed 's|/usr/local/bin|/common/bin|' <old >new
Pick one you like. As long as it's not in the string you are looking for, anything goes. And remember that you need three delimiters. If you get a "Unterminated `s' command" it's because you are missing one of them.
Using & as the matched string
Sometimes you want to search for a pattern and add some characters, like parenthesis, around or near the pattern you found. It is easy to do this if you are looking for a particular string:
sed 's/abc/(abc)/' <old >new
This won't work if you don't know exactly what you will find. How can you put the string you found in the replacement string if you don't know what it is?
The solution requires the special character "&." It corresponds to the pattern found.
sed 's/[a-z]*/(&)/' <old >new
You can have any number of "&" in the replacement string. You could also double a pattern, e.g. the first number of a line:
% echo "123 abc" | sed 's/[0-9]*/& &/'
123 123 abc
Let me slightly amend this example. Sed will match the first string, and make it as greedy as possible. I'll cover that later. If you don't want it to be so greedy (i.e. limit the matching), you need to put restrictions on the match.
The first match for '[0-9]*' is the first character on the line, as this matches zero of more numbers. So if the input was "abc 123" the output would be unchanged (well, except for a space before the letters). A better way to duplicate the number is to make sure it matches a number:
% echo "123 abc" | sed 's/[0-9][0-9]*/& &/'
123 123 abc
The string "abc" is unchanged, because it was not matched by the regular expression. If you wanted to eliminate "abc" from the output, you must expand the regular expression to match the rest of the line and explicitly exclude part of the expression using "(", ")" and "\1", which is the next topic.
A quick comment. The original sed did not support the "+" metacharacter. GNU sed does if you use the "-r" command line option, which enables extended regular expressions. The "+" means "one or more matches". So the above could also be written using
% echo "123 abc" | sed -r 's/[0-9]+/& &/'
123 123 abc
Using \1 to keep part of the pattern
I have already described the use of "(" ")" and "1" in my tutorial on regular expressions. To review, the escaped parentheses (that is, parentheses with backslashes before them) remember a substring of the characters matched by the regular expression. You can use this to exclude part of the characters matched by the regular expression. The "\1" is the first remembered pattern, and the "\2" is the second remembered pattern. Sed has up to nine remembered patterns.
If you wanted to keep the first word of a line, and delete the rest of the line, mark the important part with the parenthesis:
sed 's/\([a-z]*\).*/\1/'
I should elaborate on this. Regular expressions are greedy, and try to match as much as possible. "[a-z]*" matches zero or more lower case letters, and tries to match as many characters as possible. The ".*" matches zero or more characters after the first match. Since the first one grabs all of the contiguous lower case letters, the second matches anything else. Therefore if you type
echo abcd123 | sed 's/\([a-z]*\).*/\1/'
This will output "abcd" and delete the numbers.
If you want to switch two words around, you can remember two patterns and change the order around:
sed 's/\([a-z]*\) \([a-z]*\)/\2 \1/'
Note the space between the two remembered patterns. This is used to make sure two words are found. However, this will do nothing if a single word is found, or any lines with no letters. You may want to insist that words have at least one letter by using
sed 's/\([a-z][a-z]*\) \([a-z][a-z]*\)/\2 \1/'
or by using extended regular expressions (note that '(' and ')' no longer need to have a backslash):
sed -r 's/([a-z]+) ([a-z]+)/\2 \1/' # Using GNU sed
The "\1" doesn't have to be in the replacement string (in the right hand side). It can be in the pattern you are searching for (in the left hand side). If you want to eliminate duplicated words, you can try:
sed 's/\([a-z]*\) \1/\1/'
If you want to detect duplicated words, you can use
sed -n '/\([a-z][a-z]*\) \1/p'
or with extended regular expressions
sed -n '/\([a-z]+\) \1/p'
This, when used as a filter, will print lines with duplicated words.
The numeric value can have up to nine values: "\1" thru "\9." If you wanted to reverse the first three characters on a line, you can use
sed 's/^\(.\)\(.\)\(.\)/\3\2\1/'
Sed Pattern Flags
You can add additional flags after the last delimiter. You might have noticed I used a 'p' at the end of the previous substitute command. I also added the '-n' option. Let me first cover the 'p' and other pattern flags. These flags can specify what happens when a match is found. Let me describe them.
/g - Global replacement
Most UNIX utilities work on files, reading a line at a time. Sed, by default, is the same way. If you tell it to change a word, it will only change the first occurrence of the word on a line. You may want to make the change on every word on the line instead of the first. For an example, let's place parentheses around words on a line. Instead of using a pattern like "[A-Za-z]*" which won't match words like "won't," we will use a pattern, "[^ ]*," that matches everything except a space. Well, this will also match anything because "*" means zero or more. The current version of Solaris's sed (as I wrote this) can get unhappy with patterns like this, and generate errors like "Output line too long" or even run forever. I consider this a bug, and have reported this to Sun. As a work-around, you must avoid matching the null string when using the "g" flag to sed. A work-around example is: "[^ ][^ ]*." The following will put parenthesis around the first word:
sed 's/[^ ]*/(&)/' <old >new
If you want it to make changes for every word, add a "g" after the last delimiter and use the work-around:
sed 's/[^ ][^ ]*/(&)/g' <old >new
Is sed recursive?
Sed only operates on patterns found in the in-coming data. That is, the input line is read, and when a pattern is matched, the modified output is generated, and the rest of the input line is scanned. The "s" command will not scan the newly created output. That is, you don't have to worry about expressions like:
sed 's/loop/loop the loop/g' <old >new
This will not cause an infinite loop. If a second "s" command is executed, it could modify the results of a previous command. I will show you how to execute multiple commands later.
/1, /2, etc. Specifying which occurrence
With no flags, the first matched substitution is changed. With the "g" option, all matches are changed. If you want to modify a particular pattern that is not the first one on the line, you could use "\(" and "\)" to mark each pattern, and use "\1" to put the first pattern back unchanged. This next example keeps the first word on the line but deletes the second:
sed 's/\([a-zA-Z]*\) \([a-zA-Z]*\) /\1 /' <old >new
Yuck. There is an easier way to do this. You can add a number after the substitution command to indicate you only want to match that particular pattern. Example:
sed 's/[a-zA-Z]* //2' <old >new
You can combine a number with the g (global) flag. For instance, if you want to leave the first word alone, but change the second, third, etc. to be DELETED instead, use /2g:
sed 's/[a-zA-Z]* /DELETED /2g' <old >new
Don't get /2 and \2 confused. The /2 is used at the end. \2 is used in inside the replacement field.
Note the space after the "*" character. Without the space, sed will run a long, long time. (Note: this bug is probably fixed by now.) This is because the number flag and the "g" flag have the same bug. You should also be able to use the pattern
sed 's/[^ ]*//2' <old >new
but this also eats CPU. If this works on your computer, and it does on some UNIX systems, you could remove the encrypted password from the password file:
sed 's/[^:]*//2' </etc/passwd >/etc/password.new
But this didn't work for me the time I wrote this. Using "[^:][^:]*" as a work-around doesn't help because it won't match an non-existent password, and instead delete the third field, which is the user ID! Instead you have to use the ugly parenthesis:
sed 's/^\([^:]*\):[^:]:/\1::/' </etc/passwd >/etc/password.new
You could also add a character to the first pattern so that it no longer matches the null pattern:
sed 's/[^:]*:/:/2' </etc/passwd >/etc/password.new
The number flag is not restricted to a single digit. It can be any number from 1 to 512. If you wanted to add a colon after the 80th character in each line, you could type:
sed 's/./&:/80' <file >new
You can also do it the hard way by using 80 dots:
sed 's/^................................................................................/&:/' <file >new
/p - print
By default, sed prints every line. If it makes a substitution, the new text is printed instead of the old one. If you use an optional argument to sed, "sed -n," it will not, by default, print any new lines. I'll cover this and other options later. When the "-n" option is used, the "p" flag will cause the modified line to be printed. Here is one way to duplicate the function of grep with sed:
sed -n 's/pattern/&/p' <file
But a simpler version is described later
Write to a file with /w filename
There is one more flag that can follow the third delimiter. With it, you can specify a file that will receive the modified data. An example is the following, which will write all lines that start with an even number, followed by a space, to the file even:
sed -n 's/^[0-9]*[02468] /&/w even' <file
In this example, the output file isn't needed, as the input was not modified. You must have exactly one space between the w and the filename. You can also have ten files open with one instance of sed. This allows you to split up a stream of data into separate files. Using the previous example combined with multiple substitution commands described later, you could split a file into ten pieces depending on the last digit of the first number. You could also use this method to log error or debugging information to a special file.
/I - Ignore Case
GNU has added another pattern flags - /I
This flag makes the pattern match case insensitive. This will match abc, aBc, ABC, AbC, etc.:
sed '/abc/I' <old >new
Combining substitution flags
You can combine flags when it makes sense. Please note that the "w" has to be the last flag. For example the following command works:
sed -n 's/a/A/2pw /tmp/file' <old >new
Next I will discuss the options to sed, and different ways to invoke sed.
Arguments and invocation of sed
previously, I have only used one substitute command. If you need to make two changes, and you didn't want to read the manual, you could pipe together multiple sed commands:
sed 's/BEGIN/begin/' <old | sed 's/END/end/' >new
This used two processes instead of one. A sed guru never uses two processes when one can do.
Multiple commands with -e command
One method of combining multiple commands is to use a -e before each command:
sed -e 's/a/A/' -e 's/b/B/' <old >new
A "-e" isn't needed in the earlier examples because sed knows that there must always be one command. If you give sed one argument, it must be a command, and sed will edit the data read from standard input.
The long argument version is
sed --expression='s/a/A/' --expression='s/b/B/' <old >new
Also see Quoting multiple sed lines in the Bourne shell
Filenames on the command line
You can specify files on the command line if you wish. If there is more than one argument to sed that does not start with an option, it must be a filename. This next example will count the number of lines in three files that don't begin with a "#:"
sed 's/^#.*//' f1 f2 f3 | grep -v '^$' | wc -l
Let's break this down into pieces. The sed substitute command changes every line that starts with a "#" into a blank line. Grep was used to filter out (delete) empty lines. Wc counts the number of lines left. Sed has more commands that make grep unnecessary. And grep -c can replace wc -l. I'll discuss how you can duplicate some of grep's functionality later.
Of course you could write the last example using the "-e" option:
sed -e 's/^#.*//' f1 f2 f3 | grep -v '^$' | wc -l
There are two other options to sed.
sed -n: no printing
The "-n" option will not print anything unless an explicit request to print is found. I mentioned the "/p" flag to the substitute command as one way to turn printing back on. Let me clarify this. The command
sed 's/PATTERN/&/p' file
acts like the cat program if PATTERN is not in the file: e.g. nothing is changed. If PATTERN is in the file, then each line that has this is printed twice. Add the "-n" option and the example acts like grep:
sed -n 's/PATTERN/&/p' file
Nothing is printed, except those lines with PATTERN included.
The long argument of the -n command is either
sed --quiet 's/PATTERN/&/p' file
or
sed --silent 's/PATTERN/&/p' file
Using 'sed /patter" -
27 Jan 14
-
07 Dec 13
-
06 Dec 13
-
28 Nov 13
-
21 Nov 13
-
12 Nov 13
-
09 Nov 13
-
23 Oct 13
-
20 Oct 13
-
15 Sep 13
-
29 Aug 13
-
21 Aug 13
-
Another important concept is that sed is line oriented.
-
Note that this changed "one" to "ONE" once on each line.
-
Others use the "|" character.
-
The solution requires the special character "&." It corresponds to the pattern found.
-
If you tell it to change a word, it will only change the first occurrence of the word on a line.
-
"[^ ]*," that matches everything except a space
-
sed 's/[^ ][^ ]*/(&)/g' <old >new
-
You can add a number after the substitution command to indicate you only want to match that particular pattern.
-
sed 's/[a-zA-Z]* //2' <old >new
-
sed 's/[a-zA-Z]* /DELETED /2g' <old >new
-
If you use an optional argument to sed, "sed -n," it will not, by default, print any new lines.
-
the "p" flag will cause the modified line to be printed
-
delimiter
-
's/^[0-9]*[02468] /&/w even'
-
sed -e 's/a/A/' -e 's/b/B/' <old >new
-
sed 's/PATTERN/&/p' file
-
sed -n 's/PATTERN/&/p' file
-
sed -n '/PATTERN/p' file
-
If you have a large number of sed commands, you can put them into a file and use
-
-e 's/a/A/g'
-
sed '\_/usr/local/bin_ s_/usr/local_/common/all_'
-
sed '/^g/s/g/s/g'
-
The last example has a range that overlaps the "/start/,/stop/" range, as both ranges operate on the lines that contain the keywords.
-
sed -e 's/#.*//' -e 's/[ ^I]*$//' -e '/^$/ d'
-
"^I" is a CTRL-I or tab character.
-
pattern space.
-
sed -n '1,10 p' <file
-
sed -n '/match/ p'
-
sed -n '/match/ !p'
-
The q or quit command
-
sed '11 q'
-
These braces can be nested, which allow you to combine address ranges.
-
This inverts the address,
-
Writing a file with the 'w' command
-
Append a line with 'a'
-
Insert a line with 'i'
-
Change a line with 'c'
-
sed -n '/PATTERN/ =' file
-
like "n" or "d" cause sed to go to the top of the loop
-
After all of the commands are examined, the pattern space is output unless sed has the optional "-n" argument.
-
The "N" command does not print out the current pattern space and does not empty the pattern space. It reads in the next line, but appends
-
a new line character along with the input line itself to the pattern space.
-
The "D" command deletes the first portion of the pattern
-
space, up to the new line character, leaving the rest of the pattern alone.
-
The "P" command only prints the first part of the pattern space, up to the NEWLINE character.
-
/#$/ {
-
now read in the next line
-
s/# //
-
/ONE/ {
-
read in next line
-
The next example would delete everything between "ONE" and "TWO:"
-
mmand and starts the command cycle over again
-
delete the first line
-
sed -n '
-
print the first line
-
s/ / /
-
N - AB CD -
D - - -
P AB AB -
However, if you are inserting a new line, don't use " " - instead insert a literal new line character:
-
There is one more "location" to be covered: the hold buffer or hold space.
-
The "x" command eXchanges the pattern space with the hold buffer.
-
One use of the hold buffer is to remember previous lines. An example of this is a utility that acts like grep as it shows you the lines that match a pattern. In addition, it shows you the line before and after the pattern. That is, if line 8 contains the pattern, this utility would print lines 7, 8 and 9.
-
delete the old one, which is
-
now in the pattern buffer
-
# get the next line
-
The "x" command exchanges the hold buffer and the pattern buffer. Both are changed
-
The "h" command copies the pattern buffer into the hold buffer.
-
you can copy the hold space to the pattern space with the "g" command.
-
Instead of exchanging the hold space with the pattern space
-
use the "G" command.
-
The "b" command branches to the label.
-
If no label is there, branch to the end of the script.
-
-
12 Aug 13
-
04 Aug 13
-
26 Jul 13
-
16 Jul 13
Page Comments
Would you like to comment?
Join Diigo for a free account, or sign in if you are already a member.