back to listing index

How can I delete a newline if it is the last character in a file?

[web search]
Original source (stackoverflow.com)
Tags: bash shell-scripting
Clipped on: 2016-11-01

I have some files that I'd like to delete the last newline if it is the last character in a file. 'od -c' shows me that the command I run does write the file with a trailing new line:

0013600   n   t  >  \n

I've tried a few tricks with sed but the best I could think of isn't doing the trick:

sed -e '$s/\(.*\)\n$/\1/' abc

Any ideas how to do this?

asked Oct 31 '09 at 10:42
Image (Asset 3/22) alt=
4 upvote
  flag
newline is only one character for unix newlines. DOS newlines are two characters. Of course, literal "\n" is two characters. Which are you actually looking for? – Dennis Williamson Oct 31 '09 at 10:47
3 upvote
  flag
Although the representation might be \n, in linux is is one character – pavium Oct 31 '09 at 10:49
9 upvote
  flag
Can you elaborate on why you want to do this? Text files are supposed to end with an end-of-line, unless they are entirely empty. It seems strange to me that you'd want to have such a truncated file? – Thomas Padron-McCarthy Oct 31 '09 at 11:06
   upvote
  flag
The usual reason for doing something like this is to delete a trailing comma from the last line of a CSV file. Sed works well, but newlines have to be treated differently. – pavium Oct 31 '09 at 11:15
3 upvote
  flag
@ThomasPadron-McCarthy "In computing, for every good reason there is to do something there exists a good reason not to do it and visa versa." -Jesus -- "you shouldn't do that" is a horrible answer no matter the question. The correct format is: [how to do it] but [why it may be bad idea]. #sacrilege – Cory Mawhorter Mar 30 '15 at 17:18
up vote 141 down vote accepted
perl -pe 'chomp if eof' filename >filename2

or, to edit the file in place:

perl -pi -e 'chomp if eof' filename

[Editor's note: -pi -e was originally -pie, but, as noted by several commenters and explained by @hvd, the latter doesn't work.]

This was described as a 'perl blasphemy' on the awk website I saw.

But, in a test, it worked.

answered Oct 31 '09 at 10:55
Image (Asset 5/22) alt=
pavium
9,76041938
10 upvote
  flag
You can make it safer by using chomp. And it beats slurping the file. – Sinan Ünür Oct 31 '09 at 11:17
6 upvote
  flag
Blasphemy though it is, it works very well. perl -i -pe 'chomp if eof' filename. Thank you. – Todd Partridge 'Gen2ly' Oct 31 '09 at 13:27
9 upvote
  flag
The funny thing about blasphemy and heresy is it's usually hated because it's correct. :) – Ether Oct 31 '09 at 17:11
8 upvote
  flag
Small correction: you can use perl -pi -e 'chomp if eof' filename, to edit a file in-place instead of creating a temporary file – Romuald Brunet Aug 14 '12 at 10:26
7 upvote
  flag
perl -pie 'chomp if eof' filename -> Can't open perl script "chomp if eof": No such file or directory; perl -pi -e 'chomp if eof' filename -> works – aditsu May 1 '13 at 1:29

You can take advantage of the fact that shell command substitutions remove trailing newline characters:

Simple form that works in bash, ksh, zsh:

printf %s "$(< in.txt)" > out.txt

Portable (POSIX-compliant) alternative (slightly less efficient):

printf %s "$(cat in.txt)" > out.txt

Note:

  • If in.txt ends with multiple newline characters, the command substitution removes all of them - thanks, @Sparhawk. (It doesn't remove whitespace characters other than trailing newlines.)
  • Since this approach reads the entire input file into memory, it is only advisable for smaller files.
  • printf %s ensures that no newline is appended to the output (it is the POSIX-compliant alternative to the nonstandard echo -n; see http://pubs.opengroup.org/onlinepubs/009696799/utilities/echo.html and http://unix.stackexchange.com/a/65819)

A guide to the other answers:

  • If Perl is available, go for the accepted answer - it is simple and memory-efficient (doesn't read the whole input file at once).

  • Otherwise, consider ghostdog74's Awk answer - it's obscure, but also memory-efficient; a more readable equivalent (POSIX-compliant) is:

    • awk 'NR > 1 { print prev } { prev=$0 } END { ORS=""; print }' in.txt
    • Printing is delayed by one line so that the final line can be handled in the END block, where it is printed without a trailing \n due to setting the output-record separator (OFS) to an empty string.
  • If you want a verbose, but fast and robust solution that truly edits in-place (as opposed to creating a temp. file that then replaces the original), consider jrockway's Perl script.

answered Aug 27 '12 at 19:55
Image (Asset 6/22) alt=
mklement0
47.4k9106110
3 upvote
  flag
N.B. if there are multiple newlines at the end of the file, this command will delete all of them. – Sparhawk Feb 19 '13 at 23:52

You can do this with head from GNU coreutils, it supports arguments that are relative to the end of the file. So to leave of the last byte use:

head -c -1

To test for an ending newline you can use tail and wc. The following example saves the result to a temporary file and subsequently overwrites the original:

if [[ $(tail -c1 file | wc -l) == 1 ]]; then
  head -c -1 file > file.tmp
  mv file.tmp file
fi

You could also use sponge from moreutils to do "in-place" editing:

[[ $(tail -c1 file | wc -l) == 1 ]] && head -c -1 file | sponge file
answered Sep 25 '12 at 9:02
Image (Asset 7/22) alt=
Thor
20.1k54875
   upvote
  flag
Best solution of all so far. Uses a standard tool that really every Linux distribution has, and is concise and clear, without any sed or perl wizardry. – Dakkaron Sep 14 '15 at 17:58
head -n -1 abc > newfile
tail -n 1 abc | tr -d '\n' >> newfile

Edit 2:

Here is an awk version (corrected) that doesn't accumulate a potentially huge array:

awk '{if (line) print line; line=$0} END {printf $0}' abc

answered Oct 31 '09 at 10:59
Image (Asset 8/22) alt=
Dennis Williamson
170k43246310
   upvote
  flag
Good original way to think about it. Thanks Dennis. – Todd Partridge 'Gen2ly' Oct 31 '09 at 13:27
   upvote
  flag
the awk version removes empty lines as well.. – ghostdog74 Oct 31 '09 at 23:24
   upvote
  flag
You are correct. I defer to your awk version. It takes two offsets (and a different test) and I only used one. However, you could use printf instead of ORS. – Dennis Williamson Nov 1 '09 at 1:49
1 upvote
  flag
Using -c instead of -n for head and tail should be even faster. – rudimeier Apr 18 '13 at 18:10
1 upvote
  flag
For me, head -n -1 abc removed the last actual line of the file, leaving a trailing newline; head -c -1 abc seemed to work better – ChrisV Apr 4 '14 at 11:09

gawk

   awk '{q=p;p=$0}NR>1{print q}END{ORS = ""; print p}' file
answered Oct 31 '09 at 11:21
Image (Asset 9/22) alt=
ghostdog74
144k26155233
   upvote
  flag
Still looks like a lot of characters to me... learning it slowly :). Does the job though. Thanks ghostdog. – Todd Partridge 'Gen2ly' Oct 31 '09 at 13:35
1 upvote
  flag
awk '{ prev_line = line; line = $0; } NR > 1 { print prev_line; } END { ORS = ""; print line; }' file this should be easier to read. – Yevhen Pavliuk May 7 '13 at 8:35

If you want to do it right, you need something like this:

use autodie qw(open sysseek sysread truncate);

my $file = shift;
open my $fh, '+>>', $file;
my $pos = tell $fh;
sysseek $fh, $pos - 1, 0;
sysread $fh, my $buf, 1 or die 'No data to read?';

if($buf eq "\n"){
    truncate $fh, $pos - 1;
}

We open the file for reading and appending; opening for appending means that we are already seeked to the end of the file. We then get the numerical position of the end of the file with tell. We use that number to seek back one character, and then we read that one character. If it's a newline, we truncate the file to the character before that newline, otherwise, we do nothing.

This runs in constant time and constant space for any input, and doesn't require any more disk space, either.

answered Nov 2 '09 at 0:12
Image (Asset 10/22) alt=
jrockway
30.5k55281
2 upvote
  flag
but that has the disadvantage of not reseting ownership/permissions for the file...err, wait... – ysth Nov 2 '09 at 4:47
   upvote
  flag
Verbose, but both fast and robust - seems to be the only true in-place file-editing answer here (and since it may not be obvious to everyone: this is a Perl script). – mklement0 Apr 8 '15 at 16:12

Here is a nice, tidy Python solution. I made no attempt to be terse here.

This modifies the file in-place, rather than making a copy of the file and stripping the newline from the last line of the copy. If the file is large, this will be much faster than the Perl solution that was chosen as the best answer.

It truncates a file by two bytes if the last two bytes are CR/LF, or by one byte if the last byte is LF. It does not attempt to modify the file if the last byte(s) are not (CR)LF. It handles errors. Tested in Python 2.6.

Put this in a file called "striplast" and chmod +x striplast.

#!/usr/bin/python

# strip newline from last line of a file


import sys

def trunc(filename, new_len):
    try:
        # open with mode "append" so we have permission to modify
        # cannot open with mode "write" because that clobbers the file!
        f = open(filename, "ab")
        f.truncate(new_len)
        f.close()
    except IOError:
        print "cannot write to file:", filename
        sys.exit(2)

# get input argument
if len(sys.argv) == 2:
    filename = sys.argv[1]
else:
    filename = "--help"  # wrong number of arguments so print help

if filename == "--help" or filename == "-h" or filename == "/?":
    print "Usage: %s <filename>" % sys.argv[0]
    print "Strips a newline off the last line of a file."
    sys.exit(1)


try:
    # must have mode "b" (binary) to allow f.seek() with negative offset
    f = open(filename, "rb")
except IOError:
    print "file does not exist:", filename
    sys.exit(2)


SEEK_EOF = 2
f.seek(-2, SEEK_EOF)  # seek to two bytes before end of file

end_pos = f.tell()

line = f.read()
f.close()

if line.endswith("\r\n"):
    trunc(filename, end_pos)
elif line.endswith("\n"):
    trunc(filename, end_pos + 1)

P.S. In the spirit of "Perl golf", here's my shortest Python solution. It slurps the whole file from standard input into memory, strips all newlines off the end, and writes the result to standard output. Not as terse as the Perl; you just can't beat Perl for little tricky fast stuff like this.

Remove the "\n" from the call to .rstrip() and it will strip all white space from the end of the file, including multiple blank lines.

Put this into "slurp_and_chomp.py" and then run python slurp_and_chomp.py < inputfile > outputfile.

import sys

sys.stdout.write(sys.stdin.read().rstrip("\n"))
answered Nov 2 '09 at 19:49
Image (Asset 11/22) alt=
steveha
39.3k96589
   upvote
  flag
os.path.isfile() will tell you about file presence. Using try/except might catch a lot of different errors :) – Denis Barmenkov Feb 11 '13 at 20:06

Yet another perl WTDI:

perl -i -p0777we's/\n\z//' filename
answered Nov 1 '09 at 2:27
Image (Asset 12/22) alt=
ysth
68.7k381166
$  perl -e 'local $/; $_ = <>; s/\n$//; print' a-text-file.txt

See also Match any character (including newlines) in sed.

answered Oct 31 '09 at 10:54
Image (Asset 13/22) alt=
Sinan Ünür
93.3k13143283
1 upvote
  flag
That takes out all the newlines. Equivalent to tr -d '\n' – Dennis Williamson Oct 31 '09 at 11:03
   upvote
  flag
@Dennis Williamson: noted and corrected. – Sinan Ünür Oct 31 '09 at 11:13
   upvote
  flag
This works good too, probably less blasphemous than paviums's. – Todd Partridge 'Gen2ly' Oct 31 '09 at 11:47
   upvote
  flag
Sinan, although Linux and Unix might define text files to end with a newline, Windows poses no such requirement. Notepad, for example, will write only the characters you type without adding anything extra at the end. C compilers might require a source file to end with a line break, but C source files aren't "just" text files, so they can have extra requirements. – Rob Kennedy Nov 1 '09 at 20:45
   upvote
  flag
in that vein, most javascript/css minifiers will remove trailing newlines, and yet produce text files. – ysth Nov 2 '09 at 4:53

Using dd:

file='/path/to/file'
[[ "$(tail -c 1 "${file}" | tr -dc '\n' | wc -c)" -eq 1 ]] && \
    printf "" | dd  of="${file}" seek=$(($(stat -f "%z" "${file}") - 1)) bs=1 count=1
    #printf "" | dd  of="${file}" seek=$(($(wc -c < "${file}") - 1)) bs=1 count=1
answered May 3 '10 at 16:39
Image (Asset 14/22) alt=
cpit
211

A very simple method for single-line files, requiring GNU echo from coreutils:

/bin/echo -n $(cat $file)
answered Jun 14 at 22:24
Image (Asset 15/22) alt=
anotheral
113
   upvote
  flag
This is a decent way if it's not too expensive (repetitive). – user4401178 Aug 19 at 5:17

Assuming Unix file type and you only want the last newline this works.

sed -e '${/^$/d}'

It will not work on multiple newlines...

* Works only if the last line is a blank line.

answered May 3 '10 at 16:50
Image (Asset 17/22) alt=
5 upvote
  flag
This only works if the last line is blank. – dave4420 Sep 2 '11 at 9:03

Yet another answer FTR (and my favourite!): echo/cat the thing you want to strip and capture the output through backticks. The final newline will be stripped. For example:

# Sadly, outputs newline, and we have to feed the newline to sed to be portable
echo thingy | sed -e 's/thing/sill/'

# No newline! Happy.
out=`echo thingy | sed -e 's/thing/sill/'`
printf %s "$out"

# Similarly for files:
file=`cat file_ending_in_newline`
printf %s "$file" > file_no_newline
answered Apr 11 '12 at 13:55
Image (Asset 18/22) alt=
Nicholas Wilson
5,99711760
1 upvote
  flag
I found the cat-printf combo out by accident (was trying to get the opposite behavior). Note that this will remove ALL trailing newlines, not just the last. – technosaurus Sep 27 '13 at 20:50
perl -pi -e 's/\n$// if(eof)' your_file
answered Mar 22 '13 at 10:54
Image (Asset 19/22) alt=
Vijay
25.7k53162252
   upvote
  flag
Effectively the same as the accepted answer, but arguably clearer in concept to non-Perl users. Note that there's no need for the g or the parentheses around eof: perl -pi -e 's/\n$// if eof' your_file. – mklement0 Apr 8 '15 at 15:48

The only time I've wanted to do this is for code golf, and then I've just copied my code out of the file and pasted it into an echo -n 'content'>file statement.

answered Nov 1 '09 at 11:57
Image (Asset 20/22) alt=
dlamblin
21k165692
1 upvote
  flag
Hey now... it works. – dlamblin Nov 8 '09 at 12:53
   upvote
  flag
Halfway there; complete approach here. – mklement0 Aug 27 '12 at 19:58

I had a similar problem, but was working with a windows file and need to keep those CRLF -- my solution on linux:

sed 's/\r//g' orig | awk '{if (NR>1) printf("\r\n"); printf("%s",$0)}' > tweaked
answered Jun 22 '12 at 10:17
Image (Asset 21/22) alt=
cadrian
6,04622036
sed ':a;/^\n*$/{$d;N;};/\n$/ba' file
answered Mar 20 '10 at 7:47
Image (Asset 22/22) alt=
ghostdog74
144k26155233
   upvote
  flag
Works, but removes all trailing newlines. – mklement0 Apr 8 '15 at 15:45

ruby:

ruby -ne 'print $stdin.eof ? $_.strip : $_'

or:

ruby -ane 'q=p;p=$_;puts q if $.>1;END{print p.strip!}'
answered May 18 at 14:21
peak
5,96241429

POSIX SED:

'${/^$/d}'

$ - match last line


{ COMMANDS } - A group of commands may be enclosed between { and } characters. This is particularly useful when you want a group of commands to be triggered by a single address (or address-range) match.
answered Aug 25 at 10:02
Oleg Mazko
22425
sed -n "1 x;1 !H
$ {x;s/\n*$//p;}
" YourFile

Should remove any last occurence of \n in file. Not working on huge file (due to sed buffer limitation)

answered Nov 4 '13 at 13:46
NeronLeVelu
6,3861929

Your Answer

asked

7 years ago

viewed

56557 times

active

2 months ago

Blog

Featured on Meta

Get the weekly newsletter! In it, you'll get:

  • The week's top questions and answers
  • Important community announcements
  • Questions that need answers

Hot Network Questions

Technology Life / Arts Culture / Recreation Science Other
  1. Stack Overflow
  2. Server Fault
  3. Super User
  4. Web Applications
  5. Ask Ubuntu
  6. Webmasters
  7. Game Development
  8. TeX - LaTeX
  1. Software Engineering
  2. Unix & Linux
  3. Ask Different (Apple)
  4. WordPress Development
  5. Geographic Information Systems
  6. Electrical Engineering
  7. Android Enthusiasts
  8. Information Security
  1. Database Administrators
  2. Drupal Answers
  3. SharePoint
  4. User Experience
  5. Mathematica
  6. Salesforce
  7. ExpressionEngine® Answers
  8. Cryptography
  1. Code Review
  2. Magento
  3. Signal Processing
  4. Raspberry Pi
  5. Programming Puzzles & Code Golf
  6. more (7)
  1. Photography
  2. Science Fiction & Fantasy
  3. Graphic Design
  4. Movies & TV
  5. Music: Practice & Theory
  6. Seasoned Advice (cooking)
  7. Home Improvement
  8. Personal Finance & Money
  1. Academia
  2. more (8)
  1. English Language & Usage
  2. Skeptics
  3. Mi Yodeya (Judaism)
  4. Travel
  5. Christianity
  6. English Language Learners
  7. Japanese Language
  8. Arqade (gaming)
  1. Bicycles
  2. Role-playing Games
  3. Anime & Manga
  4. Motor Vehicle Maintenance & Repair
  5. more (17)
  1. MathOverflow
  2. Mathematics
  3. Cross Validated (stats)
  4. Theoretical Computer Science
  5. Physics
  6. Chemistry
  7. Biology
  8. Computer Science
  1. Philosophy
  2. more (3)
  1. Meta Stack Exchange
  2. Stack Apps
  3. Area 51
  4. Stack Overflow Talent
site design / logo © 2016 Stack Exchange Inc; user contributions licensed under cc by-sa 3.0 with attribution required
rev 2016.11.1.4163