[Errno 28] No space left on device - but which device?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[Errno 28] No space left on device - but which device?

Ron Leach
List, good afternoon, we're puzzled by this error (from our overnight
backup) because the backup destination has more than 350 GB free, and
the increment would likely have been somewhere between 50MB and 200MB.
  It was a 'fatal' error, and the backup ceased there and then.

The nightly backup is scheduled by cron and has been completed for a
couple of years without a fatal error.  Here's the command:

rdiff-backup --print-statistics -v3 --exclude /Data/.Trash-1000 /Data
/mnt/D7bkp100/Backups

/Data is a local 1.8TB partition, and has around 163GB spare.

/mnt/D7bkp100/ is an NFS export of a 2.6TB LVM located on another
machine, and has 367GB spare.

Here's a df of the machine doing the backup:

ron@D5s4\;13:50:09;~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              2.8G  2.8G   24K 100% /
tmpfs                 501M     0  501M   0% /lib/init/rw
udev                   10M  792K  9.3M   8% /dev
tmpfs                 501M     0  501M   0% /dev/shm
/dev/md6              1.8T  1.7T  163G  92% /Data
/dev/sda1             324M   12M  296M   4% /boot
/dev/sdb1             313M   16M  281M   6% /boot2
/dev/md5               38G  2.9G   35G   8% /home
/dev/md4              949M  4.3M  945M   1% /tmp
/dev/md2              9.4G  1.2G  8.2G  13% /usr
/dev/md3              4.7G  1.1G  3.7G  23% /var
192.168.0.100:/mnt/R1/bkp100
                       2.6T  2.1T  367G  86% /mnt/D7bkp100
ron@D5s4\;13:50:13;~$

To check that there really was (and is) enough space on the
destination machine, I copied a 4.4GB Debian iso from /Data to
/mnt/D7bkp100, and it copied without problems.

I'm a little puzzled as to why rdiff-backup encountered a problem, and
would be grateful for any suggestions as to where I might look for
some more clues.

In that spirit, I include [1] the log output (it's not too long), but
it seems to only list the hierarchy of procedure calls.  (I've never
provided logs before - if there is a more convenient way to parse or
format it let me know and I'll repost it.)

If anyone can determine from it what 'mode' rdiff-backup was in when
it hit the problem, that might help provide a clue as to which
resource was encountered as full.

Grateful for any comments,

regards, Ron

[1]
Exception '[Errno 28] No space left on device' raised of class '<type
'exceptions.IOError'>':
   File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line
304, in error_check_Main
     try: Main(arglist)
   File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line
324, in Main
     take_action(rps)
   File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line
280, in take_action
     elif action == "backup": Backup(rps[0], rps[1])
   File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line
346, in Backup
     backup.Mirror(rpin, rpout)
   File "/var/lib/python-support/python2.5/rdiff_backup/backup.py",
line 38, in Mirror
     DestS.patch(dest_rpath, source_diffiter)
   File "/var/lib/python-support/python2.5/rdiff_backup/backup.py",
line 232, in patch
     ITR(diff.index, diff)
   File "/var/lib/python-support/python2.5/rdiff_backup/rorpiter.py",
line 281, in __call__
     last_branch.fast_process(*args)
   File "/var/lib/python-support/python2.5/rdiff_backup/backup.py",
line 529, in fast_process
     if self.patch_to_temp(mirror_rp, diff_rorp, tf):
   File "/var/lib/python-support/python2.5/rdiff_backup/backup.py",
line 553, in patch_to_temp
     result = self.patch_snapshot_to_temp(diff_rorp, new)
   File "/var/lib/python-support/python2.5/rdiff_backup/backup.py",
line 582, in patch_snapshot_to_temp
     (diff_rorp, new))
   File "/var/lib/python-support/python2.5/rdiff_backup/robust.py",
line 32, in check_common_error
     try: return function(*args)
   File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py",
line 105, in copy
     if rpin.isreg(): return copy_reg_file(rpin, rpout, compress)
   File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py",
line 133, in copy_reg_file
     return rpout.write_from_fileobj(rpin.open("rb"), compress = compress)
   File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py",
line 1195, in write_from_fileobj
     copyfileobj(fp, outfp)
   File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py",
line 64, in copyfileobj
     outputfp.write(inbuf)

IOError while trying to log exception!
Exception '[Errno 28] No space left on device' raised of class '<type
'exceptions.IOError'>':
   File "/var/lib/python-support/python2.5/rdiff_backup/robust.py",
line 32, in check_common_error
     try: return function(*args)
   File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py",
line 105, in copy
     if rpin.isreg(): return copy_reg_file(rpin, rpout, compress)
   File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py",
line 133, in copy_reg_file
     return rpout.write_from_fileobj(rpin.open("rb"), compress = compress)
   File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py",
line 1195, in write_from_fileobj
     copyfileobj(fp, outfp)
   File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py",
line 64, in copyfileobj
     outputfp.write(inbuf)

Traceback (most recent call last):
   File "/usr/bin/rdiff-backup", line 30, in <module>
     rdiff_backup.Main.error_check_Main(sys.argv[1:])
   File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line
304, in error_check_Main
     try: Main(arglist)
   File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line
324, in Main
     take_action(rps)
   File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line
280, in take_action
     elif action == "backup": Backup(rps[0], rps[1])
   File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line
346, in Backup
     backup.Mirror(rpin, rpout)
   File "/var/lib/python-support/python2.5/rdiff_backup/backup.py",
line 38, in Mirror
     DestS.patch(dest_rpath, source_diffiter)
   File "/var/lib/python-support/python2.5/rdiff_backup/backup.py",
line 232, in patch
     ITR(diff.index, diff)
   File "/var/lib/python-support/python2.5/rdiff_backup/rorpiter.py",
line 281, in __call__
     last_branch.fast_process(*args)
   File "/var/lib/python-support/python2.5/rdiff_backup/backup.py",
line 529, in fast_process
     if self.patch_to_temp(mirror_rp, diff_rorp, tf):
   File "/var/lib/python-support/python2.5/rdiff_backup/backup.py",
line 553, in patch_to_temp
     result = self.patch_snapshot_to_temp(diff_rorp, new)
   File "/var/lib/python-support/python2.5/rdiff_backup/backup.py",
line 582, in patch_snapshot_to_temp
     (diff_rorp, new))
   File "/var/lib/python-support/python2.5/rdiff_backup/robust.py",
line 32, in check_common_error
     try: return function(*args)
   File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py",
line 105, in copy
     if rpin.isreg(): return copy_reg_file(rpin, rpout, compress)
   File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py",
line 133, in copy_reg_file
     return rpout.write_from_fileobj(rpin.open("rb"), compress = compress)
   File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py",
line 1195, in write_from_fileobj
     copyfileobj(fp, outfp)
   File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py",
line 64, in copyfileobj
     outputfp.write(inbuf)
IOError: [Errno 28] No space left on device
close failed: [Errno 28] No space left on device


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: [Errno 28] No space left on device - but which device?

Remy Blank-2
Ron Leach wrote on 2015-07-04 16:08:
> List, good afternoon, we're puzzled by this error (from our overnight
> backup) because the backup destination has more than 350 GB free, and
> the increment would likely have been somewhere between 50MB and 200MB.
>   It was a 'fatal' error, and the backup ceased there and then.

One possible cause is that you have run out of inodes.

> Here's a df of the machine doing the backup:
>
> ron@D5s4\;13:50:09;~$ df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md1              2.8G  2.8G   24K 100% /
> tmpfs                 501M     0  501M   0% /lib/init/rw
> udev                   10M  792K  9.3M   8% /dev
> tmpfs                 501M     0  501M   0% /dev/shm
> /dev/md6              1.8T  1.7T  163G  92% /Data
> /dev/sda1             324M   12M  296M   4% /boot
> /dev/sdb1             313M   16M  281M   6% /boot2
> /dev/md5               38G  2.9G   35G   8% /home
> /dev/md4              949M  4.3M  945M   1% /tmp
> /dev/md2              9.4G  1.2G  8.2G  13% /usr
> /dev/md3              4.7G  1.1G  3.7G  23% /var
> 192.168.0.100:/mnt/R1/bkp100
>                        2.6T  2.1T  367G  86% /mnt/D7bkp100
> ron@D5s4\;13:50:13;~$

Try "df -i".

-- Remy


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: [Errno 28] No space left on device - but which device?

Ron Leach
On 04/07/2015 15:14, Remy Blank wrote:

> Ron Leach wrote on 2015-07-04 16:08:
>> List, good afternoon, we're puzzled by this error (from our overnight
>> backup) because the backup destination has more than 350 GB free, and
>> the increment would likely have been somewhere between 50MB and 200MB.
>>    It was a 'fatal' error, and the backup ceased there and then.
>
> One possible cause is that you have run out of inodes.
>
> Try "df -i".
>

ron@D5s4\;15:25:37;~$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/md1                6816    6608     208   97% /
tmpfs                 128214      11  128203    1% /lib/init/rw
udev                  128214    1627  126587    2% /dev
tmpfs                 128214       1  128213    1% /dev/shm
/dev/md6             378583488 2627650 375955838    1% /Data
/dev/sda1              85680     110   85570    1% /boot
/dev/sdb1             171024     110  170914    1% /boot2
/dev/md5             39062400   17254 39045146    1% /home
/dev/md4              976448       8  976440    1% /tmp
/dev/md2             9765504   73568 9691936    1% /usr
/dev/md3             4882688    5022 4877666    1% /var
192.168.0.100:/mnt/R1/bkp100
                      173170688 6320288 166850400    4% /mnt/D7bkp100
ron@D5s4\;15:25:41;~$

Source and Destination look ok, don't they?
'/' looks tight, though.  (The fs is xfs.)

Does rdiff-backup create any or many files on / during a run?

Remy, thank you, regards, Ron



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: [Errno 28] No space left on device - but which device?

Remy Blank-2
Ron Leach wrote on 2015-07-04 16:37:

> ron@D5s4\;15:25:37;~$ df -i
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/md1                6816    6608     208   97% /
> tmpfs                 128214      11  128203    1% /lib/init/rw
> udev                  128214    1627  126587    2% /dev
> tmpfs                 128214       1  128213    1% /dev/shm
> /dev/md6             378583488 2627650 375955838    1% /Data
> /dev/sda1              85680     110   85570    1% /boot
> /dev/sdb1             171024     110  170914    1% /boot2
> /dev/md5             39062400   17254 39045146    1% /home
> /dev/md4              976448       8  976440    1% /tmp
> /dev/md2             9765504   73568 9691936    1% /usr
> /dev/md3             4882688    5022 4877666    1% /var
> 192.168.0.100:/mnt/R1/bkp100
>                       173170688 6320288 166850400    4% /mnt/D7bkp100
> ron@D5s4\;15:25:41;~$
>
> Source and Destination look ok, don't they?

Yes, they look fine. Maybe try "df -i" on the server side, too? I don't
know if inodes map 1:1 between the NFS server and the client.

> '/' looks tight, though.  (The fs is xfs.)
>
> Does rdiff-backup create any or many files on / during a run?

I don't think so, but if you want to be sure, run rdiff-backup under strace.

-- Remy


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: [Errno 28] No space left on device - but which device?

Ron Leach
On 04/07/2015 16:20, Remy Blank wrote:

> Yes, they look fine. Maybe try "df -i" on the server side, too? I don't
> know if inodes map 1:1 between the NFS server and the client.
>


ron@D7Gateway:~$ df -i
Filesystem                                                Inodes
IUsed     IFree IUse% Mounted on
/dev/mapper/bkp100vg-bkp100lv                          173170688
6320288 166850400    4% /mnt/R1/bkp100
ron@D7Gateway:~$

(The word wrap destroyed that, but the report for the LVM indicates 4%
inodes in use.)


>> Does rdiff-backup create any or many files on / during a run?
>
> I don't think so, but if you want to be sure, run rdiff-backup under strace.
>
I'll maybe try that.

Thank you very much for the clues, which I think have provided some
evidence for thinking the fs looks ok, with these reports, and with
the static test of a 4GB iso transfer.

I may also try to run a backup with some enhanced logging.  I'd
particularly like to see what rdiff-backup calculates the next
increment to be - perhaps it thinks it has 'lost' some tranche of data
and needs to copy or diff it again.

regards, Ron

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: [Errno 28] No space left on device - but which device?

Joe Steele-2
In reply to this post by Ron Leach
On 7/4/2015 10:08 AM, Ron Leach wrote:

> [1]
> Exception '[Errno 28] No space left on device' raised of class '<type
> 'exceptions.IOError'>':
>    File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line
> 304, in error_check_Main
>      try: Main(arglist)
>    File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line
> 324, in Main
>      take_action(rps)
>    File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line
> 280, in take_action
>      elif action == "backup": Backup(rps[0], rps[1])
>    File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line
> 346, in Backup
>      backup.Mirror(rpin, rpout)

The above line in the trace looks suspicious.  The surrounding code
implies that the line would only be executed when there was no previous
backup found (and yet this was to be an incremental backup).  More
specifically, no
"/mnt/D7bkp100/Backups/rdiff-backup-data/current_mirror.*.data" file was
found.  Normally, rdiff-backup would not proceed with a backup if it
found the rdiff-backup-data folder but no current_mirror file.  It would
also complain if the destination looked like a backup but had no
rdiff-backup-data folder.

Since rdiff-backup didn't complain about the missing current_mirror
file, the implication is that it didn't find the existing backup
destination folder.  In that case, it would attempt a full backup (and
it doesn't look like there would be enough space for that).

All of this points to a possible mount problem at the time of the backup.

> IOError while trying to log exception!
> Exception '[Errno 28] No space left on device' raised of class '<type
> 'exceptions.IOError'>':

Another error occurs while attempting to log the previous error.  This
time, it can't write to the log file
/mnt/D7bkp100/Backups/rdiff-backup-data/backup.log.  That would support
the conclusion that the destination disk is the device that is full.
(But if there was a problem with mounts, then it might be the root
device that was full.)

If you unmount the destination device, is there anything found under its
mount point on the root device?

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: [Errno 28] No space left on device - but which device?

Graeme Robinson
On 5 July 2015 at 03:10, Joe Steele <[hidden email]> wrote:
> On 7/4/2015 10:08 AM, Ron Leach wrote:
>>
>> [1]
>> Exception '[Errno 28]

<snip>

> The above line in the trace looks suspicious.  The surrounding code implies
> that the line would only be executed when there was no previous backup found


<snip>

> All of this points to a possible mount problem at the time of the backup.

Its a good idea to specify a target (LocalPath) folder for
rdiff-backup that will not be present if the destination partition is
not mounted - that way rdiff-backup will not run and will not fill up
the wrong filesystems with backups unintentionally.

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: [Errno 28] No space left on device - but which device?

Ron Leach
In reply to this post by Joe Steele-2
On 04/07/2015 18:10, Joe Steele wrote:

>
> The [..] trace looks suspicious. The surrounding code
> implies that the line would only be executed when there was no
> previous backup found (and yet this was to be an incremental backup).
> More specifically, no
> "/mnt/D7bkp100/Backups/rdiff-backup-data/current_mirror.*.data" file
> was found. Normally, rdiff-backup would not proceed with a backup if
> it found the rdiff-backup-data folder but no current_mirror file. It
> would also complain if the destination looked like a backup but had no
> rdiff-backup-data folder.
>
> Since rdiff-backup didn't complain about the missing current_mirror
> file, the implication is that it didn't find the existing backup
> destination folder. In that case, it would attempt a full backup (and
> it doesn't look like there would be enough space for that).
>
> All of this points to a possible mount problem at the time of the backup.
>
>
> Another error occurs while attempting to log the previous error. This
> time, it can't write to the log file
> /mnt/D7bkp100/Backups/rdiff-backup-data/backup.log. That would support
> the conclusion that the destination disk is the device that is full.
> (But if there was a problem with mounts, then it might be the root
> device that was full.)
>


Joe, the problem was indeed the mount.

For some reason the destination had become disconnected from the
source.  I don't know whether the source lost the connection, or the
destination lost its connectivity but the result was that the source
could not see the destination.  The error didn't exactly say that, so
I guess there is something in 'NFS' that doesn't return 'doesn't
exist' and instead returns something else (maybe free space = 0 ?),
potentially misleading rdiff-backup into issuing error 28.

The 2nd problem you saw, of logging the first error, was caused for
the same reason, the logs are stored in the destination, and the
destination wasn't there.

I've fixed the connectivity, remounted on both servers, and
successfully completed a backup.

Thanks to everyone who helped,

regards, Ron

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki