No space left/setxattr problem.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

No space left/setxattr problem.

apatca
Hi,

I need help tracking down this problem with rdiff-backup.

Today my weekly backups failed with IOError: [Errno 28] No space left
on device, as have all following attempts to --check-destination-dir.

There were 255 GB free on the backups drive and 14 GB on / (where /tmp
resides), both more than enough. Additionaly, pointing --tempdir or
the TMPDIR, TMP and TEMP variables to a different location made no
difference. I'm quite convinced lack of disk space is not actually
the problem.

Since the last line in the traceback is
"rp.conn.xattr.setxattr(rp.path, name, value, 0, rp.issym())", I
believe the error is related to extended file attributes. I have found
similar reports, like https://savannah.nongnu.org/bugs/?37415 , but in
my case I am backing up an XFS volume over SSH to an EXT4 one, and
needless to say had never encountered this problem.

Both machines are running rdiff-backup 1.2.8-7 on Debian, and have been
running the same version for at least a month withouth problems. I am
completely at a loss as to what has caused this problem and how to fix
it.

What follows is the full error message as it was mailed to me by cron.
The error took place when trying to back up my home directory, but as
you can see afterwards rdiff-backup seems to have successfully
listed and deleted and old increment from that same destination
directory (the other three messages correspond to three other
directories, which backup was skipped for and can still list
increments just fine).



Exception '[Errno 28] No space left on device' raised of class '<type
'exceptions.IOError'>': File
"/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in
error_check_Main try: Main(arglist) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in
Main take_action(rps) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in
take_action elif action == "backup": Backup(rps[0], rps[1]) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 343, in
Backup backup.Mirror_and_increment(rpin, rpout, incdir) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 51, in
Mirror_and_increment DestS.patch_and_increment(dest_rpath,
source_diffiter, inc_rpath) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 243, in
patch_and_increment ITR(diff.index, diff) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 281,
in __call__ last_branch.fast_process(*args) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 692, in
fast_process if self.patch_to_temp(mirror_rp, diff_rorp, tf): File
"/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 559, in
patch_to_temp rpath.copy_attribs(diff_rorp, new) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 181, in
copy_attribs if Globals.eas_write: rpout.write_ea(rpin.get_ea()) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 1347, in
write_ea ea.write_to_rp(self) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/eas_acls.py", line 114,
in write_to_rp rp.conn.xattr.setxattr(rp.path, name, value, 0,
rp.issym())

Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line
304, in error_check_Main try: Main(arglist)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line
324, in Main take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line
280, in take_action elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line
343, in Backup backup.Mirror_and_increment(rpin, rpout, incdir)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line
51, in Mirror_and_increment DestS.patch_and_increment(dest_rpath,
source_diffiter, inc_rpath) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 243, in
patch_and_increment ITR(diff.index, diff) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 281,
in __call__ last_branch.fast_process(*args) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 692, in
fast_process if self.patch_to_temp(mirror_rp, diff_rorp, tf): File
"/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 559, in
patch_to_temp rpath.copy_attribs(diff_rorp, new) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 181, in
copy_attribs if Globals.eas_write: rpout.write_ea(rpin.get_ea()) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 1347, in
write_ea ea.write_to_rp(self) File
"/usr/lib/python2.7/dist-packages/rdiff_backup/eas_acls.py", line 114,
in write_to_rp rp.conn.xattr.setxattr(rp.path, name, value, 0,
rp.issym()) IOError: [Errno 28] No space left on device Fatal Error:
Lost connection to the remote system Fatal Error: Previous backup
to /backups/harvey-home seems to have failed. Rerun rdiff-backup with
--check-destination-dir option to revert directory to state before
unsuccessful session. Deleting increment at time: Sat Jul 25 04:03:43
2015 Deleting increment at time:
Sun Sep 28 13:13:37 2014
Deleting increment at time:
Sat Jul 25 04:14:37 2015
No increments older than Thu Jul 30 05:04:21 2015 found, exiting.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: No space left/setxattr problem.

Andrew Engelbrecht
On 03/26/2016 12:10 PM, [hidden email] wrote:

> Hi,
>
> I need help tracking down this problem with rdiff-backup.
>
> Today my weekly backups failed with IOError: [Errno 28] No space left
> on device, as have all following attempts to --check-destination-dir.
>
> There were 255 GB free on the backups drive and 14 GB on / (where /tmp
> resides), both more than enough. Additionaly, pointing --tempdir or
> the TMPDIR, TMP and TEMP variables to a different location made no
> difference. I'm quite convinced lack of disk space is not actually
> the problem.

what is the output of

$ df -h

some filesystems reserve space for root, but if you are running this
command as root that should not be the problem.

andrew


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: No space left/setxattr problem.

apatca
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        15G  1.7G   14G  11% /
udev             10M     0   10M   0% /dev
tmpfs           396M   41M  356M  11% /run
tmpfs           989M     0  989M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           989M     0  989M   0% /sys/fs/cgroup
/dev/sda4       1.8T  1.5T  402G  79% /home
/dev/sda1       976M   32M  878M   4% /boot
tmpfs           198M     0  198M   0% /run/user/1000
/dev/sdb1       2.7T  2.3T  322G  88% /backups

I am running rdiff-backup as root indeed, both locally in the server
and in the laptop that I back up over SSH, as aside of my home
directories I also back up some system directories.

A bit of find sorcery (find -printf "%b %p\n" | sort -nr) tells me the
biggest file under my laptop's home directory is under 2 GB, so those
14 should be plenty, but I still tried custom temp directories
under /home and /backups, to no avail.

/backups has more free space than I said earlier because I deleted some
old increments just in case.

For whatever it's worth, I also tried writing to /backups and checked
its SMART status with smartctl, found nothing fishy. I also ran
"dumpe2fs -h" to make sure there were free inodes, as I found suggested
elsewhere.

> what is the output of
>
> $ df -h
>
> some filesystems reserve space for root, but if you are running this
> command as root that should not be the problem.
>
> andrew
>
>
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: No space left/setxattr problem.

Patrik Dufresne-2
Hello,

I once have issue with disk space when backing-up special files. e.g.: socket or device (like /dev or /proc). Make sure to only backup real file. Check rdiff-backup manual to exclude those files.

--
Patrik Dufresne Service Logiciel inc.
http://www.patrikdufresne.com/
514-971-6442
1-114 rue des Hautbois,
St-Colomban, QC J5K 2H6

On Sat, Mar 26, 2016 at 1:18 PM, <[hidden email]> wrote:
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        15G  1.7G   14G  11% /
udev             10M     0   10M   0% /dev
tmpfs           396M   41M  356M  11% /run
tmpfs           989M     0  989M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           989M     0  989M   0% /sys/fs/cgroup
/dev/sda4       1.8T  1.5T  402G  79% /home
/dev/sda1       976M   32M  878M   4% /boot
tmpfs           198M     0  198M   0% /run/user/1000
/dev/sdb1       2.7T  2.3T  322G  88% /backups

I am running rdiff-backup as root indeed, both locally in the server
and in the laptop that I back up over SSH, as aside of my home
directories I also back up some system directories.

A bit of find sorcery (find -printf "%b %p\n" | sort -nr) tells me the
biggest file under my laptop's home directory is under 2 GB, so those
14 should be plenty, but I still tried custom temp directories
under /home and /backups, to no avail.

/backups has more free space than I said earlier because I deleted some
old increments just in case.

For whatever it's worth, I also tried writing to /backups and checked
its SMART status with smartctl, found nothing fishy. I also ran
"dumpe2fs -h" to make sure there were free inodes, as I found suggested
elsewhere.

> what is the output of
>
> $ df -h
>
> some filesystems reserve space for root, but if you are running this
> command as root that should not be the problem.
>
> andrew
>
>
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: No space left/setxattr problem.

apatca
In reply to this post by Andrew Engelbrecht
That is good advice, and in the future I'll make sure to add
--exclude-special-files and any other sane options to prevent possible
problems. Sadly that didn't allow me to restore the damaged backup
directory, but I decided to take your advice and look for other useful
options in the manpage, and I found one. I ran:

sudo rdiff-backup --no-eas --check-destination-dir /backups/harvey-home/

And it worked. My backup directory seems to be fixed and I can now list
increments again. I think the problem is fixed, though I'll know for
sure when I try to back up again.

This is uncharted territory for me, but I figure I'm better off
disabling backing up of extended attributes with --no-eas to prevent
such problems in the future.

Everyone, thank you for your ideas and your support.

> Hello,
>
> I once have issue with disk space when backing-up special files. e.g.:
> socket or device (like /dev or /proc). Make sure to only backup real
> file. Check rdiff-backup manual to exclude those files.
>
> --
> Patrik Dufresne Service Logiciel inc.
> http://www.patrikdufresne.com <http://patrikdufresne.com/>/
> 514-971-6442
> 1-114 rue des Hautbois,
> St-Colomban, QC J5K 2H6

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki