[Errno 28] No space left on device

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[Errno 28] No space left on device

Yves Bellefeuille
I'm getting the following error:

Exception '[Errno 28] No space left on device' raised of class '<type
'exceptions.IOError'>':

In fact, there's plenty of space and plenty of free inodes.

I've run fsck and smartctl on the disk and found no problems.

Here are the details. I seem to be getting two similar error reports,
which might be a clue.

  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
324, in Main
    take_action(rps)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
346, in Backup
    backup.Mirror(rpin, rpout)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/backup.py", line
38, in Mirror
    DestS.patch(dest_rpath, source_diffiter)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/backup.py", line
232, in patch
    ITR(diff.index, diff)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/rorpiter.py",
line 281, in __call__
    last_branch.fast_process(*args)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/backup.py", line
529, in fast_process
    if self.patch_to_temp(mirror_rp, diff_rorp, tf):
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/backup.py", line
563, in patch_to_temp
    rpath.copy_attribs(diff_rorp, new)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/rpath.py", line
181, in copy_attribs
    if Globals.eas_write: rpout.write_ea(rpin.get_ea())
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/rpath.py", line
1347, in write_ea
    ea.write_to_rp(self)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/eas_acls.py",
line 116, in write_to_rp
    rp.conn.xattr.setxattr(rp.path, name, value, 0, rp.issym())

Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
324, in Main
    take_action(rps)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
346, in Backup
    backup.Mirror(rpin, rpout)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/backup.py", line
38, in Mirror
    DestS.patch(dest_rpath, source_diffiter)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/backup.py", line
232, in patch
    ITR(diff.index, diff)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/rorpiter.py",
line 281, in __call__
    last_branch.fast_process(*args)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/backup.py", line
529, in fast_process
    if self.patch_to_temp(mirror_rp, diff_rorp, tf):
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/backup.py", line
563, in patch_to_temp
    rpath.copy_attribs(diff_rorp, new)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/rpath.py", line
181, in copy_attribs
    if Globals.eas_write: rpout.write_ea(rpin.get_ea())
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/rpath.py", line
1347, in write_ea
    ea.write_to_rp(self)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/eas_acls.py",
line 116, in write_to_rp
    rp.conn.xattr.setxattr(rp.path, name, value, 0, rp.issym())
IOError: [Errno 28] No space left on device

--
Yves Bellefeuille <[hidden email]>
GPG key 837A6134 at http://members.storm.ca/~yan/pgp.asc

Reply | Threaded
Open this post in threaded view
|

Re: [Errno 28] No space left on device

ewl+rdiffbackup
Salut Yves,

On 13/11/2019 04:02, Yves Bellefeuille wrote:

> I'm getting the following error:
>
> Exception '[Errno 28] No space left on device' raised of class '<type
> 'exceptions.IOError'>':
>
> In fact, there's plenty of space and plenty of free inodes.
>
> I've run fsck and smartctl on the disk and found no problems.
>
> Here are the details. I seem to be getting two similar error reports,
> which might be a clue.

It's no clue, it's how rdiff-backup is written, it's a bit an annoyance
but no priority to fix for now...

>      rp.conn.xattr.setxattr(rp.path, name, value, 0, rp.issym())
> IOError: [Errno 28] No space left on device

The error seems to be within the xattr library of Python. Can you tell us:

1. which version of rdiff-backup (--version)?
2. which OS and version (Linux, Windows...)?
3. which file system type you are using to backup?
4. do you also get the issue if you use the `--no-eas` option?

KR, Eric

Reply | Threaded
Open this post in threaded view
|

Re: [Errno 28] No space left on device

Yves Bellefeuille
On Wednesday 13 November 2019, [hidden email] wrote:

> The error seems to be within the xattr library of Python. Can you tell
> us:
> 1. which version of rdiff-backup (--version)?

rdiff-backup 1.2.8

> 2. which OS and version (Linux, Windows...)?

CentOS 6.

> 3. which file system type you are using to backup?

Backing-up from NTFS (Windows 7) to ext4.

> 4. do you also get the issue if you use the `--no-eas` option?

Indeed, this seems to have solved the problem.

I wonder why this suddenly failed after working for several years.
Thanks for the help.

--
Yves Bellefeuille <[hidden email]>
GPG key 837A6134 at http://members.storm.ca/~yan/pgp.asc

Reply | Threaded
Open this post in threaded view
|

Re: [Errno 28] No space left on device

ewl+rdiffbackup
Hi,

On 14/11/2019 01:47, Yves Bellefeuille wrote:
>> 3. which file system type you are using to backup?
> Backing-up from NTFS (Windows 7) to ext4.

Cross-filesystem-types backup is always a bit tricky. This said if you
have the occasion, it would be nice to test the latest version from
master. We could use some real life testers.

>
>> 4. do you also get the issue if you use the `--no-eas` option?
> Indeed, this seems to have solved the problem.
>
> I wonder why this suddenly failed after working for several years.
> Thanks for the help.

Here I can possibly offer you an explanation: according to man 5 xattr,
the ext4 file system has a limit of extended attributes size (1, 2 or
4KB), and you might have now a Windows file with bigger EAs, which leads
to the issue:

touch xxx
python3
 >>> x = '0123456789' * 1024  # 10KB size
 >>> import xattr
 >>> xattr.setxattr('xxx', 'user.comment', x)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
OSError: [Errno 28] No space left on device

So you have multiple solutions:

1. switch from ext4 to XFS, which doesn't know this limit (I checked)
2. identify the culprit file and trim the EAs (somehow?!?)
3. use the --no-eas option, not knowing what you lose as information

I created an issue against pyxattr:
https://github.com/iustin/pyxattr/issues/23

Hope this helps,
Eric

Reply | Threaded
Open this post in threaded view
|

Re: [Errno 28] No space left on device

Dominic Raferd-3
On Thu, 14 Nov 2019 at 19:12, <[hidden email]> wrote:

> Hi,
>
> On 14/11/2019 01:47, Yves Bellefeuille wrote:
> >> 3. which file system type you are using to backup?
> > Backing-up from NTFS (Windows 7) to ext4.
>
> Cross-filesystem-types backup is always a bit tricky. This said if you
> have the occasion, it would be nice to test the latest version from
> master. We could use some real life testers.
>
> >
> >> 4. do you also get the issue if you use the `--no-eas` option?
> > Indeed, this seems to have solved the problem.
> >
> > I wonder why this suddenly failed after working for several years.
> > Thanks for the help.
>
> Here I can possibly offer you an explanation: according to man 5 xattr,
> the ext4 file system has a limit of extended attributes size (1, 2 or
> 4KB), and you might have now a Windows file with bigger EAs, which leads
> to the issue:
>
> touch xxx
> python3
>  >>> x = '0123456789' * 1024  # 10KB size
>  >>> import xattr
>  >>> xattr.setxattr('xxx', 'user.comment', x)
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> OSError: [Errno 28] No space left on device
>
> So you have multiple solutions:
>
> 1. switch from ext4 to XFS, which doesn't know this limit (I checked)
> 2. identify the culprit file and trim the EAs (somehow?!?)
> 3. use the --no-eas option, not knowing what you lose as information
>
> I created an issue against pyxattr:
> https://github.com/iustin/pyxattr/issues/23


Unexpected out of space errors often (or even usually) relate to the
temporary file location. Try increasing the space in this location or
specifying a different one using rdiff-backup's --tempdir or
--remote-tempdir options. Regressions (which are attempted automatically by
rdiff-backup if it finds a damaged repository) are especially
temp-intensive. I have a 60GB SSD entirely dedicated to /tmp.
Reply | Threaded
Open this post in threaded view
|

Re: [Errno 28] No space left on device

Walt Mankowski
On Fri, Nov 15, 2019 at 09:01:43AM +0000, Dominic Raferd wrote:
> Unexpected out of space errors often (or even usually) relate to the
> temporary file location. Try increasing the space in this location or
> specifying a different one using rdiff-backup's --tempdir or
> --remote-tempdir options. Regressions (which are attempted automatically by
> rdiff-backup if it finds a damaged repository) are especially
> temp-intensive. I have a 60GB SSD entirely dedicated to /tmp.

I had a similar problem a few months ago. In my case the problem was
that one of the directories I was backing up was Chrome's cache
directory. The files were relatively short lived, but they had
basically random filenames and there were new ones every day. After
several years of this I had millions of files in one of one of
rdiff-backup's control directories.

It turns out that by default there's a limit to how big a directory
can get in ext4. When you reach that limit and try to add another file
to that directory you get an out of disk error.

Look through the mailing list archives for more information.

Walt