Long path problem

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Long path problem

Ben Escoto
I mentioned the long filename problem before, and Chris Wilson
mentioned that I forgot one case:  where the path as a whole is too
long, even if each file/directory name is short.

I thought this could be solved by changing directories, but after
thinking about it this would actually be a pain:  I'd need to add
either a check or a chdir before every file operation.  And I'm sure
I'd forget some places, and would be finding random (hard to
diagnose?) errors after this.

Also a path may be more than 2x as long as the limit.  These long
paths would require 2 or more chdirs to reach.


So anyway, I'm curious when/how the long-path-problem is coming up.  I
wish rdiff-backup could successfully mirror all the files in this
case, but don't see a way of doing this that's not error-prone.
(Treating them the same way as long filename files and sticking them
in a separate directory would be a bit easier.)


--
Ben Escoto

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Long path problem

Randall Nortman-3
On Sat, Nov 19, 2005 at 08:52:30PM -0600, Ben Escoto wrote:
[...]
> So anyway, I'm curious when/how the long-path-problem is coming up.  I
> wish rdiff-backup could successfully mirror all the files in this
> case, but don't see a way of doing this that's not error-prone.
> (Treating them the same way as long filename files and sticking them
> in a separate directory would be a bit easier.)

This was my most recent error along these lines:

SpecialFileError
var/lib/apache/fastcgi/80292d047b4ad9c7faad6b676c4fe423 Socket
error: AF_UNIX path too long

I get these every now and then.  Presumably these are temporary files
(sockets, actually, in this case) being created and destroyed rapidly,
and so I only see the error when the planets align and the backup
coincides with the temporary file.  I don't worry about it in this
case.  I don't recall ever seeing it on non-socket files for that
matter.  Now that I think about it, shouldn't rdiff-backup be ignoring
sockets files?


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Long path problem

Chris Wilson-5
In reply to this post by Ben Escoto
Hi Ben,

> So anyway, I'm curious when/how the long-path-problem is coming up.  I
> wish rdiff-backup could successfully mirror all the files in this
> case, but don't see a way of doing this that's not error-prone.
> (Treating them the same way as long filename files and sticking them
> in a separate directory would be a bit easier.)

I used a simple shell script on a Linux ext3 filesystem:

  while mkdir aaaaaaaa && cd aaaaaaaa; do true; done

The path got to nearly 8000 characters long, and although cd was giving
warnings about "getcwd: path name too long", it was still going when I
killed it.

When I try to back up the root of this huge tree with rdiff-backup, I get
the following error:

Processing changed file aaaaaaaa/aaaaaaaa/.../aaaaaaaa
Exception '[Errno 36] File name too long: 'aaaaaaaa/.../aaaaaaaaa'' raised
of class 'exceptions.OSError':
   File "/usr/lib/python2.3/site-packages/rdiff_backup/robust.py", line 32,
in check_common_error
     try: return function(*args)
   File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 922,
in append
     return self.__class__(self.conn, self.base, self.index + (ext,))
   File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 669,
in __init__
     else: self.setdata()
   File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 693,
in setdata
     self.data = self.conn.C.make_file_dict(self.path)

Hope this helps,

Cheers, Chris.
--
_ ___ __     _
  / __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Long path problem

Ben Escoto
In reply to this post by Randall Nortman-3
>>>>> Randall Nortman <[hidden email]>
>>>>> wrote the following on Mon, 21 Nov 2005 09:50:51 -0500

> This was my most recent error along these lines:
>
> SpecialFileError
> var/lib/apache/fastcgi/80292d047b4ad9c7faad6b676c4fe423 Socket
> error: AF_UNIX path too long

Yes, for some reason (historical, presumably) sockets need to be
shorter than any other kind of file.

But those don't count since I think rdiff-backup already handles those
socket cases satisfactorily :)


--
Ben Escoto

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Long path problem

Ben Escoto
In reply to this post by Chris Wilson-5
>>>>> Chris Wilson <[hidden email]>
>>>>> wrote the following on Tue, 22 Nov 2005 20:47:29 +0000 (GMT)
>
> I used a simple shell script on a Linux ext3 filesystem:
>
>   while mkdir aaaaaaaa && cd aaaaaaaa; do true; done
>
> The path got to nearly 8000 characters long, and although cd was giving
> warnings about "getcwd: path name too long", it was still going when I
> killed it.

May I ask why you did this?  Is it ok if rdiff-backup just skipped
over a lot of those directories?  (As long as it didn't crash of
course.)  Or do you need that structure correctly mirrored?


--
Ben Escoto

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Long path problem

Chris Wilson-3
Hi Ben,

>> The path got to nearly 8000 characters long, and although cd was giving
>> warnings about "getcwd: path name too long", it was still going when I
>> killed it.
>
> May I ask why you did this?

For kicks? :-) I wanted to see if I could make my kernel panic? And to see
if rdiff-backup really did have a problem with long path names?

> Is it ok if rdiff-backup just skipped over a lot of those directories?
> (As long as it didn't crash of course.)  Or do you need that structure
> correctly mirrored?

I don't need it mirrored as such, it was a kind of stress test for
rdiff-backup. But I think it shows that rdiff-backup may have problems
backing up some legitimate directory structures, and that the long path
problem is not just theoretical.

Cheers, Chris.
--
_ ___ __     _
  / __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Long path problem

Ben Escoto
>>>>> Chris Wilson <[hidden email]>
>>>>> wrote the following on Wed, 23 Nov 2005 00:23:01 +0000 (GMT)
>
> I don't need it mirrored as such, it was a kind of stress test for
> rdiff-backup. But I think it shows that rdiff-backup may have problems
> backing up some legitimate directory structures, and that the long path
> problem is not just theoretical.

Oh I see, I've done something similar on my system and know errors can
be produced.  I'm not sure your example qualifies as a legitimate
directory structure though :-)

I think most people will be fine until their paths reach about 3950
chars.  So I was mainly wondering if it's ok to skip paths longer than
that, or if someone actually has data they want to save in files whose
paths are 5kB long.


--
Ben Escoto

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Long path problem

Yoav-3
On Tue, 22 Nov 2005, Ben Escoto wrote:

> I think most people will be fine until their paths reach about 3950
> chars.  So I was mainly wondering if it's ok to skip paths longer than
> that, or if someone actually has data they want to save in files whose
> paths are 5kB long.
>

If I may barge in, I think it actually does happen legitimately, when a
file-level crypto filesystem is being backed up (not the plaintext inside
but the encrypted tree).  For example EncFS pads filenames/dirnames to
make it harder to guess the original filename.  It is not uncommon to end
up with very long pathnames in this mode (which is the default).

True, shortly after the 3950 limit is reached, EncFS will itself fail due
to inability to write the encrypted files.  However, some pathnames may
fall in between so they will be written correctly by EncFS but not backed
up by rdiff-backup.

  Yoav


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Long path problem

aliceten
You can use Long Path Tool as well, it works for such issues.