rdiff-backup dies with ENOMEM due to requesting enormous amounts of memory

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

rdiff-backup dies with ENOMEM due to requesting enormous amounts of memory

Jeff White
I'm trying to get rdiff-backup working for the first time across SSH
(with sudo) on two CentOS 7 hosts.  I'm getting the following crash:

$ sudo rdiff-backup --restrict-read-only / -v5 --remote-schema 'ssh -l
rdiff -i /home/rdiff/.ssh/id_rsa -t -t %s "sudo /usr/bin/rdiff-backup
--server"' bacula-p1n01::/ /dumps/bacula-p1n01/
Thu Jun  9 15:34:42 2016  Using rdiff-backup version 1.2.8
Thu Jun  9 15:34:42 2016  Executing ssh -l rdiff -i
/home/rdiff/.ssh/id_rsa -t -t bacula-p1n01 "sudo /usr/bin/rdiff-backup
--server"
Thu Jun  9 15:34:42 2016  Client sending (0): ConnectionRequest:
Globals.get with 1 arguments
Thu Jun  9 15:34:42 2016  Client sending (0): 'version'
Connection to bacula-p1n01 closed.
Thu Jun  9 15:34:43 2016  Exception '' raised of class '<type
'exceptions.MemoryError'>':
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
304, in error_check_Main
     try: Main(arglist)
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
321, in Main
     rps = map(SetConnections.cmdpair2rp, cmdpairs)
   File
"/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py",
line 76, in cmdpair2rp
     if cmd: conn = init_connection(cmd)
   File
"/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py",
line 150, in init_connection
     check_connection_version(conn, remote_cmd)
   File
"/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py",
line 158, in check_connection_version
     try: remote_version = conn.Globals.get('version')
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py",
line 450, in __call__
     return apply(self.connection.reval, (self.name,) + args)
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py",
line 368, in reval
     result = self.get_response(req_num)
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py",
line 315, in get_response
     try: req_num, object = self._get()
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py",
line 240, in _get
     data = self._read(length)
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py",
line 210, in _read
     try: return self.inpipe.read(length)

Traceback (most recent call last):
   File "/usr/bin/rdiff-backup", line 30, in <module>
     rdiff_backup.Main.error_check_Main(sys.argv[1:])
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
304, in error_check_Main
     try: Main(arglist)
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
321, in Main
     rps = map(SetConnections.cmdpair2rp, cmdpairs)
   File
"/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py",
line 76, in cmdpair2rp
     if cmd: conn = init_connection(cmd)
   File
"/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py",
line 150, in init_connection
     check_connection_version(conn, remote_cmd)
   File
"/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py",
line 158, in check_connection_version
     try: remote_version = conn.Globals.get('version')
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py",
line 450, in __call__
     return apply(self.connection.reval, (self.name,) + args)
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py",
line 368, in reval
     result = self.get_response(req_num)
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py",
line 315, in get_response
     try: req_num, object = self._get()
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py",
line 240, in _get
     data = self._read(length)
   File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py",
line 210, in _read
     try: return self.inpipe.read(length)
MemoryError


On that host when I strace the process I see:

[pid 11602] mmap(NULL, 18118029061677056, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
...
[pid 11602] <... mmap resumed> )        = -1 ENOMEM (Cannot allocate memory)


... am I mistaken or is rdiff-backup requesting a ridiculous amount of
memory?  Why would it do that?

--
Jeff White
HPC Systems Engineer
Information Technology Services - WSU


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup dies with ENOMEM due to requesting enormous amounts of memory

Patrik Dufresne-2
Hello Jeff,

You problem is interesting to me ! Rdiff-backup doesn't have the tendency to use alot of memory (it rarely reach 100MB on my systems). It is surprising to see an out or memory error like this in rdiff-backup.

May you provide some information revargin your current environment:
1. How many memory free you have
2. May you describes how big it the stuff you are trying to backup (nb files, average file size, etc.)
3. x86 or x86_64 ?
4. What is you python version ?

Can you get your hand on the server logs too ? Causes it fail when requesting stuff from remote server...


--
Patrik Dufresne Service Logiciel inc.
http://www.patrikdufresne.com/
514-971-6442
1-114 rue des Hautbois,
St-Colomban, QC J5K 2H6

On Thu, Jun 9, 2016 at 6:54 PM, Jeff White <[hidden email]> wrote:
I'm trying to get rdiff-backup working for the first time across SSH (with sudo) on two CentOS 7 hosts.  I'm getting the following crash:

$ sudo rdiff-backup --restrict-read-only / -v5 --remote-schema 'ssh -l rdiff -i /home/rdiff/.ssh/id_rsa -t -t %s "sudo /usr/bin/rdiff-backup --server"' bacula-p1n01::/ /dumps/bacula-p1n01/
Thu Jun  9 15:34:42 2016  Using rdiff-backup version 1.2.8
Thu Jun  9 15:34:42 2016  Executing ssh -l rdiff -i /home/rdiff/.ssh/id_rsa -t -t bacula-p1n01 "sudo /usr/bin/rdiff-backup --server"
Thu Jun  9 15:34:42 2016  Client sending (0): ConnectionRequest: Globals.get with 1 arguments
Thu Jun  9 15:34:42 2016  Client sending (0): 'version'
Connection to bacula-p1n01 closed.
Thu Jun  9 15:34:43 2016  Exception '' raised of class '<type 'exceptions.MemoryError'>':
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 321, in Main
    rps = map(SetConnections.cmdpair2rp, cmdpairs)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 76, in cmdpair2rp
    if cmd: conn = init_connection(cmd)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 150, in init_connection
    check_connection_version(conn, remote_cmd)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 158, in check_connection_version
    try: remote_version = conn.Globals.get('version')
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 450, in __call__
    return apply(self.connection.reval, (self.name,) + args)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 368, in reval
    result = self.get_response(req_num)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 315, in get_response
    try: req_num, object = self._get()
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 240, in _get
    data = self._read(length)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 210, in _read
    try: return self.inpipe.read(length)

Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 321, in Main
    rps = map(SetConnections.cmdpair2rp, cmdpairs)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 76, in cmdpair2rp
    if cmd: conn = init_connection(cmd)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 150, in init_connection
    check_connection_version(conn, remote_cmd)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 158, in check_connection_version
    try: remote_version = conn.Globals.get('version')
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 450, in __call__
    return apply(self.connection.reval, (self.name,) + args)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 368, in reval
    result = self.get_response(req_num)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 315, in get_response
    try: req_num, object = self._get()
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 240, in _get
    data = self._read(length)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 210, in _read
    try: return self.inpipe.read(length)
MemoryError


On that host when I strace the process I see:

[pid 11602] mmap(NULL, 18118029061677056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
...
[pid 11602] <... mmap resumed> )        = -1 ENOMEM (Cannot allocate memory)


... am I mistaken or is rdiff-backup requesting a ridiculous amount of memory?  Why would it do that?

--
Jeff White
HPC Systems Engineer
Information Technology Services - WSU


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup dies with ENOMEM due to requesting enormous amounts of memory

Jeff White
On 06/10/2016 04:34 AM, Patrik Dufresne wrote:
Hello Jeff,

You problem is interesting to me ! Rdiff-backup doesn't have the tendency to use alot of memory (it rarely reach 100MB on my systems). It is surprising to see an out or memory error like this in rdiff-backup.

May you provide some information revargin your current environment:
1. How many memory free you have
Should be plenty:

$ free -m
              total        used        free      shared  buff/cache   available
Mem:           1869         172        1169           0         527        1538
Swap:             0           0           0
2. May you describes how big it the stuff you are trying to backup (nb files, average file size, etc.)
I'm trying to back up the root filesystem of a generic CentOS 7 box.  It doesn't output that it even hit the filesystem yet though, it dies before that from what I can tell.  It dies in the same way even when I try to back up a single empty directory.
3. x86 or x86_64 ?
x86_64
4. What is you python version ?
$ python -V
Python 2.7.5

Can you get your hand on the server logs too ? Causes it fail when requesting stuff from remote server...
Nothing in any logs other than simply showing someone logged in via SSH.  The ENOMEM is /not/ on the remote side, it's on the side starting the backup and doing a pull from another server.


--
Patrik Dufresne Service Logiciel inc.
http://www.patrikdufresne.com/
514-971-6442
1-114 rue des Hautbois,
St-Colomban, QC J5K 2H6

On Thu, Jun 9, 2016 at 6:54 PM, Jeff White <[hidden email]> wrote:
I'm trying to get rdiff-backup working for the first time across SSH (with sudo) on two CentOS 7 hosts.  I'm getting the following crash:

$ sudo rdiff-backup --restrict-read-only / -v5 --remote-schema 'ssh -l rdiff -i /home/rdiff/.ssh/id_rsa -t -t %s "sudo /usr/bin/rdiff-backup --server"' bacula-p1n01::/ /dumps/bacula-p1n01/
Thu Jun  9 15:34:42 2016  Using rdiff-backup version 1.2.8
Thu Jun  9 15:34:42 2016  Executing ssh -l rdiff -i /home/rdiff/.ssh/id_rsa -t -t bacula-p1n01 "sudo /usr/bin/rdiff-backup --server"
Thu Jun  9 15:34:42 2016  Client sending (0): ConnectionRequest: Globals.get with 1 arguments
Thu Jun  9 15:34:42 2016  Client sending (0): 'version'
Connection to bacula-p1n01 closed.
Thu Jun  9 15:34:43 2016  Exception '' raised of class '<type 'exceptions.MemoryError'>':
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 321, in Main
    rps = map(SetConnections.cmdpair2rp, cmdpairs)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 76, in cmdpair2rp
    if cmd: conn = init_connection(cmd)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 150, in init_connection
    check_connection_version(conn, remote_cmd)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 158, in check_connection_version
    try: remote_version = conn.Globals.get('version')
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 450, in __call__
    return apply(self.connection.reval, (self.name,) + args)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 368, in reval
    result = self.get_response(req_num)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 315, in get_response
    try: req_num, object = self._get()
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 240, in _get
    data = self._read(length)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 210, in _read
    try: return self.inpipe.read(length)

Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 321, in Main
    rps = map(SetConnections.cmdpair2rp, cmdpairs)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 76, in cmdpair2rp
    if cmd: conn = init_connection(cmd)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 150, in init_connection
    check_connection_version(conn, remote_cmd)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 158, in check_connection_version
    try: remote_version = conn.Globals.get('version')
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 450, in __call__
    return apply(self.connection.reval, (self.name,) + args)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 368, in reval
    result = self.get_response(req_num)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 315, in get_response
    try: req_num, object = self._get()
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 240, in _get
    data = self._read(length)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 210, in _read
    try: return self.inpipe.read(length)
MemoryError


On that host when I strace the process I see:

[pid 11602] mmap(NULL, 18118029061677056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
...
[pid 11602] <... mmap resumed> )        = -1 ENOMEM (Cannot allocate memory)


... am I mistaken or is rdiff-backup requesting a ridiculous amount of memory?  Why would it do that?

--
Jeff White
HPC Systems Engineer
Information Technology Services - WSU


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki