"Corrupted MAC on input" error

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

"Corrupted MAC on input" error

Walter Hop
Hi,

for  the  past  week,  I  seem  to  be running into a consistent error
message  during a backup run on one of the machines: "Corrupted MAC on
input".

This error is not a message of rdiff-backup but seems to be emitted by
the  underlying ssh transport. Has anyone experienced this problem and
found a workaround?

The  client's  configuration is equal to five other backup clients who
don't  have this problem. Could this be due to a large file, or a file
containing some kind of escape sequence?

I have tried the following:
- Changing the versions of ssh/sshd on the client and server;
  I have tried OpenSSH 3.8.1p1 (FreeBSD standard) and OpenSSH 4.2p1
- Changing the login shell of the rdiff user on the server to /bin/sh
  instead of tcsh
- Disabling ssh escape chars by using '-e none' parameter to ssh:
  "--remote-schema '/usr/bin/ssh %s -e none rdiff-backup ..."

Versions:   client   and   server  are  FreeBSD  5.4-RELEASE-p3  using
rdiff-backup from ports rdiff-backup-0.12.7_1.

Kind regards,
Walter Hop
Transip BV

--
  Transip BV | http://www.transip.nl/
  Hoogwaardige Innovatie | Aangename Zekerheid



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: "Corrupted MAC on input" error

Keith Edmunds
Walter Hop wrote:
> for  the  past  week,  I  seem  to  be running into a consistent error
> message  during a backup run on one of the machines: "Corrupted MAC on
> input"

Yes, I've seen exactly this error and there isn't a workaround as such.
It indicates network data corruption, and in my case a customer had a
cable modem that was corrupting fewer than 20 packets in a 1Gb file
transfer. After changing network cards, cables, etc, I finally convinced
the cable company to replace the modem, and the problem went away.

Not sure if that helps or not -

Keith

--
Keith Edmunds

+---------------------------------------------------------------------+
|  Tiger Computing Ltd  |  Helping businesses make the most of Linux  |
|  "The Linux Company"  |       http://www.tiger-computing.co.uk      |
+---------------------------------------------------------------------+


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re[2]: "Corrupted MAC on input" error

Walter Hop
[in reply to [hidden email], 12-1-2006]

>> for  the  past  week,  I  seem  to  be running into a consistent error
>> message  during a backup run on one of the machines: "Corrupted MAC on
>> input"

> Yes, I've seen exactly this error and there isn't a workaround as such.
> It indicates network data corruption, and in my case a customer had a
> cable modem that was corrupting fewer than 20 packets in a 1Gb file
> transfer. After changing network cards, cables, etc, I finally convinced
> the cable company to replace the modem, and the problem went away.

> Not sure if that helps or not -

Thanks! This could definitely be the case, so I tested some more
things.

I grabbed a 500MB file and scp'ed it repeatedly across the network
from the problem client to various hosts to see if I could recreate
the ssh error outside of rdiff-backup. The error happened here as
well, so it seems to be related to large streams of data, and the
problem is not due to some input that rdiff-backup throws at ssh.

[ What's interesting, is when I ftp that file around and back, and
diff the results, there is no difference. But scp'ing almost always
results in subnormal throughput and the "Corrupted MAC on input".
Really strange. We've already replaced the client hardware but get the
same error. I'm hoping it's just a faulty cable and not a broken port
on the switch or something... ]

Thanks for the input,

Walter Hop
Transip BV

--
  Transip BV | http://www.transip.nl/
  Hoogwaardige Innovatie | Aangename Zekerheid



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: "Corrupted MAC on input" error

Dave Kempe
Walter Hop wrote:
> I grabbed a 500MB file and scp'ed it repeatedly across the network
> from the problem client to various hosts to see if I could recreate
> the ssh error outside of rdiff-backup. The error happened here as
> well, so it seems to be related to large streams of data, and the
> problem is not due to some input that rdiff-backup throws at ssh.
>

we have solved similar problems where we had no control over some of the
lower layers, by implementing a openvpn tunnel. SSH over openvpn seems
to improve in reliability a little. I know it sounds whack, but it works.

dave


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

replacing SSH with raw socket use [was: Re: "Corrupted MAC on input" error]

Charles Duffy-6
David Kempe wrote:
> we have solved similar problems where we had no control over some of
> the lower layers, by implementing a openvpn tunnel. SSH over openvpn
> seems to improve in reliability a little. I know it sounds whack, but
> it works.
If you have a VPN, there's no point to running SSH and thus getting two
layers of encryption and authentication -- it makes sense just to
replace SSH with netcat. I've done exactly that, as follows.

On the server, I'm using runit with ipsvd and the following run script:

---- snip run
#!/bin/bash

exec 2>&1

if [ instruct.d -nt instruct.cdb ] ; then
        ipsvd-cdb instruct.cdb $(mktemp instruct.cdb.tmp-XXXXXX) instruct.d
        setfacl -m u:backup:r instruct.cdb
fi

exec tcpsvd -vv -u backup -p -C 1 -c 400 -x instruct.cdb 10.1.128.1
10873 ./rdiff-backup-server
---- end snip

---- snip rdiff-backup-server
#!/bin/sh

if [ -z "$TCPREMOTEHOST" ] ; then
        echo "$TCPLOCALIP not resolved to a hostname; exiting" >&2
        exit 1
fi

DATAPATH="/path/to/data/$TCPREMOTEHOST"
mkdir "$DATAPATH"

exec rdiff-backup \
        --server \
        --restrict "$DATAPATH" \
        --force-path-prefix "$DATAPATH" \
        $*
---- end snip

...where instruct.cdb identifies systems coming over the VPN as good and
everyone else as bad.

This isolates individual machines so that they can only see their own
backed-up content (one system can't restore data backed up by a
different system) and can use an absolute path for backups and restores
(rather than using a path that includes their hostname or which has
other knowledge of the directory structure on the server).

The clients then invoke rdiff-backup as follows:
    rdiff-backup --remote-schema 'netcat %s 10873' <other args>
...and there we go! (Obviously, I'm using GNU netcat).


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki