Backward compatibility of next beta

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Backward compatibility of next beta

Arrigo Marchiori
Dear All,

I cannot recall from my archives, if the upcoming beta of rdiff-backup
will claim to be compatible with previous versions.

What is the plan?

 1- to be compatible with rdiff-backup-1.2.8 ?
 2- to be compatible with rdiff-backup-1.3.3 ?
 3- none of the above?

To make my question more clear: if the beta is ``compatible'' then I
could use it on the _same_ source and destination directories on which
I have used the previous version for years. If it is not compatible, I
should start a ``new backup history'' into another destination
directory.

The FAQ states that compatibility is not assured, but Eric's message
from October 2019 [1] seems to suggest that it should work.

I would like to have an explicit statement from the developers before
setting up my own beta testing.

Thank you in advance!

References:

 1: https://lists.nongnu.org/archive/html/rdiff-backup-users/2019-10/msg00008.html
--
rigo

http://rigo.altervista.org

Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Frank Crawford
Rigo,
It depends on what you mean by compatible.
Python3 version of rdiff-backup is not interoperable for remote
operations with python3 rdiff-backup, so v2.0.0 will not work with
v1.2.8 or v1.3.3.  This is a python3 vs python2 issue and could not be
fixed without major changes to both versions.
However, the backup archives that are created are compatible, so if you
did a backup with an older version, you can continue using it with no
issues.
In fact, while this isn't in the planned distribution, there is some
comment by someone who has written a wrapper that invokes either a
python2 or python3 version, depending on what it is talking to.  Of
course the python2 version is either v1.2.8 or v1.3.3, not v2.0.0.
RegardsFrank
On Wed, 2020-01-22 at 11:23 +0100, Arrigo Marchiori wrote:

> Dear All,
> I cannot recall from my archives, if the upcoming beta of rdiff-
> backupwill claim to be compatible with previous versions.
> What is the plan?
>  1- to be compatible with rdiff-backup-1.2.8 ? 2- to be compatible
> with rdiff-backup-1.3.3 ? 3- none of the above?
> To make my question more clear: if the beta is ``compatible'' then
> Icould use it on the _same_ source and destination directories on
> whichI have used the previous version for years. If it is not
> compatible, Ishould start a ``new backup history'' into another
> destinationdirectory.
> The FAQ states that compatibility is not assured, but Eric's
> messagefrom October 2019 [1] seems to suggest that it should work.
> I would like to have an explicit statement from the developers
> beforesetting up my own beta testing.
> Thank you in advance!
> References:
>  1:
> https://lists.nongnu.org/archive/html/rdiff-backup-users/2019-10/msg00008.html
Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Frank Crawford
In reply to this post by Arrigo Marchiori
(Sorry about the previous formatting I'm resending this formatted
correctly)

Rigo,

It depends on what you mean by compatible.

Python3 version of rdiff-backup is not interoperable for remote
operations with python2 rdiff-backup, so v2.0.0 will not work with
v1.2.8 or v1.3.3.  This is a python3 vs python2 issue and could not be
fixed without major changes to both versions.

However, the backup archives that are created are compatible, so if you
did a backup with an older version, you can continue using it with no
issues.

In fact, while this isn't in the planned distribution, there is some
comment by someone who has written a wrapper that invokes either a
python2 or python3 version, depending on what it is talking to.  Of
course the python2 version is either v1.2.8 or v1.3.3, not v2.0.0.

Regards
Frank

On Wed, 2020-01-22 at 11:23 +0100, Arrigo Marchiori wrote:

> Dear All,
> I cannot recall from my archives, if the upcoming beta of rdiff-
> backupwill claim to be compatible with previous versions.
> What is the plan?
>  1- to be compatible with rdiff-backup-1.2.8 ? 2- to be compatible
> with rdiff-backup-1.3.3 ? 3- none of the above?
> To make my question more clear: if the beta is ``compatible'' then
> Icould use it on the _same_ source and destination directories on
> whichI have used the previous version for years. If it is not
> compatible, Ishould start a ``new backup history'' into another
> destinationdirectory.
> The FAQ states that compatibility is not assured, but Eric's
> messagefrom October 2019 [1] seems to suggest that it should work.
> I would like to have an explicit statement from the developers
> beforesetting up my own beta testing.
> Thank you in advance!
> References:
>  1:
> https://lists.nongnu.org/archive/html/rdiff-backup-users/2019-10/msg00008.html
>


Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Derek Atkins-2
Hi,

Frank Crawford <[hidden email]> writes:

> (Sorry about the previous formatting I'm resending this formatted
> correctly)
>
> Rigo,
>
> It depends on what you mean by compatible.
>
> Python3 version of rdiff-backup is not interoperable for remote
> operations with python2 rdiff-backup, so v2.0.0 will not work with
> v1.2.8 or v1.3.3.  This is a python3 vs python2 issue and could not be
> fixed without major changes to both versions.

So just to be clear, the version of "rdiff-backup --server" must match
the other version?  Right now I have a backup server that basically
runs:

rdiff-backup <excludelist> hostname_backup::<dir> <targetdir>/hostname

Where "hostname_backup" is an SSH configuration, and it then uses SSH to
connect to "hostname" where it creates and mounts an LVM snapshot and
then runs "rdiff-backup --server --restrict-read-only /mnt/snap"

Having said all this, what you're saying is that once ANY of my target
hosts get upgraded to a point where they can only run
rdiff-backup-2.0.0, then I must upgrade ALL most hosts (and backup
server) to be running rdiff-backup-2.0.0?

IMHO, that is a VERY VERY BAD THING.

> However, the backup archives that are created are compatible, so if you
> did a backup with an older version, you can continue using it with no
> issues.
>
> In fact, while this isn't in the planned distribution, there is some
> comment by someone who has written a wrapper that invokes either a
> python2 or python3 version, depending on what it is talking to.  Of
> course the python2 version is either v1.2.8 or v1.3.3, not v2.0.0.

I think this is extremely important to get into the release in order to
handle the use case stated above!  I certainly can't believe that I am
the only person who uses rdiff-backup this way to back up remote
machines.

> Regards
> Frank

-derek
--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant

Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

EricZolf
Hi Derek,

your understanding is sadly correct but was always like this, because
there is no concept of API versioning (like we know in the mean time for
REST APIs). This is something we have on our roadmap but it's not an
easy thing to do, and definitely not with this release.

See
https://github.com/rdiff-backup/rdiff-backup/milestones?direction=asc&sort=title&state=open

In the mean time, I hope we can avoid further incompatibilities, but
it's not a promise.

Sorry, Eric

On 28/01/2020 21:45, Derek Atkins wrote:

> Hi,
>
> Frank Crawford <[hidden email]> writes:
>
>> (Sorry about the previous formatting I'm resending this formatted
>> correctly)
>>
>> Rigo,
>>
>> It depends on what you mean by compatible.
>>
>> Python3 version of rdiff-backup is not interoperable for remote
>> operations with python2 rdiff-backup, so v2.0.0 will not work with
>> v1.2.8 or v1.3.3.  This is a python3 vs python2 issue and could not be
>> fixed without major changes to both versions.
>
> So just to be clear, the version of "rdiff-backup --server" must match
> the other version?  Right now I have a backup server that basically
> runs:
>
> rdiff-backup <excludelist> hostname_backup::<dir> <targetdir>/hostname
>
> Where "hostname_backup" is an SSH configuration, and it then uses SSH to
> connect to "hostname" where it creates and mounts an LVM snapshot and
> then runs "rdiff-backup --server --restrict-read-only /mnt/snap"
>
> Having said all this, what you're saying is that once ANY of my target
> hosts get upgraded to a point where they can only run
> rdiff-backup-2.0.0, then I must upgrade ALL most hosts (and backup
> server) to be running rdiff-backup-2.0.0?
>
> IMHO, that is a VERY VERY BAD THING.
>
>> However, the backup archives that are created are compatible, so if you
>> did a backup with an older version, you can continue using it with no
>> issues.
>>
>> In fact, while this isn't in the planned distribution, there is some
>> comment by someone who has written a wrapper that invokes either a
>> python2 or python3 version, depending on what it is talking to.  Of
>> course the python2 version is either v1.2.8 or v1.3.3, not v2.0.0.
>
> I think this is extremely important to get into the release in order to
> handle the use case stated above!  I certainly can't believe that I am
> the only person who uses rdiff-backup this way to back up remote
> machines.
>
>> Regards
>> Frank
>
> -derek
>


Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Arrigo Marchiori
In reply to this post by Derek Atkins-2
Hello,

On Tue, Jan 28, 2020 at 03:45:19PM -0500, Derek Atkins wrote:

> Hi,
>
> Frank Crawford <[hidden email]> writes:
>
> > (Sorry about the previous formatting I'm resending this formatted
> > correctly)
> >
> > Rigo,
> >
> > It depends on what you mean by compatible.
> >
> > Python3 version of rdiff-backup is not interoperable for remote
> > operations with python2 rdiff-backup, so v2.0.0 will not work with
> > v1.2.8 or v1.3.3.  This is a python3 vs python2 issue and could not be
> > fixed without major changes to both versions.
>
> So just to be clear, the version of "rdiff-backup --server" must match
> the other version?  Right now I have a backup server that basically
> runs:
>
> rdiff-backup <excludelist> hostname_backup::<dir> <targetdir>/hostname
>
> Where "hostname_backup" is an SSH configuration, and it then uses SSH to
> connect to "hostname" where it creates and mounts an LVM snapshot and
> then runs "rdiff-backup --server --restrict-read-only /mnt/snap"
>
> Having said all this, what you're saying is that once ANY of my target
> hosts get upgraded to a point where they can only run
> rdiff-backup-2.0.0, then I must upgrade ALL most hosts (and backup
> server) to be running rdiff-backup-2.0.0?
>
> IMHO, that is a VERY VERY BAD THING.
>
> > However, the backup archives that are created are compatible, so if you
> > did a backup with an older version, you can continue using it with no
> > issues.
> >
> > In fact, while this isn't in the planned distribution, there is some
> > comment by someone who has written a wrapper that invokes either a
> > python2 or python3 version, depending on what it is talking to.  Of
> > course the python2 version is either v1.2.8 or v1.3.3, not v2.0.0.
>
> I think this is extremely important to get into the release in order to
> handle the use case stated above!  I certainly can't believe that I am
> the only person who uses rdiff-backup this way to back up remote
> machines.

Maybe you could find a solution by using the --remote-schema
parameter? If I understood correctly, it should allow you to keep two
versions of rdiff-backup side-by-side, rather than upgrading the
program at ``system-level''.

I cannot try this, but I would suggest:

 1- on the client _and_ server, install the new version in ~/rdiff-backup-2
 2- run the version 2 client with the parameter:
      --remote-schema 'ssh -C %s ~/rdiff-backup-2/rdiff-backup --server'
   
I hope this helps,
--
rigo

http://rigo.altervista.org

Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Frank Crawford
On Thu, 2020-01-30 at 11:40 +0100, Arrigo Marchiori wrote:

> Hello,
> On Tue, Jan 28, 2020 at 03:45:19PM -0500, Derek Atkins wrote:
> > Hi,
> > Frank Crawford <[hidden email]> writes:
> > > (Sorry about the previous formatting I'm resending this
> > > formattedcorrectly)
> > > Rigo,
> > > It depends on what you mean by compatible.
> > > Python3 version of rdiff-backup is not interoperable for
> > > remoteoperations with python2 rdiff-backup, so v2.0.0 will not
> > > work withv1.2.8 or v1.3.3.  This is a python3 vs python2 issue
> > > and could not befixed without major changes to both versions.
> >
> > So just to be clear, the version of "rdiff-backup --server" must
> > matchthe other version?  Right now I have a backup server that
> > basicallyruns:
> > rdiff-backup <excludelist> hostname_backup::<dir>
> > <targetdir>/hostname
> > Where "hostname_backup" is an SSH configuration, and it then uses
> > SSH toconnect to "hostname" where it creates and mounts an LVM
> > snapshot andthen runs "rdiff-backup --server --restrict-read-only
> > /mnt/snap"
> > Having said all this, what you're saying is that once ANY of my
> > targethosts get upgraded to a point where they can only runrdiff-
> > backup-2.0.0, then I must upgrade ALL most hosts (and backupserver)
> > to be running rdiff-backup-2.0.0?
> > IMHO, that is a VERY VERY BAD THING.
> > > However, the backup archives that are created are compatible, so
> > > if youdid a backup with an older version, you can continue using
> > > it with noissues.
> > > In fact, while this isn't in the planned distribution, there is
> > > somecomment by someone who has written a wrapper that invokes
> > > either apython2 or python3 version, depending on what it is
> > > talking to.  Ofcourse the python2 version is either v1.2.8 or
> > > v1.3.3, not v2.0.0.
> >
> > I think this is extremely important to get into the release in
> > order tohandle the use case stated above!  I certainly can't
> > believe that I amthe only person who uses rdiff-backup this way to
> > back up remotemachines.
>
> Maybe you could find a solution by using the --remote-
> schemaparameter? If I understood correctly, it should allow you to
> keep twoversions of rdiff-backup side-by-side, rather than upgrading
> theprogram at ``system-level''.
> I cannot try this, but I would suggest:
>  1- on the client _and_ server, install the new version in ~/rdiff-
> backup-2 2- run the version 2 client with the parameter:      
> --remote-schema 'ssh -C %s ~/rdiff-backup-2/rdiff-backup --
> server'    I hope this helps,

No, the use of "--remote-schema" is the issue as it passes data between
the two programs in a specific format that has changed between python2
and python3 because of internal variations.

Regards
Frank
Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Derek Atkins-2
In reply to this post by EricZolf
Hi,

EricZolf <[hidden email]> writes:

> Hi Derek,
>
> your understanding is sadly correct but was always like this, because
> there is no concept of API versioning (like we know in the mean time for
> REST APIs). This is something we have on our roadmap but it's not an
> easy thing to do, and definitely not with this release.
>
> See
> https://github.com/rdiff-backup/rdiff-backup/milestones?direction=asc&sort=title&state=open
>
> In the mean time, I hope we can avoid further incompatibilities, but
> it's not a promise.

This is a MAJOR problem then, which may make me have to re-think my
backup strategy completely.

I've got a dozen machines of various vintages that I'm trying to backup
from a centralized backup server.  Not all of those machines support
Python 3, and it's quite possible that, down the road, some may support
Python 3 but not Python 2.  I absolutely, cannot guarantee that both
ends can always run the same version of rdiff-backup, and frankly I
shouldn't have to.

Luckily, right now everything has been happy with 1.2.8.  But if some
future versions of backup targets drop that support, it sounds like I'm
going to be stuck.

It's really a shame there is no consistent client-server communication
protocol.  I'm not even going to call it an API, just a consistent
communication channel between rdiff-backup client and rdiff-backup
server, that would allow different versions to communicate.

From previous discussions when talking about how to handle file names
(different string types), I was under the impression that this was
"fixed" and there was future compatibility.  I thought I asked this
clearly when compatibility discussions happened several months ago, and
I thought I heard the answer that 2.0 and 1.2.8 would be able to "talk".
I'm not sure where this miscommunication happened back then.  :(

> Sorry, Eric

Me too.  Off to find a cross-version compatible backup solution :(

-derek

--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant

Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Derek Atkins-2
In reply to this post by Frank Crawford
Frank Crawford <[hidden email]> writes:

> No, the use of "--remote-schema" is the issue as it passes data between
> the two programs in a specific format that has changed between python2
> and python3 because of internal variations.

So what data specifically changed?

> Regards
> Frank

-derek

--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant

Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Reio Remma
In reply to this post by Derek Atkins-2
On 30/01/2020 17:01, Derek Atkins wrote:
>
> I've got a dozen machines of various vintages that I'm trying to backup
> from a centralized backup server.  Not all of those machines support
> Python 3, and it's quite possible that, down the road, some may support
> Python 3 but not Python 2.  I absolutely, cannot guarantee that both
> ends can always run the same version of rdiff-backup, and frankly I
> shouldn't have to.


Shouldn't it be possible to run multiple versions on the backup machine?

iirc I had both Python 2 and 3 versions installed on a CentOS 7 machine
at one point.

Good luck,
Reio


Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Derek Atkins-2

On Thu, January 30, 2020 10:12 am, Reio Remma wrote:

> On 30/01/2020 17:01, Derek Atkins wrote:
>>
>> I've got a dozen machines of various vintages that I'm trying to backup
>> from a centralized backup server.  Not all of those machines support
>> Python 3, and it's quite possible that, down the road, some may support
>> Python 3 but not Python 2.  I absolutely, cannot guarantee that both
>> ends can always run the same version of rdiff-backup, and frankly I
>> shouldn't have to.
>
>
> Shouldn't it be possible to run multiple versions on the backup machine?
>
> iirc I had both Python 2 and 3 versions installed on a CentOS 7 machine
> at one point.

Arguably yes, this would work, but it would IMHO require that the backup
server have software that automatically detected the correct remote
version and ran the appropriate frontend.  I would be okay with that
solution for now, provided it was part of the 2.0 package (and that the
packaging is such that 1.2.8 and 2.0 can co-exist on the backup server
easily).

> Good luck,
> Reio

-derek

--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant


Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Patrik Dufresne
In reply to this post by Derek Atkins-2


I also understand the pain regarding non-backward compatibility between the
two versions. In terms of migration, it becomes a lot more complex if both
ends need to run the same version.


I didn't take a look at the technical details behind the incompatibility.
I'm trusting Frank and Eric on this topic and understand it's not an easy
fix to make v2.0.0 compatible with v1.2.8. You must understand it's a draw
back of a bad design in rdiff-backup from the start.


I'm in the same boat as many of you, I have more than two hundred servers
using rdiff-backup directly or through Minarca and it's impossible to
migrate all of them at once. I've been thinking about a migration plan for
some time now and I have a general idea that I can't wait to put in place.
I'm planning to work on a solution end-February. The general idea is to
pre-compile rdiff-backup 1.2.8 and 2.0.0 as individual executable and
deploy both of them on the central server as rdiff-backup-1.2.8 and
rdiff-backup-2.0.0. Then reconfigure the SSH server using a script to
either call rdiff-backup-1.2.8 by default and call rdiff-backup-2.0.0 based
on some condition.

On Thu, Jan 30, 2020 at 10:03 AM Derek Atkins <[hidden email]> wrote:

> Frank Crawford <[hidden email]> writes:
>
> > No, the use of "--remote-schema" is the issue as it passes data between
> > the two programs in a specific format that has changed between python2
> > and python3 because of internal variations.
>
> So what data specifically changed?
>
> > Regards
> > Frank
>
> -derek
>
> --
>        Derek Atkins                 617-623-3745
>        [hidden email]             www.ihtfp.com
>        Computer and Internet Security Consultant
>
>

--
Patrik Dufresne <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Derek Atkins-2
Patrik,

On Thu, January 30, 2020 10:29 am, Patrik Dufresne wrote:

>
> I'm in the same boat as many of you, I have more than two hundred servers
> using rdiff-backup directly or through Minarca and it's impossible to
> migrate all of them at once. I've been thinking about a migration plan for
> some time now and I have a general idea that I can't wait to put in place.
> I'm planning to work on a solution end-February. The general idea is to
> pre-compile rdiff-backup 1.2.8 and 2.0.0 as individual executable and
> deploy both of them on the central server as rdiff-backup-1.2.8 and
> rdiff-backup-2.0.0. Then reconfigure the SSH server using a script to
> either call rdiff-backup-1.2.8 by default and call rdiff-backup-2.0.0
> based
> on some condition.

I'm curious -- do you have each of your 200 servers call into the
centralized backup server?  Or do you have your backup server "reach out"
to the 200 servers?

I actually have it set up so that the backup server reaches out to the
servers being backed up.  I did it this way to limit the access to the
backups themselves, and that the servers being backed-up could be "dumb"
about their backups.  Specifically, all the "private keys" necessary to
backup a device are sitting on the backup server and not on the servers
being backed up (modulo that server's existing SSH key).  It would also
prevent someone who "breaks in" to one server to gain access to the backup
server and corrupt backups from other servers.

In my case, I could have a configuration that "knows" the version running
on the server being backed up and calls the appropriate incarnation of
rdiff-backup on the backup server.  But I would rather this be automated,
so I don't have to pay that close attention and ensure both ends are
properly synced manually.

-derek

--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant


Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Arrigo Marchiori
In reply to this post by Derek Atkins-2
Hello,

On Thu, Jan 30, 2020 at 10:28:14AM -0500, Derek Atkins wrote:

> On Thu, January 30, 2020 10:12 am, Reio Remma wrote:
> > On 30/01/2020 17:01, Derek Atkins wrote:
> >>
> >> I've got a dozen machines of various vintages that I'm trying to backup
> >> from a centralized backup server.  Not all of those machines support
> >> Python 3, and it's quite possible that, down the road, some may support
> >> Python 3 but not Python 2.  I absolutely, cannot guarantee that both
> >> ends can always run the same version of rdiff-backup, and frankly I
> >> shouldn't have to.
> >
> >
> > Shouldn't it be possible to run multiple versions on the backup machine?
> >
> > iirc I had both Python 2 and 3 versions installed on a CentOS 7 machine
> > at one point.
>
> Arguably yes, this would work, but it would IMHO require that the backup
> server have software that automatically detected the correct remote
> version and ran the appropriate frontend.  I would be okay with that
> solution for now, provided it was part of the 2.0 package (and that the
> packaging is such that 1.2.8 and 2.0 can co-exist on the backup server
> easily).

About the packaging: it should be possible to use PyInstaller to make
single-executable distributions of rdiff-backup for Linux. That is
what happens for Windows.

In this way, if you have many computers running on ``old'' Linux
distributions, you would need to install Python 3 just into one of
them, use it to build the executable, and then only copy the
rdiff-backup self-contained executable file into all the other ones.

Best regards,
--
rigo

http://rigo.altervista.org

Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Frank Crawford
In reply to this post by Derek Atkins-2
On Thu, 2020-01-30 at 10:02 -0500, Derek Atkins wrote:

> Frank Crawford <
> [hidden email]
> > writes:
>
> > No, the use of "--remote-schema" is the issue as it passes data
> > between
> > the two programs in a specific format that has changed between
> > python2
> > and python3 because of internal variations.
>
> So what data specifically changed?

Okay, to get into the technical details, as I tracked them down, and
there may be other issues over and above these, but much of the data between remote servers is passed in pickle format.

One of the options in a pickle is to not just pass the data but the
object, or more specifically the data and the function call to process
that data.  These functions are essentially assumed to be the same
function on both sides.  These functions are essentially looked up in
the python symbol table.

Unfortunately in Python2 the functions in the symbol table are in
ascii, while in Python3 they are in unicode, and so the pickle data
doesn't match on the lookup in the symbol table, and hence most objects
passed between the two return a "function not found" error.

Without somehow modifying the symbol table (which would probably break
other things) you can't get around it, and you would need to modify it
on both sides scripts.

There are a number of other items like just the strings transition from
ascii to unicode for the actual data needs to be fixed, on to new
pickle formats available in Python3, that would be a headache, but they
could be addressed.

Unfortunately they are minor compared to the symbol table issue.

> > Regards
> > Frank
>
> -derek
>


Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

EricZolf
Hi,

On 01/02/2020 11:20, Frank Crawford wrote:
> There are a number of other items like just the strings transition from
> ascii to unicode for the actual data needs to be fixed, on to new
> pickle formats available in Python3, that would be a headache, but they
> could be addressed.
>
> Unfortunately they are minor compared to the symbol table issue.

And most important of all, we don't have the man-power to address any of
those aspects without delaying the release of the next version to an
undetermined date.

KR, Eric

Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Arrigo Marchiori
In reply to this post by Frank Crawford
Hello Frank,

On Thu, Jan 30, 2020 at 10:57:09PM +1100, Frank Crawford wrote:

> On Thu, 2020-01-30 at 11:40 +0100, Arrigo Marchiori wrote:
[...]
> > Maybe you could find a solution by using the --remote-
> > schema parameter?
[...]
> No, the use of "--remote-schema" is the issue as it passes data between
> the two programs in a specific format that has changed between python2
> and python3 because of internal variations.

I may have not explained myself correctly, or I might not have
understood anything at all. I'll try again :-)

For what I understood: the "--remote-schema" is the template for a
command that is popen()-ed.

If your server _and client_ have two executables for rdiff-backup, say:
 A) ~/rdiff-backup-old/rdiff-backup
 B) ~/rdiff-backup-new/rdiff-backup

What is the problem in having the client calling:

 1) ~/rdiff-backup-old/rdiff-backup --remote-schema 'ssh -C %s ~/rdiff-backup-old/rdiff-backup --server'

or:

 2) ~/rdiff-backup-new/rdiff-backup --remote-schema 'ssh -C %s ~/rdiff-backup-new/rdiff-backup --server'

The old version on the client would call the old version on the
server, and the new version on the client would call the new version
on the server.

What ``Old'' and ``new'' are, can be chosen with total freedom: stable
and beta? Python 2 and Python 3? Script and self-contained binary?
They would be completely independent, because the only common part
would be ssh transmitting their data.

This still requires to have the same versions of rdiff-backup on the
client and servers, but does not require an upgrade, in terms of the
new version _substituting_ the old. They could coexist side by side!

I hope I could explain myself more clearly now.

Regards,
--
rigo

http://rigo.altervista.org

Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Frank Crawford
On Sun, 2020-02-02 at 22:44 +0100, Arrigo Marchiori wrote:

> Hello Frank,
>
> On Thu, Jan 30, 2020 at 10:57:09PM +1100, Frank Crawford wrote:
>
> > On Thu, 2020-01-30 at 11:40 +0100, Arrigo Marchiori wrote:
> [...]
> > > Maybe you could find a solution by using the --remote-
> > > schema parameter?
> [...]
> > No, the use of "--remote-schema" is the issue as it passes data between
> > the two programs in a specific format that has changed between python2
> > and python3 because of internal variations.
>
> I may have not explained myself correctly, or I might not have
> understood anything at all. I'll try again :-)
>
> For what I understood: the "--remote-schema" is the template for a
> command that is popen()-ed.
>
> If your server _and client_ have two executables for rdiff-backup, say:
>  A) ~/rdiff-backup-old/rdiff-backup
>  B) ~/rdiff-backup-new/rdiff-backup
>
> What is the problem in having the client calling:
>
>  1) ~/rdiff-backup-old/rdiff-backup --remote-schema 'ssh -C %s ~/rdiff-backup-old/rdiff-backup --server'
>
> or:
>
>  2) ~/rdiff-backup-new/rdiff-backup --remote-schema 'ssh -C %s ~/rdiff-backup-new/rdiff-backup --server'
>
> The old version on the client would call the old version on the
> server, and the new version on the client would call the new version
> on the server.
>
> What ``Old'' and ``new'' are, can be chosen with total freedom: stable
> and beta? Python 2 and Python 3? Script and self-contained binary?
> They would be completely independent, because the only common part
> would be ssh transmitting their data.
>
> This still requires to have the same versions of rdiff-backup on the
> client and servers, but does not require an upgrade, in terms of the
> new version _substituting_ the old. They could coexist side by side!
>
> I hope I could explain myself more clearly now.

Ahh, yes, much clearer, and yes something along that line would work.
It is just a scripting exercise of some form to get it working.

> Regards,

Regards
Frank


Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Frank Crawford
On Mon, 2020-02-03 at 22:03 +1100, Frank Crawford wrote:
...

> > For what I understood: the "--remote-schema" is the template for a
> > command that is popen()-ed.
> >
> > If your server _and client_ have two executables for rdiff-backup, say:
> >  A) ~/rdiff-backup-old/rdiff-backup
> >  B) ~/rdiff-backup-new/rdiff-backup
> >
> > What is the problem in having the client calling:
> >
> >  1) ~/rdiff-backup-old/rdiff-backup --remote-schema 'ssh -C %s ~/rdiff-backup-old/rdiff-backup --server'
> >
> > or:
> >
> >  2) ~/rdiff-backup-new/rdiff-backup --remote-schema 'ssh -C %s ~/rdiff-backup-new/rdiff-backup --server'
> >
> > The old version on the client would call the old version on the
> > server, and the new version on the client would call the new version
> > on the server.
> >
> > What ``Old'' and ``new'' are, can be chosen with total freedom: stable
> > and beta? Python 2 and Python 3? Script and self-contained binary?
> > They would be completely independent, because the only common part
> > would be ssh transmitting their data.
> >
> > This still requires to have the same versions of rdiff-backup on the
> > client and servers, but does not require an upgrade, in terms of the
> > new version _substituting_ the old. They could coexist side by side!
> >
> > I hope I could explain myself more clearly now.
>
> Ahh, yes, much clearer, and yes something along that line would work.
> It is just a scripting exercise of some form to get it working.

Just thinking about it, it is even simpler than that, you only need two
versions on the central server, the rest can have either the new or old
version, so there is no need to change the name on the remote host.

You can do something like:

#!/bin/sh

remver=`ssh remhost rdiff-backup --version`
case "$remver" in
"rdiff-backup 2.*") exec rdiff-backup "$@" ;;
*) exec rdiff-backup-old "$@" ;;
esac


Frank


Reply | Threaded
Open this post in threaded view
|

Re: Backward compatibility of next beta

Patrik Dufresne
In reply to this post by Derek Atkins-2
Hello Derek,

Sorry for the delay of response. I wanted to take the time to properly
answer your questions.

This design is built specifically for our Minarca clients. The  central
server is not aware of the location of each client. Clients might be mobile
behind a NAT, or firewall.

So our central server is receiving connections.

About the security conserns. We isolate each users repository based on the
SSH keys.
If one client is compromised, only the associated repo is compromised. Any
how, is the client is compromised, all the data is accessible to the
hacker.

To mitigate the repository corumptions, either because of a hack or any
reason, we have zfs snapshot to recover.

In you case, a hacker could eventually, do modification to rdiff-backup
code to corupmt the backup.



On Thu., Jan. 30, 2020, 11:01 a.m. Derek Atkins, <[hidden email]> wrote:

> Patrik,
>
> On Thu, January 30, 2020 10:29 am, Patrik Dufresne wrote:
> >
> > I'm in the same boat as many of you, I have more than two hundred servers
> > using rdiff-backup directly or through Minarca and it's impossible to
> > migrate all of them at once. I've been thinking about a migration plan
> for
> > some time now and I have a general idea that I can't wait to put in
> place.
> > I'm planning to work on a solution end-February. The general idea is to
> > pre-compile rdiff-backup 1.2.8 and 2.0.0 as individual executable and
> > deploy both of them on the central server as rdiff-backup-1.2.8 and
> > rdiff-backup-2.0.0. Then reconfigure the SSH server using a script to
> > either call rdiff-backup-1.2.8 by default and call rdiff-backup-2.0.0
> > based
> > on some condition.
>
> I'm curious -- do you have each of your 200 servers call into the
> centralized backup server?  Or do you have your backup server "reach out"
> to the 200 servers?
>

Our deployment of rdiff-backup is solely based on Minarca as a centralized
server. So all our "clients" are connecting to the centralized server. It's
built this way because our clients are distributed, behind NAT, some of
them are mobile and some of them are Windows. With these constraints, we
can't have a centralized server reaching the clients.
 

>
> I actually have it set up so that the backup server reaches out to the
> servers being backed up.  I did it this way to limit the access to the
> backups themselves, and that the servers being backed-up could be "dumb"
> about their backups.  


It makes sense if the computer you are backing up are always reachable
without NAT and if all of them are Linux.
 

> Specifically, all the "private keys" necessary to
> backup a device are sitting on the backup server and not on the servers
> being backed up (modulo that server's existing SSH key).


To ease the SSH key exchange, we provide Minarca Client to our users. After
installing Minarca clients, the user must enter it username and password to
establish a connection with the Minarca centralized server. During this
process, it exchanges the SSH keys through a web service over Https with
the Minarca Server.
 

> It would also
> prevent someone who "breaks in" to one server to gain access to the backup
> server and corrupt backups from other servers.
>

To mitigate this Minarca Server provide an SSH entry point that restricts
each user into its own space. In other words, if one client gets
compromised, only it's data get compromised. A malicious user cannot get
access to other backup. I'm also looking for a way to add additional layers
of security using containers.
 

> In my case, I could have a configuration that "knows" the version running
> on the server being backed up and calls the appropriate incarnation of
> rdiff-backup on the backup server.  But I would rather this be automated,
> so I don't have to pay that close attention and ensure both ends are
> properly synced manually.
>

I'm definitely looking toward a similar solution where it's seamless. The
more I'm thinking of it, we may need to change the code in rdiff-backup to
help us. What would really help is having rdiff-backup calling a different
executable in the remote schema. Instead of calling "rdiff-backup" it
should call rdiff-backup2.0.0. And rdiff-backup could be a symbolic link to
the default version, either 1.2.8 or 2.0.0. That would really simplify the
migration process for everyone.

@EricZolf <[hidden email]> Do you think it's feasible ?
It would mitigate all the migration issue and API incompatibilities. We
could also call "rdiff-backup2" (with only the major version) and bump the
major version only when an API incompatibility get introduced in the code.

 

> -derek
>
> --
>        Derek Atkins                 617-623-3745
>        [hidden email]             www.ihtfp.com
>        Computer and Internet Security Consultant
>
>
12