Examples for whole-system backup

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Examples for whole-system backup

Ricky Huang
Hello all,

My friend recommend rdiff-backup so I am currently researching it for my backup solution.  I am brand new to the system so please forgive me for any dumb questions.

What I am looking for is a FreeBSD backup solution that will duplicate my multiple mission-critical servers to a remote system in case something goes bad.  In case something bad does happen, I’d like to invoke a command and be able to restore the entire system back, not having to do a rebuild then copy files back as that will have a longer downtime.

Is rdiff-backup the correct solution for my use case?  If so, does anyone have any information on whole-system level backup examples / documentations?  I have read documentations on the site already, but they mostly talk about directories.  I am sure there are caveats when one’s backing up complete system.


Thanks in advance.

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Examples for whole-system backup

MasteRTriX
Yes, rdiff-backup maintains a copy of the files and a diff directory where it stores the changes, if you need the latest version is always ready to go. Only if you need lost/deleted files it takes some time to process the diff....

El 8/4/15 a las 18:33, Ricky Huang escribió:
Hello all,

My friend recommend rdiff-backup so I am currently researching it for my backup solution.  I am brand new to the system so please forgive me for any dumb questions.

What I am looking for is a FreeBSD backup solution that will duplicate my multiple mission-critical servers to a remote system in case something goes bad.  In case something bad does happen, I’d like to invoke a command and be able to restore the entire system back, not having to do a rebuild then copy files back as that will have a longer downtime.

Is rdiff-backup the correct solution for my use case?  If so, does anyone have any information on whole-system level backup examples / documentations?  I have read documentations on the site already, but they mostly talk about directories.  I am sure there are caveats when one’s backing up complete system.


Thanks in advance.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Examples for whole-system backup

Nicolas Jungers-2
In reply to this post by Ricky Huang
On 2015-04-08 23:33, Ricky Huang wrote:

> Hello all,
>
> My friend recommend rdiff-backup so I am currently researching it for my
> backup solution.  I am brand new to the system so please forgive me for
> any dumb questions.
>
> What I am looking for is a FreeBSD backup solution that will duplicate
> my multiple mission-critical servers to a remote system in case
> something goes bad.  In case something bad does happen, I’d like to
> invoke a command and be able to restore the entire system back, not
> having to do a rebuild then copy files back as that will have a longer
> downtime.
>
> Is rdiff-backup the correct solution for my use case?  If so, does
> anyone have any information on whole-system level backup examples /
> documentations?  I have read documentations on the site already, but
> they mostly talk about directories.  I am sure there are caveats when
> one’s backing up complete system.
>
>
> Thanks in advance.

I'm sure that you could devise some scheme to do a full metal restore
with rdiff-backup, but in my opinion, it's not the tool for the job.

Regards,
N.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Examples for whole-system backup

rhkramer
Not the OP, but what do you recommend (in the LInux world, please, as that is
what I use...)?

On Thursday, April 09, 2015 02:41:44 AM you wrote:
> I'm sure that you could devise some scheme to do a full metal restore
> with rdiff-backup, but in my opinion, it's not the tool for the job.

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Examples for whole-system backup

Chris Wilson
Hi all,

I have done bare metal restores with duplicity (same author, different tech). My previous job used this as the main emergency recovery mechanism. I believe that rdiff-backup would work just as well. You'll need to sort the comms between client and server (e.g. Ssh keys and firewalls), handle databases and other files being modified during the backup, and test it of course!

Taking LVM snapshots is a very different approach, relatively fast and low impact but requires a lot of storage and bandwidth (no diffs). In the case of bare metal, where you normally only need one backup and often keep large disks on the same site (depending on your threat model) it could also be a good solution for you.

Cheers, Chris.

Sent from my iPhone

> On 9 Apr 2015, at 12:41, [hidden email] wrote:
>
> Not the OP, but what do you recommend (in the LInux world, please, as that is
> what I use...)?
>
>> On Thursday, April 09, 2015 02:41:44 AM you wrote:
>> I'm sure that you could devise some scheme to do a full metal restore
>> with rdiff-backup, but in my opinion, it's not the tool for the job.
>
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Examples for whole-system backup

Robert Nichols-2
In reply to this post by Nicolas Jungers-2
On 04/09/2015 01:41 AM, Nicolas Jungers wrote:
> I'm sure that you could devise some scheme to do a full metal restore
> with rdiff-backup, but in my opinion, it's not the tool for the job.

If you back up the whole system, you get to restore the whole system.
It's not an automated one-click "restore and go" solution, since at a
minimum you have to manually re-create your partitioning/ LVM/encryption
structure, create your filesystems, set up your mount point directories,
and re-install the boot loader, but it will do the heavy work.

It's what I've been using, and I always try a bare-metal restore on a
new system to see what issues I run into.

--
Bob Nichols     "NOSPAM" is really part of my email address.
                 Do NOT delete it.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Examples for whole-system backup

Matt Taggart
In reply to this post by rhkramer
[hidden email] writes:
> On Thursday, April 09, 2015 02:41:44 AM you wrote:
> > I'm sure that you could devise some scheme to do a full metal restore
> > with rdiff-backup, but in my opinion, it's not the tool for the job.

> Not the OP, but what do you recommend (in the LInux world, please, as that is
> what I use...)?

My knowledge is possibly old, but it's still working for me. Maybe others
can point out if there are newer solutions to this problem.

A really long time ago there was a project that did this

http://systemimager.org/

It could boot via CD, USB, or network and then it could repartition the
(possibly new) disks and copy the data over from a server. It was also
smart about settings can you could create a master image and deploy it
across a bunch of systems and it could automatically adjust hostname,
network settings, etc. It was nice, but doing all that required distro
specific code and it's no longer maintained. I would be interested to learn
of anything modern and similar.

There are also ways of doing automated installs, in Debian based distros
it's "debian-installer preseeding" and in Fedora based it's "kickstart".
Those can automate the install process and give you a repeatable fresh
install (which isn't the same as what you asked for). Then you could
restore data to the system from backups or deploy it via a configuration
management tools like puppet/chef/etc. That is still not what you are
asking for as it wouldn't capture edits you had made to the system that
weren't backed up or weren't in the config management tool. But sometimes
that's a good thing, it allows you to leave cruft behind.

One way of getting what you are asking for is taking raw images of the
partitions or whole disk. You can boot the system via CD/USB/Network (so
that you aren't changing the drives you are trying to copy) and then use
ddrescue to make an image of the drive and then copy it to another drive or
over the network (with netcat or something). There might be automated tools
for doing this. I know there are proprietary tools for doing it but I
haven't used them.

--
Matt Taggart
[hidden email]



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Examples for whole-system backup

Ricky Huang
In reply to this post by Chris Wilson

On Apr 9, 2015, at 6:32 AM, Chris Wilson <[hidden email]> wrote:

Hi all,

I have done bare metal restores with duplicity (same author, different tech). My previous job used this as the main emergency recovery mechanism. I believe that rdiff-backup would work just as well. You'll need to sort the comms between client and server (e.g. Ssh keys and firewalls), handle databases and other files being modified during the backup, and test it of course!

Thank you, I will look into duplicity.


Taking LVM snapshots is a very different approach, relatively fast and low impact but requires a lot of storage and bandwidth (no diffs). In the case of bare metal, where you normally only need one backup and often keep large disks on the same site (depending on your threat model) it could also be a good solution for you.

Cheers, Chris.

Sent from my iPhone

On 9 Apr 2015, at 12:41, [hidden email] wrote:

Not the OP, but what do you recommend (in the LInux world, please, as that is
what I use...)?

On Thursday, April 09, 2015 02:41:44 AM you wrote:
I'm sure that you could devise some scheme to do a full metal restore
with rdiff-backup, but in my opinion, it's not the tool for the job.

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Examples for whole-system backup

Leland Best
In reply to this post by Matt Taggart
All,

Just thought I'd throw my $0.02 in ...

On Thu, 2015-04-09 at 11:29 -0700, Matt Taggart wrote:

> [hidden email] writes:
> > On Thursday, April 09, 2015 02:41:44 AM you wrote:
> > > I'm sure that you could devise some scheme to do a full metal restore
> > > with rdiff-backup, but in my opinion, it's not the tool for the job.
>
> > Not the OP, but what do you recommend (in the LInux world, please, as that is
> > what I use...)?
>
> My knowledge is possibly old, but it's still working for me. Maybe others
> can point out if there are newer solutions to this problem.

Although not "one-click" rdiff-backup has worked well for me for Linux
bare-metal restores (both test cases and real dead machine, disk,
whatever, cases).  I create a rdiff-backup backup of each partition or
major data area (typically root, boot, home, and public which may or may
not be separate partitions).  I do copy the boot sector via 'dd' but
rarely use it.  To restore to a new machine/drive I boot it with a
Debian Live DVD (or USB stick or whatever), restore root and boot,
reboot with a GRUB rescue CD and boot the restored root system in single
user as 'root', restore everything else, and finally install GRUB in the
MBR.  One tip though.  I backup root and boot with
--preserve-numerical-ids because the user/group mappings are invariably
different on the Live DVD.

Windows is a whole different can of worms, sadly.

[...]
> One way of getting what you are asking for is taking raw images of the
> partitions or whole disk.
[...]

This is what I have to do with Windows machines/partitions.  I've tried
using rdiff-backup to back up Windows partitions but on restore the
permissions/ACLs are wrong.  (I posted about this a long time ago but,
obviously, the issue was never resolved).  So instead, I image the
partitions (I use ntfs-clone from the ntfs-3g package for NTFS, and
partimage for FAT, but whatever ...) then backup the _images_ with
rdiff-backup.  Typically the rdiff-backup increments are about 10% of
the size of the space _in_use_ on the partition.  Have only done
restores on machines that dual boot with GRUB though.  No idea if it'd
work with a "pure" Windows machine.

So, yes, it takes some work but once I got the procedure down it's not
bad.  The typing time is nothing compared to the actual restore time.
And the space savings using rdiff-backup vs. other tools makes it well
worth it to me.  Of course, as always, YMMV.

Cheers


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Examples for whole-system backup

Ricky Huang

On Apr 9, 2015, at 4:36 PM, Leland Best <[hidden email]> wrote:

All,

Just thought I'd throw my $0.02 in …

This is exactly what I was looking for.  Your two-cents have helped immensely!


On Thu, 2015-04-09 at 11:29 -0700, Matt Taggart wrote:
[hidden email] writes:
On Thursday, April 09, 2015 02:41:44 AM you wrote:
I'm sure that you could devise some scheme to do a full metal restore
with rdiff-backup, but in my opinion, it's not the tool for the job.

Not the OP, but what do you recommend (in the LInux world, please, as that is
what I use...)?

My knowledge is possibly old, but it's still working for me. Maybe others
can point out if there are newer solutions to this problem.

Although not "one-click" rdiff-backup has worked well for me for Linux
bare-metal restores (both test cases and real dead machine, disk,
whatever, cases).  I create a rdiff-backup backup of each partition or
major data area (typically root, boot, home, and public which may or may
not be separate partitions).  I do copy the boot sector via 'dd' but
rarely use it.  To restore to a new machine/drive I boot it with a
Debian Live DVD (or USB stick or whatever), restore root and boot,
reboot with a GRUB rescue CD and boot the restored root system in single
user as 'root', restore everything else, and finally install GRUB in the
MBR.  One tip though.  I backup root and boot with
--preserve-numerical-ids because the user/group mappings are invariably
different on the Live DVD.

When you use terms “restore” above, you meant “rdiff-backup -r” command, correct?  So that also meant your Live DVD / USB contains the rdiff-backup tool?


Windows is a whole different can of worms, sadly.

I am dealing FreeBSD system, so I can leave the can of Windows worms to someone else.



[...]
One way of getting what you are asking for is taking raw images of the
partitions or whole disk.
[...]

This is what I have to do with Windows machines/partitions.  I've tried
using rdiff-backup to back up Windows partitions but on restore the
permissions/ACLs are wrong.  (I posted about this a long time ago but,
obviously, the issue was never resolved).  So instead, I image the
partitions (I use ntfs-clone from the ntfs-3g package for NTFS, and
partimage for FAT, but whatever ...) then backup the _images_ with
rdiff-backup.  Typically the rdiff-backup increments are about 10% of
the size of the space _in_use_ on the partition.  Have only done
restores on machines that dual boot with GRUB though.  No idea if it'd
work with a "pure" Windows machine.

So, yes, it takes some work but once I got the procedure down it's not
bad.  The typing time is nothing compared to the actual restore time.
And the space savings using rdiff-backup vs. other tools makes it well
worth it to me.  Of course, as always, YMMV.

Cheers


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Examples for whole-system backup

Mike Fleetwood
In reply to this post by Ricky Huang
On 8 April 2015 at 22:33, Ricky Huang <[hidden email]> wrote:

> Hello all,
>
> My friend recommend rdiff-backup so I am currently researching it for my
> backup solution.  I am brand new to the system so please forgive me for any
> dumb questions.
>
> What I am looking for is a FreeBSD backup solution that will duplicate my
> multiple mission-critical servers to a remote system in case something goes
> bad.  In case something bad does happen, I’d like to invoke a command and be
> able to restore the entire system back, not having to do a rebuild then copy
> files back as that will have a longer downtime.
>
> Is rdiff-backup the correct solution for my use case?  If so, does anyone
> have any information on whole-system level backup examples / documentations?
> I have read documentations on the site already, but they mostly talk about
> directories.  I am sure there are caveats when one’s backing up complete
> system.
>
>
> Thanks in advance.

Hello Ricky,

Here is a blog describing how someone used rdiff-backup as part of
their personal bare metal restore.
http://web.archive.org/web/20100217070956/http://www.nowhere.dk/articles/backup-and-bare-metal-restore-with-rdiff-backup

I do something similar myself for my Linux desktop.  To recover it, I
basically do:
1) Boot bare metal using recovery media
2) Recreate partitions and empty file systems
3) Restore using rdiff-backup
4) Reinstall boot loader
(Details documented, printed out and tested.  Recovery should not be a
memory test).
This doesn't really sound like the single restore command you are after.

Thanks,
Mike

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Examples for whole-system backup

Frank Crawford
In reply to this post by Ricky Huang
Ricky,

If you are looking at duplicity, you may also want to look at "Relax-and-recover" or rear (http://relax-and-recover.org/).  It creates bare-metal recover systems and includes links to various backup utilities, including duplicity.  Unfortunately, it doesn't yet include rdiff-backup, although that should be possible to add.

I think rear looks like an excellent bare-metal environment and may be very much what you want.

Regards
Frank

On Thu, 2015-04-09 at 15:05 -0700, Ricky Huang wrote:

On Apr 9, 2015, at 6:32 AM, Chris Wilson <[hidden email]> wrote:

Hi all,

I have done bare metal restores with duplicity (same author, different tech). My previous job used this as the main emergency recovery mechanism. I believe that rdiff-backup would work just as well. You'll need to sort the comms between client and server (e.g. Ssh keys and firewalls), handle databases and other files being modified during the backup, and test it of course!



Thank you, I will look into duplicity.



Taking LVM snapshots is a very different approach, relatively fast and low impact but requires a lot of storage and bandwidth (no diffs). In the case of bare metal, where you normally only need one backup and often keep large disks on the same site (depending on your threat model) it could also be a good solution for you.

Cheers, Chris.

Sent from my iPhone

On 9 Apr 2015, at 12:41, [hidden email] wrote:

Not the OP, but what do you recommend (in the LInux world, please, as that is
what I use...)?

On Thursday, April 09, 2015 02:41:44 AM you wrote:
I'm sure that you could devise some scheme to do a full metal restore
with rdiff-backup, but in my opinion, it's not the tool for the job.

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Examples for whole-system backup

Leland Best
In reply to this post by Ricky Huang
Think I accidentally sent this directly to Ricky directly.  Hopefully,
this will get to the list instead.  My apologies if I wind up double
posting.

Ricky,

On Thu, 2015-04-09 at 17:50 -0700, Ricky Huang wrote:

>
> > On Apr 9, 2015, at 4:36 PM, Leland Best <[hidden email]> wrote:
> >
> > All,
> >
> > Just thought I'd throw my $0.02 in …
>
>
> This is exactly what I was looking for.  Your two-cents have helped
> immensely!

Glad to be of help although you should, of course, carefully evaluate
whether it meets your needs.

[...]

> > To restore to a new machine/drive I boot it with a
> > Debian Live DVD (or USB stick or whatever), restore root and boot,
> > reboot with a GRUB rescue CD and boot the restored root system in
> > single
> > user as 'root', restore everything else, and finally install GRUB in
> > the
> > MBR.  One tip though.  I backup root and boot with
> > --preserve-numerical-ids because the user/group mappings are
> > invariably
> > different on the Live DVD.
> >
>
>
> When you use terms “restore” above, you meant “rdiff-backup -r”
> command, correct?

Yes.  Typically I do something like:

live# rdiff-backup -v 5 -r now --force --preserve-numerical-ids
root@backupserver::/backup/my_net/my_host/root /new_root_mountpoint 2>&1
| tee /path/to/root_restore.log

all on one line obviously.  The '--force' option is necessary because
rdiff-backup gets (justifiably) nervous about restoring over an existing
directory.  In this case though this is exactly what we want.

>   So that also meant your Live DVD / USB contains the rdiff-backup
> tool?
[...]

Actually no, although there may be other Linux live/rescue CD/DVDs out
there that do.  But the Debian Live DVD does contain 'synaptic' so it's
trivial to install rdiff-backup once you've booted.  Obviously, for the
Live DVD this will _not_ be persistent (i.e. you have to re-install
rdiff-backup every time you boot the Live DVD).  That's why I've moved
to USB sticks.  I can even upgrade a USB stick to 'testing' or whatever.
Of course, if your hardware won't boot from USB then you're sort-of
stuck. :(

Hmm.  IIRC, once a long time ago, I did the whole 'boot the live DVD,
restore root and boot' thing but instead of rebooting with a GRUB rescue
CD, I then did a 'chroot' to the new (restored) root and restored
everything else.  I _think_ I even installed GRUB on the new root
drive's MBR from there.  On reboot, presto(!), I had a fully restored
bootable system.  But as I said, that was a long time ago and I really
don't remember the details or whether it really worked like that.  The
field of Experimental System Administration!  Fun if you have the time.
Otherwise, as many others have said, though it absolutely _can_ be used
to restore to bare metal, rdiff-backup may not be your best choice.

Good luck whatever path you take!

Cheers


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki