Hardware Issues

Setting up Backup for Linux box (esp with DVD)
Initial Post of question to message board (hardhats) I need a reliable backup solution for my linux server. Here are methods I have explored, unsuccessfully so far. Let me first mention that our office also has a new Windows Server running our billing package. My initial thoughts were to copy my linux data to the Windows server, and then let their backup process save my data.

1. Install Services for Linux (SFU) put out by Windows onto our Windows server. I then created a NFS share. I can view the directory contents of the Windows share from my linux box, but for the life of me I can't get write permission. You have to run a Windows namemapper to map linux users to Windows users, and then give that user write permission. Sounds easy. I can't get it to work. The permission system on NTFS is a bit complex to me.

2. Run a Samba server on my linux box. When I do this, I can see the drive from a Windows laptop, and I might even be able to write to it (I can't remember). But when I try to find the file when browsing from the Windows Server, I can't find it. Not sure if I have to join Windows "domain" to work. This descends into a murky realm for me of ActiveDirectories. Also, I'm not sure if my iptables is filtering out the samba ports etc.

3. Run a Samba client on my linux box. I haven't tried this, but again I worry that I would have to discover how to join the Windows "domain" and be validated.

4. Get scp to work on my Windows server, and scp the needed data up. This would probably work, but I would need to use and 'expect' script to supply the password etc if it is going to be automated. This seems a bit klunky, and will probably be my last resort.

5. Install a DVD writer on my linux box. This is what I am currently trying to do. I successfully installed a new USB 2.0 IO card, and have now plugged the drive in. I'm lost as to what to do now. I have read multiple web sites, and I'm overwhelmed. Right off the bat the install demos talk about possible kernel upgrades etc. It seems that cdrtools is one route to use. I would have to purchase a commercial license. The money is not such an issue, but I would want to be sure it was going to work.

6. Purchase and install tape backup system for the linux box. This may well be a great options, but I have no experience with this.

If anyone has any helpful thoughts or pointers (esp regarding option 5 above), I would be very grateful.

Kevin

Samba is not my expertise, so I won't presume to give you debugging advice. I will say that I have been able to get Samba and Windows working quite well together to share files and printers on multiple occasions. When I have had problems using Linux as a server, it is usually because on the Linux side, I have not created the users with smbpasswd program (just because they exist in /etc/passwd doesn't mean that they exist for Samba - although I think there is now an option to propagate changes back and forth). You can also webmin and swat to manage Samba via a browser.

Since Windows file attributes are not the same as Linux, I only recommend using Windows to backup your Linux files by first creating a tar archive on Linux and moving the tar archive to Linux. By the way, to move a single file from Linux to Windows, you don't need Samba, just use WinSCP3 on Windows.

Getting a USB DVD burner to work on any contemporary Linux should be trivial. When I got it working, I just powered down the PC, plugged in the DVD-ROM drive (a Sony drive that Chris Richardson put inside a box that provided power and interface electronics), powered up the DVD and powered up the PC. K3b (my burning software) recognized the DVD immediately. That PC runs the Debian Sarge installed via a Knoppix 3.6 Linux live CD. What Linux distribution and release are you running and what kernel? If you connect a USB flash drive to the PC, is it recognized? [In theory, you need not power it down, but just to simplify debugging, I like to start from a cold power up.]

I recommend creating a tar archive for backup. Remember to backup your journal files too.

... -- Bhaskar

>use. I would have to purchase a commercial license. >The money is not such an issue, but I would want to be >sure it was going to work. >

As Bhaskar mentioned k3b is an excellent solution for burning CD and DVD's. On this 64 bit Debian/Sarge laptop I did


 * 1) apt-get install k3b dvd+rw-tools cdrdao

You have the option of creating a seperate group (cdrom I think) for burning - or you can assign the permissions to the various tools during the setup when you first call the application.

I am not aware of any need for a commercial license for any of the above. I would also expect the USB device to be well supported with most of the recent stock kernels. Just as an aside, with 2.6.x you don't need to use SCSI emulation for IDE CDROM drives, which is nice.

>6. Purchase and install tape backup system for the >linux box. This may well be a great options, but I >have no experience with this. > > My first experience with tape backup with GNU/Debian was with an old Colorado floppy based mini-tape backup system with the ftape drivers - it was slow, but worked perfectly (restore and backup).

If you add a SCSI tape backup you will have a huge array of software solutions - from something as simple as 'tos' (tape oriented backups-perl driven) to enterprise tools like Amanda. I have used various HP SCSI backup systems in the past and don't recall any problems.

And as mentioned, if you have questions about Samba and Windows John Terpstra's series of books and essays are excellent. BTW, he is a first rate speaker if you ever have the chance of attending one of his lectures or seminars, you will really get a huge boost up with regards to networking in heterogenous environments.

I hope this is of some help. --Ismet

Whoa, quite a bit here...

1. Windows bites, do it our way or the highway. 2. Samba - Get yourself the book by John Terpstra - Samba 3 By Example - Practical Exercises to Successful Deployment. and/or Download my smb.conf tutorial file and have a go at joining that domain.... http://oswizards.com/samba-instruct.conf If you would like I can give you access to one of my Linux Admin courses online. You can study the Samba material to see if it clarifies things a little. Samba is a huge subject, especially if you are going to deploy with 500+ uses. I highly recommend the book.

I don't think that will solve your problems though, you need to backup the Linux server. 3. See number 2 above. 4. You can share your private key/identity with the client and you will not have to supply a password when you execute your command. You could do this with linux boxes easy, windows would be a hassle. In Linux I could issue the command, scp -r /var/data/* jerome@someotherhost.org:mydata/

5. Oh yeah... you could install a DVD writer, CDR or an additional HDD or both.

Here are a couple tools for you. Mondo Resuce http://www.microwerks.net/~hugo/about/about.html

I use another command line program from a cron job called rdiff-backup - http://www.nongnu.org/rdiff-backup/ It can take a snapshot of your filesystem whenever you tell it to, it also keeps diffs on all files that have been changed since the last backup. You can actually restore a file from a specific date and time. I snapshot at 3:10am onto a spare drive in the machine, I also snapshot my databases every hour. You can use Mondo to create an entire system backup onto CD, HDD, NFS or DVD - http://www.ccp14.ac.uk/ccp14admin/linux-server/mondorescue/dvd_mondo.html

Keep yourself clear of Windows and your choices will be plentiful and your difficulty will be diminished.

Setup another Linux Samba Server for the Windows clients/users to store their data on, then back that one up with the tools above.

Have Fun!!!... and test your backups regularly to make sure you are indeed backing up your data. --Mark Street

I just recalled another tool/method that you might want to look at - namely rsync

Here is a nice summary of one method employing it with automated scripts:

http://www.tux.org/~tbr/rsync/rsynchowto.html

Rsync is a fantastic tool - especially when combined with ssh.

Ismet / Ismet B. Kursunoglu, MD, FCCP

> 1. Install Services for Linux (SFU) put out by Windows > onto our Windows server. <...snipped...>

Linux + NTFS is experimental at best still. Do not know much about the SFU deal from M$, but would not put too much trust in it for my obvious prejudices. ;-)

> 2. Run a Samba server on my linux box. <...snipped...>

Samba on the Linux box will allow the M$ boxes to see and mount it as a share potentially. Possible idea, not the greatest though.

> 3. Run a Samba client on my linux box. <...snipped...>

If you are using RH (or one of the Fedora or CentOS forks), there are some built in RH specific tools for Samba (redhat-config-samba for RH). Not too hard using them. Otherwise, there are a few HOW-To's on the Samba subject.

> 4. Get scp to work on my Windows server, and scp the > needed data up. <...snipped...>

This is actually one of the easier solutions you have presented so far that could run unattended potentially.

scp is part of the SSH package typically (along with sftp). This would be one of your easiest solutions. Tarball (with gz or bz2 compression depending on the amount of data you wish to back up), scp the tarball to remote backup location directory. You can make a simple script and cron job to do this on a regular basis.

> 5. Install a DVD writer on my linux box. <...snipped...> >

There is k3b which also noted by Bhaskar is freeware. Nice tool overall once you get the DVD up and running, can handle CD and DVD burning.

At issue, you will be starting a large DVD collection potentially in time.

> 6. Purchase and install tape backup system for the > linux box. This may well be a great options, but I > have no experience with this. >

Old school, but possible via Amanda,

And finally...another Linux based solution presented is Bacula. I personally have not played with it (yet), but one of my guys is using it for one of our clients on a regular basis. This is a better or more modern idea than Amanda, allows for secure file transfer to a remote machine, and such. http://www.bacula.org/ for their web site, and yes...it is cross platformed as well.

--- Crawford, The Linux ETC Company

Here is something else you may want to consider: Unison - a file sync tool for Unix and Windows

http://www.cis.upenn.edu/~bcpierce/unison/

--Jim Drash

For clarity, Unison, rsync, or any technique that copies files or parts of files when the database is in use will not work very well, and the copy will be corrupt. This is because GT.M is constantly updating the database when it is in use (unless you freeze updates).

Let's separate the two issues under consideration. One is to get a copy of the database and journal files (backup), and the other is moving backups to offline storage (archival).

Backup when there is no active GT.M process is trivial - just copy the database files (unison, rsync and other techniques will work). There are (at least) two ways to backup when GT.M processes are active. One is to use mupip backup and the other involves using a mirrored disk. (Promotional moment: attend my workshop in the Boston area on April 6 for hands-on training on backup - it's free but you have to RSVP!)

There are many options for archival, as discussed in this thread.

-- Bhaskar

>Thanks for your suggestions below. The one issue > about scp'ing the data, though, is that I would have > to have an "scp server" running on the windows box. I > don't know how to set that up. I have downloaded a > program for windows, pscp that can pull the data from > the linux box when run on the windows box. It would > be nice to be able to push it though. > --Kevin

Cygwin will give you the sshd part-

http://pigtail.net/LRP/printsrv/cygwin-sshd.html

and this might be a good solution to limit access to scp for your scripts -

http://www.devguy.com/fp/scp/

--Ismet

> My impression is that K3B is very smart, much smarter than I am, thank goodness! But, you generally need to start K3b as root if you want to consistently have joy in Mudville! > -- Nancy Anthracite

So true, it takes some tweaking to get permissions just right for a mere mortal user to record CD's. Let us not forget that the GUI tools are merely front ends for the command line tools from cd/DVD record tools.

http://fy.chalmers.se/~appro/linux/DVD+RW/ http://cdrecord.berlios.de/old/private/cdrecord.html

A system administrator has much more flexibility with command line tools when it comes to automating tasks.

> On Tuesday 22 March 2005 09:13 am, Bhaskar, KS wrote: > Most likely, a DVD connected via a USB port will show up as a SCSI > device, with a name like /dev/sda or /dev/sdb. However, to burn a > CD/DVD, you don't mount it (there is no file system to mount on a blank disk). > --Bhaskar

It depends on the linux distribution but most newer distributions recognize USB CD drives as SCSI devices, they are labeled as /dev/scd0

I worked with a Latitude laptop this weekend and installed Debian Sarge and Fedora Core 3, both distros recognized the attached USB CD/RW as /dev/scd0.

Your USB mass storage devices, jump drives, cameras, etc. will show up as /dev/sda, /dev/sdb, etc.

> Generally, if you power up the USB drive first, then boot the PC, k3b > will find and recognize the drive. Also, as root, you can execute > "cdrecord --scanbus" to find the "SCSI" drive number for your CD drive.

Most newer Linux distributions automatically recognize USB devices and make them available on boot. Again, it depends on your Linux distro as to how much work will be involved.

> Mark and/or Crawford may want to correct me here, or expand on the > explanation, because my understanding of how USB is mapped to SCSI, and > how CD/DVD-ROM drives are handled on Linunx is shallow.

There is some difference how CD/DVD devices are handled in the version 2.4 Linux kernel and 2.6 version, especially USB devices. 2.4 kernel has a SCSI emulation layer that was rather clunky, ide_scsi which allowed IDE devices to emulate SCSI devices, Linus didn't consider the interface elegant so it was reworked in 2.6. There are some hard feelings on both sides between Linux and the author of cdrecord. Too technical for my ears. -- Mark Street, RHCE

Tape Backup *may* be considered old school, but there's a reason why it's still popular.

I recommend to my clients that they spend the extra dollars up front on a real server, because then the majors will install a SCSI card and a DAT tape drive for doing backups.

SCSI, because every DAT drive is going to plug into it, and there are ZERO emulation issue.

DAT rather than Travan, because of the media costs.

A DAT tape is 5 bucks, a Travan is 50. With DAT tapes, you can have NERDVANA:

Two full weeks of backups ( Mon1 - Thu1 & Mon2 - Thu2 ) A MONTH of Friday backups ( FriA - FriD ) and a perpetual archive of month ends

by just pulling the tape on the first, and replacing it with a blank...

The Auditors will LOVE you for that...

And even though I'm an open source advocate, I recommend a commercial product for doing the automated nightly backups. When a harddrive dies, *I* don't want to be trying to figure out the easiest way to do a full metal restore onto a differently sized harddisk.

24 hour support on you backup/restore product is a very good thing.

Mike

mikelieman@gmail.com