Archive | January, 2012

Linux shell scripting

11 Jan

I am spending more of my time working with Linux so I’m learning some of the things you can do with it

We have a couple of NAS boxes which need to be backed up in a kind of Time Machine fashion. This is relatively easy to do with rsync (which I believe is what Apple’s Time Machine uses). I found this web site http://blog.interlinked.org/tutorials/rsync_time_machine.html which helped me get started and this is what I ended up with. You just have to run this from cron and you get new backups each day which appear to contain the full contents of the source machine but actually only contain new files and a set of symlinks to files which are in a previous backup.

#!/bin/sh

#log start time
date > /tmp/timemachine.log

#folder will be called yyyy-mm-dd
date=`date "+%Y-%m-%d"`

#backup - link it to the most recent folder
#copy from nas02
#and to a dated folder on nas01
#redirect all output to a log file
rsync -a --link-dest=/destination/current sourcemachine.fqdn.tld::source_share /destination/$date >> /tmp/timemachine.log

#remove the "current" symlink (it's not current any longer)
rm -f /destination/current

#and make a symlink to the backup we just made
ln -s /destination/$date /destination/current

#all done so log end time
date >> /tmp/timemachine.log

#and send an email
echo "Backup complete; log attached" | mutt -s "Backup complete for NAS" -a /tmp/timemachine.log -- myemail@email.tld 
Advertisements

Linux shell scripting – 2

11 Jan

My previous post was about using rsync to mimic Apple’s Time Machine; I actually set that up some time ago but just didn’t get round to writing about it.

It’s now been running successfully for over a month and I wanted to find out how much space does it save when you just create hard links to files rather than backing them up. You can’t just use the du command because it counts the files as if they were actually there but find is a command which can help.

If you type:

find . -links 1 -type f

Then you will get a list of files in the current directory which have just one link to them (ie they’re not linked from any other directory so they are “today’s” backup)

that list is very nice but tells me nothing about size. find can take a -exec option which then runs a command for each file

find . -links 1 -type f -exec ls -l {} \;

lists the files in long format so you can see their size but I don’t really want to try and add up all those sizes!

awk will allow me to do some processing on these results:

find . -links 1 -type f -exec ls -l {} \; | awk '{print $5}' 

Here, we’re piping the output of the ls command into awk and getting it to print the fifth column (the size)

Still not ideal; I really want it to add up those numbers for me!

find . -links 1 -type f -exec ls -l {} \; | awk '{ print $5 }' | awk '{total = total +$1}END{print total}'

This takes the long list of numbers and pipes it to another awk command which just adds each number to a variable called “total”. Once it’s finished, it prints that variable.

So, if you run that final command at the top of a directory tree containing new files and hard links to previous files you’ll ge a number – that’s the space taken up by files actually in that directory. Run a du-s for the same directory and you’ll get the total space Linux thinks is in use (ie including all the files that are really just hard links). The difference between the 2 is the space saved by doing this backup.