Author Archive

Linux Backups for Servers and Desktops

Posted by on Thursday, 18 June, 2009

Everyone wants to back up right? Well you will once you have totally lost the last years worth of work on a website and somebody breaks things severely!

Heres a quick and nasty backup HOWTO.

Database Dumps

mysqldump -u root -p mydatabase > mydatabase.sql

This dumps a database into a file, you can modify this to dump it offsite using ssh with this command.

su postgres -c "pg_dumpall" > pgdatabase.psql

If you use postgres you can change this to something like this

mysqldump -u root -p mydatabase > mydatabase.sql | ssh username@backup.comain.com “dd of=mydatabase.sql”

If you want to dump the entire database you can use

mysqldump -A -u root -p >entiredatabase.sql

This may take some time. To put this in a shell script and dump multiple copies and keep track of things you can use something similar to this

date=`date +%m-%h-%Y`

mysqldump -A -u root -p >${date}-fulldatabase.sql

This will expand to dump it into something like

06-Jun-2009-fulldatabase.sql

File Backup

FTP

To run a regular interactive FTP session:

lftp -u 'username,password' backup.yourdomain.com

To backup one or more files:

lftp -u 'username,password' backup.yourdomain.com -e "set ftp:ssl-protect-data true; mput /local/dir/files* /remotedir; exit"

You need to set ftp:ssl-protect-data else you will not be able to store the file. If you want to make this a default option, add it to the lftp.conf file. e.g. :

grep -qai "set ftp:ssl-protect-data true" /etc/lftp.conf || echo "set ftp:ssl-protect-data true" >> /etc/lftp.conf

To restore a file from the FTP server to your Machine:

lftp -u 'username,password' backup.yourdomain.com -e "set ftp:ssl-protect-data true;mget /remotedir/files* -O /localdir; exit" .

The -O option is not required it you wish to store to the current local directory.

To mirror a whole directory to the FTP server:

lftp -u 'username,password' backup.yourdomain.com -e "set ftp:ssl-protect-data true;mirror --reverse /local/dir/name remotedirname; exit" .

--reverse means that the ‘mirroring’ is going in the reverse direction than ‘normal’. i.e. from your server to the backup server. If you run man lftp there are a few other options to choose from. e.g. --delete to delete files on the backup server that do not exist locally. Or --continue to continue a mirror job. Or --exclude files to exclude certain files from the transfer.

To restore a whole directory from the FTP server to your machine:

lftp -u 'username,password' backup.yourdomain.com -e "set ftp:ssl-protect-data true;mirror remotedirname /local/dir/name;exit"

To create a nightly cronjob that uploads a directory to the backup FTP server, create a /etc/crond.daily/ftpbackup file like this:


#!/bin/bash
lftp -u 'username,password' backup.yourdomain.com -e "set ftp:ssl-protect-data true;mirror --reverse /local/dir/name remotedirname;exit" > /dev/null

Run

chmod +x /etc/cron.daily/ftpbackup .

Then check the files have been mirrored as you expect the next day.

Rsync

Rsync is a better option in some ways as it checks the MD5 of files and updates them if they are out of date, rather than re-copying the entire lot. Short but easy shell script to copy things over

#!/bin/bash
EXCLUDE=” –exclude *.tmp \
–exclude *.temp”
USER=username
HOST=backup.domain.com
BACKUPPATH=/backups

rsync –archive -vv –rsh=ssh $EXCLUDE $USER@$HOST:/etc/ $BACKUPPATH/$HOST/etc

Rdiff-backup

This is better again than rsync as it does versioning control and only backs up the difference in files.

To backup files

rdiff-backup /some/local-dir hostname.net::/whatever/remote-dir

To restore

rdiff-backup --restore-as-of now host.net::/remote-dir/file local-dir/file
rdiff-backup -r now host.net::/remote-dir/file local-dir/file

The -r command is the same as –restore-as-of

The main advantage of rdiff-backup is that it keeps version history. This command restores host.net::/remote-dir/file as it was 10 days ago into a new location /tmp/file .

rdiff-backup -r 10D host.net::/remote-dir/file /tmp/file

Other acceptable time strings include 5m4s (5 minutes and 4 seconds) and 2002-03-05 (March 5th, 2002). For more information, see the TIME FORMATS section of the manual page.

More examples can be found at http://www.nongnu.org/rdiff-backup/examples.html

This tutorial was compiled from several others, and props go out to http://rimuhosting.com and http://www.howtoforge.com

Category: Tutorials   , , , , , , , , , ,
Comments (0)

Linux Command Line Cheat Sheet – A list of handy commands

Posted by on Tuesday, 16 June, 2009

This is a linux command line reference for common operations.
Examples marked with • are valid/safe to paste without modification into a terminal, so
you may want to keep a terminal window open while reading this so you can cut & paste.
All these commands have been tested both on Fedora and Ubuntu.

Command Description
• apropos whatis Show commands pertinent to string. See also threadsafe
• man -t man | ps2pdf – > man.pdf make a pdf of a manual page
which command Show full path name of command
time command See how long a command takes
• time cat Start stopwatch. Ctrl-d to stop. See also sw
• nice info Run a low priority command (The “info” reader in this case)
• renice 19 -p $$ Make shell (script) low priority. Use for non interactive tasks
dir navigation
• cd – Go to previous directory
• cd Go to $HOME directory
(cd dir && command) Go to dir, execute command and return to current dir
• pushd . Put current dir on stack so you can popd back to it
• alias l=’ls -l –color=auto’ quick dir listing
• ls -lrt List files by date. See also newest and find_mm_yyyy
• ls /usr/bin | pr -T9 -W$COLUMNS Print in 9 columns to width of terminal
find -name ‘*.[ch]’ | xargs grep -E ‘expr’ Search ‘expr’ in this dir and below. See also findrepo
find -type f -print0 | xargs -r0 grep -F ‘example’ Search all regular files for ‘example’ in this dir and below
find -maxdepth 1 -type f | xargs grep -F ‘example’ Search all regular files for ‘example’ in this dir
find -maxdepth 1 -type d | while read dir; do echo $dir; echo cmd2; done Process each item with multiple commands (in while loop)
• find -type f ! -perm -444 Find files not readable by all (useful for web site)
• find -type d ! -perm -111 Find dirs not accessible by all (useful for web site)
• locate -r ‘file[^/]*\.txt’ Search cached index for names. This re is like glob *file*.txt
• look reference Quickly search (sorted) dictionary for prefix
• grep –color reference /usr/share/dict/words Highlight occurances of regular expression in dictionary
archives and compression
gpg -c file Encrypt file
gpg file.gpg Decrypt file
tar -c dir/ | bzip2 > dir.tar.bz2 Make compressed archive of dir/
bzip2 -dc dir.tar.bz2 | tar -x Extract archive (use gzip instead of bzip2 for tar.gz files)
tar -c dir/ | gzip | gpg -c | ssh user@remote ‘dd of=dir.tar.gz.gpg’ Make encrypted archive of dir/ on remote machine
find dir/ -name ‘*.txt’ | tar -c –files-from=- | bzip2 > dir_txt.tar.bz2 Make archive of subset of dir/ and below
find dir/ -name ‘*.txt’ | xargs cp -a –target-directory=dir_txt/ –parents Make copy of subset of dir/ and below
( tar -c /dir/to/copy ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) copy/ dir to /where/to/ dir
( cd /dir/to/copy && tar -c . ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) contents of copy/ dir to /where/to/
( tar -c /dir/to/copy ) | ssh -C user@remote ‘cd /where/to/ && tar -x -p’ Copy (with permissions) copy/ dir to remote:/where/to/ dir
dd bs=1M if=/dev/sda | gzip | ssh user@remote ‘dd of=sda.gz’ Backup harddisk to remote machine
rsync (Network efficient file copier: Use the –dry-run option for testing)
rsync -P rsync://rsync.server.com/path/to/file file Only get diffs. Do multiple times for troublesome downloads
rsync –bwlimit=1000 fromfile tofile Locally copy with rate limit. It’s like nice for I/O
rsync -az -e ssh –delete ~/public_html/ remote.com:’~/public_html’ Mirror web site (using compression and encryption)
rsync -auz -e ssh remote:/dir/ . && rsync -auz -e ssh . remote:/dir/ Synchronize current directory with remote one
ssh (Secure SHell)
ssh $USER@$HOST command Run command on $HOST as $USER (default command=shell)
• ssh -f -Y $USER@$HOSTNAME xeyes Run GUI command on $HOSTNAME as $USER
scp -p -r $USER@$HOST: file dir/ Copy with permissions to $USER’s home directory on $HOST
ssh -g -L 8080:localhost:80 root@$HOST Forward connections to $HOSTNAME:8080 out to $HOST:80
ssh -R 1434:imap:143 root@$HOST Forward connections from $HOST:1434 in to imap:143
wget (multi purpose download tool)
• (cd dir/ && wget -nd -pHEKk http://www.pixelbeat.org/cmdline.html) Store local browsable version of a page to the current dir
wget -c http://www.example.com/large.file Continue downloading a partially downloaded file
wget -r -nd -np -l1 -A ‘*.jpg’ http://www.example.com/dir/ Download a set of files to the current directory
wget ftp://remote/file[1-9].iso/ FTP supports globbing directly
• wget -q -O- http://www.pixelbeat.org/timeline.html | grep ‘a href’ | head Process output directly
echo ‘wget url’ | at 01:00 Download url at 1AM to current dir
wget –limit-rate=20k url Do a low priority download (limit to 20KB/s in this case)
wget -nv –spider –force-html -i bookmarks.html Check links in a file
wget –mirror http://www.example.com/ Efficiently update a local copy of a site (handy from cron)
networking (Note ifconfig, route, mii-tool, nslookup commands are obsolete)
ethtool eth0 Show status of ethernet interface eth0
ethtool –change eth0 autoneg off speed 100 duplex full Manually set ethernet interface speed
iwconfig eth1 Show status of wireless interface eth1
iwconfig eth1 rate 1Mb/s fixed Manually set wireless interface speed
• iwlist scan List wireless networks in range
• ip link show List network interfaces
ip link set dev eth0 name wan Rename interface eth0 to wan
ip link set dev eth0 up Bring interface eth0 up (or down)
• ip addr show List addresses for interfaces
ip addr add 1.2.3.4/24 brd + dev eth0 Add (or del) ip and mask (255.255.255.0)
• ip route show List routing table
ip route add default via 1.2.3.254 Set default gateway to 1.2.3.254
• tc qdisc add dev lo root handle 1:0 netem delay 20msec Add 20ms latency to loopback device (for testing)
• tc qdisc del dev lo root Remove latency added above
• host pixelbeat.org Lookup DNS ip address for name or vice versa
• hostname -i Lookup local ip address (equivalent to host `hostname`)
• whois pixelbeat.org Lookup whois info for hostname or ip address
• netstat -tupl List internet services on a system
• netstat -tup List active connections to/from system
windows networking (Note samba is the package that provides all this windows specific networking support)
• smbtree Find windows machines. See also findsmb
nmblookup -A 1.2.3.4 Find the windows (netbios) name associated with ip address
smbclient -L windows_box List shares on windows machine or samba server
mount -t smbfs -o fmask=666,guest //windows_box/share /mnt/share Mount a windows share
echo ‘message’ | smbclient -M windows_box Send popup to windows machine (off by default in XP sp2)
text manipulation (Note sed uses stdin and stdout. Newer versions support inplace editing with the -i option)
sed ‘s/string1/string2/g’ Replace string1 with string2
sed ‘s/\(.*\)1/\12/g’ Modify anystring1 to anystring2
sed ‘/ *#/d; /^ *$/d’ Remove comments and blank lines
sed ‘:a; /\\$/N; s/\\\n//; ta’ Concatenate lines with trailing \
sed ‘s/[ \t]*$//’ Remove trailing spaces from lines
sed ‘s/\([`”$\]\)/\\\1/g’ Escape shell metacharacters active within double quotes
• seq 10 | sed “s/^/ /; s/ *\(.\{7,\}\)/\1/” Right align numbers
sed -n ‘1000p;1000q‘ Print 1000th line
sed -n ‘10,20p;20q‘ Print lines 10 to 20
sed -n ‘s/.*<title>\(.*\)<\/title>.*/\1/ip;T;q‘ Extract title from HTML web page
sed -i 42d ~/.ssh/known_hosts Delete a particular line
sort -t. -k1,1n -k2,2n -k3,3n -k4,4n Sort IPV4 ip addresses
• echo ‘Test’ | tr ‘[:lower:]’ ‘[:upper:]’ Case conversion
• tr -dc ‘[:print:]’ < /dev/urandom Filter non printable characters
• history | wc -l Count lines
set operations (Note you can export LANG=C for speed. Also these assume no duplicate lines within a file)
sort file1 file2 | uniq Union of unsorted files
sort file1 file2 | uniq -d Intersection of unsorted files
sort file1 file1 file2 | uniq -u Difference of unsorted files
sort file1 file2 | uniq -u Symmetric Difference of unsorted files
join -a1 -a2 file1 file2 Union of sorted files
join file1 file2 Intersection of sorted files
join -v2 file1 file2 Difference of sorted files
join -v1 -v2 file1 file2 Symmetric Difference of sorted files
math
• echo ‘(1 + sqrt(5))/2’ | bc -l Quick math (Calculate φ). See also bc
• echo ‘pad=20; min=64; (100*10^6)/((pad+min)*8)’ | bc More complex (int) e.g. This shows max FastE packet rate
• echo ‘pad=20; min=64; print (100E6)/((pad+min)*8)’ | python Python handles scientific notation
• echo ‘pad=20; plot [64:1518] (100*10**6)/((pad+x)*8)’ | gnuplot -persist Plot FastE packet rate vs packet size
• echo ‘obase=16; ibase=10; 64206’ | bc Base conversion (decimal to hexadecimal)
• echo $((0x2dec)) Base conversion (hex to dec) ((shell arithmetic expansion))
• units -t ‘100m/9.69s’ ‘miles/hour’ Unit conversion (metric to imperial)
• units -t ‘500GB’ ‘GiB’ Unit conversion (SI to IEC prefixes)
• units -t ‘1 googol’ Definition lookup
• seq 100 | (tr ‘\n’ +; echo 0) | bc Add a column of numbers. See also add and funcpy
calendar
• cal -3 Display a calendar
• cal 9 1752 Display a calendar for a particular month year
• date -d fri What date is it this friday. See also day
• [ $(date -d “tomorrow” +%d) = “01” ] || exit exit a script unless it’s the last day of the month
• date –date=’25 Dec’ +%A What day does xmas fall on, this year
• date –date=’@2147483647′ Convert seconds since the epoch (1970-01-01 UTC) to date
• TZ=’:America/Los_Angeles’ date What time is it on West coast of US (use tzselect to find TZ)
echo “mail -s ‘get the train’ P@draigBrady.com < /dev/null” | at 17:45 Email reminder
• echo “DISPLAY=$DISPLAY xmessage cooker” | at “NOW + 30 minutes” Popup reminder
locales
• printf “%’d\n” 1234 Print number with thousands grouping appropriate to locale
• BLOCK_SIZE=\’1 ls -l get ls to do thousands grouping appropriate to locale
• echo “I live in `locale territory`” Extract info from locale database
• LANG=en_IE.utf8 locale int_prefix Lookup locale info for specific country. See also ccodes
• locale | cut -d= -f1 | xargs locale -kc | less List fields available in locale database
recode (Obsoletes iconv, dos2unix, unix2dos)
• recode -l | less Show available conversions (aliases on each line)
recode windows-1252.. file_to_change.txt Windows “ansi” to local charset (auto does CRLF conversion)
recode utf-8/CRLF.. file_to_change.txt Windows utf8 to local charset
recode iso-8859-15..utf8 file_to_change.txt Latin9 (western europe) to utf8
recode ../b64 < file.txt > file.b64 Base64 encode
recode /qp.. < file.txt > file.qp Quoted printable decode
recode ..HTML < file.txt > file.html Text to HTML
• recode -lf windows-1252 | grep euro Lookup table of characters
• echo -n 0x80 | recode latin-9/x1..dump Show what a code represents in latin-9 charmap
• echo -n 0x20AC | recode ucs-2/x2..latin-9/x Show latin-9 encoding
• echo -n 0x20AC | recode ucs-2/x2..utf-8/x Show utf-8 encoding
CDs
gzip < /dev/cdrom > cdrom.iso.gz Save copy of data cdrom
mkisofs -V LABEL -r dir | gzip > cdrom.iso.gz Create cdrom image from contents of dir
mount -o loop cdrom.iso /mnt/dir Mount the cdrom image at /mnt/dir (read only)
cdrecord -v dev=/dev/cdrom blank=fast Clear a CDRW
gzip -dc cdrom.iso.gz | cdrecord -v dev=/dev/cdrom – Burn cdrom image (use dev=ATAPI -scanbus to confirm dev)
cdparanoia -B Rip audio tracks from CD to wav files in current dir
cdrecord -v dev=/dev/cdrom -audio *.wav Make audio CD from all wavs in current dir (see also cdrdao)
oggenc –tracknum=’track’ track.cdda.wav -o ‘track.ogg’ Make ogg file from wav file
disk space (See also FSlint)
• ls -lSr Show files by size, biggest last
• du -s * | sort -k1,1rn | head Show top disk users in current dir. See also dutop
• df -h Show free space on mounted filesystems
• df -i Show free inodes on mounted filesystems
• fdisk -l Show disks partitions sizes and types (run as root)
• rpm -q -a –qf ‘%10{SIZE}\t%{NAME}\n’ | sort -k1,1n List all packages by installed size (Bytes) on rpm distros
• dpkg-query -W -f=’${Installed-Size;10}\t${Package}\n’ | sort -k1,1n List all packages by installed size (KBytes) on deb distros
• dd bs=1 seek=2TB if=/dev/null of=ext3.test Create a large test file (taking no space). See also truncate
• > file truncate data of file or create an empty file
monitoring/debugging
• tail -f /var/log/messages Monitor messages in a log file
• strace -c ls >/dev/null Summarise/profile system calls made by command
• strace -f -e open ls >/dev/null List system calls made by command
• ltrace -f -e getenv ls >/dev/null List library calls made by command
• lsof -p $$ List paths that process id has open
• lsof ~ List processes that have specified path open
• tcpdump not port 22 Show network traffic except ssh. See also tcpdump_not_me
• ps -e -o pid,args –forest List processes in a hierarchy
• ps -e -o pcpu,cpu,nice,state,cputime,args –sort pcpu | sed ‘/^ 0.0 /d’ List processes by % cpu usage
• ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS List processes by mem usage. See also ps_mem.py
• ps -C firefox-bin -L -o pid,tid,pcpu,state List all threads for a particular process
• ps -p 1,2 List info for particular process IDs
• last reboot Show system reboot history
• free -m Show amount of (remaining) RAM (-m displays in MB)
• watch -n.1 ‘cat /proc/interrupts’ Watch changeable data continuously
system information (see also sysinfo) (‘#’ means root access is required)
• uname -a Show kernel version and system architecture
• head -n1 /etc/issue Show name and version of distribution
• cat /proc/partitions Show all partitions registered on the system
• grep MemTotal /proc/meminfo Show RAM total seen by the system
• grep “model name” /proc/cpuinfo Show CPU(s) info
• lspci -tv Show PCI info
• lsusb -tv Show USB info
• mount | column -t List mounted filesystems on the system (and align output)
• grep -F capacity: /proc/acpi/battery/BAT0/info Show state of cells in laptop battery
# dmidecode -q | less Display SMBIOS/DMI information
# smartctl -A /dev/sda | grep Power_On_Hours How long has this disk (system) been powered on in total
# hdparm -i /dev/sda Show info about disk sda
# hdparm -tT /dev/sda Do a read speed test on disk sda
# badblocks -s /dev/sda Test for unreadable blocks on disk sda
interactive (see also linux keyboard shortcuts)
• readline Line editor used by bash, python, bc, gnuplot, …
• screen Virtual terminals with detach capability, …
• mc Powerful file manager that can browse rpm, tar, ftp, ssh, …
• gnuplot Interactive/scriptable graphing
• links Web browser
• xdg-open http://www.pixelbeat.org/ open a file or url with the registered desktop application
miscellaneous
• alias hd=’od -Ax -tx1z -v’ Handy hexdump. (usage e.g.: • hd /proc/self/cmdline | less)
• alias realpath=’readlink -f’ Canonicalize path. (usage e.g.: • realpath ~/../$USER)
• set | grep $USER Search current environment
touch -c -t 0304050607 file Set file timestamp (YYMMDDhhmm)
• python -m SimpleHTTPServer Serve current directory tree at http://$HOSTNAME:8000/
Category: Tutorials   , , , , , , , , ,
Comments (0)

DataSlide re-invents the Hard Drive

Posted by on Tuesday, 16 June, 2009

From: http://www.theregister.co.uk/2009/06/15/dataslide_berkeleydb/

UK-based data storage start-up DataSlide has announced potentially revolutionary hard drive technology, and a Partnership Network agreement with Oracle for the Berkeley Data Base to be embedded into the device.

DataSlide’s Hard Rectangular Drive (HRD) does not use read-write heads moving across the recording surface of a spinning hard disk drive (HDD). Instead an ultra-thin, 2-dimensional array of 64 read-write heads, operating in parallel, is positioned above an piezo-electric-driven oscillating rectangular recording surface, and delivers 160,000 random IOPS with a 500MB/sec transfer rate.

For comparison, a STEC ZEUSIOPS SSD, as used by EMC, IBM and others, with up to 320GB capacity, can provide 220MB/sec read bandwidth, 115MB/sec write bandwidth and 45,000 random IOPS. There is no read-write asymmetry, typically found with SSDs, with the HRD because it uses a standard hard disk drive recording medium and not flash memory.

Charles Barnes, DataSlide’s CEO, said: “DataSlide’s Massively Parallel architecture with 64 heads per surface could saturate a 32-lane PCI express bus. The Hard Rectangular Drive has the industry reliability and cost advantages of Hard Disk Drives with superior performance and lower power then Solid State Drives.

“The HRD uses over 60 per cent lower power than HDDs and during idle the media has zero power dissipation making it the green storage winner.”

The technology is also more shock-resistant than hard drives. This could be described as a solid-state drive with none of the well-known NAND flash problems, such as read-write asymmetry and write endurance.

Oracle’s Embedded Global Business Unit stated: “DataSlide provides a high bandwidth, low latency, magnetic storage device whose architecture lends itself to vastly improved database throughput and latency reduction.”

There is a description of the DataSlide technology here (PowerPoint deck pdf). Literally, it is non-revolutionary, using oscillations to move the magnetised bits to and fro underneath the read-write heads so that they can use magnetism value changes at the bit edge just as a read-write head on a spinning hard drive does, but where the recording layer passes continuously under the heads. There is no seek time access delay with the HRD.

The Embedded Global Business Unit at Oracle has an OEM charter and Data Slide meets its requirements by incorporating the Berkeley DB into the actual storage device to make what it calls a ‘smart’ storage device. It says potential applications are many and varied. Examples include TCP/IP-based systems and video applications requiring multiple concurrent streams. The company says media indexing, fast positioning, forward, back, skip, and scene/track operations will have significant performance improvements with its technology.

DataSlide is a privately-held company with locations in the United Kingdom, France and United States. It has affiliations with academia from Carnegie Mellon University/DSSC, and the Universities of Cambridge, Exeter, Sussex, Sheffield and Brighton in the UK, and Paris-Sud in France, and is backed by angel investors. It has a management team with experience from companies such as Seagate, Connor, Quantum, Maxtor and HP.

The technology is proven in a research and prototype sense, and the company emphasises that it uses standards-based mature process technologies from LCD, HDD and semiconductor manufacture. There is no need to design and tool-up a new manufacturing process.

DataSlide is currently in discussion with a number of storage and system OEMs and can provide more details under a non-disclosure agreement. It will be holding private meetings at the Santa Clara Hyatt Regency from June 22-25 during the Memcon 2009 conference.

Category: News   , , , , , ,
Comments (0)