We’re Back!

July 17th, 2010

Well, after a painful server move, and a discontinuation of my old server, I’m back on the net!!!

Also, could someone verify that the FTP site is working from outside?

AIDE configuration for Ubuntu 8.04

April 29th, 2009

I had trouble finding some good documentation for AIDE on ubuntu, so I figured I would write a quick post about configuring AIDE on a ubuntu system.

Installing AIDE is simple in ubuntu, just type:

sudo apt-get install aide

This will install all of the packages that you need, and create a standard configuration.

In true debian fashion, all of our configuration files are split out, which makes things confusing, but easy to manage. The global configuration is in /etc/aide/aide.conf. Take a look at the file, but don’t change anything.

Next, take a look at the /etc/aide/aide.conf.d directory. In this directory there are tons of configuration snipits that are included when aide is run. There are two changes that I make here:

First, delete 99_aide_root, 70_aide_var, and 70_aide_tmp files. I find that trying to enter /var into the database doesn’t work so well because it changes to quickly. Second, I edit the 70_aide_etc file and change this:

/etc$ VarDir

to this:

/etc VarDir

Which tells the system to enter the entire etc directory into its system, instead of just checking the inode counts in the directory itself.

Now we can run aideinit to create or update the database (which needs to be done after system changes), and aide.wrapper to run reports. Aide reports will be run every night be default, and emailed to root (which can be changed in the /etc/defaults/aide config file)

Red Hat Linux 5 Basic Training

April 9th, 2009

As promised, here are the slides for the Red Hat Linux 5 Basic Training we had today: redhat5class.

I wanted to thank everyone again for taking the time to come out. I would also love any additional feedback you would like to give about ways to improve the class.

A couple of quick follow-up items:

Using nano instead if vi to edit the crontab:

if you want to use nano to edit your crontab file instead of vi (with the crontab -e command), then run this instead:

env EDITOR=nano crontab -e

Canceling a yum download: (Thanks Renee)

Wow, this was a tough one! I didn’t find a lot if information about it, but it looks like a bug (you are supposed to be able to press CTRL-C twice quickly to cancel)

What I ended up doing was:

Pressing CTRL-Z
Then typing: kill -s 9 %1

CTRL-Z puts a process to sleep, and kill -s 9 %1 kills the first job in the current terminal with SIGTERM signal. Overly complicated in my opinion.

Send me an email, or leave a comment if you have any follow-up questions or suggestions.

Thanks again!

Example of using a wii remote as an input device.

January 23rd, 2009

After a few hours of googling, I finally got Call of Duty 4 with a Wiimote working. Check it out, I can write down the steps I went through to get it working if you would like, just post something in the comments.

CoD 4 with a Wiimote

New Year’s Eve LAN Party Announced

December 15th, 2008

We are going to be hosting a LAN party in the south end this new year’s eve. Details to come soon, but mark your Calendar’s. You can email me at nospam@ccrow.org or leave a message in the comments if you are interested in attending.

UPDATE 2008.12.23:

We are still on for this! This LAN party will be held at the Schneider residence on South Hill. Cost will be $5 for pizza, and bring drinks to share. We will start up at about 4:00pm and end when people feel like leaving, but we will try to clear out in the morning of January 1st. This LAN party will be held in a garage, and although I plan to have heaters there, you will want to bring warm clothing.

I will be making phone calls tonight with directions and these details. Send me an email at nospam@ccrow.org or leave a message in the comments and I will make sure that I will email you.

We also need some basic supplies like folding tables, chairs, power strips, and network cables.

We will be playing the following games if you want to have the pre-installed:

  • Call of Duty 4
  • Dawn of War or Company of Heroes
  • Quake 2

If anyone has additional ideas, email me at nospam@ccrow.org with ideas.

Titan Quest LAN Online Party Announced!

December 8th, 2008

We are planning a Titan’s quest LAN party using the online vpn this Friday.

We will be playing with the latest retail patch of with the Imortal Throne expansion. I thought the latest patch was 1.30, but if it is not, please let me know.

We are going to get started around 8:00pm and go until people drop off from exhaustion. I will have IM open to assist with tech support, and getting people on. I can be reached by adding nospam2@rcrow.com to your buddy list on MSN messanger. You can find the VPN download from the links on the right.

Titan Quest has some issues with VPNs on Vista, we are still testing, but have Sins of a Solar empire install just in case we have to bag Titan Quest.

Using LVM and the VI Perl Toolkit to take snapshots of an NFS LUN

November 8th, 2008

If you followed my previous howto, you should have a CentOS box set up that is ready to accept NFS connections. In this howto, we are going to walk through the steps to install a script that will correctly snapshot the lvm volume we exported in the previous howto, and serve that volume through SMB for backups.

Install the VI Perl Toolkit on Your CentOS box.

From the CentOS box, download the VI Perl toolkit from here: http://www.vmware.com/support/developer/viperltoolkit/, Click on the download Link (make sure you download the 32bit linux version) and save the file to your desktop. Right click on the file and select ‘Extract Here’

Open a terminal and type: ’su -’. Then enter your root password to get a root prompt. We need to install the prerequisites by typing:

yum install perl-Crypt-SSLeay.i386

Now we will install the VI Toolkit by typing:

[root@centos ~]# cd /home/USER/Desktop/vmware-viperl-distrib/
[root@centos vmware-viperl-distrib]# ./vmware-install.pl

Follow the text prompts to install the toolkit.

Install and configure SAMBA for snapshot access.

To install samba, type:
yum install samba

Now we need to create an a directory to hold our snapshots:
mkdir /export/snapshots

Now lets edit the /etc/samba/smb.conf file, and add the following to the very end of the file:

[snapshots]
path = /export/snapshots
writable = no

Now we need to add a local user account that our client will use:

[root@centos apps]# adduser backupuser
[root@centos apps]# smbpasswd -a backupuser
New SMB password:
Retype new SMB password:
Added user backupuser.
[root@centos apps]#

And lastly, we need to start samba:

[root@centos apps]# service smb start
Starting SMB services:
Starting NMB services:
[root@centos apps]#


Install the custom Snapshot scripts

Now that we have all of the prerequisites out of the way, you will need to download the following two files to the /usr/lib/vmware-viperl/apps directory on your CentOS box:
snapdatastore.pl
removedatastoresnap.pl

Don’t forget to set the execute flag on the programs by typing: chmod +x /usr/lib/vmware-viperl/apps/*.pl

Now download the configuration file into /root and rename it to .visdkrc (putting a . in front of a unix file makes it hidden, you can still edit the file with ‘nano .visdkrc’
visdkrc

Edit the visdkrc file and change the information to suite your environment:

VI_SERVER=192.168.235.10
VI_USERNAME=administrator
VI_PASSWORD=a
VI_DATASTORENAME=esxnas1
LVM_SNAPNAME=BackupTest2
LVM_VG=VolGroup00
LVM_LV=esxnas1
LVM_SNAPSHOTRESERVE=10G

Now invoke the script by typing:

[root@centos ~]# /usr/lib/vmware-viperl/apps/snapdatastore.pl
Operation :: Snapshot backuptmp for virtual machine SnapTest created sucessfully under host 192.168.235.12
Operation :: Snapshot backuptmp for virtual machine CentOS-Test created sucessfully under host 192.168.235.11
Logical volume "BackupTest2" created
Operation :: Remove All Snapshot For Virtual Machine SnapTest under host 192.168.235.12 completed sucessfully
Operation :: Remove All Snapshot For Virtual Machine CentOS-Test under host 192.168.235.11 completed sucessfully

End Disconnect
[root@centos ~]#

And remove the snapshot with:

[root@centos ~]# /usr/lib/vmware-viperl/apps/removedatastoresnap.pl
Logical volume "BackupTest2" successfully removed
[root@centos ~]#

You can also override the name of the snapshot, the esx datastore, and the LVM Logical Volume from the command line by typeing:

/usr/lib/vmware-viperl/apps/snapdatastore.pl --snapshotname 2008-11-08 --datastorename esxnas2 --vg LogVol1

The above will create a snapshot called 2008-11-08, it will snap all the virtual machines on the datastore labeled ‘esxnas2′, and it will create the snapshot on the Logical Volume called ‘LogVol1′.

Access the snapshot from a Windows Host

Now all we have to do is click on Start -> Run.. from a windows host, and type “\\<NAS_IP_ADDRESS>\snapshots

Enter your username and password that we created previously, and you are set.

In the above image, the only virtual machine files you need to back up are:

  • SnapTest.vmx
  • SnapTest.vmdk
  • SnapTest-flat.vmdk

The rest are log files and temporary delta files.

Keep in mind that while a snapshot is enabled, write performance on all of your virtual machines will drop by 60%, so you should use it at non-peak times.

Beginner’s guide to creating an ESX compatible NAS

November 7th, 2008

I few people have asked me about how to create a basic NAS based on RedHat / CentOS linux for use with ESX server. In this post, I am going to attempt to walk you through the basics of setting up a linux NFS server, and how to administer it.

Install CentOS

Download the CentOS ISO from http://www.centos.org/. Burn and put the CD in the drive and boot the computer. Just press <Enter> at the splash screen. CentOS will then ask if you want to test the CD Media. I usually skip the test.

Now you should be presented with the graphical installer. Follow the on screen instructions, and specify you language and keyboard type. Now you will be presented with the partitioning screen. Be default, CentOS likes to take over the entire disk. This is fine in most cases, but if you have a very large disk (say, the aggregate disk presented by a hardware raid controller). For our example, check the ‘Review and modify partitioning layout’, then click next.

We can see that CentOS decided to take over the entire disk, we are going to change that. Click on  the ‘LogVol00′ line and select edit. This will bring up the LVM volume editing, we only want to edit the LogVol00, so select it and press edit.

Now we are presented with the a screen that allows us to change the size of the root (/) partition. Change the size to 10000 (MB) and select OK.

Select OK again, and then next to continue the installation. Now we will see the boot loader configuration step. Just accept the defaults and select ‘Next’.

This screen allows us to configure the network. Configure the host name and network parameters as you wish, and select next. You will always be able to change them later.

Now select the timezone, and select next.

Set the root password, and select next. Now you will be presented with the software selection screen. Make sure ‘Server’ and ‘Desktop – Gnome’ is selected. ‘Desktop – Gnome’ is not necessary for CentOS to function as a NFS server for ESX, but it makes it easier to get around.

Select  ‘Next’ and your install will get started, you now get to watch the propaganda screens while you wait for your OS to be installed.

Configuring CentOS

After the system boots for the first time, there are a few things we need to do. First, CentOS will walk you through a setup wizard that will allow you to configure some settings. It is important that we set a couple of things up.

Disable the Firewall

Disable SELinux

Now you can finish the rest of the steps. It is wise to create a user account for yourself to log in as. At the end of the wizard, and one reboot later, you should see a log in prompt. Go ahead and log in with the user you create during the install process. You will be presented with a Desktop. The first thing you need to do is launch a terminal.

We will be using the terminal for the remainder of this howto.

Open up a terminal by clicking on ‘Applications->Accessories->Terminal’ and type:
<code>su -</code>
This will prompt you for the root password

Finishing the Disk partitioning.

Now, in this howto, I have configured a CentOS box with a 50GB hard drive, I would recommend the following partition layout for any size disk:

10 GB for Root (/)
4 GB for swap
70% of the remaining space for exports (the NFS word for Share)
the rest of the space will remain for a snapshot reserve.

Now a quick explanation of how linux handles mounts (or drives for the windows people). Every hard disk is “Mounted” under a directory under the root (/). So we are going to mount our primary data drive under /export.

From the terminal we are going to create a mount directory, create a logical volume called ‘esxnas1′ that is 20000MB, format the drive, then mount the drive

[root@centos ~]# mkdir /export
[root@centos ~]# mkdir /export/esxnas1
[root@centos ~]# lvcreate -L20000 -nesxnas1 VolGroup00
Logical volume "esxnas1" created
[root@centos ~]# mke2fs -j -m 0 -b 4096 /dev/VolGroup00/esxnas1
[root@centos ~]# mount /dev/VolGroup00/esxnas1 /export/esxnas1

To make the mount permanent, we need to add the following line to /etc/fstab:

/dev/VolGroup00/export /export/esxnas1 ext3 defaults 0 0

HINT: You can edit files by typing ‘nano <filename’, just type <CTRL>-X to save and quit

Configuring NFS

Now we can configure NFS, the first thing we need to do is edit the /etc/exports file by typing ‘nano /etc/exports’ and add the following:


/export/esxnas1 192.168.0.0/255.255.0.0(rw,no_root_squash)

The above will create the export, now we need to start the nfs server by typing the following:

[root@centos ~]# chkconfig --level 345 nfs on
[root@centos ~]# service nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]
[root@centos ~]# exportfs

Configure ESX to connect to your new NFS server

  • Fire up you VMware Infrastructure Client and connect to your ESX server, or virtual center server.
  • Click on the ESX server you want to connect, and click on the configuration tab.
  • Click on the storage link
  • Click on the ‘Add Storage…’ link
  • Select ‘Network File System’ and click next.
  • Enter in the IP address of your server, the path of the folder you exported (/export/esxnas1 in our example) and give it a name

NOTE: Keep in mind that the Datastore Name needs to be the same on all ESX servers you connect to the NFS server for vMotion and HA to work.

Trunking CentOS 5 to a Cisco 3750 using etherchannel

October 17th, 2008

Well, because of potential budget cutbacks at work, it has become necessary for me to research how to build a SAN for ESX out of a whitebox server. I figured I would spend some time in our lab figuring out how all of the parts would fit together, and it has become obvious that the best way to do this is with an NFS server that is bonding multiple ethernet connections.

In this post, I’m going to walk you though what I did for the first stage of this project, which is to get a CentOS box to preform bonding with a cisco switch. In later posts, I will talk about ESX configuration, NFS storage configuration with snapshotting, and some optimization and backup scripts.

Computer Specs:
3 Network interfaces for this test

Configure CentOS for Bonding

Install centos with the default options.

Configure one interface as the management interface (it will be excluded from bonding)

Figure out which interfaces you want to use for bonding
TIP: if you are having trouble figuring out which eth interfaces belong to which physical port, turn them all on by typing:

ifconfig eth0 up
ifconfig eth1 up
...

Then run a ‘tail -f /var/log/messages’ and start connecting your ethernet cables one at a time. As you plug cables in, you will see a message on the console telling you which ethernet interfaces come up.

In my examples, I am using the following network interfaces:
eth0 – Management Port (192.168.64.10)
eth2 – First adapter to be bonded
eth4 – Second adapter to be bonded

And on the cisco switch:
1/0/21 – connected to eth2
1/0/22 – connected to eth4

For bonding to work, it must be compiled as a module so that you can pass parameters to the module. Check that it is compiled as a module by running:


[root@test-nas-01 ~]# grep -i bonding /boot/config-2.6.18-8.el5
CONFIG_BONDING=m

Also, make sure that your ethernet drivers support link status by typing:


[root@test-nas-01 ~]# mii-tool
eth0: negotiated, link ok
eth1: no link
eth2: negotiated 100baseTx-FD, link ok
eth3: no link
eth4: negotiated 100baseTx-FD, link ok
eth5: no link

Now we need to configure the bonding interface, create a file called /etc/sysconfig/network-scripts/ifcfg-bond0 and put in the following contents:

DEVICE=bond0
BOOTPROTO="none"
STARTMODE="onboot"
USERCTL=no

Notice how I am not specifying an IP address, we will do that later when we configure vlans.

Now add this to your /etc/modprobe.conf file:

alias bond0 bonding
options bond0 mode=balance-xor miimon=100 xmit_hash_policy=layer2

Most of the above options make sense, except for the module options:

mode=balance-xor: uses XOR operators to hash the traffic, this mode directive DRASTICALLY changes the behavior of the bonding driver

miimon=100: then frequency in milliseconds to check for link state changes to bypass a failed adapter.

xmit_hash_policy=layer2: This options works in conjunction with the balance-xor option to tell the driver how to hash. The other option is layer3+4, which will hash based on IP and Port, which is the preferred option to bond with cisco switches.

See this website for more options: http://www.cyberciti.biz/howto/question/static/linux-ethernet-bonding-driver-howto.php

Now we need to set the options for the slave ethernet interfaces:
ifcfg-eth0:

DEVICE=eth4
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no
HWADDR=00:04:23:D8:44:D4

Do the above for each ethernet interface that is part of the bonded connection.

Configure Cisco for Bonding (Link Aggregation)

This is the easy part, simply jump into your switch and fire up config mode:

!
interface Port-channel8
description trunk_to_test-nas-01
switchport trunk encapsulation dot1q
switchport trunk native vlan 99
switchport mode trunk
switchport nonegotiate
!
interface GigabitEthernet1/0/21
description Trunk to test-nas-01
switchport trunk encapsulation dot1q
switchport trunk native vlan 99
switchport mode trunk
switchport nonegotiate
speed 1000
channel-group 8 mode on
!
interface GigabitEthernet1/0/22
description Trunk to test-nas-01
switchport trunk encapsulation dot1q
switchport trunk native vlan 99
switchport mode trunk
switchport nonegotiate
speed 1000
channel-group 8 mode on
!

Now some explaination, and if any body out there does cisco, I would love some feedback about whether I did this right.

Most of the above configuration directives are self explainitory, but the tricky one for me was the channel-group 8 mode on line. This configuration is telling the cisco to enable etherchannel without a negotiation process.

The other thing we need to check is the hashing algorithm on the cisco switch. Run “show etherchannel load-balance’ in enable mode to see what the policy is. You can (sort of) mix and match hashing algorithms, but you run the risk of having packets arrive out of sequence. Also, hashing policies are switch wide.

UPDATE: Apparently, you will not affect your traffic, it is just considered sub optimal. Thanks to Alec (http://www.ivoxy.com) for the correction.

VLAN tagging on CentOS

Now, after all of the above hard work, we can finally have some fun:

For each vlan you want to create, create a new file called /etc/sysconfig/network-scripts/ifcfg-vlan915 except replace the number with the vlan ID you wish to use and put the following into the file:

VLAN=yes
VLAN_NAME_TYPE=VLAN_PLUS_VID_NO_PAD
DEVICE=vlan915
PHYSDEV=bond0
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet
IPADDR=192.168.15.143
NETMASK=255.255.255.0

Now you done!

run the following to bring up your interfaces:

ifup eth2
ifup eth4
ifup bond0
ifup vlan915

As always, I would love any feedback, corrections, or discussions.

Quick and dirty script to collect Disk IO performance metrics from Linux

October 15th, 2008

We run into a performance bottleneck today with one of our whitebox nfs servers. Unfortunately for me, this nfs server was running 2 mounts points for ESX that contained all of our citrix servers. We migrated the citrix servers off of our primary SAN a few weeks back because of some severe performance problems, and using NFS was our solution. NFS works great as an ESX target, but our nfs server was is very sub-optimal for the workload.

Anyway, we ended up having a very sudden need for disk performance metrics, so 2 hours of painful debuging later, and I had a working solution.

The first step is to install iostat, this package is provided by most linux distributions, and can be installed via the sysstat package in CentOS, or Ubuntu.

Next run iostat, and have it dump to a text file:


iostat -x -m -t 15 5760 > RAWdiskio.txt

The above will dump statistics every 15 seconds for 24 hours, which gives you 5760 samples. The raw output from the command above can be painful to parse through, so I built a perl script to turn the output into a CSV file:


#!/usr/bin/perl

open (INFILE, "RAWdiskio.txt");

print '"Timestamp","device","rrqm/s","wrqm/s","r/s","w/s","rMB/s","wMB/s","avgrq-sz","avgqu-sz","await","svctm","%util"';
print "\n";

while ($input = ) {

	if ($input =~ /^Time: (\S+)/) {
		print "\"",($1);

	} elsif ($input =~ /^sda/) {
		my @array = split(/\s+/, $input);
		print "\",\"",$array[0],"\",\"",$array[1],"\",\"",$array[2],"\",\"",$array[3],"\",\"",$array[4],"\",\"",$array[5],"\",\"",$array[6],"\",\"",$array[7],"\",\"",$array[8],"\",\"",$array[9],"\",\"",$array[10],"\",\"",$array[11],"\"\n";
	}

}

close (INFILE);

Just save the code into a file, put it in the same directory as the RAWdiskio.txt file, and invoke the command with:

./parse.pl > output.csv