And if it has been two years since many of us began using Dartmouth Secure, it has also been two years since many of us downloaded our Dartmouth personal PKI certificates that are required for accessing the wireless network. And, like Cinderella at the stroke of midnight, those certificates are going to turn into pumpkins as they reach their second birthday.

Dartmouth-issued personal PKI certificates are set to expire 2 years from the date they are issued. And just as an expired driver’s license keeps you from (legally) motoring, an expired personal certificate will keep you from accessing the restricted online Dartmouth services you have come to depend on and love.

Not everyone’s certificate will expire this month — only those of you who are still using that first certificate you downloaded two years ago. But this is likely to be many of you.

We have written a help page with instructions on how to check the expiration date of your certificates, an explanation of what and how services may fail if your certificate has expired, and how to download a new certificate if you old one has passed (no eye exam required).

Visit Renewing Expired Dartmouth Personal Certificates for the nitty-gritty.

And as always, please let Your Friendly Thayer Computing Services Team know if you have any questions. Happy motoring!

Also back in December Apple released Java for Mac OS X 10.5 Update 6 and Java for Mac OS X 10.6 Update 1.

Unfortunately, like two normally tasty ingredients that should never mixed together in the same glass (I’ll never recreate another gator-ovaltine-ade, I promise!), the combination of the Java and Juniper updates has had some less-than-delicious repercussions.

Mac users who visit Dartmouth’s VPN gateway before running the Java update see no problems or issues arise. However, those who try to use VPN after applying the Java update experience a failure as the system tries to update their local VPN client:

An error occurred while extracting one of the Network Connect components

The error occurs because the Java update changed the password on the Mac’s local Java keystore, with the result that any Java applications that try to access the keystore with the old password will fail. And, yes, the Juniper VPN update is one of the unlucky offenders.

The fix is fairly straightforward, though it does involve opening the Terminal application, found in Applications/Utilities. Don’t be scared, though, the command line is your friend.

In Terminal, at the prompt copy the following command in its entirety:

sudo keytool -storepasswd -new changeit -keystore /Library/Java/Home/lib/security/cacerts -storepass changeme

Then, when asked, type in your local OS X password.

Juniper VPN should now be able to update the local client on your Mac with no further impediments. Just revisit the Dartmouth VPN gateway to complete.

And trust me, Gatorade and Ovaltine never should be mixed again.

Unfortunately, there isn’t a good way to provision accounts for users. We have the system authenticating against LDAP. You can also have an account auto created upon first login, but in that scenario, the user gets a default profile. This means everyone’s recordings will end up in one directory which quickly becomes a mess. Currently, we are set up so that recordings get sFTP’d to a web server under one account, and each user has their own subdirectory.

This means each relay account must have its own custom profile and publishing destination set up manually which is a time consuming process… lots of clicking and waiting. Techsmith has stated they plan to make it easier to automatically create users, profiles, and publishing destinations, but no clear date has been given.

There is also some hope that Techsmith has may be able to provide us with raw SQL commands to insert users directly into the database, but this solution is still unproven and won’t be ready for a while.

In the meantime, our solution is to use Chickenfoot. Chickenfoot is a firefox extension that allows you to automate web page interaction. It allows you to automatically click buttons, fill in text fields, etc. Another similar extension is iMacros.

This was my first try with Chickenfoot, which means, the script is ugly and I’m only scratching the surface of Chickenfoot’s capabilities. But the important part is that it actually works. Add the username to the script, press the start button, a couple minutes later the user is provisioned with a relay acccount, custom profile, and publishing destination.

Currently the script only sets up a single account, but could be modified to allow mass import (chickenfoot uses Javascript).

In the off chance someone else is in the same boat as us, here is the chickenfoot script. You’ll need to customize the variables at the top and possibly tweak the sleep statements. Keep in mind that this script works with Camtasia Relay 1.2.0, but will likely need tweaking with any other versions. Hopefully Relay 2 will include more robust user provisioning and completely get rid of the need for this hack.

Here is a screencast of the script in action. (Quicktime)

This term, we are recording seven courses using four different techniques. At this point in Lecture Capture, we haven’t found a suitable "one size fits all" solution.

Below I briefly touch on how we deliver the recordings to students, and details of each lecture capture method.

We’ve made the decision to not have a camera operator for any recording. We just don’t have the person resources for this to scale to multiple classes.

Delivery

The actual delivery of the video to students uses the same technique for each capture technology.

The video is stored on our fileserver. A simple web page provides a list of available videos. The page is only accessible to students enrolled in the course.

The video can be viewed through a browser (via the Flash based JW Player and Wowza streaming server). Alternatively, the video can be downloaded for viewing in QuickTime Player, VLC, etc. We’re using the JW Player Google Analytics plug-in to gather some viewing statistics.

Capture

HD Camera in back of the room

• At the beginning of class, a High Definition video camera is placed in the back of the room pointing towards the front.
• The professor wears a wireless lapel microphone.
• The video camera records to a SD flash drive.
• The recorded file is copied to our file server.
• A watch folder script watches for newly uploaded files. When it finds a new file, it re-encodes it to a lower bitrate more suitable for viewing.

Gear details

• Sanyo Xacti HD1010 Camera
• Sennheiser Evolution G2 100 Series Wireless Microphone kit
• Transcend 16 GB SDHC Flash Memory Card with reader
• 4 AA Rechargeable batteries and battery charger
• Tripod

Pros

• Cheap (under $1000)
• Can be used in any room
• Captures anything with reasonable detail – chalkboard, projector screen, professor. We record at 1280×720 resolution. The camera will actually do 1920×1080, but most students screens are smaller than that.
• Simple, quick set up. A Teacher’s Assistant can be trained in a matter of minutes

Cons

• Needs to be set up and taken down for each class
• The video needs to be processed for more reasonable file sizes. We currently transcode the video to 2 Mbps and 15 frames per second. We could probably go lower as there often is much motion in the frame. However, it is assumed that most students will be viewing the video from on campus, so we aren’t super concerned about file size.
• Cameras tend to automatically split files for long recordings. Joining them is non-trivial.
• Cameras that record in H.264 with a microphone jack are rare.
• Setting up a script to watch a folder and transcode the files is a bit messy

Wirecast

We still use Wirecast to record, and live stream if needed, courses and events in Spanos, our large lecture hall. We have a permanently installed system in the A/V booth. We also have a Wirecast system on a cart that we can roll to any room. We used to have a portable system, but the set up and take down is very invovled, so we haven’t used it in a while.

Gear details

• Mac Pro
• Wirecast license with HD add-on
• Epiphan VGA2USB LR framegrabber
• Canon VIXIA HV30 video camera

Pros

• Lots of flexibility with recording layout, bitrates, etc.
• Live streaming capability

Cons

• Requires fairly powerful computer for capture
• Requires a fair amount of training for a non-technical operator
• Has several moving parts and has had its fair share of bugs
• No automated recording, though there have been some attempts using Applescript

Techsmith Camtasia Relay

We’ve been working closely with Dartmouth’s Curricular Computing department to implement Camtasia Relay. It is now slowly being rolled out to faculty. It is a software based solution that records the screen of the lecturer’s computer, and audio from a microphone.

Gear details

• Camtasia Relay license
• Server to transcode the screencasts
• Computer (classroom or professor’s) for performing screencast
• Microphone – We are using a RevoLabs xTag

Pros

• Cheap compared to screencasting competition
• Site license can be installed on everyone’s computer
• Can also be used by a professor for other screencast needs
• LDAP authentication

Cons

• Requires professor to start, stop, and each recording (could conceivably be done by a TA).
• Doesn’t capture video. If the professor uses the chalkboard, you’re out of luck.

NCast Telepresenter M4

The NCast Telepresenter M4 is our latest addition. It is an appliance, permanently installed in a classroom. The biggest benefit of the Telepresenter is the automation it provides. Course recordings can be scheduled in advance. The recording starts, stops, and is published with no human intervention. In can also be controlled through a web interface, or through the room’s built-in A/V controls.

Gear details

• NCast Telepresenter M4
• Canon HD Video Camera (currently using component out)
• Shure wireless microphone (already a part of the room’s A/V system)

Pros

• Automatic start, stop, and publishing of videos
• Permanently installed in room means no set up and take down for each class
• Live streaming capability
• Supports HD Video cameras

Cons

• Expensive
• Management interface takes a while to get used to
• Still need to deal with lapel microphone (and batteries)
• Automation means you lose some flexibility (in case class starts late, ends early/late, etc.)

So after waiting over two years to dive into the cheap storage arena, we finally decided to go with a pretty boring solution. In this case, we hope boring will mean that it is a stable solution and easy to replace once our dream system comes along.

We chose to serve files with CIFS and NFS from a single server running Linux. Here are the details:

Aberdeen 48 bay storage server

We compared Aberdeen to several other similarly sized server sellers (Pogo Linux, Server Direct, Thinkmate, etc.) Pricing was almost identical, but Aberdeen was the only vendor to include a standard 5 year warranty.

We also looked at the price of building the system ourselves. Using all the same components as the Stirling X888, the price (before shipping) came in $2,274.95 (19%) lower. Putting systems together can be time consuming. We may consider it for the backup system, but for our primary, it is good to know that if anything goes wrong, we can call one company and get help.

We’re going to start out with twelve 2 TB “enterprise” SATA drives in a hardware RAID 6 with one hot spare. That allows us to lose two drives per 12 drive RAID group without losing data. And the hot spare will allow use to start rebuilding immediately.

The server will be running Ubuntu with an XFS filesystem. We’re a happy Ubuntu shop so this decision was easy. We’d prefer to use Hardy, the latest Long Term Support release, but due to some network driver issues, we’ll use Jaunty. XFS is mature, deals well with large filesystems, supports advanced ACLs, and has project/directory based quotas.

There will be no snapshots (LVM snapshots cause abysmal performance). However, we will back up all the data (probably via rsync) to a nearly identical system located off site. The RAID should protect us against hard drive failure, and the offsite backup should protect us against accidental deletion and server room disaster (fire, flood).

In the case of a disaster, the hope is that the near identical backup system can become the primary server… though we’ve yet to deal with the nitty gritty details of this.

Expandability – We’d love a system like clustered Samba or AFS where we can just add additional storage servers as our needs grow at unpredictable rates. Our way around it? Buy a server with lots of empty bays so we can populate it as we need the space. This system has a 96 TB raw capacity today. As drive capacity grows, that will go even higher. The system can also use additional Direct Attached Storage. There is a risk that we’ll have several clients hitting the single server simultaneously and performance will suffer. This is an unknown that we hope to test, and hope to not run into in practice.

Samba/winbind and the NFS kernel server integrated with our Active Directory environment will be the heart of the file serving. We have experience with both, but not at this scale. The hope is that because these are widely deployed, we won’t run into any major issues. At the moment we don’t intend to offer SSH/SFTP or AFP.

High Availability Failover, or the lack thereof

Jordan and Dailey spent a lot of time investigating and testing High Availability techniques (heartbeat, STONITH, DRDB, etc). We could have chosen to have two head ends, or two fully redundant servers, and if one were to fail we would automatically promote the backup to the primary role. They got things working, but in the end, the failover technology made the system more complicated. Basically both servers have to share identities (IPs, kerberos keytabs, etc.) Then you have to set up reliable monitoring of system health, and kill the failed server, while promoting the backup. There are a lot of moving parts, and you need to be very sure only one server is in charge. This is one area where NetApp has earned its money.

If our NetApp is down, the productivity of Thayer School faculty, staff, and students takes a serious nose dive. So it is easy to justify an active failover head. With our cheap storage system however, the hope is that in the event of a server kernel panic our researches can still be productive with other things while we bring the system back up. Of course, the plan with a simple system is that failures such as kernel panics are very rare.

Further down the road, hopefully I’ll update this series of posts with our experience actually receiving the system and getting it set up.

There are many reasons to pay for the reliability, rich feature set, and fail-over capabilities of our NetApp. The NetApp is an excellent product. However, there is also a significant amount of data at Thayer that just doesn’t need all the advanced features. For instance, we have research groups with multi-Terabyte data sets that just want their data easily accessible. If it were unavailable occasionally it wouldn’t be a huge deal. Other features such as snapshots are not critical.

If we can store these larger data sets on a cheap storage alternative, we can minimize the size and cost of our NetApp.

Design philosophy

After stalling for a couple years, hoping something would come along, we have decided to move forward with a plan that makes many compromises, but should meet the basic requirements for many of our storage needs.

We initially started the search to replace the NetApp altogether feature for feature. However, over time we lowered our standards to requirements that meet most of the needs for simple storage of lots of data. As our research continued, our philosophy on what product to choose evolved. In the end, the three main philosophical items we looked for when choosing a cheap storage solution were:

1. Widely used technologies and standards – If lots of people are already using the technology it is more likely to be stable. It also is likely to be around in a few years. While little start ups may have cool technology, we really don’t want to trust our data to a half-baked product. Little companies can also get gobbled up or go out business overnight, never to be heard from again.
2. Healthy communities and companies that are committed to the technology – In case we do have questions, or run into an issue, we want an active community of people that will be able to knowledgeably answers our questions. It isn’t enough that the technology says it supports CIFS and Active Directory. If it doesn’t work quite right in _our_ environment, and nobody seems to know why, we just can’t be comfortable moving forward.
3. Simplicity – Simple solutions have less chance of breaking and are easier to fix in the event that they do break. It is easy to build several layers of redundancy, but end up with a system that is more likely to break because of all the moving pieces.

What we considered

ZFS on OpenSolaris/Solaris

Link

We actually own a Sun Fire X4540 (Thor) running Solaris. We are currently using it as our online backup for the NetApp. It seems like a nice piece of hardware and the ZFS feature set is amazing. Snapshots, filesystem compression, expandability, checksumming, and the rest are all great.

• Technically a home run.
• Doesn’t score so well on our three design philosophy requirements.
• Reasonable, researched questions in forums go unanswered.
• Sun Engineers try to be helpful, but can disappear for weeks at a time.
• Sun shunned the Samba project and instead decided to implement their own CIFS server.
• Projects lack focus. There are several different editions of Solaris. They appear to be attempting to clear this up, but I really wish their “Project Indiana” broke free of a lot of the legacy issues weighing them down.
• Storage systems must be purchased fully populated.
• We could not get AD group integration working and ID mapping is complex.
• Even the latest build of OpenSolaris makes you feel like you are a Sys Admin from 1999.
• Sun laid off many employees. They have lots of projects they are trying to maintain, but seem to be having trouble keeping up with them.
• To make things worse, they got gobbled up by Oracle. How important Solaris is to Oracle remains to be seen.

Sun Storage 7000 Unified Storage (Amber Road)

Link

Sun realized there were admins like us who wanted ZFS but didn’t want to admin Solaris. So they wisely made a storage appliance. They did a great job of putting a pretty face on top of ZFS and Dtrace and then went and charged way too much for it.

• Price is lower than NetApp and EMC, but not low enough to build a decent sized userbase and build some critical mass.
• Can’t replicate from an Amber Road device to our existing Sun Fire X4540.

ZFS on Nexenta

Link

Nexenta seemed like it could be a good compromise for some of our Solaris concerns. Like Amber Road, it hides all/most of the Solaris administration and allows you to use ZFS like an appliance. We set up Nexenta on test systems on multiple occasions and it shows a lot of promise. Again a home run technically, but falls apart on our design philosophy.

• The company is very small without a very big user base.
• If you have any Solaris specific technical problems, they basically say, “go talk to Sun”.
• The depth and breadth of their documentation needs some major improvements.
• Fairly inexpensive (good educational discounts) but when building cheap storage, every additional dollar pushes up the cost per gigabyte.
• They are working like mad to add advanced features like failover.

Came very close to giving them a shot, but they just need a little more time to fully bake.

Clustered Samba

Link

I hope this project gains momentum as we are really drawn to an open system that we can grow on demand by adding another server full of hard drives as our storage needs grow in fits and starts.

• Still early and very little documentation
• Uses clustered file system which are still maturing and poorly supported on Ubuntu

Netgear ReadyNAS

Link

Netgear sells the ReadyNAS, a decent Linux based Network Attached Storage. We bought a 6 bay ReadyNAS Pro which we currently use for network based Time Machine backups for a few Macs. Bascially we wanted to get some mileage with a Linux based NAS.

• Decent web management interface with easy AD integration.
• Seems to have better throughput than most small business NAS systems.
• Tech Support is friendly but very bureaucratic. We ran into a bug when trying to add user and group permissions to a share. Getting tech support has been very difficult. You basically have to give them your first born child in order to be raised to the higher level of support. They really need to say, “yes that is a bug, we’ll fix it and get back to you”. Instead, they had me do dozens and dozens of unrelated tweaks to no avail. When I was finally suppose to get the next level of tech support, my ticket somehow got closed and I basically had to do the dozens of task all over again under a new ticket (which is still unresolved).
• The largest product they sell is 12 bays. Probably not big enough for us unless we used multiple systems.
• It uses EXT3 filesystem, so quota is user and group based instead of directory/project based. (can be worked around with lots of groups).
• Dollar per gigabyte is very good (currently around $0.53 per usable GB for the 3200 with 12 2TB drives in RAID 6 with 1 hot spare).

Btrfs

Link

A new, GPL filesystem originally developed by Oracle. All the fancy features that we want, baked into the Linux kernel. Fingers crossed that in a couple years this will have stabilized to the point that it is ready for production use. It seems like it has the most promise of competing with NetApp’s WAFL and ZFS. Oddly enough, Btrfs is now owned by Oracle. Will Oracle continue to develop both ZFS and Btrfs?

Other

We looked at numerous other technologies in varying levels of detail. None seemed to quite have the momentum at this point, and several were just not designed for a typical file server. We’ll keep an eye on them.

Some of the projects include:

• IBRIX – now owned by HP
• LeftHand Networks – now owned by HP
• Cleversafe
• Parascale
• Caringo CAStor

Read part three… what we actually chose to go with (warning, it is pretty anticlimactic).

The cost of our NetApp storage is a little fuzzy. Over the years, we’ve come up with calculations from around 2.50 – 12 dollars per gigabyte. But if you head over to NewEgg and buy a stand-alone 2 terabyte enterprise SATA drive, it is only $0.16 per raw gigabyte. Obviously there is a fair amount of additional hardware and software required to make that drive available on the network. So the question is, how can cheap can make a file server that still has respectable features, availability, and performance.

We’ve planned a system that comes out to about $0.28/GB raw and $0.37/GB usable. If the system works well, our plan is to buy a near identical system that we’ll use for backup. So backed up, the cost will be double, or about $0.84 per usable gigabyte. While this is many times more than the cost of a raw drive, it is significantly cheaper than the alternatives that we explored.

Over the next few blog posts, I’ll give some details of the options we investigated, what we’ve ultimately decided to go with, and how the implementation actually works out.

Update: The original price per GB I used was assuming we would buy a spare part of almost every electronic component (motherboard, CPU, RAM, etc.). I’ve updated it to reflect the price without any spares parts, which is still slightly misleading, as we will buy some spare parts. The price is also based on a fully populated system. However, we will initially only have a system that is 25% populated. At the beginning, this pushes the $/GB up, but because $/GB of hard drives is going down over time, by the time the system is fully populated, it will actually be lower than above quoted price.

Go to part two of the Cheap Storage Project series.

The new Thayer School Dropbox is a way to quickly and easily get your file sent. Once you visit the site and authenticate, you just choose the file and upload it to the server. You’ll then be given a link to send to the recipient – or just have the Thayer School Dropbox do it for you. The recipient can then visit the link and download your file.

Right now, files can only be uploaded by Dartmouth community members (although anyone can download). Soon, we plan to add the capability for Dartmouth community members to generate a “slot” where an outside person could upload to them.

To use the Thayer School Dropbox, just visit https://dropbox.thayer.dartmouth.edu/ and follow the easy instructions. We hope you find this to be a useful service and welcome your comments on how it could be improved and expanded.

The “no questions asked” maximum quotas for graduate students remains at 5GB and 10GB for faculty and staff. As always, if you have reasonable case for needing more space, just let us know.

Full details about ThayerFS quotas are here.