This term, we are recording seven courses using four different techniques. At this point in Lecture Capture, we haven’t found a suitable "one size fits all" solution.

Below I briefly touch on how we deliver the recordings to students, and details of each lecture capture method.

We’ve made the decision to not have a camera operator for any recording. We just don’t have the person resources for this to scale to multiple classes.

Delivery

The actual delivery of the video to students uses the same technique for each capture technology.

The video is stored on our fileserver. A simple web page provides a list of available videos. The page is only accessible to students enrolled in the course.

The video can be viewed through a browser (via the Flash based JW Player and Wowza streaming server). Alternatively, the video can be downloaded for viewing in QuickTime Player, VLC, etc. We’re using the JW Player Google Analytics plug-in to gather some viewing statistics.

Capture

HD Camera in back of the room

• At the beginning of class, a High Definition video camera is placed in the back of the room pointing towards the front.
• The professor wears a wireless lapel microphone.
• The video camera records to a SD flash drive.
• The recorded file is copied to our file server.
• A watch folder script watches for newly uploaded files. When it finds a new file, it re-encodes it to a lower bitrate more suitable for viewing.

Gear details

• Sanyo Xacti HD1010 Camera
• Sennheiser Evolution G2 100 Series Wireless Microphone kit
• Transcend 16 GB SDHC Flash Memory Card with reader
• 4 AA Rechargeable batteries and battery charger
• Tripod

Pros

• Cheap (under $1000)
• Can be used in any room
• Captures anything with reasonable detail – chalkboard, projector screen, professor. We record at 1280×720 resolution. The camera will actually do 1920×1080, but most students screens are smaller than that.
• Simple, quick set up. A Teacher’s Assistant can be trained in a matter of minutes

Cons

• Needs to be set up and taken down for each class
• The video needs to be processed for more reasonable file sizes. We currently transcode the video to 2 Mbps and 15 frames per second. We could probably go lower as there often is much motion in the frame. However, it is assumed that most students will be viewing the video from on campus, so we aren’t super concerned about file size.
• Cameras tend to automatically split files for long recordings. Joining them is non-trivial.
• Cameras that record in H.264 with a microphone jack are rare.
• Setting up a script to watch a folder and transcode the files is a bit messy

Wirecast

We still use Wirecast to record, and live stream if needed, courses and events in Spanos, our large lecture hall. We have a permanently installed system in the A/V booth. We also have a Wirecast system on a cart that we can roll to any room. We used to have a portable system, but the set up and take down is very invovled, so we haven’t used it in a while.

Gear details

• Mac Pro
• Wirecast license with HD add-on
• Epiphan VGA2USB LR framegrabber
• Canon VIXIA HV30 video camera

Pros

• Lots of flexibility with recording layout, bitrates, etc.
• Live streaming capability

Cons

• Requires fairly powerful computer for capture
• Requires a fair amount of training for a non-technical operator
• Has several moving parts and has had its fair share of bugs
• No automated recording, though there have been some attempts using Applescript

Techsmith Camtasia Relay

We’ve been working closely with Dartmouth’s Curricular Computing department to implement Camtasia Relay. It is now slowly being rolled out to faculty. It is a software based solution that records the screen of the lecturer’s computer, and audio from a microphone.

Gear details

• Camtasia Relay license
• Server to transcode the screencasts
• Computer (classroom or professor’s) for performing screencast
• Microphone – We are using a RevoLabs xTag

Pros

• Cheap compared to screencasting competition
• Site license can be installed on everyone’s computer
• Can also be used by a professor for other screencast needs
• LDAP authentication

Cons

• Requires professor to start, stop, and each recording (could conceivably be done by a TA).
• Doesn’t capture video. If the professor uses the chalkboard, you’re out of luck.

NCast Telepresenter M4

The NCast Telepresenter M4 is our latest addition. It is an appliance, permanently installed in a classroom. The biggest benefit of the Telepresenter is the automation it provides. Course recordings can be scheduled in advance. The recording starts, stops, and is published with no human intervention. In can also be controlled through a web interface, or through the room’s built-in A/V controls.

Gear details

• NCast Telepresenter M4
• Canon HD Video Camera (currently using component out)
• Shure wireless microphone (already a part of the room’s A/V system)

Pros

• Automatic start, stop, and publishing of videos
• Permanently installed in room means no set up and take down for each class
• Live streaming capability
• Supports HD Video cameras

Cons

• Expensive
• Management interface takes a while to get used to
• Still need to deal with lapel microphone (and batteries)
• Automation means you lose some flexibility (in case class starts late, ends early/late, etc.)

So after waiting over two years to dive into the cheap storage arena, we finally decided to go with a pretty boring solution. In this case, we hope boring will mean that it is a stable solution and easy to replace once our dream system comes along.

We chose to serve files with CIFS and NFS from a single server running Linux. Here are the details:

Aberdeen 48 bay storage server

We compared Aberdeen to several other similarly sized server sellers (Pogo Linux, Server Direct, Thinkmate, etc.) Pricing was almost identical, but Aberdeen was the only vendor to include a standard 5 year warranty.

We also looked at the price of building the system ourselves. Using all the same components as the Stirling X888, the price (before shipping) came in $2,274.95 (19%) lower. Putting systems together can be time consuming. We may consider it for the backup system, but for our primary, it is good to know that if anything goes wrong, we can call one company and get help.

We’re going to start out with twelve 2 TB “enterprise” SATA drives in a hardware RAID 6 with one hot spare. That allows us to lose two drives per 12 drive RAID group without losing data. And the hot spare will allow use to start rebuilding immediately.

The server will be running Ubuntu with an XFS filesystem. We’re a happy Ubuntu shop so this decision was easy. We’d prefer to use Hardy, the latest Long Term Support release, but due to some network driver issues, we’ll use Jaunty. XFS is mature, deals well with large filesystems, supports advanced ACLs, and has project/directory based quotas.

There will be no snapshots (LVM snapshots cause abysmal performance). However, we will back up all the data (probably via rsync) to a nearly identical system located off site. The RAID should protect us against hard drive failure, and the offsite backup should protect us against accidental deletion and server room disaster (fire, flood).

In the case of a disaster, the hope is that the near identical backup system can become the primary server… though we’ve yet to deal with the nitty gritty details of this.

Expandability – We’d love a system like clustered Samba or AFS where we can just add additional storage servers as our needs grow at unpredictable rates. Our way around it? Buy a server with lots of empty bays so we can populate it as we need the space. This system has a 96 TB raw capacity today. As drive capacity grows, that will go even higher. The system can also use additional Direct Attached Storage. There is a risk that we’ll have several clients hitting the single server simultaneously and performance will suffer. This is an unknown that we hope to test, and hope to not run into in practice.

Samba/winbind and the NFS kernel server integrated with our Active Directory environment will be the heart of the file serving. We have experience with both, but not at this scale. The hope is that because these are widely deployed, we won’t run into any major issues. At the moment we don’t intend to offer SSH/SFTP or AFP.

High Availability Failover, or the lack thereof

Jordan and Dailey spent a lot of time investigating and testing High Availability techniques (heartbeat, STONITH, DRDB, etc). We could have chosen to have two head ends, or two fully redundant servers, and if one were to fail we would automatically promote the backup to the primary role. They got things working, but in the end, the failover technology made the system more complicated. Basically both servers have to share identities (IPs, kerberos keytabs, etc.) Then you have to set up reliable monitoring of system health, and kill the failed server, while promoting the backup. There are a lot of moving parts, and you need to be very sure only one server is in charge. This is one area where NetApp has earned its money.

If our NetApp is down, the productivity of Thayer School faculty, staff, and students takes a serious nose dive. So it is easy to justify an active failover head. With our cheap storage system however, the hope is that in the event of a server kernel panic our researches can still be productive with other things while we bring the system back up. Of course, the plan with a simple system is that failures such as kernel panics are very rare.

Further down the road, hopefully I’ll update this series of posts with our experience actually receiving the system and getting it set up.

There are many reasons to pay for the reliability, rich feature set, and fail-over capabilities of our NetApp. The NetApp is an excellent product. However, there is also a significant amount of data at Thayer that just doesn’t need all the advanced features. For instance, we have research groups with multi-Terabyte data sets that just want their data easily accessible. If it were unavailable occasionally it wouldn’t be a huge deal. Other features such as snapshots are not critical.

If we can store these larger data sets on a cheap storage alternative, we can minimize the size and cost of our NetApp.

Design philosophy

After stalling for a couple years, hoping something would come along, we have decided to move forward with a plan that makes many compromises, but should meet the basic requirements for many of our storage needs.

We initially started the search to replace the NetApp altogether feature for feature. However, over time we lowered our standards to requirements that meet most of the needs for simple storage of lots of data. As our research continued, our philosophy on what product to choose evolved. In the end, the three main philosophical items we looked for when choosing a cheap storage solution were:

1. Widely used technologies and standards – If lots of people are already using the technology it is more likely to be stable. It also is likely to be around in a few years. While little start ups may have cool technology, we really don’t want to trust our data to a half-baked product. Little companies can also get gobbled up or go out business overnight, never to be heard from again.
2. Healthy communities and companies that are committed to the technology – In case we do have questions, or run into an issue, we want an active community of people that will be able to knowledgeably answers our questions. It isn’t enough that the technology says it supports CIFS and Active Directory. If it doesn’t work quite right in _our_ environment, and nobody seems to know why, we just can’t be comfortable moving forward.
3. Simplicity – Simple solutions have less chance of breaking and are easier to fix in the event that they do break. It is easy to build several layers of redundancy, but end up with a system that is more likely to break because of all the moving pieces.

What we considered

ZFS on OpenSolaris/Solaris

Link

We actually own a Sun Fire X4540 (Thor) running Solaris. We are currently using it as our online backup for the NetApp. It seems like a nice piece of hardware and the ZFS feature set is amazing. Snapshots, filesystem compression, expandability, checksumming, and the rest are all great.

• Technically a home run.
• Doesn’t score so well on our three design philosophy requirements.
• Reasonable, researched questions in forums go unanswered.
• Sun Engineers try to be helpful, but can disappear for weeks at a time.
• Sun shunned the Samba project and instead decided to implement their own CIFS server.
• Projects lack focus. There are several different editions of Solaris. They appear to be attempting to clear this up, but I really wish their “Project Indiana” broke free of a lot of the legacy issues weighing them down.
• Storage systems must be purchased fully populated.
• We could not get AD group integration working and ID mapping is complex.
• Even the latest build of OpenSolaris makes you feel like you are a Sys Admin from 1999.
• Sun laid off many employees. They have lots of projects they are trying to maintain, but seem to be having trouble keeping up with them.
• To make things worse, they got gobbled up by Oracle. How important Solaris is to Oracle remains to be seen.

Sun Storage 7000 Unified Storage (Amber Road)

Link

Sun realized there were admins like us who wanted ZFS but didn’t want to admin Solaris. So they wisely made a storage appliance. They did a great job of putting a pretty face on top of ZFS and Dtrace and then went and charged way too much for it.

• Price is lower than NetApp and EMC, but not low enough to build a decent sized userbase and build some critical mass.
• Can’t replicate from an Amber Road device to our existing Sun Fire X4540.

ZFS on Nexenta

Link

Nexenta seemed like it could be a good compromise for some of our Solaris concerns. Like Amber Road, it hides all/most of the Solaris administration and allows you to use ZFS like an appliance. We set up Nexenta on test systems on multiple occasions and it shows a lot of promise. Again a home run technically, but falls apart on our design philosophy.

• The company is very small without a very big user base.
• If you have any Solaris specific technical problems, they basically say, “go talk to Sun”.
• The depth and breadth of their documentation needs some major improvements.
• Fairly inexpensive (good educational discounts) but when building cheap storage, every additional dollar pushes up the cost per gigabyte.
• They are working like mad to add advanced features like failover.

Came very close to giving them a shot, but they just need a little more time to fully bake.

Clustered Samba

Link

I hope this project gains momentum as we are really drawn to an open system that we can grow on demand by adding another server full of hard drives as our storage needs grow in fits and starts.

• Still early and very little documentation
• Uses clustered file system which are still maturing and poorly supported on Ubuntu

Netgear ReadyNAS

Link

Netgear sells the ReadyNAS, a decent Linux based Network Attached Storage. We bought a 6 bay ReadyNAS Pro which we currently use for network based Time Machine backups for a few Macs. Bascially we wanted to get some mileage with a Linux based NAS.

• Decent web management interface with easy AD integration.
• Seems to have better throughput than most small business NAS systems.
• Tech Support is friendly but very bureaucratic. We ran into a bug when trying to add user and group permissions to a share. Getting tech support has been very difficult. You basically have to give them your first born child in order to be raised to the higher level of support. They really need to say, “yes that is a bug, we’ll fix it and get back to you”. Instead, they had me do dozens and dozens of unrelated tweaks to no avail. When I was finally suppose to get the next level of tech support, my ticket somehow got closed and I basically had to do the dozens of task all over again under a new ticket (which is still unresolved).
• The largest product they sell is 12 bays. Probably not big enough for us unless we used multiple systems.
• It uses EXT3 filesystem, so quota is user and group based instead of directory/project based. (can be worked around with lots of groups).
• Dollar per gigabyte is very good (currently around $0.53 per usable GB for the 3200 with 12 2TB drives in RAID 6 with 1 hot spare).

Btrfs

Link

A new, GPL filesystem originally developed by Oracle. All the fancy features that we want, baked into the Linux kernel. Fingers crossed that in a couple years this will have stabilized to the point that it is ready for production use. It seems like it has the most promise of competing with NetApp’s WAFL and ZFS. Oddly enough, Btrfs is now owned by Oracle. Will Oracle continue to develop both ZFS and Btrfs?

Other

We looked at numerous other technologies in varying levels of detail. None seemed to quite have the momentum at this point, and several were just not designed for a typical file server. We’ll keep an eye on them.

Some of the projects include:

• IBRIX – now owned by HP
• LeftHand Networks – now owned by HP
• Cleversafe
• Parascale
• Caringo CAStor

Read part three… what we actually chose to go with (warning, it is pretty anticlimactic).

The cost of our NetApp storage is a little fuzzy. Over the years, we’ve come up with calculations from around 2.50 – 12 dollars per gigabyte. But if you head over to NewEgg and buy a stand-alone 2 terabyte enterprise SATA drive, it is only $0.16 per raw gigabyte. Obviously there is a fair amount of additional hardware and software required to make that drive available on the network. So the question is, how can cheap can make a file server that still has respectable features, availability, and performance.

We’ve planned a system that comes out to about $0.28/GB raw and $0.37/GB usable. If the system works well, our plan is to buy a near identical system that we’ll use for backup. So backed up, the cost will be double, or about $0.84 per usable gigabyte. While this is many times more than the cost of a raw drive, it is significantly cheaper than the alternatives that we explored.

Over the next few blog posts, I’ll give some details of the options we investigated, what we’ve ultimately decided to go with, and how the implementation actually works out.

Update: The original price per GB I used was assuming we would buy a spare part of almost every electronic component (motherboard, CPU, RAM, etc.). I’ve updated it to reflect the price without any spares parts, which is still slightly misleading, as we will buy some spare parts. The price is also based on a fully populated system. However, we will initially only have a system that is 25% populated. At the beginning, this pushes the $/GB up, but because $/GB of hard drives is going down over time, by the time the system is fully populated, it will actually be lower than above quoted price.

Go to part two of the Cheap Storage Project series.

The new Thayer School Dropbox is a way to quickly and easily get your file sent. Once you visit the site and authenticate, you just choose the file and upload it to the server. You’ll then be given a link to send to the recipient – or just have the Thayer School Dropbox do it for you. The recipient can then visit the link and download your file.

Right now, files can only be uploaded by Dartmouth community members (although anyone can download). Soon, we plan to add the capability for Dartmouth community members to generate a “slot” where an outside person could upload to them.

To use the Thayer School Dropbox, just visit https://dropbox.thayer.dartmouth.edu/ and follow the easy instructions. We hope you find this to be a useful service and welcome your comments on how it could be improved and expanded.

The “no questions asked” maximum quotas for graduate students remains at 5GB and 10GB for faculty and staff. As always, if you have reasonable case for needing more space, just let us know.

Full details about ThayerFS quotas are here.

And each dish is hidden underneath a towel.

And written on the towel is only the broad food group that the meal underneath belongs to: Meat, Grain, Pudding — I’m pretty sure pudding is a food group, no?

And you aren’t really looking at the actual dishes, but a black & white photograph of them. And . . . well, you get the idea.

So when it comes time to actually eat the meal you’ve chosen, sometimes it doesn’t taste quite as you’d envisioned. And this isn’t always a bad thing.

Take the Tuesday afternoon session titled simply Information Classification. The paragraph description promised a tale of how Prince George’s Community College in Maryland undertook the process of classifying and assigning security rights to all data gathered and maintained by the school.

Not quite heart-quickening stuff. But I was intrigued enough by the daunting scope of such a thing to want to hear more. Also I was curious to see how relevant the take-away would be to Thayer’s own data security efforts. So I tucked in . . .

Ajay Gupta, the Director of Security at PGCC, is engaging and funny, and spoke quite eloquently about their mammoth endeavor with a deep grasp on subtle (& not so) implications that are inherent in applying a rational order to chaos and madness.

Amazingly they did it. & he showed the spreadsheet to prove it. Wow. Lots and lots of data records to classify, let me tell you.

But he said something very interesting during the question session. He had talked a lot of the classifying and assigning ownership and abstract access rights, but almost nothing on enforcing the policies. What were the plans and tools in play for this brass-tacks part of the whole affair? Because for me, as primarily technical & support-focused, this was the filling of the pie. Yes, I am very dessert-driven.

His answer was short: “We’re not allowing any data to reside outside of the ERP system.”

I swear I could hear the needle slide off the phonograph.

Oh well. So much for my dessert.

By the way, just what the heck is an ERP system anyway? (Kidding!)

And this was a group of multi-media technologists, demonstrating the Guns N' Roses method of lecture capture:

As you can see this was a popular session. (They apologized for not quite adhering to rules as Axl Rose is technically in Malibu, CA, which is not quite the same as being deceased.)

(Aw you got me! These were actually from the Lenovo-sponsored halloween end-of-conference party at an artificial bar-and-music theme-park midway called Universal CityWalk. The scene was replete with costumed stilt-walkers and stunningly bad celebrity impersonators (except the John Belushi-as-Blues-Brother, who was dead-on). Sadly, it was much to dark too take any more iPhone photos of the event. But who knew education technologists could let loose so?)

(I’ll be picking out a few sessions that particularly stand out to write about in more detail, but I didn’t want anyone to think all I was doing was napping or playing iPhone Labyrinth.)

• Generalized Exhibition Hall wanderings
• Library: REPLAY: An Integrated and Open Solution to Produce, Handle, and
  Distribute Audiovisual Lecture Recordings
• Meeting: EDUCAUSE/Internet2 Computer & Network Security Task Force (Open
  Meeting)
• Teaching and Learning: Tomorrow’s Students, Today’s K–12 Digital Learners: Are You Ready for Them?
• Teaching and Learning: The Future of Instructional Computing Labs
• Discussion Session: Openness
• Leadership and Management: Breaking Through Technology Barriers: Creating
  an Effective IT Communications Program with a Limited Budget
• General Session: Why IT Matters: A President’s Perspective on Technology
  and Leadership