The first step was getting community feedback.  We reached out to teaching faculty and students, and received several suggestions for making the rooms a better teaching and learning space.

We decided to use this feedback to upgrade a single room, Cumming 202, refine it, then replicate what works to other classrooms.  We are currently in the process of upgrading Cummings 105 followed by Cummings 118 and Cummings 200 during winter break.

Here are the changes we’ve made to Cummings 202:

Lighting

Old hanging lights

Old hanging fluorescent lights have been replaced by recessed LED lights.  Each light has occupancy and ambient light sensors and is wireless controlled using built-in mesh networking.  In the future, we can reconfigure how many switches we have, and which lights are associated with each switch.  Previously the room switches controlled the lights in a left and right group, which didn’t make sense.  We now have them set to control the front and back of the room so you can turn off (or dim) just the front while projecting.

New LED lights

Writing space

We greatly increased writing area by installing two huge 16′ x 5′ whiteboards.  So big, they had to come in through the window. We’ve turned the classroom 180 degrees, but the chalkboard remains in the back for now.

Projection

When projection screen is in use, the right half of the marker board remains available

The most common piece of feedback we received from faculty is the desire to project and write on the board at the same time.  To accommodate this request, we’ve put the new projection screen on the left, leaving a big area of whiteboard on the right.  The screen itself is now widescreen to match the widescreen projector and most laptops.

While motorized screens are not that much more expensive, we stayed with a manual projection screen in C202.  The faculty we heard from felt the manual screen is faster to put up and take down.

Furniture

Desks in a “U” configuration

desks grouped together

The huge, heavy tables have been replaced with smaller tables that have wheels and flip up for compact storage.  These tables should be much easier to reconfigure from class to class.  They can be put in groups or more traditional rows.  The chairs have also been replaced by wheeled chairs with adjustable height.

We’ve installed an adjustable height desk for the professor that allows you to sit or stand during lectures.

A/V Controls

A/V controls

We are using a custom designed touch screen interface for the room.  We spent a lot of time making the interface as easy to use as possible.  Even better, we made the projector turn on automatically and select the correct input when you connect your laptop.  There is also an occupancy sensor that will shut the system down automatically if you forget.

Power outlets

outlets all around

More power outlets for laptops was a common request in our student feedback. Installing floor outlets in Cummings wasn’t feasible, so we compromised by installing several outlets around the perimeter.

Recording appliance

Like our larger classrooms, we’ve installed a recording appliance in C202.  We are working on getting recording controls added to the touch screen

A/V Monitoring

A graph of the lamp hours on the projector, one of the several items we monitor.

We are now monitoring our A/V system.  This will enable us to provide preventative maintenance, and notice when systems are broken before you do.  We can’t monitor all components due to limitations of some hardware, but we hope our systems will be even more reliable.

Failed to perform space reclamation on machine desktop1 in Pool mypool

Once these started, we’d get them about once per hour for every desktop.

Back over on the View Connection server, we’d see the corresponding entries in its log (C:\ProgramData\VMware\VDM\logs\log-yyyy-mm-dd.txt):

WARN (08CC-09B8) [ServiceConnection25] Problem while performing VC operation: ‘Permission to perform this operation was denied.’ [com.vmware.vim25.NoPermission]
ERROR (08CC-09B8) [ServiceConnection25] Permission to perform this operation was denied.
ERROR (08CC-09B8) [PendingOperation] Error reclaiming space for VM /View/vm/view-pool-storage/mypool/desktop1 :com.vmware.vdi.vcsupport25.VmException25: Permission to perform this operation was denied.

We’d followed the instructions for giving permissions to the vCenter user used by the Connection server outlined on View Manager Privileges Required for the vCenter Server User, but there was obviously something missing. As an aside, some of the permissions given in the View documentation needed some “translating” for vSphere 5.1 – thanks to Terence Luk for his blog outlining this: Configuring vCenter role permissions for VMware vSphere 5.1 and VMware Horizon View 5.2 (View Manager and View Composer)

To figure out what was missing, we gave Administrator privileges to the vCenter user for one desktop and then initiated a manual reclamation from the View Connection server:

C:\Program Files\VMware\VMware View\Server\tools\bin>vdmadmin -M -d mypool -m desktop1 -markForSpaceReclamation

The reclamation worked, and in the event log on the vCenter server, we could see Flex-SE wipe and shrink operations for that desktop. Looking at the role permissions in vCenter, we found the culprit. The vCenter user needed to have this permission added to work properly:

Virtual Machine -> Interaction -> Perform wipe or shrink operations

Once this was added to the role for the vCenter user, all desktops were able to perform reclamation happily, and there are no more errors in the event log.

In our case we plan to have a professor doing a SolidWorks tutorial to an entire room of students.  We are likely to see all the users simultaneously do an operation that could stress many components. If 60 students were to all rotate a complex SolidWorks model at the same time, it will stress the GPU, CPU, and network.  Because of this, all the VDI sizing guides were pretty useless.  Our plan was do a little data collection, and then dive in with real equipment.

We purchased Liquidware Labs Stratusphere Fit. It comes as a VMware appliance and an agent that gets installed on each client computer you want to monitor. We installed it on our general Windows lab and our CAD Lab systems.

It has a bazillion reports you can run. You can see tables and graphs showing which applications are using resources and how much.  It did give us a general idea of how much RAM we would need, how busy our labs were, and which applications got the most use.

In the end, the data was interesting, but still not enough to have an educated guess at how VMware View would perform for us.  We needed to buy a server and just try it out.

We had no idea what the bottleneck was going to be.  So we wanted to get a very beefy system that still had room to grow.  We knew we wanted a system that supported NVIDIA’s K1 card for offloading the GPU operations.  This limited our options, as there are a limited number of systems that support the card.  Our existing servers are predominantly Dell or Supermicro based.  After evaluating the options from these two manufacturers, we ended up with the Dell PowerEdge R720.  In only 2U of rack space, it has impressive abilities. It can handle up to two NVIDIA K1 cards, has 24 DIMM slots, 2 CPU slots, and 16 2.5″ storage bays.  Along with the R720 and NVIDIA K1, we purchased a Teradici APEX 2800 card.  It basically does hardware transcoding of the video stream that gets sent over the network to the clients.  It works in conjunction with the NVIDIA K1 card.  Without these two cards, the server CPU would have to take care of all the GPU, video stream encoding, and client CPU operations.  It may work, but the number of simultaneous users could be much lower.

Our hardware stack. A Dell PowerEdge R720, NVIDIA K1, and a Teradici APEX 2800

We ended up with the following hardware:

• 2 – Intel Xeon E5-2670 2.60 GHz CPUs (Currently the fastest that Dell supports in a system with the K1 GPU card)
• 256 GB of RAM
• 6 – 200 GB SSD drives (we hope to serve everything off of local fast SSD storage)
• 1 NVIDIA K1 GRID GPU acceleration card
• 1 Teradici APEX 2800 LP card

We’ve just received the components and finished getting the VMware Horizon View environment set up.  This wasn’t exactly smooth.  Despite the software stack almost exclusively coming from VMware, there are several components involved.  We naively expected it to be a, “download this virtual appliance from VMware and start it up” operation.  Instead, there are many manual steps involved, including at least two Windows servers that orchestrate the system.  We are still wrapping our heads around the numerous buttons and knobs.

Once we get the  buttons and knobs mostly under control we’ll be back with part 3… real world(ish) benchmarking.

We offer a number of Windows labs with every application under the sun, but this is a limited resource, one that is often at capacity with a class or students trying to get a project finished. Our buildings are at capacity so adding additional labs is not an option.

We’ve guided students to install Boot Camp or a virtualization solution on their Mac (Virtualbox, Fusion, or Parallels) but it is a time consuming endeavor, both for us and the student.

We’ve long wished that we could allow students to run these big engineering applications remotely on our server infrastructure. However, we had concerns about being able to scale and user interface latency. How many servers would we need to accommodate a class of students? Would the app feel sluggish when rotating a CAD model? Until recently, the available solutions didn’t look promising.

However, with the latest release of Horizon View (VMware’s Virtual Desktop Infrastructure (VDI) solution), and NVIDIA’s GRID GPU, DirectX and OpenGL graphics operations can now be offloaded to a dedicated video card. We were intrigued. After our colleagues set us up with a non-graphics-accelerated demo on their VMware View server, were were sufficiently impressed to proceed with a pilot. We couldn’t find any solid information with estimates of how many users we could support on a single server. So we decided the best way to proceed was to buy some equipment and do our own benchmarking.

Our testing is just under way. We’ll follow this post up with our pilot set up and our findings on whether running resource intensive applications remotely is really feasible and cost effective.

Here’s an rc script that will run early in the sequence and create a multipath “link” between drives based on their serial number.

/etc/rc.d/manual_multipath

#!/bin/sh

# PROVIDES: manual_multipath
# REQUIRE: sysctl
# BEFORE: hostid

. /etc/rc.subr

name="manual_multipath"
start_cmd="${name}_start"
stop_cmd=":"

manual_multipath_start()
{
        echo "> manual_multipath script started"
        echo "> linking drives with the same serial number with gmultipath"
        counter=0
        serials=""
        devices=`/usr/bin/find /dev -maxdepth 1 -regex '.*da[0-9]*' | /usr/bin/cut -d '/' -f 3`
        for device in $devices
        do
                echo $device
                serial=`camcontrol inquiry $device -S`
                substring=`echo "$serials" | /usr/bin/sed -n "s/\|$serial\|.*//p" | /usr/bin/wc -c`
                if [ $substring -eq 0 ]
                then
                        found_multi=0
                        arg1="$device"
                        arg2="$device"
                        for newdevice in $devices
                        do
                                newserial=`camcontrol inquiry $newdevice -S`
                                if [ "$device" != "$newdevice" -a "$serial" == "$newserial" ]
                                then
                                        echo "  same as $newdevice!"
                                        counter=`expr $counter + 1`
                                        found_multi=1
                                        arg1=$arg1"$newdevice"
                                        arg2=$arg2" $newdevice"
                                fi
                        done
                        if [ $found_multi -eq 1 ]
                        then
                                gmultipath create $arg1 $arg2
                        fi
                fi
                serials=$serials"|$serial|"
        done
        echo "> manual_multipath script finished, found $counter matches"
}

load_rc_config $name
run_rc_command "$1"

Don’t forget to “chmod 555 /etc/rc.d/manual_multipath”.

Lastly, when importing a zpool from the drives you just multipathed, make sure to specify where to look for devices or you might end up importing a mix of multipath and regular devices. Make sure to “zpool import -d /dev/multipath”.

SSH

strengths: encryption / 1 command on the sender

weaknesses: slowest

command:

zfs send tank/volume@snapshot | ssh user@receiver.domain.com zfs receive tank/new_volume

NetCat

strengths: pretty fast

weaknesses: no encryption / 2 commands on each side that need to happen in sync

command:

on the receiver

netcat -w 30 -l -p 1337 | zfs receive tank/new_volume

on the sender

zfs send tank/volume@snapshot | nc receiver.domain.com 1337

(make sure that port 1337 is open)

MBuffer

strengths: fastest

weaknesses: no encryption / 2 commands on each side that need to happen in sync

command:

on the receiver

mbuffer -s 128k -m 1G-I 1337 | zfs receive tank/new_volume

on the sender

zfs send tank/volume@snapshot | mbuffer -s 128k -m 1G -O receiver.domain.com:1337

(make sure that port 1337 is open)

SSH + Mbuffer

strengths: 1 command / encryption

weaknesses: seems CPU bound by SSH encryption, may be a viable option in the future?

command:

zfs send tank/volume@snapshot | mbuffer -q -v 0 -s 128k -m 1G | ssh root@receiver.domain.com ‘mbuffer -s 128k -m 1G | zfs receive tank/new_volume’

Finally, here is a pretty graph of the relative time each approach takes:

SSH + MBuffer would seem like the best of both worlds (speed & encryption), unfortunately it seems as though CPU becomes a bottleneck when doing SSH encryption.

To say the least, Bonjour is not designed for an enterprise environment. Sill, we advertise our printers with it in an effort to accommodate Apple users.

Here is a script I wrote that will discover which printers are currently advertised on the network. We use it along with a Nagios check to make sure that our printers are present within the Bonjour noise. Writing it took me through the meanders of MDNS in Python & on Linux with multiple vlans. Let’s just say non-trivial.

Downloads

find_mdns_printers_1.0.tar.gz

Sample output

With the igb driver in FreeBSD, the mbuf cluster size needed is a mathematical formula involving the number of CPUs & the desired MTU. Unfortunately, it is currently hard set. On enterprise machines with many cores and higher MTUs, it is quite easy to reach this set limit. It will express itself with the following error message after an ifconfig:

igb0: Could not setup receive structures

This limit can be overridden with the following in /etc/sysctl.conf

kern.ipc.nmbclusters=131072
kern.ipc.nmbjumbo9=38400

These are the value that worked for 16 cores & an MTU of 9000.

While we’re at it, it took me a while to nail the exact syntax require for NIC bonding so here it is:

/etc/rc.conf

if_lagg_load=”YES”
ifconfig_igb0=”mtu 9000 UP”
ifconfig_igb1=”mtu 9000 UP”
cloned_interfaces=”lagg0″
ifconfig_lagg0=”laggproto failover laggport igb0 laggport igb1 192.168.0.123 netmask 255.255.255.0″

As far as I can tell, capitalization matters…

This is the fourth generation of hardware we’ve experimented with. Around four or five years ago, we started out with iPod Touches in custom made cases. They proved unreliable, so we moved to Linux based resistive touch screens. These took some work to get stable and are painfully slow. They’ve been working reasonably well, but we wanted to try something a little more modern and capable. The latest round uses an iPad and wall-mount enclosure made by nClosures Inc. We are still pulling data from R25 and creating a custom interface using HTML. We are using Kiosk Pro Plus as the kiosk app that displays the web page in full screen mode. If we had access to our schedule data with a standard calendar protocol, we might have used an off-the-shelf app like EventBoard.

iOS has made some simple improvements such as support for synchronizing time via NTP and locking screen rotation. However, there are still several limitations that may prevent us from deploying iOS devices widely. Remote management is still challenging or impossible.

Things that still are sub-optimal include:

• no automatic or remote rebooting
• can’t start an app upon power on
• can’t turn off or dim the display
• doesn’t always auto reconnect to wireless if the network is unavailable

We now have a Google Nexus 7 in the office. This has the advantage of being a great size and significantly cheaper than an iPad. Hopefully wall-mount enclosures for the Nexus 7 will come out soon (Armor Active has announced one coming soon).

What we do

Provide crazy good technology service.

Thayer School has a unique multidisciplinary approach to engineering education and our IT philosophy reflects this. Rather than just being back-room “crank turners”, our Computing Services team interacts extensively with faculty, students, staff, and researchers. We go out of our way to spend time with our users, deliberately encouraging community members (often with cookies) to ask questions, explain their needs, and brainstorm solutions.

We work hard to solve problems, implement solutions, and allow Thayer School to make the fullest use of computing technology. We are driven and love a challenge.

How we do it

OS: mostly Linux, some Windows and a little bit of everything else
Virtualization: Yes please. VMware is where we’re at
Scripting: A whirl of Perl, PHP, Python, with a sprinkling of Bash and dash of VBS
Authentication: CAS, AD, kerberos
Storage: NetApp, ZFS (Nexenta, FreeBSD, Solaris)
Protocols: TCP/IP, SNMP, NFS, SSH, Bonjour, VNC, CIFS, HTTP… (get your Wireshark on)

What we expect, and what you can expect

• We’ll ask for your opinion and expect you to have one
• The desire and ability to quickly learn new & old technologies
• The skills and chops to apply your knowledge building integrated systems
• Full participation within a small, collaborative, agile team
• Automation wherever possible so we can spend time on the fun stuff
• Building robust systems that won’t bug you off-hours
• Keeping up with the latest in tech and not being afraid of the bleeding edge
• Hands in all aspects of computing – full stack.
• Close collaboration with all members of the Thayer School community
• Enthusiasm for higher education; yours, ours, & everyone else’s

Work environment

• Not in a basement, daylight will reach you
• Top-notch engineering happening all around you
• Thriving engineering school within a prestigious college
• Northern New England as your backyard

Interested? Please click here to fill out the short form on the job page.