Upgrading Linux SOC firmware on Amulet DXip rack cards

I've banged on about Amulet a lot in the past - it really is the best KVM-over-IP implementation of Teradici there is. One of the reasons for this is their external DXiP cards and rack system that allows you all the benefits of Teradici without having to stick in a PCI-e card (either for reasons of turnkey compatibility or no spare slot!). We've installed hundreds of seats at broadcasters, VFX, post and educational facilities. 

The external card although compatible with all other Terdici-based gadgets is a much more complicated gadget than an internal one. The reason for this is that when installing on a PCI-e bus you have the bus and OS to host the audio and USB chips; no drivers are required (it's just another Realtek audio device and USB hub) but in the case of an external card with analogue audio i/o and USB in you have to have an additional Linux System-On-Chip machine to handle those functions.

Upgrading the Teradici firmware is a cinch, either from the web interface of each card or using the Teradici Management Console but upgrading the SOC firmware is a different matter!

The guys from Amulet took me through the upgrade on a couple of our demo cards but that was a month ago and I hadn't really made any notes and so I spend yesterday afternoon familiarising myself with the process. 

Here are my notes with particular attention to the gotchas that had me scratching my head! The principle is that you use the serial header on an external rear-card (they call it the "personality" card), and power it from a bench supply (22v with at least 0.75A - don't set the current limit any lower!).

For networking connect to the AUX port - although when the card is in it's usual mode the AUX and MAIN network ports behave exactly the same, for uploading new firmware you need the AUX. 

Position yourself such that you can hold the ident button on the front of the card and push the power button on the front of your bench-supply. You should then be able to swivel to your computer whilst keeping the ident button held in so you can reach the keyboard.

You probably don't have a serial port on your computer (definitely not if you're rocking a Mac!) and you will need wired ethernet. 

So - fire up your favorite serial-terminal program (doesn't the whole world use PuTTY?!) - CoolTerm if your on OS-X and make sure you have your USB-serial adaptor (I use the Keyspan) with the following serial parameters set;

However, this is where my first gotcha reached up to bit me in the backside! Using PuTTY (I run Windows on my Mac in Parallels) I could see the serial terminal output from the card, but could send keypresses back to it until I realised PuTTY sets it's own serial parameters and ignores the port settings in Windows!

So - make sure it hasn't re-set flow control to XON/XOFF - it has to be None. If you want a little RS232 primer then Hugh and I did an Engineer's Bench podcast on it a while ago.

So, with this all set, power up the bench supply with the ident button held in and keep it held down until you've read to the end of this section! You should see the following in the terminal window;

When it gets to the hit any key to stop autoboot do that and only then can you release the ident button on the front of the card. If you release the button any sooner it disables the networking on the card which is a bummer 'cause you need that for the card to reach out and acquire its new firmware!

Now, notice a couple of things in the terminal; the IP address of the network port on the card AND the expected IP address of the TFTP server that is going to provide the firmware images. You could reset those by using;

setenv ipaddr_GT xxx.xxx.xxx.xxx (the DXiP-2 IP address)
setenv serverip_GT xxx.xxx.xxx.xxx (the TFTP PC IP address)
setenv gatewayip_GT xxx.xxx.xxx.xxx (is the network gateway IP address)
ipc GT flash (This write the IP settings to flash)

But, for my money life is too short and I'd rather just set my computer's IP to match the requirement for the TFTP server.

So, TFTP server - there are lots to choose from including TFTP32/64 for Windows but since OS-X has one built in (although it's a faff to configure - use Fabrizio Larosa's front end)

I've got the UNZIP'ed archive of the package from Amulet in the TFTP root.  Now the best thing to do it make sure the card can see the server - although it is a very minimal kernel it does have PING;

I did this screen grab with a different card that was looking for the TFTP server at 192.168.200.161 rather than the 192.168.200.127 shown in the rest of these notes.

The card itself will not respond to PINGs whilst in this mode (which rules out using my favorite multicast PING 224.0.0.1 which everything should respond to!)

So - you're logged into the card over RS232, you've got your TFTP server running (either on the same machine you're terminal is on or some other machine on the same LAN - remember the requirement for IP addresses - in this case everything is 192.168.200.x) now you can tell the card to go and collect the new boot-loader using the command RU

This happens quite quickly - the card retrieves the new boot-loader from the TFTP server and then does a reset; as soon as you see it doing that press the ident button and keep it held in exactly like the first part of the process.
Next it's time to re-flash the Linux SOC image, to do this use the command RL

This take a few minutes. Once it's done you can verify all is well with the command V

The Image Name line should match the CN-version you got from Amulet.

CN-887 is the current version

The next thing to do is take the card, put it in a DXip chassis and test it works as a proper Amulet sender. This firmware improves USB stability for bad-behaved devices, improves audio noise and stops the requirement for the occasional card-reboot when re-starting a machine.

Amulet with the Linux Wacom driver under sub v3.8 kernel

Amulet are a great company - I believe their KVM-over-IP extender to be MUCH better than Avocent, AdderLink and ThinkLogical's offerings, but like everyone else, once you packetise up DVI and USB and send it over a network you are going to have to deal with funny effects of devices and drivers that play fast & lose with the specifications. This is from their engineering team;

Wacom tablet and Red Hat Linux has been traced to a bug in the Red Hat Linux Wacom driver. The driver has a very obvious bug in it whereby outward data is declared as inward. The Wacom tablet simply ignores this when it is connected directly, although this works it is actually because the tablet USB stack is sloppy. The rack card however cannot ignore anything else we wouldn’t be able to operate at all so it believes what the Red Hat driver is saying and waits for an event which then never happens.

As detailed below in our analysis of the Red Hat and main Linux code drivers this issue has been resolved in the latest Linux build, with the fix implementing exactly what we would expect to see. This however has not yet found its way in to Red Hat, although may do so later this year, again as per the details below.

To enable use of the Wacom tablet on the current version of Red Hat (5.3) you are using we have added a work around to the rack card firmware. It appears to work but will require testing in the lab  over the next couple of next week before we can release. The workaround has to circumvent many checks in the code that are designed to catch driver bugs just like this. As a result the workaround is very specific to Wacom devices and this exact bug so we can’t guarantee it will work if anything changes.

Red Hat 6.3 (Amulet Hotkey test version)

• Red Hat version - Red Hat 6.3 (Santiago)
• Kernel version - 2.6.32-279.el6.x86_64
• RPM package - 2.6.32-279.el6.x86_64
• Wacom driver source - wacom_sys.c
• Driver bug - Incorrect bitmask causing incorrect data direction expectation in USB driver.

Code copied below shows the addition of the correct bit mask command (in red) in the latest Linux driver

• Code in current driver - USB_REQ_GET_REPORT, USB_TYPE_CLASS | USB_RECIP_INTERFACE
• Code in latest Linux v3.3 driver - USB_REQ_GET_REPORT, USB_DIR_IN, USB_TYPE_CLASS | USB_RECIP_INTERFACE

Red Hat 5.3 (Client version)

Although versions numbers differ to Amulet Hotkey test version the Wacom driver is the same and has the same bug present

• Latest Red Hat release (6.4)
• RPM package - kernel-2.6.32-358.el6
• Wacom driver source - wacom_sys.c
• Driver bug - Still present as per RPM package 2.6.32-279.el6.x86_64

Main Linux code base

• Version - v3.2.45
• Driver bug - Present
• Version - v3.3
• Driver bug - Resolved

Future Red Hat 

• Wikipedia gives the time frame for the next major RHL release (7.x) as Q3/4 2013
• We can't be sure whether this would incorporate the fix to the Wacom driver however Wikipedia states "Red Hat Enterprise Linux 7 (?), 2013-6+ Will be based on Fedora 18,[17] which as of March 2013 uses Linux kernel 3.8.[18]". This would indicate that the upcoming release of Red Hat 7 will fix the issue, as it will use a newer kernel, v3.8

Adventures with the Raspberry Pi

Unless you've been under a rock for the last few months you can't have failed to have seen the Raspberry Pi; that credit-card sized ARM-based computer that sells for £25. Although it's modestly powered it does stand up as a small Linux computer for server, desktop or media-centre use. Remember; we had servers twenty years ago when no computer on Earth was this powerful!

So I've been monkeying around with a couple of these boards for a week or so and here are a few observations. To start with a few initial notes;

• Make sure the SD card you use is both properly formatted and has a valid OS image on it; I battled with one board for days before re-formating it and sticking a new copy of Debian on it. The board literally looks dead if the boot-loader is corrupt.
• I didn't find any of the problems a lot of online folks claim is an issue - power supplies; I've tried everything from iPhone, Kindle, no-name USB through to the USB-service port on the back of my TV. All powered the boards fine. I haven't measured it I imagine we're talking less than an amp at 5v.
• It's a lot easier to us one of the many tools to format the card and copy on the OS image. I've been using RPi-sd card builder v1.1

So - there are numerous pre-compiled OS images for download and I suppose it depends on what you want to do with it. My first port of call was a Debian build called "Wheezy" which seems to be the general Linux distro of choice that comes with the KDE desktop. The card must be partitioned into a FAT32 formatted boot partition (64 megabytes) and the rest of the card (at least two gigs) as an ext4 Linux partition. YaST allows you to max-out the main partition when it runs. So long as you know what IP address gets assigned at boot-time you can SSH into the board;

So with Debian installed you can use it as a proper desktop machine or a network server. There are two USB2 ports and so it makes for a very powerful NAS head for a regular USB drive.

The other application that seems to attract the most attention is XMBC for which there are several builds. I stuck OpenElec onto another SD card (this is the joy - you can switch £5 SD cards around and you've got a new machine).

The XMBC builds don't come with a desktop or even an X-client (it writes straight into the display buffer - and it has MPEG2 & H.264 decoding on the board). But it is just the job for a Media Centre.

Another feature of the OpenElec build is support for the Apple Airplay protocol so you can "throw" media playback to it from iTunes or iPad/iPhone;

Several folks online have commented about the poor output quality of the analogue audio 3.5mm jack. The HDMI audio is fine and although the mini-jack ain't great I discovered a couple of things that improved it to tolerable;

• Don't use a USB hub to power the Pi - most don't have a great regulation of the 5v rail which also tend to be noisy. A Kindle or Apple wall-wart suffices.
• Use a good-quality USB cable; I had a cheapie cable powering the board and it covered the audio output in hiss. A Kindle cable did the job.