Max's Chips and Dips: The USPS uses Opal Kelly's FPGA USB 2.0 Modules in a Real-Time Process Control ApplicationThe problem lay in creating a sophisticated control unit that could perform real-time data-processing and control operations at blinding speed; to create this unit in such a way that it could be linked via a hardware interface to the mail processing machine on one side and via USB 2.0 to a PC running Linux on the other; and to achieve all of this in a very tight timeframe...
In these hard economic times, everybody is trying to do more with less, and the United States Postal Service (USPS) is no exception. Recently, it became necessary to create a new type of mail processing machine to be installed at over 200 USPS mail hubs across America. One solution would have been to create a new device from the ground up at an estimated cost of around $400K per unit.
As an alternative, it was decided to refurbish existing mail-processing equipment that had originally been created for quite a different task. These existing units had already been decommissioned and warehoused. Instead of scrapping these machines, it was realized that they could be modified to perform the new task for only $60K per unit, thereby saving the USPS (and the American postal customer) close to $70M.
The problem lay in creating a sophisticated control unit that could perform real-time data-processing and control operations at blinding speed; to create this unit in such a way that it could be linked via a hardware interface to the mail processing machine on one side and via USB 2.0 to a PC running Linux on the other; and to achieve all of this in a very tight timeframe...
The Increasing Demand for Speed and Efficiency
From watching old films on television, many of us continue to cherish a mental image of our local postmaster casually sorting the day's mail by hand into wooden cubbyholes in preparation for a mail carrier to deliver it to our homes. Things just don't work this way anymore. Today, the USPS is the third-largest employer in the country, behind the Defense Department and Wal-Mart, with more than 34,000 facilities, a fleet of over 220,000 vehicles (the largest civilian fleet in the world), and over 600,000 employees.
Processing an average of around 670 million pieces of mail a day, USPS carriers deliver to more than 149 million residences, businesses, and post office boxes. Not surprisingly, performing this task quickly and efficiently demands hard work and dedication from USPS employees backed up by the latest and greatest in technological infrastructure.
When you post a letter, it is quickly transported to the nearest major mail hub. As soon as it arrives at the hub, the mail is fed into a machine that captures an image of its handwritten or printed address. This image is transmitted to powerful computers that perform incredibly sophisticated Optical Character Recognition (OCR) in real-time. Based on this, additional software determines the destination the originator of this piece of mail had in mind and accesses a national database to "fill in the gaps". The result is translated into a POSTNET barcode that is printed onto the mail as it passes at high speed through the machine. This barcode is subsequently used to guide the mail throughout the remainder of its journey.
Next, the mail is sorted with respect to whichever major hub is closest to its ultimate destination and it is transported to that hub. In the not-so-distant past, the mail would eventually end up at some main local mail facility. Here, the mail would be fed into a Carrier Sequence Bar Code Sorter (CSBCS) machine, which would read its barcode and sort the mail into the Carrier Route Sequence – that is, the mail would be sorted based on the order in which it would be delivered by the carriers serving that community.
The old CSBCS machines were around 6 feet wide, 4 feet tall, and 12 feet long. Approximately 4,000 of these units were deployed at local facilities around the country. These machines performed their duties admirably for many years, but time moved on, and they were eventually superseded by new technological developments. Now, the task of sorting the mail into the Carrier Route Sequence is performed at the central hubs. Each hub is equipped with an advanced Delivery Barcode Sorter (DBCS) machine. The size of these monster devices varies depending on the installation, but they can easily be several hundred feet in length.
Figure 1. The original, decommissioned CSBCS machines in the warehouse
Following the introduction of the DBCS machines, the original CSBCS units were decommissioned and stored in a warehouse facility in Topeka, Kansas. Destined for the scrapheap, their future looked bleak until...
We Have a Problem
Although the modern DBCS machines are capable of sorting 40,000 pieces of mail per hour, they can perform this task only if they can actually read the barcode labels that have been sprayed onto the mail. Unfortunately, around 4% of the time, the barcodes are unreadable. In some cases this is because a piece of mail has images and/or too much writing on it, which results in too little space to spray the barcode. Alternatively, the mail may be colored, which means that there isn't enough contrast to read the barcode without error.
This failed mail has to be processed by hand at a rate of only 300 pieces per hour. In addition to being time-consuming, prone to error, and – let's face it – not a lot of fun, this hand-sorting has to be formed again and again at the various hubs and local mail facilities – typically four or five times before the mail reaches its ultimate destination.
In order to address this issue, it was decided to create a new machine. The task of this machine would be to process the rejected mail; to re-scan the original address and transmit it to the computers that perform the optical character recognition and determine the destination; to apply a sticky label to the mail, and to spray the POSTNET barcode onto this sticky label. This sounds easy if you say it quickly, but it becomes a little trickier when you discover that this is all occurring while the mail is being transported at 4 meters-per-second.
Initially, the thought was to build this new machine from the ground up. But early evaluations showed that this would cost around $400K per machine, which would have been far too expensive...
The "Green" Solution
As an alternative, Brent Raney, Manager of Technology Development and Applications, and Mike Amato, Manager of Engineering Software Management at USPS Headquarters Engineering, championed the idea of retrofitting the existing CSBCS machines. In addition to providing major cost savings to the USPS, this "green" solution would also be much kinder to the environment. A team of "in-house" engineers was commissioned to design the upgrade. Meanwhile, the USPS Central Repair Facility in Topeka, KS, was tasked with performing the refurbishing effort and applying the upgrade to the old equipment.
The project required a mixture of mechanical, electronic, and software design disciplines. Much of the original CSBCS machines – the feeder, transport belt, optical reader, and sorter – could be reused. A new section was to be added to the existing machine to perform the enhanced functions. This new section was to have its own dedicated electronics and computer for control.
Jerry Pender, Electronic Engineer, was tasked with the electronic portion of the design, which was to be based on a Field Programmable Gate Array (FPGA). This was because the massively parallel nature of the FPGA could satisfy the requirements of the computationally intensive control and data processing algorithms in real-time. Meanwhile, Jerry's colleague, Bill Storey, was to develop the computer control software. "We needed a PC platform computer interface, but didn't want to spend a lot of time and money designing one," says Jerry. Jerry had designed specialized ISA and PCI bus interface cards in the past, but felt that there should be an easier way. He had been toying with the idea of using USB 2.0 as a machine control interface for quite some time. In researching the idea, however, Jerry found that there are a lot of pesky little details involved with USB. "On the surface USB seemed so simple, but once you got into it, it is so much more involved," he notes. Jerry knew that if he could somehow couple USB 2.0 with an FPGA, it would be the ultimate implementation platform and a match made in heaven.
Jerry started to research the problem, and he quickly discovered Opal Kelly. Opal Kelly's FPGA Integration Modules connect a user's hardware design implemented in an FPGA, directly to the user's software implemented on a PC. Opal Kelly handles all of those pesky little USB details, such as the enumeration process, low level drivers, and bus timeouts. They even provide a control panel software package called FrontPanel, which allows testing your hardware design (to some extent) at your desk without using the actual application software, and building some level of confidence before putting it into your actual application. Opal Kelly even used Xilinx platform FPGAs, which Jerry had been working with for years, so there was no learning curve. "I realized within seconds that this is exactly what I had been searching for. If I had written a wish list of everything I wanted, I could not have done better," comments Jerry.
Jerry used Opal Kelly's XEM3050 module in his interface to the PC, and Bill used Linux Fedora for the application software operating system. The system was required to track mail as it moved through the various processes on a transport running at 4 meters-per-second. Each piece of mail needed to be monitored in real-time so that the correct mail address was used.
Figure 2. Jerry Pender shows the Opal Kelly XEM3050 module in the new machine.
Linux is not a real time operating system, and even high-speed USB 2.0 has inherent latencies, so how is this combination capable of performing real time process control? "The answer lies in the design of the application," says Jerry. "We made sure that all incoming events are time-stamped (or in this case, position-stamped) in the FPGA before they are passed along to the control software as inputs. All outgoing data and control signals from the control software are sent to the FPGA in advance, where external real time signals are then used to trigger the functions at the appropriate time."
"The only requirement is that the CPU and control software be fast enough to process these events without missing any deadlines," notes Jerry. "Our transport moves at the rate of 10 mailpieces per second, and there can be up to 20 mailpieces in the processing queue at any given time. Bill has run timing analysis software along with his control application software, and has found the processing speed to be significantly more than adequate for the application." The FPGA also allowed the team to build a simple mailpiece image capture system which was transferred to the Linux system via the USB interface.
One really useful aspect of the Opal Kelly module is that it allows both data and control signals to pass into and out of the computer, all over a single twisted pair of USB wires. As Jerry points out, "We never even had to take the cover off of the computer." Opal Kelly also provides simple mechanisms to monitor and/or control signals, monitor and/or control events, and receive and/or output data.
Figure 3. The team with one of the retrofitted machines (from left: Jerry Pender, Brent Raney, and Mike Amato)
Jerry continues, "Without this system, we would have had to purchase more expensive hardware and develop two special purpose boards." Jerry estimates that that going with the Opal Kelly solution reduced design and debugging time by six months. The team commenced the project in September 2008, and they had a fully working prototype up-and-running by January 2009. At a cost of only $60K to retrofit an existing CSBCS machine, this solution – when applied to the 200 units that will be deployed in USPS mail hubs across America – will save the USPS (and the American postal customer) close to $70M. (For those who are interested, the Opal Kelly XEM3050 module costs just $750, and the FrontPanel software – which runs on the Linux, Mac, and Windows operating systems – is included for free.)
As Jerry says, "There is beauty in simplicity. Using the Opal Kelly module made things really, really simple. It has been a pleasure (even fun) to work with this solution! (Whoops please don't let my boss hear that!)"
Clive (Max) Maxfield is the author and co author of a number of books, including Bebop to the Boolean Boogie (An Unconventional Guide to Electronics), The Design Warrior's Guide to FPGAs, and How Computers Do Math; featuring the pedagogical and phantasmagorical virtual DIY Calculator. In addition to being a hero, trendsetter, and leader of fashion, Max is widely regarded as being an expert in all aspects of computing and electronics (at least by his mother). Max was once referred to as "an industry notable" and a "semiconductor design expert" by someone famous who wasn't prompted, coerced, or remunerated in any way.