There's a disturbing statistic widely accepted today that two-thirds of all designs fail at first silicon. Of these failures, the primary failure mode in over 70 percent of the designs is functional failure. These failures can come despite weeks of testing and verification work, billions of vectors, and thousands of man-hours of effort to locate and repair all possible bugs and failures. The costs of failure reside not only in generating new masks and additional engineering time, but also in the opportunity costs of missing the largest part of the market window for a product. The later you offer a product in the market, the smaller the selling price and the lower the total product volume. All of this, quite naturally, is driving an intensifying need for verification tools and technology.

Although simulation is still the primary verification tool and hardware-accelerated simulation is picking up, many other tools are available to help complete the process. However, simulators are focused on gate events and clock-to-out details, not system functionality--a growing part of the verification challenge. Increasingly, designers need to work with the architectural models using higher-level languages such as SystemC. These trends are combining to cause overdesign and reduced performance metrics not predictable at the system level. Therefore, a good set of verification tools must be interoperable with RTL, software, and other languages--and be able to examine implementation details at the post-silicon level to evaluate real traffic. All of these combined as the factors currently playing into overall verification flow and strategies.

Claims to fame

The main claim to fame for all of the current verification-specific tools is the ability to perform tasks that are cumbersome or very difficult in the simulator. Increasingly, the simulators are running out of steam in handling the largest designs, even those targeted for programmable logic implementation. Hardware acceleration is one way to address the need for ever-more vectors, but few companies can afford the people resources needed for an emulation team, to say nothing about the costs of the hardware. (See Chip Design December/January 2004 for a Focus Report on emulation and acceleration, "Hardware Tools for Design" on p18.)

The explosion in the gate count due to device scaling has driven the functional verification phase of design to exceed three quarters of the design cycle. A part of the problem is the ability of the design engineer to create a large number of gates with relative ease. The designer has to prove that the design work. So in the case of a 4-input NAND gate, for instance, the designer would check one or two input combinations for a "0" output, and all "0" inputs for a "1" output. The verification engineer then has to check all 16 of the possible input combinations. Meanwhile, the high fan-outs generated by synthesis only makes the problem harder to solve, due to the increasing number of gates affected by a single output.

As designs grow in gate count, therefore, the number of vectors necessary to cover the design increase as a power of the gate count. Unfortunately, just getting the code coverage to an adequate percentage of instances doesn't ensure state coverage, or even pattern sensitivity checks. The addition of random (directed) vectors in an attempt to capture more cases adds incremental coverage, but still can miss many bugs, especially those associated with multi-cycle paths and cross domain race conditions. In addition, simulation-based verification has great difficulty in covering extended sequences of transactions.

One way companies have tried to address the growing bug problem is to increase the number of test cases in the regression suites. These sets of tests are ad-hoc tests for known (past) problems, while new cases are added almost continuously to the test suite every time another issue is exposed. The usual technique for adding to the regression suite is to take an existing test that covers a similar problem and add a new test for the latest bug. This process for developing new regression suites may add hundreds of millions of vectors, but unfortunately only adds a small increment in overall verification coverage.

In addition to the factor of growing regression suites, the verification teams have to check the design at multiple levels--systems at the ESL, and blocks at the RT, gate, and netlist levels. Many designs are checked at every level, further increasing duplication without improving verification coverage. Although formal equivalence tools can confirm functionality across transforms, there's a growing need today to track implementation issues at the various design levels; layout and congestion, test, manufacturability, yield analysis, as so forth. Engineers must remain wary; just because a block works at the block-level simulation doesn't mean that it can be built--or that it will work in the overall system.

General classes of verification tools include code analysis-formerly just lint, which is equivalent to grammar and spell check in a word processor, but now includes other checks as well-code coverage, debuggers, viewers for timing analysis, hierarchy, and logical interconnections (schematics), formal checkers, formal analysis and property and assertion checkers, and many other types of tools as well. The needed tools must work on everything from gates and clocks to hardware-software co-verification.

What are we looking for here?

Gulam Nurie, vice president of marketing at Atrenta Corp. (San Jose, CA) says there's a growing need for predictive analysis in the verification flows. Verification is expanding from simulation, and some accelerated hardware verification now includes formal verification in the form of assertions and properties. Companies need to do more checking up front, according to Nurie, before they spend too much effort in the implementation phases (synthesis, DFT, physical design and analysis, mask enhancements, and the likes). They need to be checking the design at the ESL, where many of the implementation issues can be addressed--is the design synthesizable, testable, and so one.

Agreeing with Nurie and adding emphasis to another area of concern, Steve Glaser, vice president of marketing at Verisity, Inc. (Mountain View, CA) comments that the design community is moving towards verification process automation as a true infrastructure for verification. Glaser says the idea is to drive the device under test (DUT) in hand, plus all of the specifications, towards closure. The problem, of course, is that the DUT and the specifications continue to change right up to tape-out.

The goal of verification is to solve these problems with one simple answer, but actual results are never quite that simple. Good verification requires a combination of technologies: functional coverage, assertions, code coverage, directed, random directed. The dispersed design and verification teams also need the ability to have distributed functionality.

Prakash Narain, CEO at Real Intent, Inc. (Santa Clara, CA) says the verification process consists of taking a lot of tools and attempting to cover as many paths to defect detection as possible. He says a combination of tools is more likely to find issues than a single tool, while there are many classes of design situations that aren't (easily) testable with a simulator, including race conditions or metastabilty across clock domains.

These changes in verification emphasis illustrate the magnitude of the problems that face large IC verification teams. One challenge for companies working in or near state-of-the-art technology is to modify the concept of verification from finding (and fixing) bugs to building assurance that the design space has no more existing bugs. Those companies know that a major difficulty in making this change is the need to alter the underlying philosophies and methodologies.

In today's verification flows, the principal problem lies in the fact that the testbenches are more difficult to develop, validate, and debug than the base design. The test environment has to not only check for functionality in the nominal case, but also for any anomalies related to layout or process variations. For small blocks, designers need to find a way to develop simple testbenches and model checking to allow the verification to be reused at the chip level. When the design size is small enough, the simulation with test vectors will suffice for most of the design. However at the full chip level, the strategies have to change to incorporate other tools into the verification mix. Some think more tools will solve the problem.

Janick Bergeron, scientist at Synopsys, Inc. (Mountain View, CA) and moderator of the on-line Verification Guild (http://verificationguild.com/) observes that the tools needed for verification for the most part already exist. He says a complete tool set would include simulator(s), hardware verification language(s) (HVLs), formal property checkers, and semi-formal tools to handle the cases where the formal tools run out of capacity. In addition, functional coverage is important to check the regression suite and to start moving towards a coverage-driven methodology. Going without all of these tools is like trying to drive across the U.S.A. without a map. The greatest problem is that the users don't use the tools properly, or to their full capabilities, assuming that the users even invoke the tools in the first place.

Tool Box oR Pandora's Box

David Kelf, vice president of marketing at Novas Software, Inc (San Jose, CA) proposes the use of automated testbench tools and model checking to help find areas that are hard to find with direct vector simulation. Adding assertions into the mix can help to check the testbenches and helps direct the model checkers. And Kelf says various types of coverage tools are necessary: code coverage tells that the code was used, functional coverage tests the interactions between blocks, and specification coverage ensures the system meets the definitions of overall functionality. One has to combine all these coverage types to make sure that the design is covered as well as possible. One key issue is the incorporation of good debug facilities into the overall tool suite, whether a cluster of point tools or a verification platform from a single vendor.

As the tool environment changes with the technology, some vendors are creating additional point tools, others are collaborating with partners to create an instrument cluster, and a few big companies are trying to create a complete verification environment for use in-house.

Ideally, it should be possible to exchange coverage data across tools, something we're not able to do yet. This transparent data exchange should provide an integrated way with which to develop a process environment, in a manner equivalent to the first standards for VHDL. Users need to be able to exchange assertions, verification IP, generators across platforms and design levels. For example, if a tool gathers information in one stage of the design, it should generate appropriate data for the downstream tools. So a test analysis tool should forward test coverage metrics and constraints to the DFT tools. And, any front-end tool should link to other tools wherever possible.

Front-end analysis is important to check the code entry. The tools have changed from just a lint function to check spelling and grammar, to offerings that now including design rules and reusability. In addition, there is a paradigm shift to automated checks and tools, and whole methodologies like assertion-based verification (ABV) to ameliorate the size and complexity issues.

Meanwhile, the latest entry into the bag of verification tricks currently is the move to verification process automation. These functions integrate a group of tools under a single user interface, enable iterative extensions, and operate with a common data model to reduce switching times.

Bergeron notes that assertions in both simulation and formal verification are helped by a single language for the testbench and the design. But he also insists that this isn't a critical issue, just a reduction in inconvenience. Language is only the vehicle for expressing the design intent, not the key.

Verification IP

The positioning and value of verification IP is in its ability to save time by eliminating the rewriting of tests at the different design levels. The use of IP is growing, especially third-party IP. So today, users need IP qualification and checks for interoperability and compatibility such as positive or negative edge clocking, naming conventions, and so on. The ability to reuse some verification IP helps confirm functional correctness and reduces the risks.

Bergeron claims that verification IP needs to be domain specific. The generic portions of the IP are already built into the languages. Additionally, verification IP has to be application or protocol specific, and also needs to move up in abstraction level from bus-functional models to transactions and full operational functions--those such as a USB hub.

David Lin, vice president of marketing at Denali Software, Inc. (Palo Alto, CA) notes one key is the move towards standard interfaces for IP, both the silicon and verification IP. At the chip level, verification IP fills the need for specification coverage, which in turn drives the number of assertions to confirm the analysis. First, a compliance suite covers all of the rules, standards, and specifications for an interface. Second, dynamic assertions enable checking the actual performance of the interface to a real operating environment. Third and finally, the IP encapsulates domain-specific knowledge, so the user doesn't have to become an expert in the specifics of the interface.

Are we done yet?

The $10 million question is always, "Have I completely verified my design?" The unfortunate answer is, you never know when you're actually done with the verification process. Verification is related to the final application. For instance, some applications such as mathematics units have too many combinations and permutations of external stimulus to exhaustively check. What you have to do, instead, is to check as completely as possible using vectors, directed random vectors, and negative tests such as bad data or control sequences to try to find all of the problems.

Narain notes there is no theoretical way to guarantee completeness. In the past, designs were much simpler. Back then, it was possible to identify all items that needed to be checked. This led to the use of a more formal structure of checklists in combination with accumulated experience to identify a complete verification. Now, a similar approach defines those aspects of a design that need coverage. Narain elaborates on this concept by stating that verification IP is able to cover one or more items on the checklist, although IP is just another component in the verification toolbox.

Another path to completeness is experience. Seasoned vendors can help because their user base gives them exposure to a large number of problems and resulting solutions, which have been checked by many users--similar in a way to the open source software movement.

And finally, there's the strategy of expanding the checklist coverage. For instance, in addition to the checkers and monitors that users have invoked, they're now adding assertions as well. The use of formal tools encourages the creation of more assertions, which improves coverage.

There is a paradigm shift towards automated checks and tools today. Now a change to ABV is underway. The many vendors in the ABV area are showing success in reducing the amount of work it takes to write assertions and, therefore, improving design coverage.

Change as the only constant

Meanwhile, advanced verification processes will find issues with the tools at the human level. Engineers want to measure something, so they use measurements that the tools are focused on--for instance, clocks per second or number of gates. Today, users need to have good and useful metrics for verification. Over 90 percent of verification cycles are wasted in running redundant regression suites, redoing verification across hierarchy levels, and other less-than-useful procedures. This situation has lead to an overload of testing that's actually produced little added coverage. Most development teams simply don't know how to run 24/7.

Glaser offers this advice: metrics should include such items as maximum coverage per day, rather than gates verified per man-month. He says a good set of metrics includes the amount of verification reuse, and how well available human resources are used. In addition, predictability issues such as convergence as a percentage of the whole design and the quality of the verification should include measurements of total coverage across hardware and software, due to the costs of a re-spin.

Technology always exists as a matrix of functions. Users need a repeatable process and domain-specific verification IP and they want to be able to incorporate different types of building blocks including assertion libraries at the lowest levels. A formal way to capture the test plan, therefore, should include various tools such as Lint, and coverage tools to rebuild blocks and construct the total test plan. The overall plan should apply an operating environment to the specifications, generate coverage through directed random, system, formal verification, and so on down the list.

In a perfect world, verification should not be a source of great frustration or fear. Instead, verification should be seen as a complex, but satisfying portion of the overall design flow. Experts in verification today are in great demand. However, the wealth of tools available now and visible on the immediate horizon, allow even those below the "expert" class to have the resources needed to verify their design with a certain level of confidence. Although there's still room for improvement, the current situation should provide hope to the entire industry.