By Ed Sperling
Estimates of how much time it takes to verify a complex SoC are still hovering around 70% of the total non-recurring engineering costs, but with more unknowns and more things to verify it’s becoming harder to keep that number from growing.

Verification has always been described as an unbounded problem. You can always verify more, and just knowing when to call it quits is something of an art. Moreover, with software now thrown into the mix, engineering teams have to decide what’s good enough for tapeout and what can be fixed once the chip is already in the market.

Making that decision is becoming tougher, though. The amount that has to be verified is less clear, in part because of the growing amount of outside IP that is now included in designs. Of the 70% or 90% of IP that is used or re-used in a complex SoC, less than 50% is now commercially purchased with the remainder internally developed, often for previous projects. The amount of commercially generated IP is expected to rise over the next few years, though, basically creating a series of black boxes that companies didn’t create internally.

While much of this commercial IP will be sold as pre-verified, what works in one design may not work exactly the same way in another. That’s particularly true with different process technologies. A general-purpose process built for speed may cause IP to behave completely differently than one optimized for low power. And in stacked die, two known good die may no longer work when they are packaged together.

“The new world is a broader supply chain for chips,” said Mike Gianfagna, vice president of marketing at Atrenta. “There is a need for better visibility in the supply chain, including everything from early predictions to yield to the track record of the supplier. There are multiple points of failure. For data management, planning, thermal and mechanical analysis you need fundamental enabling technologies. At the same time there is a re-invention of the industry into smaller, more niche markets.”

Knowing what to verify
Just knowing how much to verify is a challenge. Taher Madrawala, vice president of engineering at Open-Silicon, said this is not a simple decision because file sizes for verification are becoming enormous. That means what gets left out of the verification process may be as strategic as what gets included, because all of this can affect time to market. Verification budgets remain tight, both from a manpower and equipment standpoint.

“On top of that you don’t always have access to all of the functionality,” Madrawala said. “That’s especially true in 3D stacks or system-in-package. You don’t always have access to increased functionality because some things are encapsulated inside the package.”

He noted that from an NRE perspective, the percentage spent on verification has remained constant from 90nm down to 45nm. That has been helped by more standards, including modeling of IP in C or C++, an increased use of emulation, and the ability to run tests on multiprocessing computers. But with compressed schedules and greater complexity, those numbers can change.

There also are differences of opinion about what works, what will continue to work, and what needs to be changed in the future, both from a physical and a functional standpoint. Tools vendors insist that most of the capabilities are already there to do verification, even though they will need to be speeded up through better modeling at a higher level of abstraction with a greater reliance on multiprocessing servers. They also say that verification teams need to learn to use the tools that are out there better.

Chipmakers generally acknowledge the need for better training on the tools, but they say the growth in complexity will create the need for additional testbenches. In particular, there will need to be new tools for partitioning designs and verifying the results once stacked die become more mainstream.

“As complexity grows, integration will be the issue,” said Prasad Subramaniam, vice president of design technology at eSilicon. “You will need specialists for each part of the design. People’s specialties will get narrower. And then you will need people to manage more specialties. The generalists, who will be the architects and higher-level engineers, will define the problem. Once they have made the decision about what to do, then the specialists will take over. But there will also be a lot of feedback. This will be an iterative process. There will be meetings where you need to reconcile differences and make adjustments. There will be a lot of collaboration, and verification will start from the get-go.”

Verification strategies
There are two main approaches to verification. One is to verify the pieces. Another is to verify the system. Both are necessary, but the order in which they need to be done as part of a verification flow can vary greatly for even derivative chips.

Samta Bansal, 3D IC lead and silicon realization digital project manager at Cadence, said that in stacked die an incremental approach will be needed to do verification. “If you analyze it all together it overcomplicates the process,” Bansal said. “For one thing, not all of the pieces will be available at the same time. A more feasible approach will be to verify each chip in a stack as part of a verification flow, then focus on the microbumps, TSVs, LVS and DRC for alignment and ultimately create a single file.”

That’s not so simple, of course. In stacked die there are physical verification issues that can complicate the functional verification, notably stress and power. And there is now software that needs to be considered in the mix, with the trend toward an increasing portion of the stack.

“Functional and physical verification are both important but independent tasks,” said George Zafiropoulos, vice president of solutions marketing at Synopsys. “In both cases, verification is moving up in system complexity. We’ve gone from blocks to lots of blocks to lots of processes and I/O, and there is more stuff coming. Complex interface IP at the periphery of the chip has gone up by an order of magnitude. The design team can’t verify everything, though.”

Zafiropoulos said design teams used to think there was not enough time to do verification at the block level. He said that putting 100 blocks together increases the challenge exponentially.

“A lot of this is bottom up,” he said. “You build sub-circuits up to the chip and then in multiple chips. You can’t afford to have errors inside these blocks. But you also need to change the scope of what has to be done. In the past, one engineer could comprehend everything on a chip. Now we’ve gone from the guy who knows everything about a chip to teams that are in different companies and maybe different countries.”

The result, he said, will be a gradual change in three areas. First, more and more engineers will do verification, rather than just specific verification teams. Second, all engineers will become more software savvy. And third, new kinds of tools will be introduced, including formal approaches.

By Ed Sperling
System-Level Design sat down to discuss verification with Albert Camilleri, engineering director for Qualcomm’s QCT Division; Jim Kenney, marketing director for Mentor Graphics’ emulation division; George Zafiropoulos, vice president of solutions marketing at Synopsys; Ran Avinun, marketing group director for Cadence’s system design and verification group; Charles Janac, CEO of Arteris; and Luc Burgun, CEO of EVE. What follows are excerpts of that conversation.

SLD: What needs to change when we get into networking the signal around the chip? Is it mindset, is it adding something new to verification, or are we dealing with a whole different problem than we’ve dealt with in the past?
Janac: One of the things that makes verification easier is that the networking techniques can be used to isolate the individual blocks and IPs from the data communication. You can then start to verify with verification IP as a proxy for the traffic. You can verify the communications subsystem in isolation. That makes the integration a lot easier. It gives you the ability to run a bunch of verification tests for the chip communication subsystem by itself. Before, when it was all mixed, it was much more difficult. The verification of the top level gets easier.

SLD: So there appear to be two trends going on here. One is a discrete way of looking at this functionally. The other way is to put it all together into a high-level system approach. But something isn’t working because it still costs too much.
Janac: What you’re dealing with is a race between extreme growth of complexity and functionality and improving verification—emulation that’s able to be used earlier in the design cycle, divide-and-conquer module verification, software-hardware co-simulation ability. Do you make verification easier or keep up with complexity?
Kenney: Just being able to keep up is a win. We’re doing well just by keeping up.
Mentor internally has a culture of hardware and software. Not enough software developers are using the tools we’ve developed, so we aren’t solving the problem for everyone. But internally there’s a lot of cross-pollination between the embedded tool developers and the functional verification tool developers. That’s how we’re trying to solve the hardware-software issue. And now you see power management software being layered on top of that, tying power consumption to various software routines. All of that has to come together to make the system to work.

SLD: Will subsystems, which are basically larger blocks, make verification easier or more difficult?
Camilleri: It depends. Re-use is always helpful, and the re-use of a subsystem would be very helpful. But it’s never simple. The demand may be something that isn’t quite what the previous one was. There may be demand for a version that works at a lower tier or a higher tier, or one that does more video. Much as we’d like to re-use more at the system level, we’re more likely to re-use at the core level. But verifying subsystems is the way to go, and hopefully we will develop methodologies that allow us to tolerate smaller changes in subsystems and be able to re-use the bulk of it. A lot of the testbenches can be re-used. But the whole concept is a challenge.
Avinun: If you’re creating verification IP, there’s probably more functionality than you need at the system level. You need it at the block level. We’ve made tremendous progress to scale those and to be able to re-use verification IP for hardware-software verification as well as TLM. But now the question is whether you really want to re-use them because 90% of what you did doesn’t really require re-use at the system level because you’re wasting your cycles. And then you have other kinds of tests that you need to run at the system level that you can’t run at the block level because you can’t re-use it. And beyond this, when you look at verification IP that is driven by software like Android or operating systems, that will look totally different. So the question is whether you want to re-use everything. As an industry we’re trying to figure out how to automate block-level verification and verification IP that has been used for blocks should be used for systems, and which other things need to be created. There may be verification IP that addresses H.264 and MPEG vs. PCI. We’re working on this, but it isn’t defined yet.

SLD: How are you defining re-use?
Avinun: From block level to system level, and not so much from one design to another one.
Zafiropoulos: In the abstract, you can’t possibly design everything from scratch on big chips. The challenge is what’s the quality and can it be deployed and how long does it take to be able to absorb somebody else’s design content—and be able to re-deploy it. In the typical SoC today half of the blocks on the top are easily supplied by third parties. That gives the designer the ability to focus on IP that’s differentiating for them. The downside is, how many different people are you going to be buying this IP from? You could put 60 or 70 IP cores on an SoC. If you’re buying that from 20 different vendors, that’s immense. Just absorbing all of this third-party content, and then understanding the quality of that content, becomes very important.
Camilleri: One thing we haven’t touched upon is debug. There’s the verification aspect of this, which is the proactive work we have to do. But then there’s the debug, and this depends on your meaning of re-use. If I interpret the word re-use to include the portability of test as well as design, being able to debug quickly and efficiently and on the best platform for finding a bug is very important to our schedules. That’s why you need a seamless methodology across platforms. If a software person finds a bug, you need to be able to find the root cause as quickly as you can. It’s very difficult to do that if they’re using a completely different set of tools than what we’ve got, and we can’t replicate it to see what’s going on. Being able to leverage my testbenches across all the different platforms is paramount.
Zafiropoulos: But debug to a hardware guy and a software guy are quite different. On the hardware side, debugging a hardware interconnect has to be done at the protocol level. You can’t look at it using bits and bytes because there’s too much to look at. You have to look at it at the protocol level or at the application-specific level to understand what is really going on.
Camilleri: And you have to root-cause it to the block you’re working on.
Zafiropoulos: It’s almost like there’s a debug stack where you look at it using multiple levels of abstraction.
Kenney: We have a customer who’s helping us solve this problem. If the software guy has a problem here, how does he communicate that to the hardware guy? They came up with an interesting way of dealing with this. They say, ‘go solve this using this particular piece,’ and then we can synchronize where the software failed and what that’s doing in hardware. It allows you to get to the same place quicker.

SLD: Does NoC technology make it easier to communicate between the hardware and software?
Janac: Yes, because the data traffic is carried by the NoC. You can put in things like latency probes, observation probes and statistics collectors that give you information in the software about what is happening in the hardware. You can use it to trap errors. That’s an area that’s just starting up, but the networking technology enables that.
Burgun: That’s a good way to write an abstraction. In the end the designers don’t need to deal with huge waveform files to understand what is going on in the design.
Janac: That information can be processed in the emulation systems much more effectively.
Avinun: There are multiple platforms. Some are better to run higher performance. Some are better to run debug. Our philosophy is that the way to solve it is to use the platform for what it was created for. Or you write for one platform and then you have 100% correlation to another platform and do the debug there. There are different methods to help you debug on top of this.

SLD: Do the existing verification methods and tools work in stacked die?
Zafiropoulos: We don’t know for sure, but we have some suspicions. One suspicion is there will be thermal variations. What will the impact on power be if it’s not uniform across the die? We don’t know. But if you look at power, functional verification was perfectly good in the 1 and 0 world when low-power techniques became prevalent. Functional started to go a little more analog, and we’re wondering if there will be an effect from thermal on things like timing. It might be there are more corner cases to be simulated.
Burgun: 3D will enable more complexity. This is where network-on-chip tools will be beneficial. But the architecture will create constraints and we don’t know what the end result will be. Will it create more issues? We don’t know.
Kenney: From where we sit, functional verification hits a lot more function in the 3D space.

SLD: Doesn’t this redefine what a system is?
Camilleri: Yes, it’s more and more integration. There will be a whole set of electrical issues. Will there be more functionally than we’re already dealing with? We don’t know. Electrical is certainly harder, and there are more physical issues so there is more strain on the hardware-software interdependency.

SLD: Do we start changing our perception of what is good enough?
Janac: The only solution is divide and conquer. If you add a die that has been completely verified, as long as you engineer the interface correctly then you only have to verify the interface.
Zafiropoulos: But if you can’t verify the full end-to-end functionality of the device it may be different.
Janac: You will have to do system-level verification, but you shouldn’t have to delve down into that die.
Zafiropoulos: Yes, the question is whether you have to do a gate-level, transistor-level verification of everything? Hopefully not.
Kenney: If we can equate it to separate chips today, they settle for verifying the chips separately. Maybe they’ll do the die individually and assume they’re going to go together okay. We don’t know yet.

By Ann Steffora Mutschler
Verifying SoCs from a functional perspective pushes the limits of already lean resources, driving verification teams to seek out new ways to improve productivity of verification tasks. Of course, with the verification task being a time-bound one, the challenge is daunting.

It is well understood that consumer electronics is pushing the technology envelope in terms of the amount of technology that is packed into a single SoC, while also putting huge scheduling pressures on engineering teams. They are tasked with delivering increasingly complex SoCs in the same time (or less) as their previous project, while meeting low-power requirements and generating a higher amount of software content.

RTL verification continues to be the workhorse for verifying designs in the most cost effective and most debug-efficient way because at RTL a design can quickly be simulated at multiple levels. “They can do it at block level, they can do it at a cluster level where there are multiple blocks, and it can be done at the SoC level. There is a lot of flexibility in terms of the hierarchy at which you can verify. You can run the simulations quickly, you can find problems you can do the deep to debug triaging quickly,” explained Swami Venkat senior director of marketing for Synopsys’ verification group.

For this reason, RTL verification continues to be one of the most important focus areas for verification engineers to verify SoCs.

Directed test and randomized constrained verification are two of the most common approaches used in functional verification. But Jayaram Bhasker, architect at eSilicon, observed that the latter is not used nearly as often because engineering teams are limited in time, and therefore end up just doing the directed testing as part of their verification plans.

Synopsys’ Venkat disagrees. “More and more engineers are deploying constrained random based verification. There used to be a time when an engineer had to think about the actual behavior of the design and conjure up scenarios in his head and write what used to be directed tests that are targeted at specific functionality of a particular design. One of the inherent limitations there is that it depends on the ability of an engineer to think of all the possible scenarios, and obviously because it is manual, it is going to be slow.”

In constrained random verification, the software generates various scenarios rather than the engineer creating them manually. “As long as you have multiple different stimuli and each representing different scenarios, you would be able to run them on the machine and you can have lots and lots of simulations running at any point in time,” he said.

In this usage scenario, the efficiency and the effectiveness of the constraint solver becomes extremely important and the ability of the tool to perform coverage analysis becomes equally important because there a lot of random tests running. “The engineer would want to know the additional coverage that has been achieved by each individual test and compare that against the spec to get some sort of measurable metric to know the progress that the environment is making or the verification engineer is making,” Venkat added.

But eSilicon’s Bhasker said there are problems with this approach. “You never know when verification is complete, and that’s one of the challenges. You’re really time-bounded. Otherwise you can go on verifying for as long as you can. The problem is you cannot bound it, you cannot say, ‘Can you test the chip in one year?’ There’s only so much you can do in one year. I doubt if there is any chip out there that does not have a bug in it. The question is what is the probability that a customer will notice that bug.”

To achieve the highest overall verification productivity, Cadence prescribes a solution that includes emulation, acceleration, simulation and FPGA prototyping tools, as well as an ‘enterprise manager’ that helps the engineering team create a verification plan and then serves as the window into the metrics that the tools produce in order to analyze them.

“Verification is a case of doing just enough. If you try to verify everything, it is an infinite process. It’s a case of is it good enough? Is the risk contained enough? That risk of, ‘Is it good enough?’ is usually measured by coverage. Enterprise manager or approaches like that create plans and aggregate all the coverage and other metrics from all the tools that lets the manager engineering manager–the one who says it’s ready to release,” said Mitch Weaver, general manager of Cadence’s advanced verification solutions group.

And this fits into the company’s Time-to-Integration strategy, which includes the following tenants:

1. Use IP that is integration-ready. It is pre-certified and is delivered with associated software, not just the hardware IP. It is pre-verified, comes with the related testbench and is ready for integration.
2. Employ a disciplined, mixed-language-capable re-use methodology using UVM as the base, which is a standard verification methodology that promotes reuse.
3. Use tools that are optimized for SoC verification.

Weaver noted that verification environments often include more lines of code than the design itself, so re-use of the verification environment is a huge, huge deal.

Methodology is the foundation
SystemVerilog, VMM, OVM and now UVM are widely adopted methodologies that provide a uniform, holistic path for different engineers to stitch together verification environments, whether they are developing verification IP, a verification methodology, mixing and matching multiple verification components from an earlier project or from a derivative project.

The move to smaller geometries is another driver for approaching verification in a holistic manner. Fortunately, many engineering teams today do spend the time up front to create proper and thorough verification plans, Synopsys’ Venkat observed. Also, as the number of IPs on an SOC continues to grow, engineering teams are using more verification IP. “If you look at an SoC today compared to similar designs about 10 years back the number of protocols on a chip has gone up so much—in order to improve their overall productivity we see people using a lot of verification IP.”

Bernard Murphy, CTO at Atrenta, pointed out that there are some interesting ideas emerging around assertion synthesis. One involves looking at the simulations that already have been run and building assertions based on that information. That’s not just what the behaviors should be. It also includes what behaviors have not been tested. When you use it in the application you’re straying into an area that was never tested.

“At the integration level, one thing people think about when they think about static checking is formal verification. Can I formally verify that things are hooked up correctly? And certainly that’s one method. Another method is path tracing—just walking along paths from the pin on one IP to another IP and asking if these things are connected correctly,” Murphy said.

The trick in all of this is that it’s not so much how you check it, it’s how you represent what you want to check, Murphy said. But as for the best way to represent what you want to check, there isn’t a well-defined answer to that right now. “Assertions are certainly part of the answer because they are at least well defined. The problem with assertions is they are primarily functional. They don’t have any architecture content, so if I mistakenly connected an interrupt to an error state reset then without doing simulation there’s no obvious way I could tell that that was a mistake because they’re both signals, they both toggle, they do stuff, but architecturally it doesn’t make sense. You can’t really capture that in assertions as we know them today. There needs to be some new ways of representing these things, and God forbid we should talk about standards for what it is we’ve got to represent.”

Raising the bar
To develop the most effective methodology for improving functional verification the use cases for the device must be completely understood, but the approach to the verification is also critical.

“At the IP level there is a certain comfort in terms of how to get reasonable productivity levels from functional verification,” according to Andy Meyer, verification architect at Mentor Graphics. “The bigger challenge for our customers is when they try to integrate those IPs into an SoC. What we try to bring to the party is the productivity, the measurement of that productivity, how to measure the results in a quantitative fashion, and the effort level–so you have a way of asking if you are in fact approaching it right, and how do you know when you’re improving it.”

“Where people have been and where we are trying to encourage people away from is the [thumb-in-the-air guesstimate] based on the last project into something that you can actually measure. That means the use and collection of metrics and the analysis of metrics—how am I doing, how many regressions did I run, how many bugs did I find per regression hour or per simulation cycle per engineer time? These are the kinds of things that give you at least some way of measuring and getting a feel for where you are, even though you never really have an accurate understanding of what’s happening in an SoC because it’s too big and too complex,” he continued.

Mentor calls this task verification management, and includes the collection of coverage data. “It really started with the focus being able to rank and assess how well you are progressing in your verification through your coverage progression. Over the last few years it has grown into other areas as well because verification management is not just the coverage, it’s the results you get–you have to actually analyze the results of the simulation,” explained Steven Bailey, marketing director at Mentor.

After the results are analyzed, they must be collected over a number of products to see how results are trending. It isn’t as simple as just coverage data, he pointed out, it must include metrics on the churn rate on the code of their design as well as how well they’re progressing on reaching their coverage, how many verification cycles it’s taking.

This kind of tracking helps engineering teams understand the numbers of verification cycles they need to perform, and project the compute capabilities that will be needed. “When they start tracking that that’s also when they start noticing they need horsepower like emulation and they start realizing that emulation is actually cheap compared to tera cycles of verification and all this gets brought together,” Bailey added.

Meyer believes this is the beginning of a trend that actually begins to look at where results come from and where the effort was allocated — it’s those two together that give an actual measure of productivity.

One of his favorite examples is a company that employed this methodology for an entire category of testing and was able to determine, ‘What do we mean by results?’ One of the obvious results was, ‘Did we in fact see an RTL error ever in the last four generations of this product show up based on this entire category of stimulus?’ The answer in some cases was ‘no.’”

“I think that’s where you begin to see the serious productivity gain of taking this approach…the point is if you don’t measure, you don’t know,” Meyer concluded.

By Ed Sperling
System-Level Design sat down to discuss verification with Albert Camilleri, engineering director for Qualcomm’s QCT Division; Jim Kenney, marketing director for Mentor Graphics’ emulation division; George Zafiropoulos, vice president of solutions marketing at Synopsys; Ran Avinun, marketing group director for Cadence’s system design and verification group; Charles Janac, CEO of Arteris; and Luc Bergun, CEO of EVE. What follows are excerpts of that conversation.

SLD: If verification takes so long, are we looking at this correctly?
Janac: You need a verification strategy at the system level, but then you have to have the verification done at the module or unit level. And once that’s done you have that you have to go back up and make sure it still works. It’s top-down and bottom-up.
Camilleri: We are starting to look at our designs from more than just a system-level. It’s also a bottom-up perspective.
Zafiropoulos: We’re seeing a lot of customer feedback where their pain point is at that level. More and more of our users tell us that designing and verifying the block level is really not a problem. They’re quite good at that. It’s when you start stitching things together that they really start to fall apart. The sooner you can get the architecture right the better, so that you don’t have to find later on that you don’t have the right buffer sizes or bandwidth or power considerations.
Camilleri: And the trick is to find a balance so you can do as much of this as you can in parallel. You can’t afford just to do the architecture. You really have to have flows in place so you can do everything.
Zafiropoulos: That’s the feedback we get, as well, but you can’t afford to do everything twice. People still want to do things the way they’ve always done them. So we need to figure out ways to do it more effectively further up.
Avinun: There also are cultural issues. Software engineers still don’t work in lock step with hardware engineers. That costs their company a lot of money, but it still takes time to change those organizations. Another issue is that companies used to evolve work at the previous generation as hardware engineers started on the next generation. Even if you give them all the tools, they will say they’re still working on the previous generation and they don’t have the software for the new generation. As a result, they can’t work in lock-step the way they’re organized right now. Some companies in the networking domain are late adopters because their product cycles last four years. Other companies in the consumer market are moving to this faster because they have no choice. They can’t afford to wait six or nine months. If the software is not ready they start to use artificial software to integrate it.
Kenney: I agree. We’ve been making software-hardware co-simulation tools for 12 years. In many cases, the customer said, ‘We didn’t get a lot of value because we didn’t get the software until the last two months due to the fact that they were busy working on the last project.’ So when they bring the software guys in four months early they get more value. But if you look at all the stories for system-level verification and ESL, it’s about doing the architecture abstractly. The software guys aren’t there yet. Tools aren’t the only issue.

SLD: What’s happening at Qualcomm? Are the hardware and software teams working together?
Camilleri: We’re certainly trying to make sure we have the right architectures and that they get implemented and integrated from the start. So we’re looking at both the software and hardware architecture from the beginning. That’s a big part of a successful recipe for the aggressive timelines we’re after.

SLD: But are they working together and speaking the same language, or are they functioning independently?
Camilleri: We do try to work more closely. There’s always room for improvement, and each organization is extremely busy. But we’re all beginning to see that working closely from the start is key. We’re a systems company, so we have to provide the entire solution. That’s both hardware and software. The timelines needed to develop software are getting more demanding. Software challenges are becoming huge. Operating systems need to be supported and drivers have to be developed. There’s enormous pressure to start software much earlier. That’s the first conflict. Hardware want to verify longer to make sure there is the right quality, and software wants to start earlier. We use different platforms to get software started sooner so hardware can continue to verify for tapeout.
Kenney: It’s the tools, also. I’ve seen situations where there were hardware-software integration issues. Both teams looked at their tools and concluded there was no problem. Then we sat them down behind both and they solved the problem in half an hour. It’s getting the teams to work together, but it’s also providing tools that work for both of them rather than trying to debug with a waveform display.
Zafiropoulos: That gets to your point about the system validation being more critical. It’s when you bring all the parts together and you see where the root cause of the problem is. But without that validation point of bringing the systems together you’re just hoping it’s the other guy’s problem and not yours.
Kenney: I think it’s up to us to provide tools that look native to both sides.
Avinun: Do you really see there is a demand for software guys to look at things early? My experience is that when we deliver the tools for hardware, they say the software guys are not ready.
Camilleri: It has to happen. If you’re a semiconductor company you may have different needs. In our case we’re more of a systems company. We’re beginning to realize that we both benefit—both hardware and software—by being enabled earlier. Some of the applications that run on these chips are getting very demanding. There have to be certain kinds of compliance cases that make it difficult to run the number of frames or tests to ensure the architecture will meet the compliance tests. You need drivers and compilers to do that, and the sooner you can get software the better.
Zafiropoulos: We’ve seen a big uptick in doing software-driven functional verification, where even the guys using just simulation tools would like those multiple blocks to be stimulated by design content that was generated through the software running on the device. Having at least a driver layer and some code doing something to stimulate the hardware is moving into the hardware verification guy’s world today. Ten years ago you didn’t really think about that.
Camilleri: Functional correctness is only one angle to it. You also have performance and power, and you have to run the tests and the benchmarks that will tell you whether your architecture meets these performance requirements. You can’t wait until you have silicon to determine whether you have an issue. What can you do every step of the way, as you find issues with integration, to verify core performance and look at bandwidth and latency and how that works with busses and memory? What are the different traffic patterns and use cases? All your platforms and tools come together to make this possible.
Zafiropoulos: It also becomes an important factor in making later-stage modifications to the architecture. We’ve had several customers pick a certain processor core and then look to add more functionality. So do they add more code and another processor? Do they build an accelerator for that function? They’d like make that decision without just making a guess.
Camilleri: And the answers to those questions are not trivial sometimes. You do have to model it and see where your bottlenecks are. Just increasing megahertz won’t always solve your performance problem.

SLD: Is there more interest in emulation by software teams these days? Historically they refused to pay for emulation hardware.
Kenney: The answer is yes, but the software guys still aren’t paying for it. The hardware teams are paying for it. But these companies definitely are running software on the emulator. The full functionality isn’t there unless you’re running software, and for that you need the code.
Avinun: If you look at the verification process, the person who is responsible for verification—both hardware and software—is attached to the hardware team. He is working with the software team, but his responsibility is to the hardware team. Unlike in the past, where the hardware team would take whatever it got from the software team, now it has to look at different tests as part of the regression testing. If the hardware has to work on five or six operating systems, it’s part of your hardware responsibility.
Zafiropoulos: Part of it’s a budgetary issue. Those teams aren’t budgeting adequate dollars for the tools needed to solve these problems. The spend per software engineer is orders of magnitude different than for a hardware engineer.
Avinun: It may be a legacy perception, where software engineers used to get everything free and hardware engineers used to have to pay for tools.
Burgun: In our case the money is coming from the software team and not the hardware team. On the other hand, the ideal thing is for these companies to have a platform to validate both hardware and software and understand where the problems are.

By Ed Sperling
System-Level Design sat down to discuss verification with Albert Camilleri, engineering director for Qualcomm’s QCT Division; Jim Kenney, marketing director for Mentor Graphics’ emulation division; George Zafiropoulos, vice president of solutions marketing at Synopsys;Ran Avinun, marketing group director for Cadence’s system design and verification group; Charles Janac, CEO of Arteris; and Luc Burgun, CEO of EVE . What follows are excerpts of that conversation.

SLD: Why is verification still such a big problem? It still takes too long.
Burgun: Now we have a new element with the software, which is adding to the complexity. It’s not just the hardware anymore. We have to integrate the software and make sure we can provide a product with the right level of quality for both hardware and software.
Janac: Chips are getting more complex. There’s an issue of system verification, an issue of module verification and an issue of unit verification. We’re on the IP provider part, where the module is assembled out of individual unit IPs. Those have to be assembled very carefully. If you have errors in those, you introduce them up to various levels of the design. Then there has to be a tool for the module verification. The IP design has to come with this verification, because you don’t want to leave this to someone else. With system verification, that’s challenging both top-down and bottom-up. On one hand you want to verify all the modules and do the test one level up, but you need to consider the architecture in the verification strategy. The problem is complex and has to be solved at multiple levels.
Camilleri: Complexity is the No. 1 driver and it continues to increase. The systems of 10 years ago are now on a single chip. There’s also the expectation of the integration that has to be done early and chips are expected to work right off. You have to address performance, power and compliance and it all has to be validated early on.
Avinun: Everything is being driven now by applications. Initially we started to see this with smart phones. Now it’s moving to other market segments. Everything is being measured and has to be relevant to the applications running on the device. Those applications obviously are increasing the complexity and the number of processors running on the SoC, and they require not only the integration of hardware and software but also the integration of multiple processors with multiple sources of software. This is something new that wasn’t there 10 years ago. Going back to the 1990s, constraint-driven random generation techniques were introduced. Those solved problems at the block level. But with full integration of hardware and software at the system level, those techniques don’t work anymore. Simulation is running too slowly. Virtual prototyping still hasn’t been totally solved. And beyond this the techniques used at the system level are different. Even if a technique can be scaled, you still have to look at hardware-software power and compliance.
Zafiropoulos: The systems today are actually systems of systems. The typical product today involves multiple execution units, so you have an interaction problem that expands the state space. You can’t debug it readily and you need techniques to go after that. There’s no generally agreed upon single way to do that, either. It may depend upon the application domain so the tools can be applied to the problem. In addition to that, when you get up to the system level you need more domain experience. If you’re looking at the bus contention and interconnectivity of a cell-phone architecture you’ve got to understand the application. It’s not a general problem anymore. It’s a domain-specific problem. That’s a challenge because design engineers and verification engineers at some point also need to have knowledge of the application—much more than they’ve had in the past.
Kenney: Verification has gotten much better. We are now able to run the complete device with software, do power analysis, do power island/software management, and connect up to physical and virtual peripherals. And customers are able to get this done first pass and get the silicon pretty quickly. It’s gotten harder, but the tools have gotten better. We’re verifying a lot more than we were before.

SLD: Then why is it still 70% of the NRE? Is it just keeping pace? If that’s the case, then the problem hasn’t been solved.
Kenney: Agreed. Verification is accomplishing a lot more than in the past. The effort per chip hasn’t gone down. Has the problem gone away? No. But verification has grown to encompass software and power.
Zafiropoulos: The point of the problem changes. The problem we had 20 years ago has been solved. The problem we had 10 years ago has been solved. And today’s problem will eventually get solved. The ball is always moving. That’s true on the physics side and the implementation side, as well.
Camilleri: One of the things that is changing is the integration. In the past it was divide and conquer. Each group would verify their particular responsibility. Someone would be doing firmware. Someone else would be doing a USB core. Now you have to verify and validate all at once. That’s a new challenge. We are verifying better than before. One challenge that I do see is that the more tools we get as part of our arsenal, the more we need to integrate these tools seamlessly. It’s one thing to have the same group using multiple tools. Once you have different teams using different tools and different platforms, you suddenly have different challenges. How do you know that it’s all going to come together? From a user perspective, that’s a challenge.
Zafiropoulos: Verification is different from implementation. In implementation there’s a very clear path. You must do synthesis, place and route, timing analysis, extraction. In verification, besides your simulator, everything is optional. There’s a wide variety of opinion about what is the appropriate tool, app, strategy to take. Some people believe very strongly in anything that’s not simulation—formal verification, prototyping emulation, etc.—and there are others who don’t see the value in those tools. There’s no general agreed upon tool suite to create that commonality, so it’s an open-ended problem.
Avinun: We have some new numbers from analysts. HALs implementation takes 20% of the overall hardware development cost. Hardware verification is 50% of the cost. At 28nm, it’s actually 51%. Software verification at 28nm is 56% of the overall development cost. The number of options you have in verification is infinite. The key is how you bound it in a way where you will get sufficient confidence in what you get at the end will not deviate significantly from your spec. You’ll never get to 100% like you do with place and route. In verification, you need to assess your risk. Are you comfortable where you are you or do you need to verify more. We are focusing on integration of our tools and those other tools that are based on standards. We aren’t completely there, but we are looking at this as a complete picture. The other area that’s changing is that with formal verification and constraint-driven random generation, those were ill-defined over the past 15 years. We see now a need to figure out which portion should go to system level and which portion cannot go to system level, both for performance and complete verification.
Camilleri: One of the initial challenges that system integration is bringing is the need to do system validation as well as verification from the start. We cannot afford anymore to get to a stage where we find out a chip doesn’t do what it was intended to do. It’s not just verification. It’s how we validate throughout the whole design cycle. There won’t be time to do it again.
Zafiropoulos: And you see that even more now with virtual platforms driving a system model.
Camilleri: You have all these virtual platforms becoming available. I need a seamless flow moving from one platform to another to take advantage of all of these, and not just for the verification. How does software get started so much earlier in the process and move in tandem with the hardware?

By Ed Sperling
It’s probably a combination of post-DAC, post-holiday and maybe even pre-Semicon, but the design world was remarkably quiet this week. In fact, you could probably hear a needle drop into a haystack and actually find it.

Mentor Graphics created a verification academy Web site to provide information and training on UVM and OVM. Given the percentage of SoC NRE spent on verification, this certainly can’t hurt. What’s interesting is this comes on the heels of Cadence’s decision to donate its UVM Web site to Accellera. There’s a lot of activityy in the verification Web site space these days.

MDV is a key to Silicon Realization, part of the greater EDA360 vision. EDA360 helps both design creators and design integrators close the “productivity gap” (through improved approaches to design, verification, and implementation) as well as the “profitability gap” (by providing new capabilities for IP creation/ selection/integration and system optimization). Silicon Realization represents everything it takes to get a design into working silicon, including the creation and integration of large digital, analog, and mixed-signal IP blocks. Advanced verification capabilities, such as MDV, are a must.

The functional verification landscape has changed beyond all recognition over the last 10 years, and while design paradigms have become mature and stable, verification methodologies and technologies have continued to evolve and new flows and tools are still being invented. Against this rapidly changing background, designs have been steadily growing with more IP and more complex IP being integrated into larger and larger SoCs
.
Realizing silicon in the face of these challenges requires new approaches and a very flexible workforce capable of adapting and changing on a regular basis. This paper outlines a new approach to managing verification complexity—metric- driven verification—and a valuable resource to back up this new approach, the Cadence Incisive Verification Kit, which together will enable you to plan and embrace new methodologies and approaches in a safe and controlled way. The kit provides clear and realistic examples at each stage to guide you and your teams in their new projects. To download this paper, click here.

By Ed Sperling
System-Level Design sat down to talk about changes in verification with Russ Vreeland, emulation technologist in Mentor Graphics’ Emulation Division; Ran Avinun, product marketing group director for system design and verification at Cadence, and Brian Bailey, consultant and member of Accellera’s Interface Technical Committee. What follows are excerpts of that conversation.

SLD: Do customers get what you’re trying to do with emulation and verification?
Vreeland: Sometimes it’s really hard to convert customers. We have a mantra in EDA that the customer is always right, but that’s really not true—having been a customer myself. Some customers get into overly complex, proprietary environments that are dead ends. Then they have a massive task, which they tend to procrastinate, where they can use IP and new technologies like SCE-MI that accelerate tasks. Then other customers can naturally develop their testbench environment. With the old OVM-UVM being transaction-based, and preaching a clear separation of the testbench and the DUT, all of a sudden we have the SCE-MI development area and the OVM-UVM community, and both have discovered the same thing independently. They can take their testbenches and run them in simulation, or go to a hardware acceleration platform.

SLD: What happens in 2.5D and 3D? Do these same techniques apply?
Bailey: We’re not dealing with the physical aspects of the chip, so we don’t care how it’s fabricated. What we care about is bringing multiple execution models together.

SLD: But doesn’t it matter if you have multiple voltages, different IP, physical effects?
Avinun: No, because we are focused on functional verification here. You don’t take those effects into account when you’re doing functional verification. It doesn’t mean those aren’t important or that you won’t have a poor chip if you don’t handle those issues. But to some extent we’re trying to divide and conquer the problem. We had this argument 15 years ago with timing accelerators. Customers kept asking for timing accelerators, but the industry decided to separate static timing and functional verification. If you try to do it all together sometimes you’re wasting time. If you’re looking at 2.5D and 3D, the majority of problems will come from thermal, noise and other physical effects. They may take over as the primary needs from resources and investment, but so far we haven’t seen it. Verification and hardware/software integration are still consuming most of the resources.
Bailey: Probably 99.99% of simulations don’t even think about voltage. It’s just about ones and zeroes.

SLD: Is that because they’re under a certain speed threshold, though?
Bailey: No, it’s the separation of concerns. You use logic simulation for functional verification. You abstract out the whole notion of voltage. You just assume that everything is powered on.
Vreeland: We have enough problems with ones and zeroes simulation. Imagine how long we would spend with SPICE-level simulation.
Bailey: Every design today, with all the power issues, is really a mixed-signal simulation. But who’s going to do mixed-signal simulation all the time? It’s not reality to consider all of the effects in every run. You have to pick and choose.

SLD: Are software teams using emulation more frequently now?
Vreeland: Yes, because it gives the embedded teams so much of a head start in embedded silicon.
Avinun: If you don’t have a dependence on the hardware you can run it standalone. The moment you are dependent on the hardware it changes.

SLD: Can these services be offered via a cloud model?
Avinun: We went back and forth on the business model. At the end of the day it doesn’t matter if we provide a service from a Cadence site or whether the hardware is at the customer site. Hardware has to be allocated. We’re not yet in the same volume of cloud computing where we can build rooms of computers and wait for customers to dial in. We tried that several times where customers pay per hour, and today we still have that offering. But large companies more often will buy the hardware and share it internally. Even small companies like to have the hardware on-site.
Vreeland: Many companies mix the co-modeling paradigm in ICE, which is hard to do on a cloud.

SLD: How do standards like SCE-MI affect all of this?
Bailey: One of the challenges we face is that when things like DPI for Verilog or TLM get created, they’re really thinking about simulators. They have no restrictions. As soon as you start making this work with a hardware-assisted platform, you have to consider how you implement that interface in hardware. We have to physically synthesize DPI or TLM because we have to get half of that interface onto the emulation platform.

SLD: There was a move to push verification further into the front end, and some companies have architects working on verification. Is that working?
Avinun: It does help. But part of what you see is complexity is 10x and this may help 2x or 3x. You still have a gap. In general we see two use models. One is the TLM-virtual use platform. Customers start to create TLMs that are synthesizable and then verify them up front. We don’t see this method will eliminate the need to run RTL verification at the system level because you have all the state machines, the cache and the details that you can’t verify when you start. You also have multiple processors that you need to verify together, and you need to verify the firmware that goes into a lower level of abstraction. You won’t eliminate this, but it will help.
Bailey: We’re at the very tip of an iceberg here. Everything we know as an industry has been built using bottom-up verification. Simulators provide plenty of performance for block-level simulation. As we start integrating more blocks together, though, we need longer and longer sequences to do useful things with those subsystems. And when we assemble the entire system together, the simulators do not have the capability to use the sequence lengths we need to come up with anything useful. That’s where emulation comes in—or, you start doing it earlier in the models. With abstract models we can put in these long sequences and start it earlier in the process. Are we there yet? A few companies are just pushing the envelope in that area. If you don’t have the models, you can’t do these abstract, long simulations. The big companies have the resources to do these models. The smaller companies have to wait for commercial models to become available.

By Ed Sperling
System-Level Design sat down to talk about changes in verification with Russ Vreeland, emulation technologist in Mentor Graphics’ Emulation Division; Ran Avinun, product marketing group director for system design and verification at Cadence, and Brian Bailey, consultant and member of Accellera’s Interface Technical Committee. What follows are excerpts of that conversation.

SLD: Do customers really know which tools to use and when?
Avinun: There is a lot of ambiguity, and it becomes even more complicated when you’re dealing with multiple tools from multiple vendors. A lot of the burden is on the customer. We’re trying to improve that and change it. But when you get a model that’s supposed to be compatible with SCE-MI (Standard Co-Emulation Modeling Interface) and you mix that with another model that’s supposed to be SCE-MI-compatible from another vendor, the burden is on the customer.
Vreeland: SCE-MI increases the complexity level. To understand it well you have to be a hardware and a software engineer. You need to combine both sides, and a lot of times those kind of people are hard to find in a verification team. They tend to be more compartmentalized, whether it’s for historical or some other reasons. It’s changing, but it’s still a problem.
Bailey: In the early days it was design, test and SCE-MI in the middle. It was a testbench. But now, with the directions we’re heading in, who says that has to be a testbench? And who says that has to be in the design? You can start doing testbenches in the emulator and spreading design across the software and the emulator. It’s no longer a testbench and design. It can be any combination in any area. I’m finding people using SCE-MI that I’ve never heard of before. They may have a weird execution engine built into hardware and they’ve used SCE-MI to integrate it into the rest of the execution world. It’s one of those standards no one really hears about, but I’m amazed at where it’s getting used.

SLD: Is it confined to any geographical area?
Bailey: No, it’s worldwide. And it’s any execution engine to another execution engine.
Vreeland: It has good synthesis with the upcoming OVM-UVM standards, because that’s transaction-based. SCE-MI is naturally transaction-based.
Avinun: We see several directions here. One is where advanced verification—OVM or UVM—communicates with the hardware. Customers can’t finish their verification in simulation. In the past it was an option to use hardware-software verification. Now it’s a must-have. It’s not a choice. They can’t finish their verification when it involves all the software, the interfaces and the design complexity. We used to argue smart verification vs. fast verification. Today, customers need the performance to get the job done.
Vreeland: Emulation also includes FPGA-based, right?
Bailey: Absolutely.
Avinun: Yes. And along the lines of advanced verification we don’t see it being used for FPGA prototyping yet, mostly because the debug time is long and the bring-up time is long. By the time you find a bug, you need to wait a long time. That might be the case in the future, but we don’t see it yet. But the other emerging use model we see is connection to a high-level of abstraction or TLM. You implement a portion of your design in TLM or a high level of abstraction and you put another portion in the hardware. In this case, you could have one processor being implemented in TLM, such as an ARM core, and the other processor running inside the hardware. We haven’t seen this used much in FPGA prototyping, either. An FPGA prototyping platform is good for performance, but it’s not good when you’re design isn’t mature. A third use model is in-circuit emulation. That’s been there for years. FPGA prototyping dominates that use model. You don’t have a testbench. You have software and target interfaces.
Vreeland: FPGA prototyping is only used by customers who must have the 50MHz or 100MHz speed. They lose all the debug capability that the emulators attempt to provide. We’re going to have a migration to the transaction-based testbenches in the future because it takes a lot of effort to get a target board up and running. Testbenches are going to be more complicated in the future and they’re going to want to use soft models that they can hook into the emulator through a transaction-based channel like SCE-MI. We see that as a major growth opportunity for emulation, as well as the best route for customers who really want to get an enormous number of cycles implemented before tapeout. Otherwise they’re going to have a big problem. As chips get to a billion gates and above, they’re going to have to move some of the testing into virtual models running on host workstations.
Bailey: The biggest penetration for FPGA prototyping that I’ve seen so far is where you need to deploy a number of copies of the hardware for software verification purposes. The emulators are expensive and you don’t need the debug capabilities of the hardware. If you find a problem in the software then you go back to emulation to investigate it. But for actual execution, then the FPGA enables you to get multiple copies out.

SLD: So it’s find a problem and then investigate, rather than search for problems throughout the design?
Bailey: Yes. With verification these days, you’ve got to be very targeted. If you don’t plan for emulation or prototyping it’s going to cost you twice as much. It’s part of the whole development cycle.

SLD: One of the problems voiced by a number of chipmakers is in the area of constraints.
Vreeland: Higher-end emulators can handle the constraints and functional coverage, which are unavailable in prototyping solutions.
Avinun: The original SCE-MI was invented as an extension of emulation. It wasn’t created by people who had simulation in mind. Initially you couldn’t even simulate the environment. It supported only C, and it was rather primitive. In the last few years, SCE-MI 2.0 started to look at it from a simulation point of view.

By Ann Steffora Mutschler
It’s common for semiconductor design teams to spend 60% to 70% of product development time on verification, which is why verification has bubbled to the top of the management chain as a concern. Executives worry about the predictability of their product development cycle because so much of it is dependent on successful execution of verification, the ability to achieve coverage closure and the ability to predict a schedule based the confidence level they have on verification.

Corner cases are an integral part of this verification process. But as simple as the issue might sound, defining what a corner case is can be elusive.

“It is a simple question, but there is unfortunately no simple answer to it primarily because there is no standard, agreed-upon industry definition of what a corner case is,” said Swami Venkat, senior director of marketing for the verification group at Synopsys. “Corner cases are really a combination of multiple inputs like stimulus, the state of the design, and certain conditions that the software may be executing on the chip. All of these conditions need to occur so that it triggers some sort of misbehavior and unexpected behavior on the design. If you have such a behavior then obviously you want to be able to find it in RTL because finding it post silicon becomes extremely expensive both from a time point of view as well as from dollars point of view.”

Consider Intel’s latest corner case problem. It can be extremely challenging for design teams to think of all of the various scenarios and situations that will lead up to a corner case, particularly as much more functionality is getting integrated into SoCs today. “There are various conditions, multiple simultaneous operations that are happening on the design and now to be able to simulate all of that requires extensive amount of planning, requires a lot of experience and obviously the right technology to be able to uncover this,” he said.

Standards help in this regard, and the more that can be standardized presumably the fewer corner cases. But even with standard interfaces there are plenty of corners to consider.
Verification IP (VIP) also can check that each core as well as the system interconnect supports the same signals and speaks the same language.

“We have to speak to all cores with all kinds of interfaces,” said James Mac Hale, vice president of Asia operations at Sonics. “Each core might conform to a single standard, but we have to interface to all of them all together. Our challenge is often not just interfacing to all the cores out there, but also managing the translation from one to another. We spend an inordinate amount of time and resources verifying the functionality of our IP that they’ll seamlessly connect to all of the cores and function correctly. Most customers are coming to appreciate that once they try to put together anywhere from 50 to 100 cores in the same chip. With design re-use, it may very well be true that they have used the same cores previously and therefore know in theory that all those cores are good, but the problem is when you hook them together, something doesn’t work.”

To identify corner cases Synopsys’ Venkat said design teams are simulating as much as possible in the RTL and are also using a combination of techniques. He said more and more semiconductor teams are bringing architects, verification leads and design leads together to create an extensive verification that includes all of the various scenarios that would have to be verified in the RTL before they commit a design. More and more people are deploying constrained random verification where, instead of an engineer imagining the right sequences that would lead to a bug, they let the constraint solver come up with interesting sequences and combinations of inputs.

“When you do that, they need to have good coverage that will track how much of the design has been covered, what combination of the input specification has been covered and verified. There are customers that spend a lot of time creating extensive up-front plans. Then they use constrained random and then coverage to find out how much of the design has been verified. As you continue to execute through the plan the level of confidence about the functionality of the design and about its ability to meet the requirements goes up. Essentially, you have to be able to create stimulus that will replicate the real-life behavior of the design, and you need to be able to do it in RTL so you can find these problems very early in product development,” he added.

“We see people investing more and more in simulations where they try to run a lot of simulations in parallel. They deploy a lot of constrained random technology, and that’s where newly announced methodologies like Accellera UVM and the VMM methodology can help. The language offers various constructs, but the methodology can help the user leverage the constructs, put together a very effective state-of-the-art environment and create scenarios to verify the design,” Venkat said.

And due to the serious repercussions of not having adequate coverage for corner cases, the world’s leading semiconductor companies are partnering closely with EDA companies to create specific technology just for them.