Published on March 04th, 2008

If you can’t measure progress against your plan, you have no plan!

There are verification technologies available today that use markup languages to capture functional verification plans, enabling measurement of completion throughout a project.

Ilana GolanFunctional verification requires hundreds of careful tradeoffs and decisions to achieve high quality tapeout. Those important decisions are difficult to retain without capturing and measuring them against your original intent. Today’s complexities require measurement capabilities that allow for tracking, measuring, and managing the thousands of details they represent. The good news is that there are verification technologies available today that use markup languages to capture functional verification plans, enabling measurement of completion throughout a project. This article describes the critical details of such a markup verification plan, technology for creating the plan and the viewing status of those measurements.

Almost every project has some sort of initial plan. It can be in the form of a long list of (mostly directed) tests – called test plans – or a very thoroughly articulated and well thought out verification plan derived from the spec. These plans can take between a few days and a few weeks of work to create. Why invest the time to create such integrate plans up front?

Author’s note: “If you fail to plan, you plan to fail?

So, what benefits do we actually get out of the verification plan?

  • Alignment on the interpretations of the desired product functionality by the various constituents, or broadly communicate refinement of existing specifications, if needed.

  • Development of a common description of desired functionality.

  • Common understanding of tasks, goals, and complexity of the project.

  • Define and capturing project milestones.

  • Definition of the engineering view of the plan, assigning resources to features or feature sets.

  • Better risk analysis when needed: early tapeout, lack of resources, new requirements and spec changes.

Of course, all these benefits are easier said than done. There are various complexities that may complicate the matter when trying to create and maintain a verification plan. Some of these complexities include:

  1. How do we know we wrote a thorough plan?

  2. The specification itself is updated often and is hard to maintain which can lead to an outdated and useless plan.

  3. Need for plan reuse, between projects and among different owners in the team, especially at the system level.

  4. How do we track our progress against this plan?

Let’s take a closer look at how we can help overcome these complexities and achieve the goals of the verification plan mentioned above.

1. Write a thorough plan

When creating a verification plan, it is best to have a brainstorming session with all the relevant stakeholders. The session should include a discussion of the various features and components. It is always best to be near a white board for brainstorming and going over the various aspects of the spec and making sure everything is captured. That’s not necessary for every stakeholder, but the idea is to get alignment on the various interpretations of the functionality, and have the time for refinement of existing specifications if necessary.

In order to go over the spec in a systematic way, it is best to use markup languages to capture the details directly from the spec to the plan. By using a markup language verification planners can go over the spec and decide exactly what sections are the most important. They would use an annotator to mark desired sections or attributes in the spec to be automatically captured in the plan. It is good practice to use a tool that assists in resolving un-annotated areas of spec, so you can make sure your plan is thorough and complete.

2. Update the plan based on the spec

Keeping the plan up to date is probably the biggest challenge as spec changes often continue throughout the project. Some teams try to manually figure out what changed and update the plan on the fly, but this is very tedious and prone to introduce errors.

Planning software that is available today allows you to actually connect the spec with the plan. By annotating sections from the spec to the plan, you ensure that when spec changes occur in these sections, you will be notified about the change. The update will happen automatically after your approval of the change so your plan is relevant and always up to date – which is a huge benefit.

Figure 1

3. Plan reuse

Many benefits can be gained through reuse of verification plans from block-level to system-level, between projects and among team members. Both methodology and technology reuse play a key role. From a methodology standpoint, when writing the verification plan it is best to split the functional and design requirements to enable reuse between projects, and split the functional interface and core features to enable reuse to the system level.

The plan at this stage may look something like the following:

Figure 1

From a technology standpoint there are few basic capabilities that the tools need in order to enable reuse:

  • Import verification plans into a hierarchy of plans.

  • Pass parameters to a verification plan. For example: a bus protocol plan may look almost the same whether it is a 32 bit or a 64 bit bus. There’s no need to maintain two plans – just have the ability to execute the same bus plan with a bus-width parameter.

4. Tracking progress against the plan

Author’s Note: Lord Kelvin said in 1883, “When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.? With statements like that, I think Lord Kelvin would have been a fantastic verification engineer, and I would have wanted him on my team!

Now let’s take a deeper look at the two main phases of planning.

Phase1: This is the “defining the metrics? stage. In this phase we look at the data and metrics that will be used in the verification process and begin capturing the project milestones, resources needed, and other important data. In the verification plan itself, we define the metrics needed to track progress of each feature and section, including various forms of coverage, assertions, and directed tests, if relevant.

We also want to make sure we are paying close attention to the needs of the various stake holders so that everyone can focus on the information that is interesting and most useful for them (i.e. the systems person will not want to see the same information as the designer). Next, we want to define the milestones to be achieved by a specific date.

In this first phase the verification engineers start implementing the coverage model according to the plan. It is helpful to have automation-based technology to help track the implementation progress so no items are left behind. Some technologies available today will even generate the coverage code based on the plan and help map between existing coverage code and coverage items in the plan.

Phase2: This is really the “let’s get down to business and begin tracking? phase of verification. We want to run simulations and use management solutions to begin tracking the metrics defined in the plan. These will help analyze the project's status with respect to the plan. Newer management solutions allow users to easily capture the various metrics defined in the plan and show progress. They also deliver up-to-date and accurate reports that can be shared among the team to give complete picture of how the overall design is progressing.

By following these brief guidelines and utilizing good planning with some of the newer verification management solutions, your team will truly benefit. You will understand if you have reached the milestones you planned. And you will be better equipped to make changes and shift priorities, if you haven’t reached your milestones. Perhaps you will need to shift resources, change a release date, or develop a better way to observe the risk analysis and roll the information up to management. In any case, you will feel much more confident in your overall design process, and have a much higher rate of project success with a little bit of time invested up front on planning.

Figure 1

Ilana Golan is a Solution Engineer at Cadence Design Systems responsible for Verification Planning, Methodology and Management solutions. She has worked with several industry-standard protocols – including ARM, PCI-E, PCI, and SPI – and she has supported many key customers.

Prior to Cadence, Ms. Golan was a Logic Verification and EDA engineer at Intel, Haifa. Prior to Intel, she served as Lieutenant, Commander of the F16 Flight Simulator instructors, where she led training of air force pilots. Ms. Golan holds a B.Sc. in Computer and Software Engineering from the Technion - Israel Institute of Technology, Haifa, Israel.

Tech Videos

©2018 Extension Media. All Rights Reserved. PRIVACY POLICY | TERMS AND CONDITIONS