Test visualization with Unity

Do results from regression testing overwhelm you? Does the current result overview hide all details needed for a deeper analysis? We present a proof-of-concept visualization using the city metaphor. The tool is developed using the Unity game engine and offers convenient interaction.

Finally, there is a published paper describing our work on cityscape visualizations of regression test results! While both Daniel Hansson at Verifyter and myself have shared the content and shown demos at several occasions the last year, this is the first proper publication that describes our work. It has already been presented as a poster at DVCon US in San Jose and won us the best tool demonstration award at ICST in Västerås Sweden. We have also presented it for the Swedish Association for Software Testing and at workshops organized by the research projects TOCSYC and TESTOMAT. At last, there is a publication to refer interesting readers to!

This work originated in me taking two MOOCs on game development to learn Unity. I really enjoyed the course and realized I had found a tool that could be used to quickly create fairly impressive demos and prototypes. While working on the course, Daniel Hansson (at that time involved in the TESTOMAT project application) happened to see my work. I took the chance to ask him if he had any ideas for things to visualize – and he returned shortly with a great idea: results from regression testing in ASIC verification. We turned this into a MSc thesis project proposal, and found the creative KTH student Andreas Brytting to start working on the concept. A first version of this paper was sent to DAC, the premier ASIC conference (even older than ICSE), but despite a promising rebuttal phase, visualization research is probably too strange for that forum. Now we are happy to share the results with DVCon Europe!

ASIC verification vs. software testing

My first step in this project was to understand ASIC design and verification better. There are plenty of similarities between ASIC design and software engineering, and the ASIC domain is surely influenced by the softer side. Not strange, since ASIC design means writing code in hardware description languages such VHDL or Verilog. Nowadays, hundreds of ASIC engineers collaborate in large code repositories using git, continuous integration (CI), and test automation with coverage metrics (combined with constrained random testing). What I as a software developer find particularly interesting is that the laws of nature become evident. An ASIC design must consider things such as temperature and electrical interference.

The figure below shows an overview of ASIC design (below the arrow) and verification (above). During the design, ASIC engineers develop the code and the comes up with the physical design. The result from the design phase is called a “tapeout”, which is sent to a factory for manufacturing. The ASIC verification starts early (analogous to good practices in software testing) and tests for functional correctness in a CI pipeline, but on simulated hardware. When there is a physical chip, testing commences, encompassing probing and various measurements on the physical thing. At the same time validation takes place, which means trying to run customer applications on the chip. All these activities constitute one development “spin”, and typically issues are found that requires changes to the design. This is called a “respin” and the entire process is reiterated. Respins are avoided at all costs, since they delay shipping the final ASIC. Still a few respins are normal, and each spin takes months.

The results from Verifyter’s tool PinDown

Not only are respins costly, they are also increasingly difficult to avoid. ASIC designs are getting more complex (multicore CPU designs with multithreading etc.), ASICs must be smaller, strike the right balance between high performance and low energy consumption, and ever-shorter time-to-market. This results in a high demand for tools that support ASIC design and verification.

In this work, we visualize results from PinDown, a commercial tool developed by the TESTOMAT partner Verifyter. PinDown integrates in the CI pipeline and analyzes results from regression testing to cleverly revert commits that cause test cases to fail and notifies the responsible ASIC designer. Running PinDown over long projects enables collection of large amounts of data on commit history and regression test results. But how can this data be presented in a way to support analysis? Enter visual analytics.

Cityscape visualization using Unity

Visual analytics is “analytical reasoning facilitated by interactive visual interfaces”. The idea is to let the user get to new insights by smart information visualization that is combined with very flexible interaction – let the user twist and turn the data, zoom and filter whatever is on the screen. Inspired by CodeCity’s pioneering work on visualizing source code as cities, we illustrate regression test results using a cityscape. This is not so far-fetched… everyone knows that close-ups of electronic chips look like miniature cities!

Using Unity, a state-of-the-art game engine and IDE, to enable visual analytics of testing results is a novel contribution of our work – although several researchers have used Unity for software visualization lately, e.g., Merino (2017). What Unity offers us is very productive development environment, basically drag and dropping 3D scenes combined with some C# coding. It also scales really well, Unity renders the cityscape fast and we believe really large datasets could be visualized. On top of this, we get a lot of interaction right out-of-the-box. Games are all about interaction and the physics engine is there for  us to use complete with mouselook and WASD navigation – the default scheme for 3D games.

Helping verification engineers to focus

To provide a proof-of-concept, we collected 6 months of regression test data from an industrial ASIC development project. The project consists of some 60,000 files and roughly 1,500 commit sets by 95 design engineers. The corresponding test suite consists of 500 test cases that are executed 3-8 times per day. Finally, we use the red-yellow-green color scheme to illustrate how frequently commits to the specific file was involved in failed regression test execution. For read building, commits to the file was always followed failing test executions. A green building, on the other hand, shows that commits to the file never has caused any test cases to fail.

The human eye is good at finding anomalies, and we believe our approach can let humans do that in a convenient way. We argue that the cityscape visualization supports ASIC verification engineers in (at least) three use cases: 1) general exploration of the ASIC design under test, 2) localization of error-prone parts of the design – perhaps these parts could be stabilized somehow?, and 3) detection of potential coverage holes – are there really enough test cases targeting this part of the design? We added options for the user to hide specific commit sets and to filter the data based on the time dimension. Also, we added searchlights from the sky (see the figure on top of the page) that can be configured based on user-defined thresholds such as the minimum number of commits.

We have organized an initial evaluation of our prototype tool with two senior verification engineers involved in the ASIC development project. We let the engineers try our tool and assess the potential coverage hole we identified as well as the particularly error-prone area. The engineers confirmed that both findings appeared valid, and decided to continue investigations of there parts of the design. They also confirmed that the cityscape provides a new and interesting perspective on the design under test. But the evaluation also revealed that the 3D navigation using the WASD keys and mouselook is not necessarily intuitive to users. We learned a loot during the initial evaluations, and look forward to implement some of the feedback in a future version of the tool – and to run a larger evaluation.

Implications for Research

• A proof-of-concept that Unity is a feasible platform to enable visual analytics.
• An argument that some challenges of 3D visualization, especially occlusion, can be overcome by using an easily navigable game engine.
• The paper shows some similarities between ASIC design verification and software testing.

Implications for Practice

• Visualization of regression test results can reveal new insights beyond standard result matrices and dashboards.
• Bringing visual analytics to the change and testing history can help engineers identify error-prone parts and holes in the test coverage – even without coverage metrics.
• Since software is invisible, progress is difficult to show. A cityscape can be used to illustrate both development and testing effort.

Markus Borg, Andreas Brytting, and Daniel Hansson. Enabling Visual Design Verification Analytics – From Prototype Visualizations to an Analytics Tool using the Unity Game Engine. In Proc. of the Design and Verification Conference Europe (DVCon EU), Munich, Germany, 2018. (preprint, slides)

Abstract

The ever-increasing architectural complexity in contemporary ASIC projects turns Design Verification (DV) into a highly advanced endeavor. Pressing needs for short time-to-market has made automation a key solution in DV. However, recurring execution of large regression suites inevitably leads to challenging amounts of test results. Following the design science paradigm, we present an action research study to introduce visual analytics in a commercial ASIC project. We develop a cityscape visualization tool using the game engine Unity. Initial evaluations are promising, suggesting that the tool offers a novel approach to identify error-prone parts of the design, as well as coverage holes.