ICES-ENEA Seminar: Debugging of Embedded Systems
The ICES debugging seminar took place at Enea in Kista on 14th October 2010 with 55 participants from 8 companies, 4 KTH Schools - and our international keynote speaker, Kees Goossens from Eindhoven Technical University. The seminar followed the traditional ICES seminar set up by encompassing viewpoints from different domains and from academia. This was the first ICES seminar on the theme of debugging, and consequently it was natural to focus on a problem inventory for debugging overall.
The academic keynote talk focused on research in debugging for multicore systems, whilst the industrial speakers addressed problems and solutions for debugging in automation, telecom, suppliers and consultancy. Together, the presentations drew up a nice problem inventory for debugging, the essence of which is summarized here!
Presentations are available to ICES members only, via the link to the Members Area (below).
For single processor systems, the debug environments are today rather mature. However, with new applications and new electronic platforms, system complexity is growing. Today, embedded systems are usually distributed systems, making debugging more difficult. The emerging multicore processors provide another scale of complexity.
Debugging in embedded systems is intrinsically hard for a number of reasons including:
* Limited and intrusive observability: It is not trivial to get access to embedded systems and if you do, you often impact their behaviour. There is therefore a need for non-intrusive hardware level debug support.
*No consistent global state: Because of multiple clock domains, it is difficult to obtain a global state in distributed and multicore systems.
*Non-determinism: modern processor systems are non-deterministic because of time delays, transient errors and race conditions.
In industry, management is often reluctant to invest in testing and debugging. Yet a large portion of industrial development efforts (often reported as greater than 50%) are spent on debugging. Furthermore, it is rare to find testing and debugging taught in universities - so far, "debugging" has certainly not been established as an academic discipline in its own right. The participants agreed with speaker Alf Larsson (Ericsson) that industry needs to take a more proactive approach in addressing debugging. The participants also agreed that debugging deserves a more thorough treatment in academia. The area is full of challenges which closely relate to those faced in embedded systems design -namely that of dealing with the increasing complexity.
Other questions addressed at the workshop included the following:
* Model-based development relies on proper abstractions. To be useful for debugging, proper abstractions of faults and failures need to be developed to suitable abstraction levels. The devil lies in the detail - there is a need to harmonize different levels of abstraction to be able to zoom in and out of the system.
* A huge number of analysis and verification techniques have been proposed, however it is not so clear how they can best be combined in order to verify systems and to support debugging.
* Developing deterministic systems is costly, yet architectures and protocols can provide support for developing sufficiently predictable systems. Here techniques from distributed systems can be employed for multi-core systems, which are now facing similar problems. The differences between systems on chips and distributed systems are diminishing, and the experiences from this could be better shared. Debugging efforts can be reduced by proper design (fault-avoidance, correct by design), by introducing some level of regularity (order) and by reducing the accidental complexity. In this regard, one can note increasing complexity of modern processors, and ask whether there is room for improved processor architectures that can provide (to some extent) both performance and predictability?
* In debugging, a key issue is what to monitor and what to log. Debugging embedded systems requires developing a good understanding of the essential information part of the system - its state. This information, which is equally important for system design, can be used to perform smart logging and guide debugging.
* While project planners can estimate the time required for developing software, there is much less understanding and basis for even beginning to estimate the time required for debugging!
* In logging data from an embedded system during actual runs, testing or simulations, large amounts of data is generated and it is difficult to get a useful overview. There is a need to extract relevant information for different users, conforming to the users’ context. This also relates to providing system abstractions useful for debugging.
* How can one approach debugging in education? What is the key competence required by a debugger? How can debugging systematically be taught? and at which level should it be introduced?
*There is also room for improvement in the area of tools: SW & HW debug support is often provided separately. Without a solid foundation in hardware level debugging, it is difficult to support proper system level debug tools.
Moreover, tool support is mainly provided for system development, with less support for later product stages. Debugging in the field is a major challenge. – enabling post-mortem debug solutions which allow developers to debug the system as if it was up and running for real "replay in the lab" would be a big step forward.
In closing, the participants in the seminar advocate a proactive approach for debugging and for V&V in general. Planning resources for debugging needs to be seen more of as an investment - this is an immediate challenge for organizational cultures and management.