Requirements Engineering for Machine Learning: Perspectives from Data Scientists

Products and services that rely on machine learning are everywhere. The shift from source code to data has enabled some important breakthroughs. But what changes for the requirements engineer?

This work in this paper was initiated when I reached out to Prof. Andreas Vogelsang, TU Berlin, after having seen pictures from an interesting keynote he gave at an industry conference in Germany. He talked about novel challenges when engineering machine learning (ML) based systems. When I encouraged further work on the topic, he invited me to collaborate. This is our first paper together on the topic.

Why are ML-based systems different?

ML appears everywhere nowadays. It might not be the best solution for every problem, but it appears every domain wants to give it a try – after all, it has enabled some major technological breakthroughs. We have all heard about outperforming humans in image recognition, natural language processing, and playing games.

The key enablers for the ongoing ML disruption are based on the combination of the computational power of modern GPUs and the availability of large amounts of data. Some call it Internet-scale data. The frameworks needed to get going with ML mature at a high pace and they get increasingly user (developer) friendly. It is now much easier to get started with ML, including the often so tempting deep learning. It is clear that more and more companies use ML to improve their products and services.

In this paper, we argue that requirements engineers need to adapt their work. ML engineering constitutes a paradigm shift compared to conventional software engineering. Andrej Karpathy, Director of AI at Tesla, even refers to the new era as “Software 2.0” – no longer does all behavior emerge from a set of manually coded rules. Instead, ML approaches generate rules based on a set of training data. In addition, a recent survey by Ishikawa and Yoshioka (2019) suggests that requirements engineering is the most difficult activity when developing an ML-based system.

Insights from four interviews with data scientists

We designed an interview study to begin exploring how data scientists approach expectations on the output of their ML models, i.e., requirements engineering. Three research questions guided our initial study:

How do data scientists elicit, document, and analyze requirements for ML systems?
What processes do data scientists follow and which parts relate to requirements?
What are requirements-related challenges that data scientists face?

While the results from this exploratory study obviously is only preliminary, we did find quite some interesting results related to RE for ML.

New and amplified challenges for ML systems

There are some new challenges that arise when doing RE for ML systems. One example is how to best specify requirements on the predictions. We argue that the resulting predictions can be considered a functional requirement. But it is a functional requirement that requires the stakeholders to understand ML metrics – this can be an issue.

Explainability is important for any software system that offers automated decision-making. However, ML systems are different compared systems implemented in human-written source code. To update the decision-making of conventional software, the developers must understand the source code. For ML systems, on the other hand, a developer usually doesn’t change the code of the ML algorithm to influence the output, but instead manipulates the training data. The interplay between data and algorithms must be explained.

Freedom from discrimination is crucial, but what it means is highly context-dependent. First of all, ML systems are designed to discriminate, i.e., they identify recurring patterns in data stereotypes and apply them when judging new input. However, some forms of discrimination are unacceptable in society – or even forbidden by law. Filtering loan applications by race or gender is not allowed, but the same features might be essential in a medical system. Requirements engineers must understand when and how to discriminate.

Related challenges are connected to legal and ethical aspects. When conducting the interviews, the General Data Protection Regulation (GDPR) was still a fairly new concept. GDPR is a challenge to data scientists, as the law mandates that personal data can be used only in ways specified by explicit consent. Basically, the data scientist must know the model before developing it. If an organization receives consent to collect 100 features from users but in the end, only a subset is used to train a model – then the organization collects “good to have data” that is not used, which is illegal. Requirements engineers need to balance the need for exploratory data analytics and the constraints enforced by the regulations. Also, data lineage will be critical to ensure that no illegal features have influence the ML system.

Data requirements is fundamental to RE for ML, as the training data is an integral part of any ML system. One interviewee compared ML training to compilation, where the source is both code and training data. We believe that training data needs specified and validated requirements similar to the counterparts for source code. Requirements on training data might have to cover both quantity (the sheer volume) and quality (such as completeness, consistency, and correctness). Data requirements must also specify how the data was collected and how it was preprocessed before being used as training data.

Changes to the RE activities

We discuss the impact on the four core RE activities 1) elicitation, 2) analysis, 3) specification, and 4) validation. The complete list is found in the paper, but some key adaptations that RE needs include:

Elicitation: All relevant sources of data must be identified. The data scientists pop up as important stakeholders. The importance of legal experts increases – some data might be directly illegal to use.
Analysis: The performance of the ML system will be expressed in non-trivial metrics. Requirements engineers must facilitate discussions between data scientists and customers.
Specification: Requirements specifications for ML systems must evolve to better cover details related to data requirements, e.g., the data collection, the data formats, the ranges of data – and also requirements on quantity and quality. Novel types of quality requirements will be in focus, e.g., explainability and freedom from discrimination.
Validation: The requirements engineer must specify actions to validate that the training data still match real data encountered in operation – and specify when to retrain the ML models. Since data characteristics may change over time, requirements validation becomes a continuous activity.

Implications for research

RE for ML systems is special due to the different paradigm used to develop data-driven solutions.
We present a first step toward an RE methodology for ML systems.
ML systems will always be enabled by conventional software, thus future research on RE for co-existence of ML and source code is needed.

Implications for industry

Requirements engineering must adapt to fit the ML era, i.e., data-driven products and services.
The output from ML systems is described in technical measures that customers might not understand. Requirements engineers must bridge this gap.
Data scientists use plenty of techniques to balance, clean, validate, and explain data. Requirements engineers must relate these to the needs and context of the customers.

Andreas Vogelsang and Markus Borg. Requirements Engineering for Machine Learning: Perspectives from Data Scientists, In Proc. of the 6th International Workshop on Artificial Intelligence for Requirements Engineering (AIRE), 2019. (link, preprint)

Abstract

Machine learning (ML) is used increasingly in real-world applications. In this paper, we describe our ongoing endeavor to define characteristics and challenges unique to Requirements Engineering (RE) for ML-based systems. As a first step, we interviewed four data scientists to understand how ML experts approach elicitation, specification, and assurance of requirements and expectations. The results show that changes in the development paradigm, i.e., from coding to training, also  demands changes in RE. We conclude that development of ML systems demands requirements engineers to: (1) understand ML performance measures to state good functional requirements, (2) be aware of new quality requirements such as explainability, freedom from discrimination, or specific legal requirements, and (3) integrate ML specifics in the RE process. Our study provides a first contribution towards an RE methodology for ML systems.