Accepted Industry Papers and Presentations

Session Wednesday, 25th, 11:00-12:30, I1: Best Paper Award Session

Session chair: Sigrid Eldh

An Overview of Numalis Software Suite for Reliable Numerical Computation. Arnault Ioualalen, Matthieu Martel and Nicolas Normand.

Industrial Evaluation of Search-Based Test Generation Techniques for Control Systems. Florian Hauer, Alexander Pretschner, Maximilian Schmitt and Markus Grötsch.

Reliable Inspection of an Autonomous System at System Runtime with Built-in Data Probes. Joachim Fröhlich and Christoph Stückjürgen.

Session Tuesday, 24th, 02:00-03:30, I2: Test and verification

Session chair: Adnan Causevic

Gherkin Syntax Extension for Parameterization of Network Switch Configurations in Test Specification. Tianyao Li, Shigeru Tsubota and Koji Hirono.

CrashOS : Hypervisor testing tool. Anaïs Gantet.

A Configurable Validation Environment for Refactored Embedded Software: an Application to the Vertical Transport Domain. Goiuria Sagardui, Leire Etxeberria, Joseba Andoni Agirre, Aitor Arrieta, Carlos Fernando Nicolás and Jose María Martín.

Session Tuesday, 24th, 02:00-03:30, I3: Profiling Industrial Code

Session chair: Rix Groenbom

Assert Use and Defectiveness in Industrial Code. Steve Counsell, Tracy Hall, Thomas Shippey, David Bowes, Amjed Tahir and Stephen MacDonell.

A static analyzer for Industrial robotic applications. Raoul Jetley, Avijit Mandal, Sreeja S Nair and Meenakshi D'Souza.

GUI-Profiling for Performance and Coverage Analysis. Nico Beierle, Peter M. Kruse and Tanja E. J. Vos.

Session Tuesday, 24th, 04:00-05:30, I4: Validation of Security & Safety

Session chair: Veena Mendiratta

IV&V Case: Empirical Study of Software Independent Verification and Validation based on Safety Case. Kazuki Kakimoto, Kensuke Sasaki, Hiroki Umeda and Yasushi Ueda.

Appwrapping Providing Fine-Grained Security Policy Enforcement Per Method Unit in Android. Sung-Hoon Lee, Seung-Hyun Kim, Soohyung Kim and Seung-Hun Jin.

A Practical Approach towards Validating HIL Simulation of a Safety-critical System. Athanasios Stratis and Adnan Causevic.

Session Tuesday, 24th, 04:00-05:30, I5: Defect Prevention and Anomalies

Session chair: Alf Larsson

Getting Defect Prediction into Industrial Practice: The ELFF Tool. David Bowes, Steve Counsell, Tracy Hall, Jean Petric and Thomas Shippey.

Rich Network Anomaly Detection Using Multivariate Data. Veena Mendiratta and Marina Thottan.

Diagnosing Development Turbulence in Agile Processes. Ram Chillarege.

Session Wednesday, 25th, 02:00-03:30, I6: Cloud & Telecom Reliability

Session chair: Kristoffer Ankarberg

Experience Report: Testing private cloud reliability using a public cloud validation SaaS. Aniket Malatpure, Faraz Qadri and John Haskin.

BRACE: Cloud-based Software Reliability Assurance. Kazuhira Okumoto, Abhaya Asthana and Rashid Mijumbi.

SRC Ratio Method: Benchmarking Software Reliability. Pete Rotella and Sunita Chulani.

Session Wednesday, 25th, 02:00-03:30, I7: Automotive & Transport

Session chair: Eric Jenn

A Comparative Study of Static Analysis Tools for AUTOSAR Automotive Software Components Development. Vladimiro Vacca, Stefano Scala, Raffaele Rodolfo Maietta and Alfredo Imparato.

Principles for Systematic Development of an Assurance Case Template from ISO 26262. Thomas Chowdhury, Chung-Wei Lin, Baekgyu Kim, Mark Lawford, Shinichi Shiraishi and Alan Wassyng.

Control Parameter Optimization for Autonomous Vehicle Software Using Virtual Prototyping. Siyuan Dai, Takato Masuda, Yusuke Kashiba, Nikos Arechiga, Shinichi Shiraishi, Joseph Hite, Scott Eisele, Jason Scott and Ted Bapty.

Session Wednesday, 25th, 04:00-05:30, I8: Reliability issues in Industry

Session chair: Manrico Fedi Casas

Visualization of Specification Coverage: A Case Study of a Web Application Development in Industry. Hiroyuki Nakagawa, Toshinobu Hasegawa, Shori Matsui and Tatsuhiro Tsuchiya.

Applying software reliability engineering process to software development in Korea defense industry. Taehyoun Kim, Samjoon Park and Taeho Lee.

Towards Embedded System Agile Development Challenging Verification, Validation and Accreditation : Application in a Healthcare Company. Clément Duffau, Bartosz Grabiec and Mireille Blay-Fornarino.

 

Abstract of Industry papers and presentations

 

Arnault Ioualalen (Numalis), Matthieu Martel (Université de Perpignan Via Domitia) and Nicolas Normand (Numalis).

An Overview of Numalis Software Suite for Reliable Numerical Computation

Abstract: Numerical algorithms are used in many areas but they rely on approximate computations due to the finite precision computer arithmetic. As critical systems perform more and more calculations, needs for verification and validation techniques and for assisted development increase, the computer arithmetics being particularly not intuitive. It is then necessary to provide tools to the programmers, to help them to validate and increase the numerical quality of their codes and, broadly, to develop more fastly more reliable numerical codes. In this article, we give a description of the main problems concerning numerical accuracy encountered in industry at the software engineering level and we give an overview of the solutions proposed by the software suite developed by the Numalis Company. This suite contains tools for verification and validation by static and dynamic analysis as well as assisted development tools. The latter tools optimize programs in order to make them compute more accurate results and they also infer the least formats, in terms of bit size, in order to fulfill accuracy requirements.

Florian Hauer (Technical University of Munich), Alexander Pretschner (Technical University of Munich), Maximilian Schmitt (Jungheinrich Moosburg AG & Co. KG) and Markus Grötsch.

Industrial Evaluation of Search-Based Test Generation Techniques for Control Systems

Abstract: The system and software of industrial trucks get more and more complex due to the rapidly increasing demand for more functionality. The effort of manual verification rises even faster. Automated testing techniques need to be introduced into the verification process. In this work, we adapted a search-based test generation technique for the requirements-based black box test of control systems of counterweight forklift trucks. Fitness functions are derived from industrial requirement documents. Test cases consiting of multiple input signals are generated. We evaluated our approach on such a control system and our results show that faults can be found in the system and in the environment model on which the verification process is based on.

Joachim Fröhlich (Siemens AG) and Christoph Stückjürgen (Siemens AG).

Reliable Inspection of an Autonomous System at System Runtime with Built-in Data Probes

Abstract: Dynamically programmable data probes assist in solving verification issues of autonomous systems. Data probes can be programmed to monitor timed sequences of system data, to check system properties, and to stimulate and control autonomous systems. We present several probe programs to inspect and check various aspects of a safety invariant during a mobile robot’s task execution. We report on experience and identify open issues.

Tianyao Li (Hitachi, Ltd.), Shigeru Tsubota and Koji Hirono.

Gherkin Syntax Extension for Parameterization of Network Switch Configurations in Test Specification

Abstract: We applied test automation in the style of Behavior-Driven Development (BDD), an agile software development technique. In applying BDD, the problem is that a large amount of similar configurations of network switches have to be written in the BDD test specification files. To solve this problem, parameterization of configurations is considered effective. However, Gherkin, the BDD test specification language, does not provide syntax for parameterization. In this paper, we propose a syntax extension of Gherkin, which enables parameterization of configurations. From experiments on 11 configurations, parameterization using the proposed syntax extension can reduce the number of the lines used to describe configurations by 39%.

Anaïs Gantet (Airbus).

CrashOS: Hypervisor testing tool

Abstract: Hypervisors are software more and more widespread because of the increasing usage of virtualization, especially in the Cloud. But they are not bug-free. CrashOS is developed to test them and contributes to find security vulnerabilities in hypervisors.

Goiuria Sagardui (University of Mondragon), Leire Etxeberria (University of Mondragon), Joseba Andoni Agirre (University of Mondragon), Aitor Arrieta (University of Mondragon), Carlos Fernando Nicolás (Ik4-Ikerlan) and Jose María Martín.

A Configurable Validation Environment for Refactored Embedded Software: an Application to the Vertical Transport Domain

Abstract: As systems evolve, their embedded software needs constantly to be refactored. Moreover, given the different needs of different customers, embedded systems require to be customizable. The variability of these systems is large, and requires automated testing solutions. In this paper we propose a methodology that automatically generates validation environments for highly configurable embedded software that is being refactored. The method has allowed for systematically testing a real-world industrial case study involving the software in charge of controlling the doors of an elevator. Finally, we extract the lessons learned from its application.

Steve Counsell (Brunel University), Tracy Hall (Brunel University), Thomas Shippey (University of Hertfordshire), David Bowes (University of Hertfordshire), Amjed Tahir and Stephen MacDonell.

Assert Use and Defectiveness in Industrial Code

Abstract: The use of asserts in code has been a recognized programming construct for many decades. In theory, liberal use of asserts should be encouraged and the physical position of asserts in the class should make no difference to their effectiveness. A previous empirical study by Casalnuovo et al., showed that methods containing asserts had fewer defects than those that did not. In this paper, we analyze the test classes of two industrial telecom Java systems to lend support to, or refute that finding. We also analyze the position of asserts in methods to determine if there is a relationship between assert placement and method defect-proneness. Finally, we explore the role of test method size and the relationship it has with asserts. In terms of the previous study by Casalnuovo et al., we found only limited evidence to support the earlier results. We did, however, find that defective methods with one assert tended to be located at significantly lower levels of the class position-wise than non-defective methods. Finally, method size seemed to correlate strongly with asserts, but surprisingly less so when we excluded methods with just one assert. Methods with just a single assert appear to be different in terms of their link with defects than methods with multiple asserts.

Raoul Jetley (ABB Corporate Research), Avijit Mandal, Sreeja S Nair and Meenakshi D'Souza.

A static analyzer for Industrial robotic applications

Abstract: In this paper, we describe a static analysis approach to detect potential runtime errors for a programming language that is used to program industrial robots. The language we deal in this paper is RAPID, a high level programming language for programming ABB industrial robots. The presence of real-time interrupts, exception handlers and complex data-types makes it a difficult language for general purpose static analyzers. The properties of interest include some {generic} programming errors as well as some domain-specific properties that the robot system must comply with. {Generic} programming errors include properties like integer overflow, array access out of bounds and division by zero. An example of a domain-specific property is defining boundary limits for robotic arm movement. We have developed a tool to detect these errors successfully in the presence of real-time interrupts.

Nico Beierle (Assystem Germany GmbH), Peter M. Kruse (Assystem Germany GmbH) and Tanja E. J. Vos (Open Universiteit).

GUI-Profiling for Performance and Coverage Analysis

Abstract: Existing software analysis methods for performance and coverage analysis are typically tied to the source code of software applications. In this work, we extend theses methods to the Graphical User Interfaces of modern applications, motivated by the desire to bring the user perspective into focus of software quality assurance and testing at the GUI level. We present and discuss various profiling procedures, their advantages and disadvantages, the arising challenges and the identified solutions. The identification and classification of the GUI components recorded during the monitoring process posed particular problems. The monitoring and collection of data could be well implemented, while detailed improvements in the evaluation of results are still necessary.

Kazuki Kakimoto (Japan Aerospace Exploration Agency), Kensuke Sasaki, Hiroki Umeda and Yasushi Ueda (Japan Aerospace Exploration Agency).

IV&V Case: Empirical Study of Software Independent Verification and Validation based on Safety Case

Abstract: We describe a framework for constructing an IV&V case, which is an expanded safety case, in order to apply Independent Verification and Validation (IV&V). The objectives of IV&V at the Japan Aerospace Exploration Agency (JAXA) are to reduce and explain the risks of products. Although a safety case is a useful framework to explain the risks to customers, virtually all past studies on safety cases focused on safety standards that provide risk assurance in the software development process. Therefore, we provide a framework to construct IV&V cases based on the risks of a software product. In order to evaluate our framework, we compared the results of past IV&V and current IV&V. Our comparison found that current IV&V pointed out four times more valid issues than past IV&V. Moreover, the time costs of current IV&V were about half the costs of past IV&V.

Sung-Hoon Lee (University of Science and Technology), Seung-Hyun Kim, Soohyung Kim and Seung-Hun Jin (ETRI).

Appwrapping Providing Fine-Grained Security Policy Enforcement Per Method Unit in Android

Abstract: Enterprise mobility management (EMM) solution is widely used to securely protect confidential information stored on an individual's smartphone, while increasing the efficiency because of BYOD policy. The application wrapping (Appwrapping) technology is one way to be applied EMM solutions, by modifying binary applications without the original source code. In the past, Appwrapping was performed to control permissions or APIs to protect privacy on Android. This method is applied collectively to the whole section, not a specific section of the app, so it is difficult to control the section (flow) desired by the user or the manager. In addition, system overhead can occur because the control is applied to the whole section of the app. In this paper, we propose a method to insert an additional security policy code at a certain interval position in the intermediate code of a binary app, so that it can be controlled at a specific interval rather than the whole interval of the app. The proposed method extracts and saves the security policy intermediate code and the related file in advance and then adds the security policy code to the specific method on the intermediate code of the specific activity acquired by decompiling the target app. Finally, the additional security policy code is modified to avoid errors caused by the additional code. We create an automation tool for performance verification, experiment with five commercial office apps, and confirm that the apps work properly with the added EMM security functions.

Athanasios Stratis and Adnan Causevic (Mälardalen University).

A Practical Approach towards Validating HIL Simulation of a Safety-critical System

Abstract: In order to perform efficient testing of software intensive safety-critical products, organisations often utilise hardware-in-the-loop simulation of the system under test surrounding environment. This way, the system could be invoked and its behaviour could be observed in a controlled setting rather than in the field. However, what effect simulation quality might have on the effectiveness of the testing process still remains an open question. An answer to such a question is rather critical for organisations who are obliged to safety certify their products. In this paper, we are presenting an approach used by Bombardier Transportation to validate their hardware-in-the-loop simulation of a safety-critical system, by executing test cases both in the control setting (lab) and on the real product (train). The process is intended to be used when certifying the simulation which is a necessary step in order to certify the complete system. In addition, we are also presenting some observations from the pilot study and lessons learned.

David Bowes (University of Hertfordshire), Steve Counsell (Brunel University), Tracy Hall (Brunel University), Jean Petric (University of Hertfordshire) and Thomas Shippey (University of Hertfordshire).

Getting Defect Prediction into Industrial Practice: The ELFF Tool

Abstract: Defect prediction has been the subject of a great deal of research over the last two decades. Despite this research it is increasingly clear that defect prediction has not transferred into industrial practice. One of the reasons defect prediction remains a largely academic activity is that there are no defect prediction tools that developers can use during their day-to-day development activities. In this paper we describe the defect prediction tool that we have developed for industrial use. Our ELFF tool seamlessly plugs into the IntelliJ IDE and enables developers to perform regular defect prediction on their Java code. We explain the state-of-art defect prediction that is encapsulated within the ELFF tool and describe our evaluation of ELFF in a large UK telecommunications company.

Veena Mendiratta (Nokia Bell Labs) and Marina Thottan (Nokia Bell Labs).

Rich Network Anomaly Detection Using Multivariate Data

Abstract: Telecommunication networks are designed for high reliability but, given their complexity, when problems do occur they are difficult to detect and diagnose. Anomaly detection approaches typically provide cryptic results, resulting in extensive human effort for diagnosis. Using data from a 4G network, we focus on non-parametric change detection algorithms for anomaly detection and evaluate the performance of the algorithms with two variables: procedure duration and percent failing events. When an anomaly is detected, visual analytics are applied to infer the root cause. The impact of our work is the proactive detection and cause analysis of anomalies and significantly reducing the number of dropped and degraded calls and sessions (9% to 27%).

Ram Chillarege (Chillarege Inc.).

Diagnosing Development Turbulence in Agile Processes

Abstract: When the smooth flow of an Agile development project is disrupted by one or more offending process problems, turbulence sets in. Diagnosing the exact source of turbulence and remedying it promptly is the key to success. The challenge is that process problems are often multi-fault and therefore hard to diagnose or prioritize. Furthermore similar symptoms can originate from different process problems, and intuition is inadequate for accurate diagnosis. This paper illustrates how turbulence needs to be diagnosed and addressed.

Aniket Malatpure (Microsoft), Faraz Qadri (Microsoft) and John Haskin (Microsoft).

Experience Report: Testing private cloud reliability using a public cloud validation SaaS

Abstract: Private clouds (aka on-premise, virtualization clusters offering public cloud-like functionality) are an increasingly important part of the enterprise IT infrastructure. These clouds must be reliable, performant and feature rich. These private clouds are required to meet user Service Level Agreements (SLAs) just like public clouds but without the benefit of a large-scale redundant infrastructure and larger dedicated engineering teams to support them. Given this situation, SLA verification for private clouds throughout their lifecycle (from pre-production validation to maintaining in-production tenant services in the presence of failures, patches, hardware upgrades etc.) becomes critical to ensuring adequate corporate ROI.
In this presentation, we propose an agile, light-weight SaaS based approach to private cloud SLA verification that works for pre-production validation and in-production deployments. This approach is based on 3 pillars viz.: SaaS validation service with an on-premise execution agent, cloud simulation engine covering tenant services, hardware-software faults, cloud admin workflows like patching, upgrade etc. and continuous state monitoring using diagnostic logs analyzed by a big data analysis engine.
We will present results from an implementation of the above approach (i.e. an Azure SaaS service named VaaS – Validation as a Service) meant for testing Microsoft private clouds. We discuss design choices made for the SaaS service, on-premise execution agents and their applicability in partner and customer (aka non-Microsoft) environments. We suggest approaches for designing a Cloud Simulation Engine and discuss the approach used, as well as the test artifacts implemented in the SaaS service we built. We discuss the diagnostic log collection, and the insights-analysis pipeline used for continuously evaluating the state of the private cloud.
We then evaluate the success of this approach through an empirical analysis of the 100+ issues uncovered by implementing it during the release cycle of Microsoft AzureStack.

Kazuhira Okumoto, Abhaya Asthana and Rashid Mijumbi (Nokia Bell Labs).

BRACE: Cloud-based Software Reliability Assurance

Abstract: The evolution towards virtualized network functions (VNFs) is expected to enable service agility within the telecommunications industry. To this end, the software (or VNFs) from which such services are composed must be developed and delivered over very short time scales. In order to guarantee the required levels of software quality within such tight schedules, software reliability tools must evolve. In particular, the tools should provide development teams spread across geography and time with reliable and actionable insights regarding the development process. In this paper, we present BRACE – a cloud-based integrated, onestop center for software reliability tools. BRACE is home to tools for software reliability modeling, testing, and defect analysis each of which is provided as-a-service to development teams. Initial implementation of BRACE includes a software reliability growth modelling (SRGM) tool. The SRGM tool is currently being used to enable real time prediction of the total number of defects in software being developed, and for providing the required analytics and metrics to enable managers make informed decisions regarding resource allocation for defect correction so as to meet set deadlines.

Pete Rotella (Cisco Systems, Inc.) and Sunita Chulani (Cisco Systems, Inc.).

SRC Ratio Method: Benchmarking Software Reliability

Abstract: Software Reliability Classes (SRCs) have been developed in order to compare the field reliability performance of a sequence of software releases for a cluster of similar hardware products. A specific cluster is characterized by the type of market the hardware supports, and the software releases for the cluster have similar functionality, complexity, size, and customer expectations. SRCs are a normalized form of an already normalized customer experience metric, 'software defects (encounters) per million usage hours' referred to as SWDPMH. Different hardware devices, even though running identical software, can experience up to three orders of magnitude variation in SWDPMH values. The SRC method enables us to compare the best-in-class SWDPMH value, for each cluster, to the current field SWDPMH value, and this enables us to use the same SRC ratio calculation across all reliability classes to assess the reliability health of all software releases. The overall reliability health of a business unit's software, for all hardware devices supported, can thereby be accurately calculated, trended, and goaled, with particular attention paid to improving release-over-release reliability.

Vladimiro Vacca (Fiat Chrysler Automobiles Italy S.p.A.), Stefano Scala, Raffaele Rodolfo Maietta and Alfredo Imparato.

A Comparative Study of Static Analysis Tools for AUTOSAR Automotive Software Components Development

Abstract: In order to reduce runtime errors of software, the use of static analysis is clearly recommended in automotive safety standard ISO26262. By analyzing production code of AUTOSAR application software components for an Instrument Panel Cluster, a comparative study of top performance tools is provided. A quantitative analysis has been carried out based on an alert classification model and performance metrics. The goal of this paper is to define the best combination of commercial static analysis tools to minimize defects in the context of automatic code generation and the AUTOSAR software component development. Moreover, through the results of this analysis we would like to suggest further improvements of static analysis tools.

Thomas Chowdhury (McMaster University), Chung-Wei Lin (Toyota InfoTechnology Center), Baekgyu Kim, Mark Lawford, Shinichi Shiraishi (Toyota InfoTechnology Center) and Alan Wassyng (McMaster University).

Principles for Systematic Development of an Assurance Case Template from ISO 26262

Abstract: A failure in a critical system can cause death, injury, financial loss, and environmental damage. To develop safe and trustworthy systems, we need to plan the development and assessment of system functionality in advance. Assurance Cases are a generalization of Safety Cases, and are gaining momentum as a preferred way of demonstrating assurance of critical properties in complex software-intensive systems. To cope with the lack of standardized assurance structures, and to encourage safety assessment prior to development, we previously proposed the use of an assurance case template. The principles presented here can be used to build an assurance case template that complies with the functional safety standard, ISO 26262 in a cost-effective way. In the future, such principles may lead to semi-automated development of these templates.

Siyuan Dai, Takato Masuda, Yusuke Kashiba, Nikos Arechiga, Shinichi Shiraishi (TOYOTA InfoTechnology Center), Joseph Hite, Scott Eisele, Jason Scott and Ted Bapty.

Control Parameter Optimization for Autonomous Vehicle Software Using Virtual Prototyping

Abstract: Reliability and safety are important properties in the development of complex cyber-physical systems such as autonomous vehicles. Achieving a reliable autonomous vehicle is a challenging problem, as the unpredictability of the environment demands a reliable design methodology. Additionally, current testing procedures for ADAS features on vehicles are exhausting and time-consuming, so that a better way to do testing is required. In order to address these challenges, we propose a co-simulation tool-chain which integrates multiple simulation environments, optimizes the control parameters of autonomous vehicle software based on metrics, and visualizes the vehicle behavior using a video game engine.

Hiroyuki Nakagawa (Osaka University), Toshinobu Hasegawa, Shori Matsui and Tatsuhiro Tsuchiya (Osaka University).

Visualization of Specification Coverage: A Case Study of a Web Application Development in Industry

Abstract: Specifications for a software system are usually verified using test cases in the testing phase. As is code coverage, specification coverage is expected to help comprehend how many specifications to be implemented are tested. In this study, we apply a visualization process for specification coverage to a web application development project in industry. This process provides two views for visualization from different viewpoints. The process also provides a classification of specifications according to the assurance degree of test execution. The results of the case study demonstrate that the process provides effective measures to comprehend how many specifications are covered and discovers trends related to the project.

Taehyoun Kim (Agency for Defense Development), Samjoon Park (Agency for Defense Development) and Taeho Lee.

Applying software reliability engineering process to software development in Korea defense industry

Abstract: Since the role of weapon system software is emphasized, the importance of software reliability in Korea defense industry has been gradually increased. However, required activities in the defense industry are not sufficient to develop reliable software. For Korea weapon system software development, there is a manual which defines the process for software development, support and management. Although static and dynamic testing of source code is required for developers to comply this manual, these are not sufficient for developing reliable software. Therefore undefined activities in the manual, such as modeling software reliability and metric data collection, need to be specified. In this paper, we apply software reliability engineering process to the software development in Korea defense industry. We define several additional reliability activities for each software development lifecycle phase. It is based on the international standard IEEE 1633 which covers software reliability engineering. In addition, we suggest models and metrics for assessing and analyzing the reliability of software products based on the international standard IEEE 982.1 which deals with software reliability metrics. We hope that our research help developers to produce more reliable software.

Clément Duffau (AXONIC and I3S, CNRS, Université Côte d'Azur), Bartosz Grabiec (AXONIC) and Mireille Blay-Fornarino (I3S, CNRS, Université Côte d'Azur).

Towards Embedded System Agile Development Challenging Verification, Validation and Accreditation: Application in a Healthcare Company

Abstract: When Agile development meets critical embedded systems, Verification, Validation and Accreditation activities are impacted. Challenges like tests increase or accreditation documents production have to be managed in terms of time and resources. In this paper, we highlight these challenges and present an extended continuous integration ecosystem that aims to tackle these issues. We show how this approach has been applied in a research and development healthcare company named AXONIC.