| KEYWORD |
Computer vision techniques for mobile testing
Thesis in external company
keywords ARTIFICIAL INTELLIGENCE, HUMAN-MACHINE INTERFACES, MACHINE LEARNING, MOBILE APPLICATIONS, SOFTWARE TESTING
Reference persons FABRIZIO LAMBERTI, LIA MORRA
Research Groups DAUIN - GR-09 - GRAphics and INtelligent Systems - GRAINS
Description Many End-to-End (E2E) testing tools allow developers to create repeatable test scripts that mimic a human user's interaction with the application and evaluate its response. Various paradigms of testing tools have been defined that are differentiated based upon the way the widgets are located on the GUI (e.g., text, images, layout properties). This thesis falls within the visual GUI testing paradigm scope, where screen snapshots used as visual locators:
- identify the widgets, and
- replicate an existing test suite across multiple devices or versions of the same app.
This technique leverages image recognition algorithms.
In particular, the thesis focuses on testing mobile applications where the same widgets have to be identified across multiple devices, with different screen sizes and characteristics. In this context, marginal variations in the graphical rendering can invalidate widgets' recognition if the image recognition algorithm is not sufficiently robust. This thesis aims to extend an existing prototype for visual GUI testing by leveraging deep learning. In particular, the following activities are envisioned:
- Training a detector to identify widgets and their classes with high recall.
- Using the features extracted by the neural network to match the same widgets when the app is rendered across multiple devices.
- Investigating the possibility of integrating text extraction with image analysis.
The solution will be characterized and possibly optimized in terms of accuracy and execution speed. The developed techniques will be included in an open-source library.
The student should possess or should be willing to acquire these skills:
- programming skills (Python, deep learning frameworks);
- experience in training deep neural networks;
- fundamentals of mobile development (Android GUI, the Android Studio development environment).
See also http://grains.polito.it/work.php
Deadline 31/07/2023
PROPONI LA TUA CANDIDATURA