| KEYWORD |
Adaptive Design-to-Code Model: Reinforcement Learning Approach Using Automatic Evaluation Metrics
keywords ADAPTIVE MODELS, AUTOMATIC EVALUATION METRICS, DESIGN-TO-CODE, HUMAN COMPUTER INTERACTION, MACHINE LEARNING, REINFORCEMENT LEARNING, USER EXPERIENCE, WEB DEVELOPMENT
Reference persons LUIGI DE RUSSIS
External reference persons Tommaso Calò
Research Groups DAUIN - GR-10 - Intelligent and Interactive Systems - ELITE
Thesis type EXPERIMENTAL, RESEARCH
Description Recent advancements in AI-assisted web development, particularly in the Design2Code domain, have shown promising results in automatically generating HTML and CSS code from webpage screenshots. However, current approaches often lack adaptability to individual design preferences and struggle with complex layouts. This thesis proposes to leverage reinforcement learning (RL) techniques to create an adaptive Design2Code model that can learn and improve from its own generations using the automatic evaluation metrics developed in the Design2Code benchmark paper.
By framing the code generation process as a sequential decision-making problem and using the automatic metrics as a reward signal, we can potentially create a more flexible and adaptive system. This approach could lead to models that not only generate accurate code implementations but also learn to adapt to specific design styles and preferences over time, potentially surpassing the current state-of-the-art in terms of layout accuracy and stylistic nuance.
The thesis will aim to:
- Develop a reinforcement learning framework for the Design2Code task, using the automatic evaluation metrics (Block-Match, Text, Position, Color, and CLIP similarity) as a multi-objective reward function.
- Implement an adaptive Design2Code model that can learn and improve its code generation capabilities through RL-based fine-tuning.
- Investigate the impact of different RL algorithms (e.g., PPO, A2C, SAC) on the model's performance and adaptability.
- Evaluate the RL-enhanced model against state-of-the-art baselines on the Design2Code benchmark, focusing on improvements in layout accuracy and style fidelity.
If satisfactory, the result of the thesis will be released as an open-source project.
Required skills Strong programming skills in Python.
Experience with deep learning frameworks (e.g., PyTorch, TensorFlow).
Familiarity with reinforcement learning concepts and algorithms.
Understanding of web development technologies (HTML, CSS).
Basic knowledge of computer vision and natural language processing.
Notes Beneficial experience:
- Prior work with large language models or vision-language models
- Experience in implementing RL algorithms
- Familiarity with UI/UX design principles
- Knowledge of human-computer interaction concepts
Deadline 21/10/2025
PROPONI LA TUA CANDIDATURA