Model Science: getting serious about verification, explanation and control of AI systems

TL;DR

This paper advocates for a new discipline called Model Science that centers on analyzing, verifying, explaining, and controlling AI models to ensure their safety, alignment, and reliability across various contexts.

Contribution

It introduces a conceptual framework for Model Science with four key pillars: Verification, Explanation, Control, and Interface, shifting focus from data to models themselves.

Findings

Proposes strict, context-aware evaluation protocols for models.

Defines explanation methods exploring internal model operations.

Suggests interactive tools for human-AI calibration.

Abstract

The growing adoption of foundation models calls for a paradigm shift from Data Science to Model Science. Unlike data-centric approaches, Model Science places the trained model at the core of analysis, aiming to interact, verify, explain, and control its behavior across diverse operational contexts. This paper introduces a conceptual framework for a new discipline called Model Science, along with the proposal for its four key pillars: Verification, which requires strict, context-aware evaluation protocols; Explanation, which is understood as various approaches to explore of internal model operations; Control, which integrates alignment techniques to steer model behavior; and Interface, which develops interactive and visual explanation tools to improve human calibration and decision-making. The proposed framework aims to guide the development of credible, safe, and human-aligned AI systems.

Full text

\paperid

123

1 From Data Science to Model Science

The emergence of Data Science as a distinct discipline is deeply rooted in the intellectual developments of the second half of the 20th century. Two landmark works shaped the foundation of this transformation are: John W. Tukey’s 1962 essay “The Future of Data Analysis” [Tukey, 1962] and William S. Cleveland’s 2001 paper “Data Science: An Action Plan for Expanding the Technical Areas of the Field of Statistics” [Cleveland, 2001]. Together, these works redefined the scope of statistical practice and laid the groundwork for what is now considered modern data science.

Tukey argue that data analysis should be treated as an autonomous scientific discipline, distinct from mathematical statistics. In his view, analysis was not merely the application of statistical tests, but a broader process encompassing data collection, exploratory examination, and the iterative refinement of research questions. He advocated for exploratory data analysis (EDA) as a central paradigm, stressing that it is often preferable to obtain “an approximate answer to the right question rather than an exact answer to the wrong one.” Tukey foresaw the transformative role of computers and graphical methods, predicting that advances in computation and visualization would reshape how scientists extract knowledge from data. His ideas encouraged statisticians to engage directly with real-world problems, collaborating with domain experts and embracing the growing complexity and volume of data.

Nearly four decades later, Cleveland extended Tukey’s vision by providing a concrete institutional and methodological framework for the new discipline. He proposed renaming the expanded field of statistics as “Data Science”, arguing that traditional statistics should evolve to meet the demands of large, complex, and interdisciplinary data problems. Cleveland proposed a curriculum for Data Science based on six-areas: (1) multidisciplinary investigations, (2) models and methods for data, (3) computing with data, (4) pedagogy, (5) tool evaluation, and (6) theory. This plan explicitly integrated computing, software development, and algorithmic thinking as core components of data science and prioritized collaboration with domain scientists as a fundamental driver of methodological innovation. Cleveland emphasized that every technical contribution should ultimately be evaluated by its direct benefit to the practicing data analyst.

The development of Data Science has been driven by the growing capabilities to collect and process large datasets (Big Data). The discipline adopted a highly pragmatic approach to the technical aspects of data cleaning, efficient storage, and data management. Its primary driving force has been data—cheap to collect, process, analyze, and extract value from.

We are now on the brink of another revolution, one driven by large models (Big Models). Large not only in terms of the number of parameters (rapidly exceeding tens of gigabytes) and the number of users (reaching millions for a single model), but also in terms of the range of tasks that can be built upon a single base model (so-called foundation models or frontier models). This paper aims not only to discuss whether a new scientific discipline or sub-discipline—Model Science—might emerge around these models, but also to reflect on the major challenges this discipline will need to address.

Figure 1 illustrates the complementary nature of Data Science and Model Science in building models that describe and replicate specific phenomena and processes in the real world, based on collected data, trained algorithms, and the confrontation of these algorithms with reality. The central focus of Data Science is the data itself, along with the challenges of collecting, cleaning, and selecting it. Data is the King is the guiding principle of this discipline, where problems are typically formulated around a single well-defined dataset (or a collection of datasets) describing a given phenomenon, on which multiple models are built—usually to select the best-performing one.

The nature of Model Science is complementary, as it focuses on a specific model that is subjected to analysis. The model is the central object, while data becomes a variable: training, validation, test, monitoring, or out-of-domain datasets may not even be fully defined at the training stage but become crucial for verification or analysis. In Model Science, data does not necessarily have to come from real-world measurements; it can also consist of synthetically generated samples, used for additional validation, explanation, or model improvement. The constant element is the model under investigation.

Of course, comprehensive analysis often requires a solid understanding of both the data and the model; therefore, Data Science and Model Science frequently overlap—although they pursue different goals and employ different tools. The following sections discuss four main pillars of Model Science.

2 Four Pillars of Model Science

Since Model Science focuses on an already trained model, its primary interests concern what can be done after the model has been developed: verification, exploration, control, and communication.

In model analysis, the constant reference point is the model itself, or sometimes several trained models that are compared against each other. The variables in this analysis are the data—these may include data available during training, data generated later (e.g., in out-of-time or out-of-place scenarios), or even synthetic data.

Figure 2 graphically summarizes the four main pillars of model analysis, which will be discussed in greater detail in the following sections.

3 Verification: does it work?

We will begin this section by discussing recently discovered weaknesses in the most popular AI models currently available. Certainly, newer versions of these models will fix (some of) these errors over time, and new errors will appear in newer versions of these models. For this reason, the discussion below does not claim to be universal, but only to emphasize the role of verification, which is neither obvious nor taken seriously enough.

Is the AI model evaluation broken? AI models and in particular foundation models (FMs) such as large language models (LLMs), speech-to-text (STT) systems, and text-to-image generators are increasingly deployed in domains where errors may have significant ethical, legal, and social consequences. Although these models achieve state-of-the-art (SOTA) performance on standard benchmarks, recent studies reveal that their real-world behavior can deviate significantly from reported metrics, with additional validation often exposing severe flaws. We start with a short review of recent evidence of underperformance and harmful biases in high-stakes contexts, highlighting the importance of rigorous verification protocols prior to deployment. Please note that the cases mentioned below are not intended to stigmatize a particular research group or company for providing an erroneous model; this happens everywhere regardless of the amount of resources involved. This revision is intended to emphasize that systematic research is ongoing to verify large and popular models.

The study “Careless Whisper: Speech-to-Text Hallucination Harms” [Koenecke et al., 2024] demonstrates that OpenAI’s Whisper—a widely used STT model—produces hallucinated sentences absent from the original audio in approximately 1% of transcripts. Alarmingly, 38% of these hallucinations contained harmful or violent language. The issue was exacerbated for people with aphasia, where the model incorrectly “filled in” long pauses with fabricated statements. These findings reveal the need for domain-specific auditing, as such hallucinations could lead to false legal evidence or medical miscommunication.

Similarly, LLMs have been shown to produce high rates of legal hallucinations. The paper “Large Legal Fictions” [Dahl et al., 2024] reported that ChatGPT-3.5, PaLM-2, and LLaMA-2 invented non-existent case law in 69–88% of tested legal queries. Even specialized tools such as Lexis+ AI hallucinated in 17–34% of responses. This contradicts news-media articles of LLMs passing bar exams, underscoring the mismatch between benchmark performance and real-world reliability.

No area is free from these types of problems. In healthcare, a study focused on evaluation of the GPT-4 published in Lancet Digital Health [Zack and et. al., 2024] revealed persistent gender and racial biases in this model. In 37% of paired clinical scenarios differing only by gender or race, GPT-4 produced distinct diagnoses or treatment suggestions. The model overestimated sarcoidosis prevalence in Black patients and hepatitis B in Asian patients reflecting past historical biases in healthcare. Such errors pose serious risks when models are integrated into diagnostic workflows.

A large-scale analysis of GitHub projects [Majdinasab et al., 2024] demonstrated that over 35% of code suggestions generated by GitHub Copilot contained security weaknesses, from diverse group of 42 distinct MITRE’s Common Weakness Enumerations (CWEs), including command injection and insecure randomness. Eleven of these ranked in the CWE Top-25 most dangerous vulnerabilities. This raises concerns about automated code generation in safety-critical software (e.g., medical devices or financial systems).

In education, ChatGPT-3.5 failed to generate reliable medical multiple-choice questions, with only 32% of items being fully correct and explanatory [Ngo et al., 2024]. Using such unverified material risks propagating misconceptions among medical students. Moreover, audits of LLMs and text-to-image models revealed adultification bias [Castleman and Korolova, 2025], where Black girls were depicted as older and more sexualized than White peers, potentially reinforcing harmful stereotypes in educational or disciplinary systems.

Adversarial analysis of models and their explanations has become increasingly recognized as essential for ensuring reliability in AI systems. Even models considered "intrinsically interpretable" may have hidden vulnerabilities, for example a recent stody on PIP-Net and Proto-ViT models [Baniecki and Biecek, 2025] demonstrated that a prototype-based classifier could be deceived by subtle input manipulations, causing a bird image to be “explained” using car prototypes and thus undermining the model’s reasoning transparency. Such examples contradict the notion that interpretability alone guarantees robustness. Likewise, large language models (LLMs) may exploit latent linguistic patterns to influence users: recent studies such as [Mieleszczenko-Kowszewicz et al., 2024] exposed how LLMs adjust their style to a reader’s Big Five personality traits, effectively producing personalized persuasive (or manipulative) outputs that could go unnoticed without adversarial scrutiny. More broadly, recent studies emphasize that explanation tools themselves can be targets of attack. A survey [Baniecki and Biecek, 2024] catalogs numerous ways to manipulate or “fairwash” model explanations – for instance, by tweaking inputs or training data to hide biased reasoning – and it reviews defense strategies to secure explainability methods. These findings underscore the need to rigorously evaluate and challenge both models and their explanations.

Right for the Wrong Reason AI models can sometimes achieve high accuracy with predictions that are right for the wrong reason, meaning they make correct predictions but rely on spurious or misleading patterns rather than genuine, generalizable features. This issue is epitomized by the Clever Hans effect, named after a horse that appeared to perform arithmetic by picking up on its trainer’s cues. For example, the Unmasking Clever Hans paper [Lapuschkin et al., 2019] showed that an image classifier could correctly recognize horses not by identifying horse-specific visual features, but by latching onto incidental cues like a copyright tag frequently present in horse photos. Similarly, the LIME paper [Ribeiro et al., 2016] used their new interpretability technique to reveal that a seemingly accurate “wolf vs. husky” classifier was basing its decisions on the presence of snow in the background of the images — effectively detecting the background context rather than the animals themselves. These are prime examples of a model being right (getting the correct class) for the wrong reason (using a shortcut that coincides with the correct answer in the training data). While the above instances involve supervised classifiers, recent work indicates that unsupervised learning models are also vulnerable to such Clever Hans-style behavior. Recent studies [Kauffmann et al., 2025] demonstrate that even state-of-the-art unsupervised representation learning methods can form spurious associations, for instance grouping or explaining images by irrelevant features like embedded text logos or high-frequency noise patterns. In other words, an unsupervised model might “explain” or organize data in a way that accidentally aligns with the task at hand, yet fails when those coincidental cues change.

These findings demonstrate a recurring pattern: foundation models, and AI models in general, perform worse than initially reported when subjected to targeted, context-specific validation. High-stakes domains require not only standard benchmark evaluation but also rigorous verification frameworks is therefore crucial for responsible adoption of foundation models, particularly in healthcare, legal systems, education, and other safety-critical sectors.

Model Evaluation Levels AI models are being applied to more and more important decisions, thus ensuring the reliability and trustworthiness of AI models stands as a critical endeavor. Model validation—an essential step in this process—serves as the gatekeeper between well performing and faulty model. Regardless of which metrics are used to evaluate a model, it is critical to define the level at which the evaluation is conducted. In this section, we propose five levels of model evaluation, highlighting how rarely models are examined at the higher levels.

When evaluating a model, both the evaluation criterion and the selection of the data sample to assess the quality of the model are important. In this section, we will look at the issue of data used for validation. Figure 3 provides a graphical summary for model evaluation levels.

Model Evaluation Level 0 — no explicit model evaluation. It also includes the use of models for a new task for which they were not designed (unintended use) without any verification.

Model Evaluation Level 1 — for model evaluation we use the same data on which we trained the model. This is a common practice in statistical modelling based on linear models or other models where we have strong assumptions about the mechanism generating the data and the representativeness of the sample. For linear models, evaluation can be based on $R^{2}$ or p-values for the significance of the coefficients, typically calculated on the training sample.

Model Evaluation Level 2 — to evaluate the model, we use a separate data set with a distribution similar to the training data. The most common scenario is to randomly divide the data into two parts training and testing. Sometimes used in more sophisticated variants such as cross-validation or an out-of-bag data for bootstrap sampling. The characteristic of this evaluation is that the distribution of training and test data is close, which corresponds to the assumption that the reality in which the model will operate is not different from the reality in which the model was trained. This assumption definitely simplifies model validation, although covid pandemic, wars or other impactfull events, show that future data can differ from past data.

Model Evaluation Level 3 — to evaluate the model, “disjoint” data is used for training and testing. If the data contains a timestamp, the disjointness may mean using a different time range (so called out-of-time validation [Elena et al., 2021]). If the data contains a geographic tag, validation may mean using a different region (out-of-region validation). Virtually any attribute can be used to define a test sample, usually this is done to test for generalizability. For data collected from different devices, out-of-device validation can be considered, for AutoML systems working for many datasets, the out-of-dataset validation [Ho et al., 2020] and so on. This testing scheme is used by audit teams in (for example) financial institutions, especially for large-scale, high impact models, such as credit scoring models.

Model Evaluation Level 4 — it is time to actively look for cases in which the model does not work, even if they are rare according to the training data. There are situations in which users of the model may want to adversarially influence the model’s decisions. For example, for an insurance pricing model, the user is tempted to manipulate certain parameters so as to evaluate the price of the premium. Parameters such as mileage, a statement about the number of drivers or number of days being abroad may be given incorrectly by the user if he or she thinks they will reduce the price of the premium. Another example is models that detect hate speech in online forums, users who want to post offensive comments will actively test scenarios in which the models do not work. In these situations, the adversary does not know the structure of the model, and can only reflect it and thus look for model vulnerabilities.

Model Evaluation Level 5 — In a more extreme scenario, the adversary has full access to the model. Whether it has been stolen or the model is publicly available is of little relevance. Analysis at this level allows tracking decision paths, calculating and using gradients. An example is the use of the Llama 2 [Touvron et al., 2023] language model, which is publicly available, in an AI system, or the Gandalf Challenge111https://gandalf.lakera.ai/ designed to deceive LLM models in order to steal a secret, i.e., perform undesirable behavior.

Standards and Checklists How to reach higher MEL? Clear verification procedures, standards, and checklists are essential for building trustworthy AI systems. Studies on medical imaging models for COVID-19 show that many errors came from poor data handling and weak validation, which led to misleading results; a structured checklist was proposed to guide responsible model development in such high-stakes domains [Hryniewska et al., 2021]. In finance, transparency and auditability are critical for credit scoring models, and without clear validation rules, advanced algorithms cannot be trusted by regulators [Bücker et al., 2022]. With the rapid growth of generative AI, experts call for new standards to protect the information ecosystem: public release of generative models should require built-in detection mechanisms [Knott et al., 2023]. Media and technology companies are also expected to follow new obligations to detect AI-generated content and ensure responsible communication [Knott et al., 2024]. These procedures, tools, and regulations are not just formal requirements; they are key to making AI systems safe, transparent, and reliable in sensitive areas such as healthcare, finance, and public communication.

4 Explanation: how it really works?

In recent years, the field of explainable machine learning has been booming, resulting in numerous papers proposing new methods and applications of these methods in relevant areas, both from the perspective of the end user (BLUE-XAI) and the model developer (RED-XAI) [Biecek and Samek, 2024]. Below, we review some advances in interpretability and explainability, focusing on (i) representation and attention analysis, (ii) feature attribution, (iii) knowledge extraction, (iv) case-based explanations, and (v) model auditing (red teaming).

This overview does not replace a systematic surveys (such as [Holzinger et al., 2022, Guidotti et al., 2019, Biecek and Burzykowski, 2021]), but aims to show the synergies between the perspectives used in explanatory model analysis and in Model Science, which is more focused on exploration of specific models. Let’s see how this is relevant for foundation models, such as CLIP for language-image pairs, LLaMA for language, AlphaFold for genetic sequences, or AlphaZero for chess-like games.

Representation and Attention Analysis. CLIP use case. Several studies decomposed CLIP’s image-text representations to discover human-interpretable concepts. For example, [Gandelsman et al., 2024] revealed that individual attention heads specialize in distinct semantic roles (e.g., object localization, shape detection), enabling the removal of spurious features and even zero-shot segmentation. Sparse coding methods such as SpLiCE [Bhalla et al., 2024] and Matryoshka Sparse Autoencoders (MSAE) [Zaigrajew et al., 2025] expressed CLIP embeddings as a small set of semantic basis vectors, exposing $>100$ latent concepts useful for bias detection. Large Language Models (LLMs) use case. Mechanistic interpretability identified functional sub-circuits in transformers. Gould et al. [Gould et al., 2023] described successor heads responsible for incrementing sequences (e.g., “Monday” $\rightarrow$ “Tuesday”) across multiple LLM families. Causal Head Gating (CHG) [Nam et al., 2025] automatically classified attention heads as helpful, disruptive, or redundant, highlighting low modularity and overlapping circuits in LLaMA. AlphaFold use case. Analyses showed that Evoformer attention mechanisms capture coevolutionary couplings, similar to Direct Coupling Analysis in bioinformatics [Lupo et al., 2022]. Simplified single-layer attention models can recover contact maps, suggesting AlphaFold implicitly learns evolutionary constraints. AlphaZero use case. Probing studies revealed that its value network encodes classical chess concepts (e.g., king safety, pawn structure) despite training without supervision [McGrath et al., 2022]. Furthermore,  [Pálsson and Björnsson, 2023] extracted novel strategic concepts from AlphaZero, successfully transferring them to human grandmasters.

Feature Attribution Feature attribution moved from raw inputs to latent representations. For CLIP, [Dreyer et al., 2025] combined sparse autoencoders and attribution patching to quantify the influence of each latent component, revealing hidden biases (e.g., textual artefacts strongly influencing predictions). In LLMs, causal ablations, activation patching, and influence functions traced specific neurons and training examples affecting a given output [Meng et al., 2023a]. ExplainableFold [Tan and Zhang, 2023] introduced counterfactual explanations for AlphaFold by generating hypothetical sequence mutations altering the predicted 3D structure, resembling in-silico biological experiments.

Case-Based Explanations Case-based XAI helps interpret single predictions. CLIP-InterpreT [Gandelsman et al., 2024] provides interactive per-head nearest neighbors, showing which attention heads focus on which concepts. In AlphaFold, pLDDT confidence scores and sequence-level contributions (MSA evidence) guide biologists in assessing reliability. For AlphaZero, active neurons linked to strategic motifs (e.g., “passed pawn”) offer human-readable rationales for specific moves.

Exploring with synthetic data Synthetic samples, may be use to probe and explain black-box models. A popular scheme for such exploration are so colled counterfactual explanations, that adress the question “what if?” and generating plausible data variations that reveal the causal factors behind predictions. Recent work extends this paradigm to different modalities and scales. For time-series, [Płudowski et al., 2025] introduce MASCOTS, a model-agnostic method that operates in a symbolic space to replace influential subsequences with minimal changes, ensuring both fidelity and interpretability of counterfactuals while highlighting pivotal temporal patterns in classification decisions. In the vision domain, [Sobieski et al., 2025] proposes region-constrained visual counterfactuals, restricting alterations to user-defined regions, which yields clearer causal evidence and prevents misleading global modifications. Moving beyond local explanations, [Sobieski and Biecek, 2025] introduces Global Counterfactual Directions, latent-space transformations that consistently flip predictions across many instances, providing a global view of decision boundaries and exposing systemic biases. Together, these approaches demonstrate that generating synthetic observations is not merely a post-hoc explanation tool but a stress-testing mechanism that enhances transparency, robustness, and trust.

Auditing and Red Teaming Systematic red teaming exposes failure modes. Anthropic’s large-scale red teaming [Ganguli and et. al., 2022] showed that RLHF-trained LLMs become increasingly robust to harmful prompts with scale, while “raw” models remain vulnerable. Vision-language models are susceptible to adversarial patches exploiting global attention in ViTs [Lovisotto et al., 2022], whereas AlphaFold overconfidently predicts plausible but incorrect folds in proteins lacking evolutionary homologs [Wang et al., 2024]. Identifying such weaknesses informs model debugging and safety improvements.

Together, these advances demonstrate that complex foundation models can be probed, decomposed, and audited to extract meaningful scientific and strategic knowledge. From mechanistic circuits in LLaMA to novel chess strategies in AlphaZero, XAI research evolves from justifying predictions to questioning and understanding models as knowledge-bearing systems. Figure 4 shows that explainable model analysis can be an integral part of the model development process.

5 Control: how it should work?

Recent advances in alignment research have substantially improved the reliability of large language models (LLMs) and multimodal models (e.g., CLIP). Among the recent studies, Ouyang et al. (2022) introduced InstructGPT [Ouyang et al., 2022], which combined supervised fine-tuning (SFT) on high-quality instruction-response pairs with Reinforcement Learning from Human Feedback (RLHF). This approach significantly reduced hallucinations and toxic outputs, establishing RLHF as a standard for aligning LLMs.

Building on this, Constitutional AI [Bai and et. al., 2022] replacing most human feedback with AI-generated critiques guided by a set of ethical principles. This RL from AI Feedback (RLAIF) achieved comparable safety while scaling alignment to larger models at lower cost. To address the complexity of RLHF, Direct Preference Optimization (DPO) was introduced [Rafailov et al., 2024], a closed-form supervised objective that aligns models directly on human preference data, matching RLHF performance without training separate reward models. Similarly, Lee et al. (2024) empirically validated RLAIF vs. RLHF [Lee et al., 2024], confirming that AI feedback can substitute human preference data for many tasks.

Recent methods directly extracted symbolic or conceptual knowledge. In LLMs, ROME and MEMIT [Meng et al., 2023a, Meng et al., 2023b] demonstrated that factual triples (subject–relation–object) are stored in localized MLP weights and can be edited or retrieved. CLIP’s semantic basis vectors serve as a conceptual vocabulary, enabling bias audits [Zaigrajew et al., 2025]. For reinforcement learning, NUDGE [Delfosse et al., 2023] distilled logical rules from neural policies, a potential path to symbolic AlphaZero strategies.

Finally, multimodal alignment gained attention. Zhou et al. (2023) demonstrated universal adversarial attacks on CLIP (AdvCLIP) [Zhou et al., 2023], prompting robust fine-tuning strategies for vision encoders. Recent extensions (e.g., layer-wise PPO on VLMs) adapt RLHF-style techniques to vision-language models, reducing harmful outputs triggered by adversarial images.

AI model control is still a developing field, responding both to the different expectations of different user groups (the need to personalize model responses) and to the growing number of identified model misbehaviours. This is a natural continuation of the verification and explanation pillars, because once we know that something is not working and understand why, the next step is to apply mechanisms to improve model behaviour, preferably without complete retraining.

6 Interface: how to interact with it?

Foundation models such as large language models and vision transformers have achieved unprecedented performance on diverse tasks, but their internal reasoning remains largely opaque to users [Bommasani and et. al., 2022]. This opacity raises concerns about trust and accountability, especially as these models are deployed in high-stakes domains. To address this, researchers are exploring explanation interfaces that leverage interaction and visualization to help humans comprehend complex model behavior. Rather than treating a foundation model as a “black box,” interactive interfaces aim to open up the model’s decision process, allowing users to probe why a model produced a given output and under what conditions it might change its decision [Miller, 2019]. These approaches build on the notion that effective explanations can improve users’ mental models of the AI system and calibrate their trust accordingly [Kulesza et al., 2015].

Model exploration An important challenge in model exploration is combining multiple explanatory methods into coherent, user-driven workflows. The Grammar of Interactive Explanatory Model Analysis (IEMA) formalizes such workflows, enabling sequential exploration of diverse methods and supporting structured human–model dialogues [Baniecki et al., 2023]. Such multi-perspective analysis improves understanding and decision-making but requires careful design to avoid overwhelming users. Another challenge concerns aligning explanations with human semantics and enabling user guidance. Methods like LIMEcraft allow analysts to craft meaningful superpixels and inspect how predictions change, revealing model biases that purely algorithmic segmentation might miss [Hryniewska et al., 2022]. Similarly, animated visualizations offer richer insight into complex behaviors: Spyrison et al. demonstrate that dynamic linear projections can expose local decision boundaries of nonlinear models, supporting intuitive “what-if” analyses [Spyrison et al., 2025]. Finally, conversational interaction is emerging as a promising paradigm. Studies on human–model dialogues show that users ask diverse, context-dependent questions—about reasons, counterfactuals, and uncertainties—that static tools often fail to address [Kuzba and Biecek, 2020]. Building systems that respond naturally and faithfully to such queries remains an open problem.

Interactive explanation tools enable a two-way exchange between the user and the model, transforming explanation from a static output into an ongoing dialogue. One line of work integrates explanations into chat-based interactions with large language models. For instance, an LLM can justify its answer or clarify specific steps when asked, providing on-demand self-explanations. More structured interfaces allow users to manipulate model inputs or reasoning and see the effects. Kulesza et al.’s early concept of explanatory debugging highlighted how users could iteratively correct a model by giving feedback on its explanations [Choo and Liu, 2018]. Modern interfaces are bringing this vision to foundation models. For example, [Wu et al., 2022] introduced PromptChainer, a visual interface for building and debugging chains of LLM prompts. It lets users break a complex task into a sequence of prompt transformations, making the overall reasoning process more transparent and controllable. Other systems support interactive prompt refinement: [Strobelt et al., 2022] lets users visually edit and experiment with prompt phrases and observe changes in the LLM’s output in real time. Similarly, ScatterShot is an interface that helps users curate in-context examples for few-shot learning by interactively selecting and revising examples, which in turn updates the model’s behavior [Wu et al., 2023]. By allowing users to test “what if” scenarios—e.g. “What would the model do if we tweak this prompt or example?”—these interfaces help users identify the model’s decision boundaries and failure modes. Such iterative exploration can not only improve task performance but also give users a sense of agency and deeper understanding of the model.

Visualization-based explainability plays a complementary role, especially for vision and transformer-based models. Complex models often produce high-dimensional internal states that defy simple explanation; visualization techniques translate these into human-interpretable forms. For vision foundation models, spatial heatmaps and feature visualizations are common: e.g. Grad-CAM highlights which regions of an image influence a CNN’s prediction [Selvaraju et al., 2017], helping users verify if the model is “looking” at relevant features (like the suspected tumor area in a medical image). Such visual explanations can be integrated into interfaces where a user hovers over or clicks on parts of an input to reveal the model’s attention or confidence for those parts. [DeRose et al., 2020] present attention flow diagrams that trace how information propagates between words across the layers of a transformer. Their interface allows users to select a token and see which other tokens it attends to and influences, illuminating the model’s latent reasoning chain. Other tools like AttentionViz [Yeh et al., 2023] provide a global view of attention patterns, summarizing where the model focuses across multiple inputs and layers. This helps in spotting broad trends, such as whether a language model disproportionately attends to certain prompt keywords or if a vision transformer’s attention aligns with meaningful image regions [Li et al., 2023]. By interacting with these visualizations (for example, filtering attention by layer or threshold, or comparing attention before and after fine-tuning), users can investigate model behaviors at different levels of granularity.

User trust and understanding are central metrics for the success of these model exploration interfaces. The goal is not only to inform the user, but to do so in a way that calibrates their trust—users should trust the model when it is correct and be appropriately cautious when the model is uncertain or makes an error. Interactive and visual explanations can aid in this calibration by revealing the rationale behind model outputs. Recent studies shows that poor interface design or misleading explanations can backfire. [Kaur et al., 2020] found that data scientists often misinterpreted outputs from interpretability tools and became overconfident in the model’s correctness even when the explanations were superficial. This highlights that simply providing an explanation interface is not enough; the explanations themselves must be faithful and understandable. [Bansal et al., 2021] similarly showed that while explanations (such as highlighting important features or providing model confidence) can improve human-AI team decision performance in some cases, they can also decrease performance if the model is wrong and the explanation leads the human to over-rely on the AI. To build trustable interfaces, designers are adopting UX approaches: conducting user studies to see how people actually use and react to explanations, and iterating on the interface to better support users’ decision-making and sensemaking processes. In some cases, this means allowing users to challenge the model’s reasoning (e.g. by asking “why not?” questions or providing counterexamples) and having the system respond with revised explanations or outputs, in others, it means clearly conveying model uncertainty.

7 Conclusions

We are witnessing the emergence of new AI models that are continuously trained on diverse datasets, which are often no longer accessible afterwards. These models will become the foundation for numerous tools that rely on their representations in relation to dozens of other important tasks. It is therefore crucial to develop methods, processes, and exemplary analyses of such models, addressing both their efficiency and safety.

In this article, we introduced the concept of Model Science—a subfield of research focused on the analysis of models. We identified four key areas essential for such analysis and outlined initial research directions already underway in these domains. By providing a structured framework for this field, we hope to foster the further development of methods for analyzing models. We also invite readers to engage in both shaping and promoting the field of Model Science.

{ack}

Work on this project is financially supported from the SONATA BIS grant 2019/34/E/ST6/00052 funded by Polish National Science Centre (NCN) and INFOSTRATEG-I/0022/2021-00 grant funded by Polish National Centre for Research and Development (NCBiR).