From Images to Decisions: Assistive Computer Vision for Non-Metallic Content Estimation in Scrap Metal

TL;DR

This paper introduces an assistive computer vision system that objectively estimates non-metallic contamination in scrap metal from images, improving safety, accuracy, and workflow integration in steelmaking.

Contribution

It develops a novel multi-task and multi-instance learning pipeline for contamination estimation and scrap classification, with real-time deployment and active learning for continuous improvement.

Findings

MAE of 0.27 and R2 of 0.83 for contamination regression

F1 score of 0.79 for scrap classification

System operates in near real-time within industrial workflows

Abstract

Scrap quality directly affects energy use, emissions, and safety in steelmaking. Today, the share of non-metallic inclusions (contamination) is judged visually by inspectors - an approach that is subjective and hazardous due to dust and moving machinery. We present an assistive computer vision pipeline that estimates contamination (per percent) from images captured during railcar unloading and also classifies scrap type. The method formulates contamination assessment as a regression task at the railcar level and leverages sequential data through multi-instance learning (MIL) and multi-task learning (MTL). Best results include MAE 0.27 and R2 0.83 by MIL; and an MTL setup reaches MAE 0.36 with F1 0.79 for scrap class. Also we present the system in near real time within the acceptance workflow: magnet/railcar detection segments temporal layers, a versioned inference service produces railcar-level estimates with confidence scores, and results are reviewed by operators with structured overrides; corrections and uncertain cases feed an active-learning loop for continual improvement. The pipeline reduces subjective variability, improves human safety, and enables integration into acceptance and melt-planning workflows.