Vision Language Model for Interpretable and Fine-grained Detection of Safety Compliance in Diverse Workplaces (2025)

2.1 Personal Protective Equipment Detection

The detection of PPE has been significantly advanced through the application of deep learning and computer vision technologies.Early methods for PPE detection primarily relied on sensor-based systems, such as RFID technology (Barro-Torres etal., 2012).For example, one approach involved equipping each PPE component with RFID tags and using scanners at job site entrances to ensure workers were wearing the appropriate gear (Kelm etal., 2013).Another method investigated the use of short-range transponders embedded in PPE items, with a wireless system verifying compliance with safety regulations (Naticchia etal., 2013).Similarly, (Barro-Torres etal., 2012) investigated a local area network to monitor RFID tags on PPE continuously, ensuring compliance throughout the workday.Additionally, GPS technology has also been utilized to enhance PPE detection. For instance, (Zhang etal., 2015) employed GPS devices attached to workers’ safety helmets to provide supplementary safety monitoring on construction sites. In a more recent advancement, (Pisu etal., 2024) introduced an operator area network system that leverages Machine Learning and RSSI analysis to detect improper PPE usage robustly. This system uses an SVM model with a customizable post-processing algorithm, reducing false positives by 80% and detecting issues within seven seconds. Furthermore, (Yang etal., 2020) designed an automated PPE-tool pairing detection system based on a wireless sensor network, which effectively monitors the wearability status of PPE.However, despite their effectiveness in certain scenarios, these sensor-based approaches were limited by their dependency on physical sensors attached to PPE items and necessitated significant investment in purchasing, installing, and maintaining complex sensor networks, which could hinder their practical implementation (Nath etal., 2020).

With the advancement of computer vision technology, vision-based systems have revolutionized PPE detection due to their lower cost, simpler configuration, and broader application scenarios compared to sensor-based technologies. These systems utilize cameras to capture real-time images and videos of workers, which are then analyzed using computer vision techniques. The extensive deployment of cameras, combined with significant progress in computer vision technology, has established a foundation for effective PPE detection. For instance, (Wu and Zhao, 2018) proposed a color-based hybrid descriptor that combines local binary patterns, human movement invariants, and color histograms to extract features of hardhats in various colors, which are then classified using a hierarchical support vector machine (HSV). (Li etal., 2018) developed a method for detecting workers utilizing the ViBe and the C4 framework, leveraging the HSV color space for hardhat classification. (Mneymneh etal., 2019) introduced a framework that first detects moving objects using a standard deviation matrix and then classifies them with an aggregate channel feature-based object detector. This approach integrates a histogram of oriented gradients-based cascade object detectors to identify hardhats in the upper regions of detected personnel, which are subsequently processed by a color-based classification component. Despite their effectiveness, these multi-stage methods heavily depend on hand-crafted features and encounter difficulties in complex scenes with varying conditions, different viewpoints and occlusions.

Thus, convolutional neural networks (CNNs) (LeCun etal., 2015) have become the backbone of the systems due to their robust image recognition capabilities. Recent studies employing CNNs for object detection have primarily utilized faster region-based CNNs (R-CNNs) (Ren etal., 2015) and you only look once (YOLO) (Redmon etal., 2016). These models facilitate the recognition of target objects, such as persons and helmets. When trained with sufficiently large datasets, they exhibit robust and improved performance, significantly enhancing the reliability of vision-based PPE detection systems.Faster R-CNN incorporates a region proposal network to improve detection speed and accuracy. For example, (Saudi etal., 2020) leverages Faster R-CNN to check workers’ safety conditions based on PPE compliance. Also, (Chen etal., 2020) introduced retinex image enhancement to improve image quality for the outdoor complex scenes in substations based on Faster R-CNN.On the other hand, YOLO is known for its speed and efficiency making it suitable for real-time applications. YOLO divides the image into a grid and predicts bounding boxes and class probabilities directly from full images in one evaluation. Many researchers have utilized variants of YOLO for PPE detection. For instance, (Wu etal., 2019) utilizes the advantage of Densenet in model parameters and technical cost to replace the backbone of the YOLO V3 network for feature extraction, forming a YOLO-Densebackbone convolutional neural network to improve helmet detection and ensure safe construction. (Chen etal., 2023) introduced a lightweight WHU detection model called WHU-YOLO, which enhances YOLOv5s with a Ghost module and Bi-FPN. Additionally, (Benyang etal., 2020) used a multi-scale training strategy to enhance the adaptability of the YOLO V4 from different scales of detection.To enhance detection accuracy, researchers have increasingly integrated object detection models with classification models, resulting in more precise and reliable safety assessments. For instance, (Lee etal., 2023) proposed a method that combines the strengths of YOLO and CNN-based approaches. In this two-stage process, YOLO is first employed to detect whether a helmet is being worn. Subsequently, a CNN-based classification model is used to distinguish between helmets, heads, and hats, allowing for a more nuanced evaluation of PPE compliance.

Despite these advancements, most research has focused on specific, high-supervision environments, such as construction sites, industrial plants, and nuclear facilities, where clear safety protocols and structured activity schedules are in place (Chen and Demachi, 2020; Önal and Dandıl, 2021). In dynamic and less supervised environments, such as manufacturing laboratories or outdoor construction sites, the requirements for PPE usage can vary significantly depending on the time of day, specific tasks being performed, and changing environmental conditions. Consequently, there is a pressing need for interpretable detection methods that can adapt to diverse scenarios and provide insights into the specific safety requirements of each context. This necessitates a system capable of not only detecting PPE but also understanding the contextual factors influencing its usage, thereby ensuring comprehensive safety compliance across different settings (Fang etal., 2018; Hung and Su, 2021).

2.2 Vision Language Model

Recent years have witnessed substantial success in extending pre-trained vision language models to support new applications. Among the most successful efforts, models like Flamingo (Alayrac etal., 2022), OpenAI CLIP (Radford etal., 2021), and OpenCLIP (Cherti etal., 2023) have exhibited impressive performance in handling image-text matching, owing to their semantic knowledge and understanding of content that spans both modalities. These models have been applied successfully in downstream applications such as object detection (Shi etal., 2022), image captioning (Mokady etal., 2021), action recognition (Wang etal., 2021), task-oriented object detection (Chen etal., 2024), anomaly segmentation (Jeong etal., 2023), semantic segmentation (Liang etal., 2023), and dense prediction (Zhou etal., 2023). However, existing CLIP-based algorithms primarily focus on matching image patches with nouns in text, which poses challenges in understanding different people and objects within the images. Therefore, additional modules are needed to facilitate the matching between the visual attributes of image patches and the adjective phrases describing different people and items.

To enhance the alignment between images and text, a recent application of VLMs for PPE detection introduced a three-step zero-shot learning-based monitoring method (Gil and Lee, 2024). First, it detects workers on-site from images and crops body parts using human-body key points. Next, the cropped body images are described with image captioning. Finally, the generated text is compared to prompts describing PPE-wearing body parts, determining safety based on cosine similarity. However, this method still has limitations in accurately distinguishing between different types of PPE and their specific features in diverse environments.

Alongside these advancements, significant progress has been made in question-answering based VLMs, which is usually called visual question answering models (VQA). (Bulian etal., 2022) pioneered the creation of the first free-form and open-ended VQA dataset, where human workers were asked to create questions that a smart robot might not answer and then collect human answers per question. The follow-up work, VQAv2 (Goyal etal., 2017), enhanced the previous dataset to reduce statistical biases in the answer distribution. Instead of relying on human workers, the GQA dataset (Hudson and Manning, 2019) employed scene graph structures from visual genome (Krishna etal., 2017) to generate question-answer pairs. These graph structures allowed the authors to balance the answer distributions for each question, reducing the dependency on answer statistics. Both datasets have been widely used for various VQA tasks (Alayrac etal., 2022; Dai etal., 2023). Due to the question-answer format of VQA models, it aligns more naturally with the process of inspecting safety equipment. (Ding etal., 2022) formulated a "rule-question" transformation and annotation system, turning safety detection into a visual question answering task, and leverages the strengths of VQA models to enhance the accuracy and efficiency of PPE compliance checks.

Despite these advancements, a notable tradeoff exists between achieving high-performance VQA models and maintaining acceptable inference times. High-performing VQA models typically require significant computational resources and time, leading to longer inference times. However, real-time safety detection applications impose strict requirements on inference times to ensure timely and effective responses. Thus, making a new model for such applications necessitates a careful balance between performance and inference speed.

3.1 Scene Recognition Module

To address the challenges posed by the diversity of real-world scenes and the corresponding variety of safety gear requirements shown in Fig.1(a), we developed a scene recognition module to identify scenes in images. As illustrated in Fig.2 ❶, this module generates high-quality image captions that provide concise summaries of the visual content. However, this abstraction process can result in the loss of specific visual details, potentially affecting our framework’s performance. To tackle this issue, we conducted a comprehensive survey of existing image captioning models, prioritizing those capable of generating more accurate scene descriptions while also considering our computational resource constraints. Ultimately, we selected Salesforce’s BLIP2-OPT-2.7B model as our image captioning model due to its advanced multi-modal capabilities, which integrate both vision and language understanding, allowing it to generate more nuanced and context-aware descriptions.

To further improve the performance of the scene recognition module while adhering to computational resource constraints, we employed Low-Rank Adaptation (LoRA) (Hu etal., 2021) to provide accurate scene recognition by fine-tuning a minimal number of weights.For this purpose, we used the text prompt "in the [scene]" paired with the respective image as the textual input.LoRA works by introducing small additive weights into existing linear weight matrices. These weights retain the original weight matrix’s dimensions and provide a parallel trainable pathway for the incoming feature map when multiplied. The core idea is that during adaptation the updates to the weight matrices have low intrinsic ranks.

The modification involves adding a rank-constrained product of matrices $\bm{A}$ and $\bm{B}$. For a weight matrix $\bm{W}\in\mathbb{R}^{d\times k}$, the update is represented as $\bm{W}^{\prime}=\bm{W}+\bm{B}\bm{A}$, where $\bm{B}\in\mathbb{R}^{d\times r}$ and $\bm{A}\in\mathbb{R}^{r\times k}$, and $r\ll\min(d,k)$. Here, $d$ and $k$ are the dimensions of $\bm{W}$, and $r$ is the rank of the adapters. During fine-tuning, $\bm{W}$ remains unchanged, while the weights of $\bm{A}$ and $\bm{B}$ are updated. Following the initialization method in Hu etal. (2021), $\bm{B}$ is initialized to zeros, and $\bm{A}$ with small random values from a Gaussian distribution. The updated matrix $\bm{W}^{\prime}$ is given by:

3.2 Visual Prompt Module

In the visual prompt module, we address the diversity of safety requirements by employing a large language model to generate the necessary safety items based on the scene, as depicted in Fig.2 ❷. To bridge the semantic and reasoning gap between the fine-grained requirements and the safety items, we also employ the LLM to generate scene-relevant visual attributes, as illustrated in Fig.2 ❺.In Fig.3, we provide a detailed description of the prompts utilized to guide the LLM in generating the appropriate safety items and their visual attributes. To illustrate this process, we take a seafood factory as an example. The specific procedure unfolds as follows:

3.3 Safety Items Detection Module

Here, $\bm{P}_{i}$ represents the prediction of whether the person is wearing the item. To optimize our results, we also adopted an LLM-driven approach, where the affinity matrix is provided to the LLM, allowing it to make the decision for us. As shown in Table 3, using GPT-4o as the decision-making LLM yielded better results.

3.4 Fine-grained Verification Module

In the fine-grained verification module, we leverage the result from the safety items detection module. If the person in the image wears the item, we utilize the same open vocabulary object detection model to generate the bounding boxes of the items and extract the image patches, as is shown in Fig.2 ❼. As illustrated in Table 1, three distinct types of attributes are generated for each item within the visual prompt module.

Here $\bm{P}_{j}$ is the prediction of whether the item worn by the person meets the attribute requirements. We can also leverage the LLM-driven approach to optimize the result. However, this step is different from 3.3. We will generate a similarity list for all the items’ corresponding attributes, and then we give the list to GPT-4o to make desicion for us.

3.5 Evaluation

Ensuring compliance with safety item requirements involves detecting whether a person is wearing the necessary safety items and verifying that these items meet specified attribute standards. We structure this task into three stages: safety items detection, feature verification, and overall evaluation. For each stage, we define different metrics to comprehensively evaluate model performance.

The first stage, safety items detection, focuses on identifying the presence of required safety items, such as hard hats, safety goggles, and high-visibility vests, on individuals. We choose five required safety items for each environment, generated from LLM, as shown in 3.2.This stage leverages advanced object detection algorithms to accurately pinpoint and label each safety item within the visual input. The detection accuracy is measured by determining the proportion of safety items correctly identified by the model relative to the total number of items that should be present.To express this more precisely, the total number of individuals is considered, along with the actual number of safety items each person is wearing. The detection accuracy is then calculated by comparing the number of safety items the model correctly identifies with the expected number of safety items across all individuals.

The second stage, feature verification, involves validating that the detected safety items adhere to specific attribute requirements. For example, it is not only important to detect the presence of gloves but also to ensure that they comply with standards for chemical resistance or thermal protection, depending on the operational context. As detailed in table 1, the attributes are classified into three distinct categories: Directly Observable (DO), Situationally Observable (SO), and Inferentially Observable (IO), with each attribute being assessed individually.To assess the accuracy of the feature verification for each attribute type, we compare the number of correctly detected attributes within each category (DO, SO, and IO) against the total number of detected items. For Directly Observable attributes, the accuracy is calculated by determining the proportion of correctly identified attributes out of the total items detected. The same approach is applied to Situationally Observable and Inferentially Observable attributes, where the accuracy is measured by comparing the correctly identified attributes in each category to the total number of detected items.

4.1 Dataset

To meet the diverse scene requirements of our model, we collected and integrated data from four sources due to the lack of a suitable comprehensive public dataset. From these sources, we selected five distinct scenes, and each dataset is described in detail as follows:

As shown in Fig.4, the proposed dataset is an attribute-based detection dataset, which covers 6 scenes, and 9 object categories, encompassing 49 attributes across image datasets. The attributes used to describe objects that are reported in table 1. As shown in Table 1, to address the varying levels of complexity in visual attribute recognition, we classify attributes into three categories based on the level of context and reasoning required to identify them accurately. Directly observable attributes are those that can be immediately identified from the visual features without the need for additional context or interpretation. Situationally observable attributes require some understanding of the scene’s context but still have a significant visual component that aids in their identification. Finally, Inferentially observable attributes are those that cannot be directly seen but must be inferred through context, usage, or additional knowledge about the object. These three categories correspond to color, material, and functionality.

4.2 Implementation Details

For the scene recognition module, we used BILP 2 with configuration of blip2-opt-2.7b. For the object detection model, we experiment with Yolo-world. The implementation of Yolo-world utilizes the yolov8l-worldv2 configuration in Ultralytics. Regarding the large VLM, we employ Open-CLIP. Specifically, the vision transformer encoder configuration is ViT-L/14@224px and the text encoder is OpenAI’s tokenizer. We generate required safety items and visual feature attributes for each item using OpenAI ChatGPT 4o. All experiments are conducted with a single NVIDIA RTX 4090 24GB GPU.

4.3 Compare to state-of-art methods

To align with our model’s process of first verifying whether the required items are worn and then checking if the items have the required attributes, we chose the VQA model as our baseline. In Fig.5, we use LLaVA-1.6-7b to show an example of how we do our task on VQA models. Similar to our model, we divide the questions to two steps. In the first step, we utilize yes/no questions form interrogative sentences to check whether the person wearing the items. In the second step, we use wh-question form interrogative sentence to check whether the item has the required attributes. To evaluate the correctness of an answer, it is required to compare a VQA model’s output with an answer. We begin with the simplest evaluation metric, called ExactMatch (EM). It involves preprocessing the model’s output p, a free-format text that resembles natural language with humanlike fluency, and the corresponding answer c by removing non-alphanumeric characters, applying lowercase conversion, and normalizing spaces. However, for EM a prediction is considered correct only if the model’s output exactly matches the correct answer. This strict criterion may not accurately evaluate the model’s performance as it does not account for semantically correct but syntactically different answers. A less restrictive option is to consider a response correct if the prediction contains the true answer after preprocessing, named as Contains (Cont) (Xu etal., 2023).As shown in Fig.5, the ground truth answers are highlighted in yellow. If the output of the VQA models contains the true answer, it is considered correct. For example, in response to the question, "What functionality does the mask worn by the person in the image have?" LLaVA-1.6-7b correctly identifies that the mask can prevent airborne particles, bacteria, and viruses, and the spread of infections. These functionalities are included in the ground truth answer, so the response is marked as correct.

In Table 3, we present a comparison of Clip2Safety’s performance with other state-of-the-art VQA models, highlighting results for Step 1 and three observable levels in Step 2. We selected one model fine-tuned on VQAv2, BLIP (Li etal., 2022); one multi-purpose model evaluated in a zero-shot manner, BLIP-2 (Li etal., 2023); and two conversational models, InstructBLIP (Dai etal., 2023) and LLaVA (Liu etal., 2024). All models consist of an image encoder and a text encoder based on the transformer architecture, along with a fusion model. These models leverage pretrained unimodal encoders and train the fusion model using image-text data.BLIP fine-tunes the encoders together with the fusion module using common losses such as image-text contrastive loss (ITC), image-text matching loss (ITM), and image-conditional language modeling loss (LM). In contrast, BLIP-2, InstructBLIP, and LLaVA keep the unimodal encoders frozen and train only the fusion module. Notably, part of the training data for InstructBLIP includes the VQAv2 dataset. This comparative analysis underscores the architectural differences and training strategies employed by each model, providing context for their performance metrics.In Table 2, we detail the configurations of various models. Each model’s image resolution, prompt type, visual encoder, text encoder, and model size are specified. The BLIPvqa model uses a 480-pixel image resolution with a Vit-B/16 visual encoder and a BERT text encoder, totaling 0.361 billion parameters. BLIP-2 OPT employs a 224-pixel resolution, Vit-g/ EVA-CLIP visual encoder, and an unsupervised OPT text encoder, with a model size of 3.745 billion parameters. Similarly, BLIP-2 T5 uses the same visual encoder but with a FlanT5 XL text encoder, resulting in a 3.942 billion parameter model. The InstructBLIP T5 and InstructBLIP V models also utilize the Vit-g/ EVA-CLIP visual encoder but differ in their text encoders, using FlanT5 XL and Vicuna-7B, with sizes of 4.023 billion and 7.913 billion parameters, respectively. The LLaVA-1.5-7B and LLaVA-1.6-7B models feature the ViT-L/14 CLIP visual encoder and text encoders LLaMA 7B and Vicuna-7B, with sizes of 6.743 billion and 7.06 billion parameters, respectively, both at a 224-pixel resolution. Finally, the Proposed Model combines the ViT-L/14 CLIP visual encoder with OpenAI’s tokenizer, resulting in a model size of 0.428 billion parameters and a 224-pixel resolution.

In Table 3, we show the results of comparing our model with other VQA models.In Step 1, LLaVA-1.5-7b and LLaVA-1.6-7b models performed notably well, achieving accuracy of 74.4% and 91.2%, respectively. Our Clip2Safety matched InstructBLIP T5 with an accuracy of 76.8%, while using GPT4o to make decision on our Clip2Safety scored 77.2%.In Step 2, focusing on directly observable attributes, we can see that in step 1, the powerful LLaVA is not as good as BILPvqa. Our Clip2Safety can achieve an accuracy of 76.9%, and the accuracy of using GPT4o to make decisions can be improved by 1.8%.For situationally observable attributes, our model is 1.4% lower than the best LLaVA-1.5-7b. When we use GPT4o to make decisions, the accuracy can be improved to 66.6%. In the inferentially observable (IO) category, LLaVA-1.5-7b and LLaVA-1.6-7b excelled with 82.0% and 78.0%. However, in terms of average accuracy, LLaVA-1.6-7b is ahead of other VQA models. But it is still 0.1% lower than our Clip2Safety. Clip2Safety+GPT4o can achieve the highest average accuracy of 72.3%. And we also compare the inference time in Table 4. We can see that no matter step 1 or step 2, the inference time of our model is much lower than that of other VQA models.

4.4 Ablation Study

4.4.3 Comparison of Performance Across Different Threshold Values

In this section, we investigate the effect of different threshold values on our model’s performance. Viewing detection as a binary classification problem, Fig.6 presents the ROC curves for various steps of our model evaluation, emphasizing the impact of varying thresholds on the true positive rate (TPR) and false positive rate (FPR) across different observable levels. We explore the range of threshold values, $\delta$, from 0.58 to 0.65, as all the similarity scores fall within this interval.The color gradient in each plot represents different threshold values within this range, showing their significant impact on model performance. Higher thresholds generally result in fewer false positives but may also reduce the TPR, as indicated by the tighter clustering of points near the lower end of the FPR axis. Conversely, lower thresholds increase the TPR but also raise the FPR, illustrating the trade-off between sensitivity and specificity.

The upper left plot presents the ROC curve for Step 1, which involves detecting whether a person is wearing the required item. The model achieves an area under the curve (AUC) of 0.76, indicating a strong ability to distinguish between true positives and false positives at this initial stage. The curve shows that the model maintains a high TPR while keeping the FPR relatively low, especially at moderate threshold values, which suggests that the model is effective in detecting the presence of required safety items with fewer errors.

The upper right and lower left plots show the ROC curves for the directly observable and situationally observable steps, with AUCs of 0.64 and 0.76, respectively.The decrease in AUC for the directly observable step compared to Step 1 indicates that as the observables become more challenging to detect, the model’s performance drops. The curve for the directly observable step reveals a wider spread of points along the FPR axis, indicating increased false positives at certain thresholds.Despite the increased difficulty in observables compared to the directly observable step, the situationally observable step maintains a similar AUC to Step 1. This indicates the model’s robustness in handling situational contexts where observables may vary depending on the scenario. The curve demonstrates that the model can still achieve a high TPR with manageable FPR at optimal thresholds, which suggests that the model is adaptable and performs well even when the context of observables is less straightforward.

The lower right plot presents the ROC curve for the inferentially observable step, which involves assessing the functionality of the item. This step has the lowest AUC of 0.55. This significant drop in performance indicates that the model struggles to accurately classify true positives and false positives when the observables require inference rather than direct detection. The curve shows a broad distribution of points, reflecting high FPRs even at higher threshold values, which illustrates that the model’s sensitivity is greatly challenged in inferential scenarios.

4.5 Visualization and Discussion

Fig.7 presents our model’s detection results on various image samples. Red bounding boxes label the person and blue bounding boxes label the items. Specifically, Fig.7 (a) showcases an example where our model performs well, with correct results in both step 1 (Safety Items Detection) and step 2 (Feature Verification).

However, Fig.7 (b) highlights an instance where our model’s performance is suboptimal. While the results are accurate in step 1, identifying the required safety items, the model fails in step 2, incorrectly verifying the attributes of shoes. This example illustrates a common challenge faced by our model: accurately verifying specific attributes of safety items in diverse and complex environments.

Additionally, Fig.7 (c) demonstrates a scenario where the model underperforms in both steps, failing to detect the required safety items and verify their attributes. This case underscores the difficulty of ensuring comprehensive safety compliance in highly variable settings, highlighting the need for further refinement and enhancement of the model to improve its robustness and accuracy across different scenarios.