Realigning Goals and Uncovering Hidden Patterns to Drive Impact

The WatchGuard project was initially designed to address non-compliance behaviors with the goal of improving drug quality, and the focus was placed on the accuracy of the AI model. However, upon further analysis, I recognized that this approach was deviating from the true objective—preventing non-compliance behaviors. I realized that the key measure should not be solely the accuracy of detection, but rather how many non-compliance behaviors are actually caught and prevented.

I shifted the focus from asking "How much is right?" to "How much is caught and avoided?". The true goal should be to identify and prevent non-compliance behaviors, rather than simply improving AI accuracy. This shift in perspective helped align the project with its intended purpose of enhancing compliance and ultimately improving drug quality.

Through user research, I uncovered previously unidentified user behavior patterns. Staff like Barry and Zach, although trained, occasionally forget or fail to adhere to compliance policies. Human nature—forgetfulness and carelessness—plays a significant role in this. To address this, I introduced a "Alert, Vigilant, Slack off, Re-alert" user behavior model (Picture 1) to ensure compliance is consistently reinforced.

Picture 1: The Alert, Vigilant, Slack off, Re-alert User Behavior Mode

Introducing the Lighthouse User Targeting Strategy

As faith in the project has significantly diminished across the six drug production plants, I adopted a lighthouse strategy—"start small, expand broadly" —(Picture 2)to turn the situation around. Recognizing the users' urgent need for positive outcomes, I consulted with stakeholders and selected Drug Production Plant A, which has maintained the most positive attitude toward the project, as the starting point for its revitalization.

Picture 2: The lighthouse strategy will scale WatchGuard from plant A to plant B,C,D,E

One challenge we faced was the involvement of too many overlapping stakeholders, which complicated decision-making and progress. By narrowing our focus to Plant A, we streamlined stakeholder involvement and concentrated efforts on achieving tangible improvements. These successes will then be scaled to the other production plants.

Simultaneously, I am ensuring that users from the remaining five plants remain engaged and do not perceive the project as abandoned. This is achieved by consistently sharing updates, actions, and successful cases from our progress at Plant A, keeping the broader user base informed and optimistic.

Redesign The User Flow

The WatchGuard project aims to assist Talia, the Quality Control Technician, in using the model to alert workers who are not complying with regulations. However, the project is facing two main issues. First, the AI model’s results are difficult for humans to review, making it challenging to assess the accuracy of its detections due to a lack of interpretability. Second, there is no mechanism in place for users to provide feedback on the model's performance, hindering its continuous improvement.

To address these challenges, I redesigned the user flow. The human verification process has been streamlined, reducing the previous 7 steps to 4, saved 5.5 hours per 1,000 AI outputs(Picture 3). I also added a feedback mechanism to facilitate ongoing AI model improvement(Picture 5).

Picture 3: AI Output Verification Flow Optimized—Before and After

After consulting with users, I also redesigned the alert flow. Initially, alerts were directed to workers, expecting them to immediately adjust their behavior. However, this approach was disruptive, as all workers in the room had to stop their tasks and assess whether they had breached the rules.

To resolve this, we decided to redirect the alerts to someone like Talia, the Quality Control Technician(Picture 4). She can identify the worker in question, provide a targeted reminder, and maintain a record of non-compliant behavior for future reference. This approach reduces disruption and allows for more effective monitoring and correction.

Picture 4: Redesigned The Alert Flow: From Alert All Workers To Alert Only The Quality Control Technician

Shifting Focus to Data While Enhancing the Model

While the key measure should focus on how many non-compliance behaviors are caught and prevented rather than solely on detection accuracy, improving our model's accuracy remains a top priority. False alarms from the AI model disrupt the workflow and require significant human effort to verify. Through a root cause analysis, I identified that the primary issue behind the low accuracy is poor data quality. After a thorough discussion with Alex, our AI engineer, we concluded that instead of concentrating solely on optimizing the AI model, our efforts must prioritize improving the quality and accuracy of the data.

Classify and Fix Incorrect Data Types

"Data tells the biggest truths and the biggest lies." WatchGuard was supposed to be a reliable tool based on real-time data streaming from cameras, but it ended up generating many false alerts. I took the overcrow model as an example and reviewed its false results. The overcrowding model is originally designed to trigger alerts when the number of people in a room exceeds the compliance limit. After a careful review of over 7,000 pictures evenly sampled from the live cameras over past three months, I identified seven main types of errors(Picture 5):

Picture 5: AI Output Human Varification Optimized Interface

Phase results: I shared these findings with Alex, the AI engineer, who focuses on improving the AI model configuration, and asked if we could address each of these issues specifically. The response was yes! But we need more of those datasets accordingly.

Enrich the datasets in variety

In the WatchGuard system, the AI model processes footage captured in seconds from live-streaming cameras located in the drug production rooms. Me and Alex, the AI engineer, examined the dataset used to train the model. We found that while the dataset is massive, it lacks sufficient diversity. For example, we have the gowning AI model, which ensures workers comply with hygiene standards by preventing contamination from human sources.

The gowning scenario was included in the project due to its high importance as humans can be a major source of microbial contamination in drug manufacturing. This scenario is also crucial for protecting workers by ensuring they don’t expose any skin to possible hurts from reagents or solutions. However, this is also one of the behaviors where workers tend to comply most consistently. As a result, we don’t have enough footage to train the model on issues like improper gowning type, miswearing masks, gloves, shoe covers, or head covers. To address this, I drafted a Gowning Data Gathering Manual for the drug department, requesting their assistance in capturing more varied footage, 600 wrong gowning pictures for each type after consulting with Alex.

Phase Results: Due to the tight production schedule, we couldn't find an open window to collect the necessary data. However, we were directed to use footage from the factory’s maintenance downtime, when maintenance staff were seen mis-gowning.

Communication Strategy: Building User Alignment

In the face of major disappointment with the project, users struggled to understand what went wrong. Many simplified the failure, concluding that the model's performance was poor. However, I recognized that the AI model works iteratively with data and requires ongoing training based on evolving datasets. One key issue was whether the current dataset covered all business scenarios. Clearly, the existing model, particularly for gowning (identifying if workers are properly dressed with masks, gloves, shoes, and proper coverage), was incomplete. To address this, we need at least 600 negative images for each scenario, but in reality, we only had a few. Additionally, if new business scenarios arise, these need to be added to the dataset, and the model retrained. Before any new training, it’s premature to conclude that the model's performance is low.

I identified that users needed to adopt an iterative mindset and understand their role as key contributors to the project. They must provide representative datasets and ensure continuous updates. To foster user collaboration toward a common goal, I spoke with Alex and suggested regular progress meetings with users, where we could showcase the successes of the model outputs.

Model Progress Meetings with Users

I helped Alex organize and outline these meetings, which were designed to transparently share the model's progress with users. The goal was to provide key information about the model, including the number of datasets, metrics of the model's accuracy, the model's sensitivity, and the model's performance(Table 1). By doing this, we empowered users with a deeper understanding of the project’s status. This not only helped clarify the importance of a continuously evolving dataset but also encouraged active participation from the users. These meetings also highlighted the benefits of transparency and collaboration, ensuring that users see the direct impact of their contributions.

Table 1: Overcrowd Model Performance Comparison

Regaining Trust Through Gradual Model Outcome Email Reports

In addition to providing hard data on the model’s performance, I introduced visually engaging graphic email reports(Chart 1) to make the results more accessible and impactful. These reports combined detailed statistics with clear, attractive visualizations, vividly illustrating the model’s trajectory. They highlighted how much non-compliance behaviors were detected in each production room and the accuracy of those detections. Over four months, the model identified hundreds of behaviors with only three errors, offering a clear demonstration of its effectiveness. This approach not only gave users transparent insights into the model's progress but also helped rebuild their trust. The benefits of this initiative included fostering confidence in the model’s capabilities, reinforcing the project’s positive momentum, and showcasing the tangible impact of their involvement.

Chart 1: Overcrowd model v5.7 Correct Output Distribution in 4.5 months

Driving Technology Innovation: Optimizing Model Selection and Dataset Preparation

To improve the accuracy of AI model detections, I recommended that the team focus on creating a more representative dataset and iterating continuously, which led to significant success with the over-crowding model. Simultaneously, my team and I prioritized evaluating the performance of various computer vision models to determine the most suitable one for our specific use case.

I spearheaded a comparative analysis of various models, including human label image via LabelImg and object detection (YOLOv7) vs instance segmentation (Mask R-CNN) and positive/negative classification (Inception V3 via MATLAB). As the key point of contact, I conducted an ad hoc review of several thousand data points to inform our decision-making. Through this process:

Benefits of the Ad Hoc Analysis:

This innovative approach not only addressed technical challenges but also highlighted my ability to align technological decisions with business goals effectively.