What a fintech solutions architect learned building a PPE detector in three evenings

Why a fintech SA spent three evenings on computer vision

For seven years my job has been taking a hard technical platform (identity verification, KYC, fraud) and making it work inside someone else’s environment. Discovery calls, proof of concept, integration, go-live. The domain was regulated and the interface was an API.

I wanted to know whether vision work has the same shape. Does “can your platform do this for us?” resolve the same way when the input is a camera instead of a JSON payload?

So I built one. A PPE compliance detector, checking whether the person on camera has a hard hat and a hi-vis vest, trained and run end to end on a laptop over three evenings. Below is what I got wrong, and the two things I would now say to a customer before anyone writes code.

I had been here once before. My MSc dissertation fine-tuned Mask R-CNN on DeepFashion2’s thirteen garment categories across 23,424 training images, and the segmenter finished at about 6 AP overall, with eight of the thirteen categories at zero and small-object AP at zero across the board. I could see precisely what was wrong with the data and had no practical way to fix it, which is roughly the problem that annotation and retraining tooling exists to solve.

The twenty-two minute model

• A 479-image hard-hat and hi-vis dataset from Roboflow Universe, plus 34 frames of my own kitchen added as hard negatives: a bike helmet, a plain cap, a black gilet, a hi-vis jacket.
• Three augmentations, 1,228 images.
• RF-DETR Nano, 22 minutes on hosted GPUs, 83.6% mAP@50.

None of that is the interesting part.

Absence is not an object

The dataset had four classes: hat, vest, no hat, no vest. Two of them describe things. Two of them describe the absence of things. That distinction caused every problem I hit.

First it broke auto-labelling. I ran SAM 3 auto-label over my 34 frames with descriptions like “person’s torso without a high-visibility vest.” It labelled my hi-vis jacket no vest. Text-prompted models key on nouns and mostly ignore negation, so “without” barely registers. It also fired no hat only on the bike helmet and never on a bare head, because a bare head has no helmet-shaped thing to find. I re-labelled all 34 by hand.

Then it broke the model. Trained and pointed at my kitchen, it tagged four bar stools as no vest at 0.57 to 0.65 confidence. Of course it did. I had taught it that no vest means a torso-shaped region without hi-vis, and a chair back qualifies.

Then it broke the logic. In one frame the same torso carried vest at 0.54 and no vest at 0.63 at the same time. The classes are not mutually exclusive at the box level, so “is this person compliant?” cannot be answered by reading the highest-confidence box.

A better model would not fix any of this. The design has to change: detect the things that exist (person, hard hat, hi-vis) and compute compliance as logic on top. A person box with no hi-vis box overlapping it is a violation. That is a rule you can explain to a safety officer, test, and change without retraining.

In integration work the equivalent lesson is agreeing the schema before anyone writes a mapping. Here it is agreeing the label taxonomy before anyone draws a box. Is a bump cap a hard hat? Is a hi-vis jacket with sleeves a “vest”? I had to decide both, and the model only ever learned what I decided.

Everything has two states

Three things cost me an evening between them, and not one of them was difficult.

• Save is not publish. I built the workflow, saved it, ran it locally, and got 30 fps with zero detections. The API was serving an empty spec while my graph sat in the draft field. An unpublished workflow runs successfully and does nothing, which is the part worth remembering: on a customer's line, process monitoring would show green. Monitor detections per minute instead.

  

• There are three model-ID formats, across the docs, the Python library and the workflow spec. The one in the docs raised InvalidModelIDError.
• The split percentages are post-augmentation. The version page reported 88/8/4, which looks like a starved validation set, for what was actually a 70/20/10 source split.

Every platform has edges like these. Knowing where they are is most of what a solutions architect is for.

Why my laptop numbers are worthless (and what I’d ask instead)

I got about 2 fps. That number transfers to nothing.

RF-DETR Nano is built for 384×384 and my dataset version was 640×640, so the model ran on roughly 2.8× the pixels it was designed for. The platform warned me and I proceeded anyway, which cost me half my throughput against the pretrained baseline. On Apple silicon, ONNX Runtime also split the transformer graph into about 100 partitions across CoreML and CPU. On a Jetson with TensorRT, none of that applies.

So “will it run in real time?” is not a question I can answer yet. It is two questions I would ask back:

1. What is the edge hardware, and what frame rate does the decision actually need? A reject gate on a moving line and a shift-end compliance report are different products.
2. What happens on a detection? Alert, log, or stop the line. That single answer decides whether a false positive on a bar stool is an annoyance or a production outage.

Roboflow’s published predictive-maintenance case study is the shape I would expect from a real deployment: a purpose-built model trained on roughly 400 images, a network camera, lightweight edge compute, and no internet connection. The model was the small part of that job, and so was mine.

What transfers

The vocabulary is new. The job isn’t.

Taxonomy before labelling is schema before integration. “Good enough for ROI” is the same conversation as a fraud threshold, where nobody ships 100% precision and the question is always what a miss costs against what a false alarm costs. And the thing that decides whether a proof of concept becomes a deployment is still the boring one: does someone on the customer’s side own the hardware, the network, and the retraining loop.

Three evenings didn’t make me a computer vision engineer. They did make the next conversation with a plant manager a lot more honest.