YOLOv5 + EA: Three-Channel Visual Quality Inspection and Direct DO Control on a Production Line

1. Introduction

With the rapid development of AI and edge computing, industrial vision systems are evolving toward multi-channel collaboration and integrated inspection-decision-making. The traditional discrete architecture of "camera + industrial PC + PLC" has a long chain and high cost, while cloud AI solutions struggle to meet the compliance requirement that production line data must not leave the factory. A multi-channel visual inspection system based on a lightweight detection model (YOLOv5) and on-device inference acceleration (TensorRT FP16) can simultaneously perform multi-station parallel analysis, defect determination, and direct actuator control on a single edge AI box. This solution implements three-channel real-time detection (labels and device-side screws) on a 157 TOPS edge AI box, millisecond-level control, and local large-model quality daily reports—with all inference closed offline. It provides a deployment paradigm balancing accuracy, real-time performance, and data security for industrial inspection, automated sorting, and production line monitoring.

2. Technical Solution Selection

2.1 Detection End: Why Still YOLOv5 Instead of YOLOv8

The criterion for selecting a detection model is not "newer," but "whether accuracy is saturated, whether the deployment chain is shortest, and whether the saved compute is more useful elsewhere." screw_v2's mAP50 has already reached 0.995; switching to YOLOv8 would not significantly improve accuracy. YOLOv5n across three channels takes only about 84 ms per cycle, and the remaining GPU headroom is exactly enough to deploy a 7B local large model.

2.2 Inference End: Why TensorRT FP16

The EA is a Tegra platform, and JetPack 6 includes TensorRT 10.3. FP16 was chosen over INT8 because: a YOLOv5n 640-input FP16 engine runs 28 ms per frame, totaling 84 ms for three channels—ample margin for a 10 fps inspection takt (150 ms per channel). INT8 would require a calibration set, and the gain over the 28 ms baseline would not exceed 10 ms—not worth introducing another layer of quantization error.

3. Implementation Steps

3.1 Environment Preparation

The EA ships with numpy, OpenCV, and TensorRT 10.3; the only missing piece is the Python binding for GPU buffer management.

Engines are built directly with trtexec:

#Screw model 640 input

/usr/src/tensorrt/bin/trtexec \

  --onnx=screw.onnx --saveEngine=screw.engine --fp16

# Label model 416 input (reuses accessory model, no retraining)

/usr/src/tensorrt/bin/trtexec \

  --onnx=label.onnx --saveEngine=label.engine --fp16

Output: screw.engine 6.2 MB, label.engine 5.4 MB. Deserialization loading takes about 1 second.

3.2Core Functional Module Design

Each detection channel is an independent process with four internal layers that do not block each other:

[Grabber thread]  cv2 V4L2 + MJPG + 10fps frame limit, queue depth 1 (keep latest frame only)

        │

        ▼

[TrtYolo  inference]      letterbox 640/416 → cuda-python H2D → FP16 inference ~28ms → D2H

        │

        ▼

[Decision state machine]        numpy NMS → detection count → 5 consecutive frames same-direction flip OK/NG (~0.5s debounce)

        │

        ├──────────────► [ModbusCoil] FC5 → iolib:50200 → Y-board DO

        │               NG = coil constantly energized / OK = release / active release on power-up

        │

        └──────────────► [Preview thread] atomically writes framed image to /dev/shm every 300ms

                         → preview_server(:8081) pushes MJPEG to browser

Three channels + one preview service = four systemd template instances:

# /etc/systemd/system/ea-detector@.service（template, front/back/label instances）

[Unit]

Description=EA visual QC channel %i

After=iolib.service

Wants=iolib.service

[Service]

Type=simple

User=bliiot

EnvironmentFile=/home/bliiot/ea_detector/%i.env

ExecStart=/usr/bin/python3 /home/bliiot/ea_detector/ea_detector.py

Restart=always

RestartSec=5

All field variables go into env files; changing parameters does not touch code:

# front.env —— front screw channel

CHANNEL=front

DEVICE=4                                  # /dev/videoN

ENGINE=/home/bliiot/ea_detector/screw.engine

IMGSZ=640

MODE=screw                                # screw = screws / absent = check presence

COIL=1000                                 # FC5 coil address → Y-board DO1

EXPECT=4                                  # minimum screws expected

CONF=0.50

3.3 Key Code 1: TensorRT 10 Tensor API

import tensorrt as trt

from cuda import cuda as cu, cudart as crt

crt.cudaFree(0)                    # establish primary context

self.engine = trt.Runtime(logger).deserialize_cuda_engine(f.read())

self.ctx = self.engine.create_execution_context()

# RT10 uses num_io_tensors property (get_nb_io_tensors is the C++ interface name)

for i in range(self.engine.num_io_tensors):

    name = self.engine.get_tensor_name(i)

    self.ctx.set_tensor_address(name, int(dev_ptr[name]))

# Three-stage asynchronous inference:

cu.cuMemcpyHtoDAsync(self.d_in, blob.ctypes.data, blob.nbytes, self.stream)

self.ctx.execute_async_v3(stream_handle=int(self.stream))

cu.cuMemcpyDtoHAsync(self.h_out.ctypes.data, self.d_out, self.h_out.nbytes, self.stream)

cu.cuStreamSynchronize(self.stream)

ost-processing does not require torch: YOLOv5 output (1, 25200, 6), obj*cls threshold + numpy IoU NMS in under 30 lines.

3.4 Key Code 2: FC5 Coil Write for Direct DO Control

ioLib's Modbus TCP server only accepts function code 5 (write single coil); addresses 1000–1003 correspond to the Y-board's 4 DO channels. Raw socket implementation with reconnect:

def write_coil(self, addr, on, retry=3):

    pdu = struct.pack('>BHH', 5, addr, 0xFF00 if on else 0x0000)

    for i in range(retry):

        try:

            self.tid += 1

            mbap = struct.pack('>HHHB', self.tid, 0, len(pdu) + 1, 1)

            self.sock.sendall(mbap + pdu)

            r = self.sock.recv(256)

            if len(r) >= 12 and r[7] == 5:      # transaction ID + function code echo match

                return True

        except Exception:

            self._connect()                     # Disconnection reconnection

    return False

During NG, the coil stays energized; it releases only when back to OK. On program startup, a release is actively written—driving the production line actuator (buzzer/rejection cylinder).

3.5 Key Code 3: Three-Channel Debounce State Machine

Determination is not output frame by frame; it flips only after 5 consecutive frames (about 0.5 s at 10 fps) in the same direction. This is the key to false-alarm prevention:

ng_frame = (count < EXPECT) if MODE == 'screw' else (count == 0)

if ng_frame:

    ng_streak += 1; ok_streak = 0

else:

    ok_streak += 1; ng_streak = 0

if ng_streak >= STABLE_N and state != 'NG':

    state = 'NG'

    mb.write_coil(COIL, True, retry=5)          # constantly energized

elif ok_streak >= STABLE_N and state != 'OK':

    state = 'OK'

    mb.write_coil(COIL, False, retry=5)         # release

Measured behavior: about 1 second after replacing the missing screw device, it flips to NG (5-frame debounce + frame interval); about 1 second after restoring, it returns to OK and releases DO.

Detection results:

3.6 Web Preview: Shared Memory Instead of WebSocket

# Preview  thread, every 300ms

cv2.imencode('.jpg', img, [cv2.IMWRITE_JPEG_QUALITY, 80])[1].tofile(tmp)

os.replace(tmp, '/dev/shm/ea_%s.jpg' % CHANNEL)   # atomic replace; readers never get a half image

The preview service reads these three images and pushes MJPEG on port 8081, one page for three channels. It saves 33% bandwidth compared with WebSocket + Base64 (no Base64 expansion), and the detection process is fully decoupled from the preview process—a preview crash does not affect quality inspection.

4. Deploying Qwen2.5-7B on Jetson Orin NX: Automatic Quality Daily Report Generation

4.1 Why Add This Layer

After running three YOLO channels, the EA has enough compute left to support a 7B-level local large model. Quantized to INT4, it occupies only 4.7 GB of memory and part of the GPU's idle compute, without competing with the three detection channels. The inspection system upgrades from a "sensor" to a "production line brain": perception (YOLO) → cognition (Qwen) → execution—all three layers on one device, fully offline.

But one thing must be clear first: the large model is not used for real-time alarms. NG determination, DO actuation, and buzzer alarms must be millisecond-level and deterministic—the detector and controller already handle these. Letting the large model intervene would add only uncertainty and duplicate messages, with no benefit. So this layer is positioned as "summarization for people to read after work"—shift daily reports, abnormal period identification, quality trends, natural language word counts, and incidentally archiving NG frames as samples for later fine-tuning.

4.2 Model Selection

Visual understanding is left to YOLO; the LLM only does text-layer summarization and routing. When the agent needs image information, feeding it detection boxes and confidence scores as JSON is enough—do not make the large model look at images again.

Numbers such as NG count and duration are all precisely aggregated by Python and written into the prompt. The model only turns numbers into natural language, fundamentally preventing the large model from fabricating statistics.

4.3 Data Foundation: State Event Persistence

The detector adds an event persistence line at state flips (JSON, O_APPEND single-line atomic write; three processes writing the same file will not tear):

def log_event(**kw):

    kw['ts_full'] = time.strftime('%Y-%m-%d %H:%M:%S')

    kw['ch'] = C['CHANNEL']

    with open('/home/bliiot/ea_detector/events.jsonl', 'a') as f:

        f.write(json.dumps(kw, ensure_ascii=False) + '\n')

Every NG/OK flip carries a timestamp, channel, detection count, and detection boxes—this JSONL is the sole data source for all intelligent functions below.

4.4 Three Agent Working Modes

A ~100-line pure standard library script (even the Ollama HTTP call uses urllib), three modes, only two of which use the LLM:

ea_agent.py archive  daemon: incrementally watch events.jsonl → NG event → archive current frame of that channel to datasets/pending_review/ (no LLM)

ea_agent.py report   one-shot: aggregate daily statistics + abnormal periods + LLM writes daily report → reports/

ea_agent.py ask ...  one-shot: statistics → LLM answers (numbers and judgments provided by script)

Archiving only moves files: same channel saves only one image within 30 seconds, with filename carrying detection count (HHMMSS_n2.jpg). Accumulated images become material for a round of "deployment-perspective supplementary sampling," preparing to widen the confidence boundary for fine-tuning—this chain does not require the large model at all.

Daily report is the main function. The statistics part is all computed in Python: NG count per channel, cumulative duration, hourly distribution, abnormal periods with 3 occurrences within 5 minutes, comparison with yesterday—then handed to the LLM to write a 120-word conclusion. Measured output:

The key is the "abnormal period" line: front reporting 10 times within 3 minutes—a human would spend half an hour digging through logs, but the script calculates it in one second. This is the real increment of the agent over "printing a count table."

Production line Q&A measured: "Which channel had the most NG today? Is any channel still in NG state?"—the model correctly cited statistics to answer, without fabrication.

4.5 Deployment Form

[Three ea-detector@*]──state flip──► events.jsonl

                                                              │

                        ┌──────────────┴──────────────┐

                        ▼ Incremental watch (NG)                                  ▼ Scheduled 08:00 / 18:00

                  [ea-agent archive]                                                [ea-report]

                        │ No LLM                                                              │ Aggregate → LLM writes conclusion

                        ▼                                                                         ▼

              datasets/pending_review/                 reports/YYYY-MM-DD_HHMM.md

              (fine-tuning material)                                       │

                                                                                      ▼

                                                           [Ollama 127.0.0.1:11434] ← qwen2.5:7b q4

Three stabilization configurations: Ollama side OLLAMA_NUM_PARALLEL=1 (LLM is a soft task, detection is hard real-time; limiting concurrency prevents GPU contention from causing determination jitter); daily report triggered by ea-report.timer with Persistent=true to ensure missed time points are caught up after boot; archive process resident but not touching GPU, zero impact on detection.

4.6 What This Layer Changes

• Quality data has a "mouth": shift daily reports automatically land in reports/, abnormal periods visible at a glance—no more grep through logs.
• Real-time actions are not disturbed: DO alarms remain millisecond closed-loop by the detector; the large model only intervenes afterward—a few seconds slower does not affect any execution action.
• Model iteration has material: NG frames are automatically archived; next-round fine-tuning samples are "accumulated" rather than requiring a dedicated supplementary sampling exercise.
• Fully offline:images, quality data, and LLM inference all closed locally—zero data leaves the factory.
• Marginal cost ≈ 0:the agent is a 100-line script + an already-installed Ollama, with no new hardware and no new cloud service bills.

5. Summary

The real value of this solution based on the Beilai EA250 (Jetson Orin NX) is not "running a few models," but using one edge AI box to close the loop of "perception, cognition, and execution" in industrial quality inspection.

Practical recommendations: