本項目的題目為: 石頭剪刀布猜拳識別。本項目實現了基本猜拳識別、根據識別 猜拳結果，與機器內的手勢對比，根據結果控制舵機的行為，并且將識別的結果同步顯示在上位機。

目錄

項目概述

硬件清單

舵機控制引腳

軟件與運行環境

固件編譯

創建數據集

模型訓練

代碼結構與主要函數

使用方法

常見問題

項目源碼

1 項目概述

目標: 在3秒以內識別猜拳的結果，并且根據猜拳輸贏控制舵機旋轉，將識別結果顯示在上位機

平臺: 瑞薩visionboard(攝像頭、GPIO)

核心思路:

通過瑞薩visionboard采集照片數據作為數據集進行模型訓練

將采集的數據集通過edge impulse進行訓練

使用mqttx接受來自瑞薩visionboard的識別數據

根據識別猜拳結果控制舵機行為

使用手冊: RA8D1 Group User’s Manual: Hardware（https://www.renesas.cn/zh/document/mah/ra8d1-group-users-manual-hardware?r=25456556）

使用燒寫工具: Renesas Flash Programmer V3.12（https://en.freedownloadmanager.org/Windows-PC/Renesas-Flash-Programmer.html）

openmv固件: openmv固件（https://github.com/RT-Thread-Studio/sdk-bsp-ra8d1-vision-board）

2 硬件清單

瑞薩visionboard開發板(攝像頭)

攝像頭(RGB565，工作分辨率 320×240，ov5640)

360度 SG90舵機

3 舵機控制引腳

P008 —> 舵機

P008地址: 0x4040_0000 + 0x0020 × m(參考RA8D1 Group User’s Manual: Hardware 655頁)

4 軟件與運行環境

openmv固件（https://github.com/RT-Thread-Studio/sdk-bsp-ra8d1-vision-board）

主要依賴: sensor、time、tf、network、uctypes

5 固件編譯

從github（https://github.com/RT-Thread-Studio/sdk-bsp-ra8d1-vision-board）倉庫上將代碼拉取到本地

運行鏈接腳本后，進入到 sdk-bsp-ra8d1-vision-board-master\projects\vision_board_openmv，打開 RT-Thread Env Tool，輸入 scons —target=mdk5

項目生成后在C++ 預處理加上MICROPYTHON_USING_UCTYPES 的 define

(可以在sdk-bsp-ra8d1-vision-board-master\projects\vision_board_openmv\packages\micropython-v1.13.0\port\mpconfigport.h查看定義)，

編譯后在objects文件夾下得到rtthread.hex的openmv固件

使用Renesas Flash Programmer V3.12 燒寫固件，將 Enable address check of program file 選項去除勾選

6 創建數據集

使用瑞薩visionboard進行訓練圖片采集，這樣可以保持輸入輸出 的一致，一定程度上提高了識別的準確率。

importsensor, image, time, ossensor.reset()sensor.set_pixformat(sensor.RGB565)sensor.set_framesize(sensor.QVGA) #320x240sensor.set_windowing((240,240))sensor.skip_frames(time=2000)img_counter=0whileTrue: img= sensor.snapshot() filename="/dataset/scissors/scissors_img_%03d.jpg"% img_counter img.save(filename) print("Saved:", filename) img_counter+=1 time.sleep_ms(500)ifimg_counter >=550: # 停止條件break

運行時需要插入sd卡，并且提前創建對應的文件夾

7 模型訓練

使用edge impulse訓練模型， 使用Transfer learning，圖片大小為240*240，訓練時將圖片輸入改為灰度，減少干擾

8 代碼結構與主要函數

mqttx：

def publish(self, topic, msg, retain=False, qos=0): 發送消息到mqttx

defpublish(self, topic, msg, retain=False, qos=0): pkt= bytearray() # MQTT publish header header=0x30 ifretain: header|=0x01 ifqos ==1: header|=0x02 elifqos ==2: header|=0x04 pkt.append(header) # 計算剩余長度 remaining_length=2+ len(topic) + len(msg) ifqos >0: remaining_length+=2 # 包含packet id # 先編碼剩余長度 defencode_len(length): encoded= bytearray() whileTrue: digit= length %128 length= length //128 iflength >0: digit|=0x80 encoded.append(digit) iflength ==0: break returnencoded pkt+= encode_len(remaining_length) # 主題 pkt+= struct.pack("!H", len(topic)) + topic # qos>0時需要packet id ifqos >0: pkt+= struct.pack("!H",1) # packet id固定為1，可改 pkt+= msg self.sock.write(pkt)

def connect(self, clean_session=True): 連接到mqttx

defconnect(self, clean_session=True): addr = socket.getaddrinfo(self.server, self.port)[0][-1] self.sock = socket.socket() self.sock.connect(addr) pkt =bytearray(b"\x10") # CONNECT packet type var_header =bytearray(b"\x00\x04MQTT\x04") # Protocol Name + Level flags =0 ifclean_session: flags |=0x02 var_header.append(flags) var_header += struct.pack("!H", self.keepalive) payload = struct.pack("!H",len(self.client_id)) + self.client_id remaining_length =len(var_header) +len(payload) # MQTT剩余長度編碼（可能大于127字節，需要多字節編碼） defencode_len(length): encoded =bytearray() whileTrue: digit = length %128 length = length //128 iflength >0: digit |=0x80 encoded.append(digit) iflength ==0: break returnencoded pkt += encode_len(remaining_length) pkt += var_header pkt += payload self.sock.write(pkt) resp = self.sock.read(4) ifnot resporresp[0] !=0x20orresp[1] !=0x02: raiseMQTTException("Invalid CONNACK") ifresp[3] !=0: raiseMQTTException("Connection refused, code: %d"% resp[3])

WIFI

def connect_wifi(SSID, PASSWORD): WIFI連接

def connect_wifi(SSID, PASSWORD): wlan = network.WLAN(network.STA_IF) wlan.active(True) wlan.connect(SSID, PASSWORD) connect_times =0 whilenot wlan.isconnected(): print('Trying to connect to "{:s}"...'.format(SSID)) time.sleep_ms(1000) connect_times +=1 if connect_times >5: print(f"Connect to {SSID} failed.") return False print("WiFi Connected ", wlan.ifconfig()) return wlan.ifconfig()

手勢識別

初始化

def__init__(self): self.net =None self.lables =None self.WIFIConnectStatus = GestureRecoginze.connect_wifi("IQOO Neo 6","x31415926y") self.MqttxClient =None self.MqttxConnectStatus =False ifself.WIFIConnectStatus: self.MqttxClient = MQTTClient("openmv","broker.hivemq.com", port=1883) self.MqttxConnectStatus =True try: self.MqttxClient.connect() self.MqttxClient.subscribe("openmv/test") except: print("connect to MQTTx failed.") self.MqttxConnectStatus =False self.MqttxClient.set_callback(lambdatopic, msg:print(topic, msg)) self.servo = Servo360(0x40400000,8) """ 初始化攝像頭 """ sensor.reset() # Reset and initialize the sensor. sensor.set_pixformat(sensor.RGB565) # Set pixel format to RGB565 (or GRAYSCALE) sensor.set_framesize(sensor.QVGA) # Set frame size to QVGA (320x240) sensor.set_windowing((240,240)) # Set 240x240 window. sensor.skip_frames(time=2000) # Let the camera adjust. """ 加載模型 """ try: # load the model, alloc the model file on the heap if we have at least 64K free after loading self.net = tf.load("trained.tflite", load_to_fb=uos.stat('trained.tflite')[6] > (gc.mem_free() - (64*1024))) exceptExceptionase: print(e) raiseException('Failed to load "trained.tflite", did you copy the .tflite and labels.txt file onto the mass-storage device? ('+str(e) +')') try: self.labels = [line.rstrip('\n')forlineinopen("labels.txt")] exceptExceptionase: raiseException('Failed to load "labels.txt", did you copy the .tflite and labels.txt file onto the mass-storage device? ('+str(e) +')')

識別主體

defMainAction(self, comparetimes): clock = time.clock() compare_times =0 start_time =None CompareResultShow =None compare_result =None while(compare_times clock.tick() img = sensor.snapshot() results = self.net.classify(img, roi=(0,0, img.width(), img.height()), scale_mul=0, x_overlap=0, y_overlap=0) obj = results[0] scores = obj[4] predictions_list =list(zip(self.labels, scores)) predictions_max =0 predictions_num =None foriinrange(len(predictions_list)): label, score = predictions_list[i] ifscore > predictions_max: predictions_max = score predictions_num = label print("%s = %f"% (label, score)) img.draw_string(0,0,"Predictions: %s"% predictions_num, mono_space=False, scale=2) ifstart_timeisNone: start_time = time.ticks_ms() iftime.ticks_diff(time.ticks_ms(), start_time) >5000: ifpredictions_max >0.90: machines_gesture = random.randint(0,2) ifmachines_gesture ==0:# rock ifpredictions_num =="rock": compare_result ="draw" elifpredictions_num =="paper": compare_result ="win" else: compare_result ="lose" elifmachines_gesture ==1:# paper ifpredictions_num =="rock": compare_result ="lose" elifpredictions_num =="paper": compare_result ="draw" else: compare_result ="win" else:# scissors ifpredictions_num =="rock": compare_result ="win" elifpredictions_num =="paper": compare_result ="lose" else: compare_result ="draw" ifself.WIFIConnectStatus: self.MqttxClient.publish("openmv/test", ujson.dumps({"compare_times": compare_times, "machine_label":self.RPS[machines_gesture], "label": predictions_num, "score": predictions_max, "compare_result": compare_result})) ifcompare_result =="win": self.servo.run(1,1)#正轉一秒 else: self.servo.run(-1,1) print(compare_times) start_time = time.ticks_ms() CompareResultShow = time.ticks_ms() compare_times +=1 else: print("get_ready......") ifCompareResultShowisnotNoneandtime.ticks_diff(time.ticks_ms(), CompareResultShow)

9 使用方法

前置條件

在光線充足，設置一個白色的識別背景

確保wifi、mqttx可以連接

使用

將開發板置于手的正上方20-30cm處，經過5s將會識別一次，比對后會將結果顯示在 屏幕上2.5s，對比5次之后識別結束

10 常見問題

在某些情況下識別錯誤: 確保光線充足，與開發板的距離適當

無法在mqttx上獲取識別結果: 確保WIFI是否存在，密碼是否正確