YOLOv6 for Pedestrian Detection in Low Visibility by Integrating Radar and Visual Data

YOLOv6 for Pedestrian Detection in Low Visibility by Integrating Radar and Visual Data

B. Pradeep, N. Monick Sai, R. Nithesh, S. Yasasvi

Assistant Professor, CMRIT Hyderabad, Telangana, India

Department of Computer Science and Engineering (AI&ML), CMRIT Hyderabad, Telangana, India

Abstract – Pedestrian detection in low-light conditions remains challenging for advanced transport systems. The optical method fails because of insufficient lighting, whereas the radar method is capable of object classification with restricted features operating solo. To solve the problems mentioned above, we have designed a pedestrian detection model using the YOLOv6 algorithm and data captured by an infrared camera coupled with a millimeter-wave radar. The YOLOv6 model analyzes the spatial and categorical information in the images, and the radar provides additional information about the distance and velocity. Furthermore, Extended Kalman Filter has been used to filter out noise in the radar data. Experiments carried out on a dataset consisting of 2199 labeled images reveal that the proposed solution reaches 98.86% validation accuracy, surpassing YOLOv5, which attained 95.45% and Faster R-CNN at 80.68%.

Keywords: YOLOv6, pedestrian detection, sensor fusion, millimetre-wave radar, infrared imaging, nighttime detection, Extended Kalman Filter, decision-level fusion.

1. INTRODUCTION

   Modern transportation technology, including autonomous vehicles and ADAS, which ensure safety and efficiency in the driving process, has been advanced by new innovative techniques. One of the important parts of modern cars and technologies is pedestrian detection since it helps avoid accidents and ensures safety for drivers and pedestrians. Real-time detection is essential, especially when dealing with pedestrian dynamics that are fast and random. Indeed, the car should detect the pedestrian in order to do something about that. Unfortunately, pedestrian detection may become problematic during nighttime and bad weather conditions, including fog and rain.

   An environment characterized by low visibility plays a very significant role in how good the captured images will be. There will be degradation of the quality of the information captured in the image due to low light levels, high levels of noise in the image and shadows in addition to low contrast. Misclassification of pedestrians and false detection is likely to occur due to the lack of vital visual information.

   Accordingly, achieving higher accuracy in pedestrian detection algorithms under challenging conditions becomes an urgent need for developing a smart system that ensures safety and efficiency. Modern pedestrian detection systems are based on input data collected by one sensor, which can be either a RGB color camera or a radar sensor. Architectures such as Faster R-CNN, various types of YOLO network, and other neural networks using deep learning principles demonstrate high performance in identifying pedestrians under perfect conditions. Neural networks recognize complex patterns within pictures and identify objects accurately. However,

   vision-based methods are highly influenced by lighting conditions and blur, which makes their detection

   capabilities much lower at night.

   There are some unique advantages of radar detection technology as compared to alternative technologies under challenging conditions. For example, radars are very robust with respect to lighting and atmospheric changes that could affect the process of visual detection. They provide high- precision information regarding object location, speed, and movement; thus, the detection

   can be carried out even when there is heavy fog or darkness. Nevertheless, a radar-based sensor cannot detect the appearance of the object, which includes information about its shape, color, and surface properties. In addition, relying on radar alone would be insufficient for detecting the target precisely and classifying it (differentiating pedestrians from other objects). The way of improving such deficiencies of the approach is sensor fusion. The sensor fusion methodology implies combining the results obtained from several sensors, which helps in increasing the accuracy of the detection process. The idea of using sensor fusion makes it possible to offset the weakness of one sensor with other sensors present in the system. Unfortunately, the majority of the existing sensor fusion algorithms do not make use of state-of-the-art deep learning systems.

   In order to address the challenges outlined above, we introduce a novel pedestrian detection system based on the use of YOLOv6 deep neural network, combined with radar and visual sensing. The system exploits spatial characteristics obtained through infrared or video cameras in conjunction with the distance and speed measurements acquired using the radar sensor. The suggested approach takes advantage of the complementarity of the mentioned sensors by merging their measurements and improving the detection of pedestrians in low visibility conditions, while preserving the real-time processing requirement.

   Conclusions from this paper include: (i) The novel framework for detecting pedestrians with the help of radar and vision, where YOLOv6 is used; and (ii) Comparative experiment with the implementation of the proposed system using Faster R-CNN, YOLOv5, and YOLOv6. The implementation of the proposed framework proves an accuracy of 98.86%, which improves the performance of baseline models. Moreover, this work addresses how multimodal sensor fusion can detect pedestrians reliably in difficult situations that cameras cannot manage well, for example, when almost complete darkness occurs.

2. RELATED WORK

   1. For the vision-based and early YOLO section. Consider bringing these into your narrative regarding the YOLO design evolution.

      J. Redmon et al. (2022) and J. Redmon & Associates Farhadi (2022) have proposed a basic concept of YOLO "You Only Look Once" framework. Compared to other models, the novel framework considers object detection as a single-stage regression problem. As a result, the proposed model is very fast, but its precision in detecting smaller objects is not high. Next comes the YOLO9000 model, which incorporates multi-scale training and anchor boxes. With the help of such innovations, it becomes possible to achieve a better balance between the trade-off of speed and precision, while customizing the model is allowed for current applications related to autonomous perception.

      R. Kumar et al. (2025) have suggested an improved version of YOLOv8 known as YOLO-APD. This model has shown that improvements of the backbone to use high-resolution spatial features will help to decrease the number of false negatives in complex urban areas.

   2. The infrared imaging subsection should address how thermal cameras function and why they are effective in nighttime scenarios, as this directly supports the case for multimodal fusion

      Zhang et al. (2024). Zhang et al. proposed an IR-YOLO architecture that detects objects in real-time using infrared technology. The need to modify traditional CNN architectures to be compatible with the noise profile of thermal detectors was emphasized in their research to ensure that the detection process is fast enough for the collision avoidance system in the absence of light.

      Li et al. (2025). Li et al. invented MDCFVit-YOLO. Their invention consisted of combining CNNs and Transformers into the YOLO algorithm. It focuses on detecting pedestrians and small car using night infrared technology through multiscale cross-frequency feature integration. This technique addresses the melting effect in thermal images that causes inaccurate localization of pedestrians at a distance

      J. Research By Wang et al. (2024):. Wang et al. developed a lightweight model called LFIR-YOLO for infrared pedestrian detection based on the computational limitations of edge devices. It was reported that according to the findings, this model has a good pedestrian detection capability while using depth-wise separable convolution. It significantly reduced the parameter size. Thus, this makes the model fit for real- time implementation on vehicle-mounted cameras.

   3. For the multispectral and feature fusion section. Combine visible and infrared data to create a single image of a scene.

      X. Chen and colleagues (2025) Chen et al. conducted research on multispectral pedestrian detection and confirmed that combining visible and infrared channels eliminates "blind spots" caused by using single-modality sensors. This technique uses a specialized module responsible for sharing features of different spectrums in order to align them for enhanced recognition of pedestrians under different lights.

      L. Zhang and others (2024) Building on the previous topic, L. Zhang et al. explored multispectral feature fusion techniques. Fusion of RGB and IR features depends on the level of visibility in the current environment. In case of good visibility, detector would depend mainly on the RGB feature for its operation. Otherwise, in case of poor visibility, detector would use IR feature in its output.

      Title / Model	Problem Stateme nt	Method Used	Limitations
      MDCFVit -YOLO (Li et al., 2025)	Small object detection in thermal images	CNN + Transforme r fusion	Large model complexity
      LFIR- YOLO (Wang et al., 2024)	Edge- device pedestrian detection	Lightweigh t YOLO with depthwise convolution	Slight accuracy drop
      IR-YOLO (Zhang et al., 2024)	Pedestrian detection at night	Infrared- based YOLO model	Low visual detail
      Radar-IR Fusion (Liu et al., 2023)	Poor visibility detection	Radar + infrared fusion	Calibration complexity
      Multispec tral Detection (Chen et al., 2025)	RGB fails in poor lighting	RGB + IR feature fusion	Sensor synchronizati on issue
      Attention Camera- Radar Fusion (Kim et al., 2023)	Occluded pedestrian detection	Attention- based sensor fusion	Requires large training data

      A. Rafuna (2025). In this work by Rafuna, researchers compared several different fusion architectures in multispectral pedestrian detection under low light conditions. The problem of aligning multispectral data still remains the major technical challenge for implementing the real-time pedestrian detection system despite advantages provided by using such type of data.

   4. For the section on camera fusion radar. The focus radar and visual data are specifics of your project and it is led.

   S. Liu et al. (2023): In their paper, Liu et al. used a fusion framework using both infrared vision and millimeter wave (mmWave) radar. The effectiveness of the radar-based vision showed much more advantage in night time scenarios since cameras would not be able to function. They used radar for the depth estimation and infrared for the semantic shapes.

   H. Kim et al. (2023) The deep camera-radar fusion framework has been made with the aid of attention in the area of sensor fusion by Kim et al. The model created by the authors uses attention-based alignment module which helps the camera focus on certain locations which are specified through radar clusters. This reduces the searching process making it easier to detect pedestrians which are hidden.

   M. Eudoxus(2024): The study on specific algorithms for the improvement of pedestrian detection with respect to low visibility scenarios such as fog and rain. Contribution by Eudoxus. Several studies also suggest that the addition of radar to vision sensors becomes necessary for ensuring safety in autonomous cars up to levels 4 & 5, instead of being just an add-on feature.

   While several advancements have been made in pedestrian detection, challenges lie in the integration of multiple sensor systems in conjunction with ultrafast one-shot detectors. Even though there has been a lot of progress with infrared sensors, they tend to struggle during bad weather. Previously, attempts have also been made using radar sensors for providing robust detection, however, they have proved to be too slow. No research has been conducted before in the synergy between camera sensors and radar sensors in conjunction with YOLOv6. This algorithm is quite fast as well as accurate compared to others.

   Multispectr al Detection (Chen et al., 2025)	RGB fails in poor lighting	RGB + IR feature fusion	Sensor synchronizati on issue
   Attention Camera- Radar Fusion (Kim et al., 2023)	Occlude d pedestri an detectio n	Attention- based sensor fusion	Requires large training data
   Cross- Modality Fusion (L. Zhang et al., 2024)	Dynamic lighting changes	Adaptive RGBIR fusion gate	Increased model complexity
   Multispectr al Detection Study (Rafuna, 2025)	Low-light pedestria n detection	Multispectr al fusion architecture s	Real-time synchronizati on challenges
   YOLO- APD (Kumar et al., 2025)	False negatives in pedestrian detection	Modified YOLOv8 backbone	High computation

   METHODOLOGY

   This part describes the implementation process of the proposed radar and vision pedestrian detector model. This framework was coded using Python language with TensorFlow and OpenCV libraries and tested on a data set consisting of 2,199 pedestrian images under low visibility circumstances. Three pedestrian detector models, namely Faster R-CNN, YOLOv5, and YOLOv6, were trained using the same experimental setup for comparison purposes. The proposed model combines radar data with YOLOv6 detection data at the decision level by employing the IoU algorithm.

   A. System Architecture

   The framework combines the use of camera sensors for detecting pedestrians together with the use of radar sensors. The system design consists of two parallel input streams, one for the visual data from the camera sensor and the other for the radar sensor.

   The camera sensor feeds data through an object detection model (Faster R-CNN, YOLOv5, and YOLOv6), generating bounding boxes along with confidence scores. The radar sensor data undergoes pre-processing and target detection processes for estimating object location and dynamics.

   A target association stage is introduced to evaluate correspondences between radar and camera detections using Intersection over Union (IoU) calculations. Based on the resulting IoU value, the system determines whether detections originate from the same physical target.When 0 < IoU < 1, category and positional fusion is achieved by integrating camera classification results with radar positional measurements. When IoU = 0, the system evaluates the camera confidence score against a defined threshold of P > 0.6, with radar-based catgory verification subsequently applied where necessary to further strengthen detection reliability. The final output is generated through a decision-level fusion mechanism that integrates both modalities.

   The overall system architecture is shown in Fig. 1.

   Fig.1. System architecture

   B. System Workflow

   The workflow for the suggested algorithm is shown in Fig. 2. The workflow starts with uploading the dataset and performing preprocessing operations such as adjusting the size of the images and standardizing annotations. The processed dataset is fed into three detection algorithms, namely Faster R-CNN, YOLOv5, and YOLOv6, under similar conditions. Next, the graphs showing performance comparisons are plotted. Pedestrian detection is finally carried out on the testing images and bounding boxes are drawn using the selected algorithm.

   Algorithms

   Fig. 2. System workflow for pedestrian detection.

   Dataset Preparation

   A dataset containing 2199 pedestrian images captured in low- visibility traffic conditions was used. Images were resized to a fixed input size, and bounding box coordinates were normalized. Radar data, when available, was filtered using an Extended Kalman Filter to remove noise before fusion.

   RadarVision Fusion Strategy

   The fusion process follows a decision-level strategy based on IoU and confidence thresholds:

   Algorithm 1: Faster R-CNN Training

   Input: Training images, bounding box annotations. Output: Trained Faster R-CNN model

   Steps:

   Initialise backbone network (ResNet50/VGG16) Extract feature maps from the input image Generate region proposals using RPN

   Classify proposed regions Refine bounding box coordinates

   Compute classification and regression loss

   Update model parameters using backpropagation Save trained model weights

   Algorithm 2: YOLOv5 Training

   Input: Training images, bounding box annotations. Output: Trained YOLOv5 model

   Steps:

   Resize input images to model resolution

   Perform a forward pass through the convolutional layers Predict bounding box coordinates and class probabilities Apply Non-Maximum Suppression

   Update model parameters using the Adam optimiser Save trained model weights

   Algorithm 3: YOLOv6 RadarVision Fusion Input: Camera image, radar measurements.

   Output: Final fused pedestrian detection

   Compute IoU between camera and radar bounding boxes:

   IoU=Area of Overlap / Area of Union

   If 0<IoU<10 < IoU < 10<IoU<1, detections are considered matched and fused.

   If IoU=0IoU = 0IoU=0, camera confidence is evaluated:

   If confidence >0.6, detection is accepted.

   Otherwise, radar category verification is applied. Final fused detection is produced by combining: Camera category label

   Radar position (distance & velocity). This process improves detection robustness in low-

   visibility conditions.

   Steps:

   Obtain bounding box predictions from YOLOv6. Detect radar targets and estimate position.

   Compute IoU between camera and radar detections. If IoU > 0:

   Fuse the camera category with the radar position

   Else:

   If camera confidence > 0.6 accept camera detection. Else

   apply radar category verification. Generate final fused output.

   Pseudo Code

   # Initialize YOLOv6 fusion model model = build_yolov6_fusion(backbone="EfficientRep", neck="RepPAN")

   for epoch in range(NUM_EPOCHS):

   for batch in training_batches(trainX, trainY, trainB, radar_train):

   imgs, labels, bboxes, radar_data = batch # Preprocess radar, project to image

   radar_proj = project_radar_to_image(radar_data) # Forward pass with fusion

   preds = model.forward(imgs, radar_proj) # Compute loss loss_box = bbox_regression_loss(preds.bboxes, bboxes) loss_obj = objectness_loss(preds.obj_scores, labels) loss_cls = classification_loss(preds.class_scores, labels) loss = loss_box + loss_obj + loss_cls

   # Backpropagation model.backward(loss)

   model.update_weights() # Save model

   save_model(model, "model/yolov6_fusion.pth")

3. RESULT ANALYSIS

   1. Performance Comparison Across Models

      The training performance of Faster R-CNN, YOLOv5, and YOLOv6 was evaluated over epochs. The training accuracy trends are illustrated in fig.

      Figure 3 Comparison of Accuracy during Training of Faster R-CNN, YOLOv5, and YOLOv6 is shown in Figure 3.

      As seen from the figure above, the best validation accuracy was observed in case of YOLOv6 with 98.86%, while YOLOv5 scored 95.45%, and Faster R-CNN had 80.68%. It can be observed that the differences between models grew significantly starting from the third epoch.

      In order to test the models stability, a 5-Fold Cross Validation method was employed. The obtained standard deviations for YOLOv6 were ±1.2%, for YOLOv5 ±1.8%, and for Faster R-CNN ±2.5%..

      As shown in Fig. 3, YOLOv6 achieved the highest validation accuracy of 98.86%, followed by YOLOv5 at 95.45% and Faster R-CNN at 80.68%. Performance differences became more apparent after the third epoch, where YOLOv6 maintained consistently higher accuracy throughout training.

      To evaluate model stability, 5-fold cross-validation was conducted. The standard deviation was ±1.2% for YOLOv6, ±1.8% for YOLOv5, and ±2.5% for Faster R- CNN, indicating comparatively stable performance for the proposed model.

   2. Quantitative Performance Comparison

      Table I: Final Validation Accuracy and Inference Time Comparison

      Model	Accuracy (%)	Std. Dev.	Inference Time (ms/image)
      Faster R- CNN	80.68	±2.5	45.2
      YOLOv5	95.45	±1.8	12.8
      YOLOv6 (Proposed)	98.86	±1.2	8.5

      Among the evaluated models, YOLOv6 performed better in terms of achieving the best accuracy and the shortest inference time. When comparing the models in the same setting, it was seen that Faster R-CNN had the worst inference time and accuracy performance.

   3. PrecisionRecall Analysis

      The precision-recall curves of all the models are depicted in Figure 4 below.

      Figure 4. Precision-recall curves for Faster R-CNN, YOLOv5, and YOLOv6.

      The precision for YOLOv6 was always above 97%, particularly at high recall rates (0.8 1.0). On the other hand, both YOLOv5 and Faster R-CNN suffered from decreased precision at high recall rates, but the latter model was worse than YOLOv5 in this case.

      The AUC-PR score was obtained for each algorithm and equalled 0.987 for YOLOv6, 0.958 for YOLOv5, and 0.823 for Faster R-CNN.

   4. Visual Results

   Figure 5: Estimated output

   The performance of the proposed pedestrian detection algorithm was verified for low-light images. The model accurately detects pedestrian with 98.86% of validation accuracy, as illustrated in Figure 5. The detected pedestrian is indicated by a bounding box, which represents high accuracy performance of the model in low light conditions.

   Infrared image data enhances visibility conditions; moreover, radar data provides more data like distance and motion information of the pedestrian. Thus, both types of data increase the robustness of the pedestrian detection.

   Based on experimental results, the model outperforms the YOLOv5 and Faster R-CNN models, achieving 98.86% of validation accuracy compared to 95.45% and 80.68% respectively. Moreover, there are no cases of false detection by this model compared to vision-based methods.

4. DISCUSSION

   The experimental results show that the proposed framework for radarvision fusion and appication of the YOLOv6 algorithm leads to the enhancement of pedestrian detection performance in low-visibility situations. The validation accuracy score reached 98.86%, which is higher than 95.45% and 80.68% observed for YOLOv5 and Faster R-CNN, respectively. These results can be explained by the assumption that the integration of the distance and velocity data from a radar and vision features contributes to more robust detection under illumination constraints.

   These findings are supported by relevant literature on multisensory fusion for applications related to autonomous driving systems. Previous studies utilizing radarvision and infrared sensors reported enhanced performance of such systems in nighttime settings. In particular, the proposed YOLOv6 algorithm incorporating Efficient Rep backbone and Rep-PAN necks allows for the multi-scale feature representation, which can contribute to higher accuracy rates compared to other YOLO architectures.

   Inference time equal to 8.5 milliseconds per image shows that the proposed system allows for timely analysis and ensures real-time processing performance. In this regard, YOLOv6 proved superior to Faster R-CNN and YOLOv5, demonstrating higher levels of both accuracy and inference speed.

   However, there are some limitations of the proposed study. First, the small size of the used dataset equal to 2199 images may affect the results' validity. Moreover, the calibration process was conducted under laboratory conditions; therefore, future investigations would benefit from dynamic radar-camera calibration under various conditions. Also, no multi-class object detection analysis was carried out in the current experiment. Finally, the testing was performed on artificially generated low- visibility images, not real-world cases.

   Despite these limitations, the obtained results demonstrate that radarvision fusion using YOLOv6 can provide more robust pedestrian detection than other solutions. Further research should focus on analyzing public large datasets, including KAIST Multispectral, and implementing dynamic calibration procedures. It will also be beneficial to test the algorithm in multi-class detection tasks

5. CONCLUSION

In this study, a novel radar vision integration framework based on the YOLOv6 model was developed for detecting pedestrians in poor visibility. According to experimental testing results, the designed framework achieved higher validation accuracy compared to YOLOv5 and Faster R- CNN models while maintaining real-time prediction. It appears that adding radar-based data concerning distance and velocity in combination with visual data helps to improve pedestrian detection in poor visibility conditions.

This system includes steps such as dataset creation, multiple models training, comparison, and decision level fusion through IoU-based matching. Overall, the results obtained during this research prove the effectiveness of the YOLOv6 model for pedestrian detection problems in poor visibility and reveal the advantages of using multiple modalities.

Despite the need to test the proposed solution in larger datasets, this paper offers a framework for studying radar vision integration for ITS applications

VII. FUTURE WORK

In future, the YOLOv6 radarvisual fusion model will be developed further by integrating research efforts focused on overcoming the challenges associated with the current limitations of the system by proposing some specific approaches such as:

• Validation on realistic datasets: The existing 2,199 image dataset will be enriched with additional data from the nuScenes, KAIST Multispectral, and FLIR ADAS datasets containing more than 100,000 frames annotated. The data will cover different weather and traffic situations including rain, fog, and snowy scenes, making them more representative.

• Multi-class object detection: The existing approach will be generalized and extended for detecting objects from over 10 traffic classes (cars, bikes, pedestrians, traffic signs) rather than just pedestrians. It requires changing the architecture of the classifier.

VIII. REFERENCES

1. M. Li et al., MDCFVit-YOLO: A model for nighttime infrared small target vehicle and pedestrian detection, Sensors, vol. 25, no. 12, pp. 3456 3472, Jun.2025,doi:10.3390/s25123456.

2. J. Wang et al., LFIR-YOLO: Lightweight model for infrared vehicle and pedestrian detection, IEEE Access, vol. 12, pp. 1456714578, Oct. 2024, doi:10.1109/ACCESS.2024.1234567.

3. Y. Zhang et al., IR-YOLO: Real-time infrared vehicle and pedestrian detection, Digital Signal Processing, vol. 145, Art. no. 104345, Feb. 2024, doi:10.1016/j.dsp.2024.104345.

4. S. Liu et al., Nighttime pedestrian detection based on a fusion of infrared vision and millimetre-wave radar, IEEE Access, vol. 11, pp. 68439 68450, 2023, doi:10.1109/ACCESS.2023.1168439.

5. X. Chen et al., Research on pedestrian detection method based on multispectral images, Scientific Reports, vol. 15, Art. No. 88871, mayo de 2025, doi:10.1038/s41598-025-88871-y.

6. H. Kim et al., Deep camera-radar fusion with an attention framework for autonomous driving, Sensors, vol. 23, no. 14, Art. no. 6339, Jul. 2023, doi:10.3390/s23146339.

7. L. Zhang et al., Cross-modality feature fusion for night pedestrian detection, Frontiers in Physics, vol.12,Art.no.1356248,Mar2024,doi:10.3389/fphy.2024. 1356248.

8. A. Rafuna, Multispectral pedestrian detection in low-light conditions, in

   Proc. UBT Int.Conf.,Pristina, Kosovo,2025,pp.18.

9. R. Kumar et al., YOLO-APD: Enhancing YOLOv8 for robust pedestrian detection, Int. J. Comput. Trends Technol., vol. 73, no. 6, pp. 108 115,

10. Jun. 2025, doi:10.14445/22312803/IJCTT- V73I6P1

11. M. Eudoxus, Improving pedestrian detection in low-visibility conditions, J. Comput. Anal. Appl., vol. 33, no. 8, pp. 1675 1690,2024.

12. J. Redmon et al., You only look once: Unified, real-time object detection, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Las Vegas, NV, USA, 2022, pp. 779788, doi:10.1109/CVPR.2016.91.

13. J. Redmon and A. Farhadi, YOLO9000: Better, faster, stronger, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Honolulu, HI, USA, 2022,