RT-DETR: When Detection Transformers Finally Got Fast Enough

If you've ever tried to deploy a transformer-based object detector in a real-time pipeline, you know the pain. DETRs have a reputation for accuracy, but they've historically been too slow to compete with YOLO in latency-sensitive applications. RT-DETR is the official implementation of the paper that set out to change that—and with the v2 and v4 releases, the project keeps pushing the boundary further.

What It Does

RT-DETR is a real-time end-to-end object detector built on the Detection Transformer architecture. The core claim, straight from the paper title, is that DETRs can beat YOLOs on real-time object detection—not just in accuracy, but in the speed-accuracy tradeoff that matters when you're running inference on a live video stream or an edge device.

The repository hosts the official PyTorch implementation of two papers: the original DETRs Beat YOLOs on Real-time Object Detection (CVPR 2024) and RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer. It includes pretrained models, training and tuning code, and deployment discussions covering ONNX Runtime, TensorRT, and OpenVINO. There's also a PaddlePaddle version referenced in the updates, along with PyTorch weights converted from it.

As of late 2025, the family has a new member: RT-DETRv4, which the README describes as leveraging Vision Foundation Models (VFMs) to boost lightweight detectors and improve the full-size model's performance without adding inference latency. That one lives in a separate repository linked from the main project.

Why It's Cool

• It's an actual answer to a real bottleneck. Transformer detectors have been the "interesting research direction" for a while, but the practical objection was always latency. This project takes that objection seriously and publishes numbers—like RT-DETRv2-S hitting 48.1 mAP, a 1.6 point improvement over RT-DETR-R18. That's not a rounding-error bump.

• The v2 improvements are "bag-of-freebies." The v2 paper's subtitle says a lot. These are improvements that don't cost you extra inference time. For anyone deploying models, that's the kind of progress that actually matters—you get better accuracy without renegotiating your latency budget.

• Practical tooling is baked in. There's a run_profile.py script for parameter and FLOPs statistics, which is the kind of thing you usually end up writing yourself. Sliced inference support was added for small object detection, and there's a hubconf.py for torch hub loading. These aren't headline features, but they're the difference between a research repo and something you can actually use.

• Small object detection got attention. Sliced inference (added in October 2024) addresses a problem that plagues most detectors in real deployments—small objects vanish at typical input resolutions. Seeing this addressed directly is refreshing.

• The architecture options are wider than you'd expect. RegNet and DLA34 backbones were added for RT-DETR in September 2024, alongside the existing ResNet and other variants. More backbone choices means more flexibility in matching a model to your compute constraints.

• Custom dataset training was made easier. The remap_mscoco_category logic was upgraded specifically to facilitate training on custom datasets, which is usually the first wall you hit after the demo works.

How to Try It

The repository is organized into separate directories for each version, so start by picking your target. For the v2 release, you'll want the rtdetrv2_pytorch directory.

1. Clone the repository:

   git clone https://github.com/lyuwenyu/RT-DETR
   cd RT-DETR
   

2. Navigate to the version you want (for example, rtdetrv2_pytorch) and follow the installation instructions in that directory's README. The project uses PyTorch, and you'll need the usual suspects—torch, torchvision, and likely pycocotools for evaluation.

3. Check the pretrained weights. PyTorch weights converted from the Paddle version were uploaded back in September 2023, and the v2 release includes improved checkpoints. There's also a hubconf.py if you prefer loading via torch hub.

4. For deployment, look at the discussion thread linked in the updates covering ONNX Runtime, TensorRT, and OpenVINO support. The README doesn't inline the export commands, but the discussion is referenced directly.

5. If you're training on your own data, read the Train custom data section in rtdetr_pytorch—that's where the remap_mscoco_category details live.

6. For the newest RT-DETRv4 work, head to the separate repository: https://github.com/RT-DETRs/RT-DETRv4

Final Thoughts

RT-DETR is best suited for developers who need real-time detection and are willing to move past the YOLO default. The project has been actively maintained since 2023, with a steady stream of updates—custom dataset support, backbone additions, deployment discussions, and now a v4 that brings in vision foundation models. If you're evaluating detectors for a production system, this is worth benchmarking against whatever YOLO variant you're currently using. The honest caveat is that the README points you elsewhere for several key details (deployment specifics, v4 architecture), so expect to do some clicking. But for a research-backed detector that's actually designed with latency in mind, the trajectory here is worth following.

Follow @githubprojects for more developer tools and open source projects.