Shaocong (Daniel) Xu

LiDAR-based semantic scene understanding is an important module in the modern autonomous driving perception stack. However, identifying Out-Of-Distribution points in a LiDAR point cloud is challenging as point clouds lack semantically rich features when compared with RGB images. We revisit this problem from the perspective of selective classification, which introduces a selective function into the standard closed-set classification setup. Our solution is built upon the basic idea of abstaining from choosing any known categories but learns a point-wise abstaining penalty with a marginbased loss. Synthesizing outliers to approximate unlimited OOD samples is also critical to this idea, so we propose a strong synthesis pipeline that generates outliers originated from various factors: unrealistic object categories, sampling patterns and sizes. We demonstrate that learning different abstaining penalties, apart from point-wise penalty, for different types of (synthesized) outliers can further improve the performance. We benchmark our method on SemanticKITTI and nuScenes and achieve state-of-the-art results. Risk-coverage analysis further reveals intrinsic properties of different methods. Codes and models will be publicly available.

In this study, we tackle the challenging fine-grained edge detection task, which refers to predicting specific edges caused by reflectance, illumination, normal, and depth changes, respectively. Prior methods exploit multi-scale convolutional networks, which are limited in three aspects: (1) Convolutions are local operators while identifying the cause of edge formation requires looking at far away pixels. (2) Priors specific to edge cause are fixed in prediction heads. (3) Using separate networks for generic and fine-grained edge detection, and the constraint between them may be violated. To address these three issues, we propose a two-stage transformer-based network sequentially predicting generic edges and fine-grained edges, which has a global receptive field thanks to the attention mechanism. The prior knowledge of edge causes is formulated as four learnable cause tokens in a cause-aware decoder design. Furthermore, to encourage the consistency between generic edges and fine-grained edges, an edge aggregation and alignment loss is exploited. We evaluate our method on the public benchmark BSDS-RIND and several newly derived benchmarks, and achieve new state-of-the-art results.

Motion forecasting is an important component in autonomous driving systems. One of the most challenging problems in motion forecasting is interactive trajectory prediction, whose goal is to jointly forecasts the future trajectories of interacting agents. To this end, we present a large-scale interactive trajectory prediction dataset named INT2 for INTeractive trajectory prediction at INTersections. INT2 includes 612,000 scenes, each lasting 1 minute, containing up to 10,200 hours of data. The agent trajectories are auto-labeled by a high-performance offline temporal detection and fusion algorithm, whose quality is further inspected by human judges. Vectorized semantic maps and traffic light information are also provided. Additionally, the dataset poses an interesting domain mismatch challenge. For each intersection, we treat rush-hour and non-rush-hour segments as different domains. We benchmark the best open-sourced interactive trajectory prediction method on INT2 and Waymo Open Motion, under in-domain and cross-domain settings. The dataset, code and models will be made public.

CONTEXT: Bees play a crucial role in terrestrial ecosystems. However, Beekeepers are unable to monitor the state of beehives (bees and environment) all the time, which often results in Colony Collapse Disorder (CCD). Currently, some researchers provided the scheme of intelligent beehive monitoring system equipped with the Internet of Things (IoT). There remain two challenges: Accurately monitor the environmental status around the hive and accurately track and monitor bees in real time.
OBJECTIVE: With the development of the IoT and computer vision algorithms, we hope to provide an automated and efficient system to meet the above challenges. In this paper, we propose a hive monitoring system and build a visualization module in the cloud to monitor the activity of bee colonies and the environmental dynamic changes.
METHODS: We proposed a multi-bee tracking algorithm to solve the problem of monitoring bees at the door of the hive. We construct a dataset containing various complex scenes, named BEE22, for training and testing the performance of our algorithm. We design a bee counting rule, based on results of multi-bee tracking algorithm, to reasonably count the bees entering or leaving the beehive. We have deployed multiple sensors around (center, margin, door, and environment) the hive to accurately reflect the changes in the environment around the hive.
RESULTS AND CONCLUSIONS: Experimental results demonstrate the effectiveness and superiority of our system. In particular, the tracking performance of the multi-bee tracking algorithm reaches 83.5% ± 0.7% accuracy (MOTA) and 77.3% ± 0.2% MOTP, speeds up to 16 frames per second, compared with other algorithms, MOTA and IDF1 are improved by 24.4% and 50.4% respectively. Moreover, our counting algorithm also achieved excellent results, with root mean square error (RMSE) of 0.17 ± 0.01, 1.55 ± 0.05, and 1.25 ± 0.06 in counting the number of bees entering, leaving, and the current scene in an episode, respectively. After that, the system will be deployed and monitored for a long time in the actual scenario, it was found that the activity of bees decreased significantly under heavy rainfall conditions. Additionally, the activity of the bee colony will also increase accordingly, when the amplitude is 500dB to 2000dB, the temperature of the center of the beehive is 25°C to 37°C, and the humidity is 48% to 67%.
SIGNIFICANCE: Our system can provide valuable information for bee farmers to make control decisions on hives.

Artificial general intelligence (AGI) used to be a sci-fi word but recently the surprising general-ization capability of foundation models have triggered a lot of attention to AGI, in both academiaand industry. Large language models can now answer questions or chat with human beings,using fluent sentences and clear reasoning. Diffusion models can now draw pictures of unprece-dented photo-realism, according to human commands and controls. Researchers have also madesubstantial efforts to explore new possibilities for robotics applications with the help of founda-tion models. Since this interdisciplinary field is still under fast development, there is no clearmethodological conclusions for now. In this tutorial, I will briefly go through best known prac-tices that have shown transformative capabilities in several sub-fields. Specifically, there are fiverepresentative paradigms: (1) Using foundation models to allow human-friendly human-car in-teraction; (2) Using foundation models to equip robots the capabilities of understanding vaguehuman needs; (3) Using foundation models to break down complex tasks into achievable sub-tasks; (4) Using foundation models to composite skill primitives so that reinforcement learningcan work with sparse rewards; (5) Using foundation models to bridge languge commands andlow-level control dynamics. I hope these best known practices to inspire NLP researchers.