Abstract: The problems of real-time tracking of table tennis in the visual system of table tennis robot are focused on. Two completely different algorithms are designed to track the table tennis under two different photography conditions of high-speed and low-speed. Under high-speed photography, the five kinds of major characteristic information of table tennis are mainly used to set the threshold for identification which include roundness, circumference, area, X-distance and Y-distance. In the low-speed photography, the machine learning and matching methods are used to achieve a smearing table tennis recognition. A kind of attention-based image segmentation algorithm is used to pre-process the image, which can effectively solve the problem of pixel distortion due to environmental interference and other factors.The algorithm of planning region of interest(ROI)area is proposed. The possible position of table tennis in the next frame image is determined in advance by the movement characteristics of table tennis, which can be used to reduce the calculation amount, shorten the calculation time, and achieve the real-time tracking.