Xilei Temple is located to the west of Puji Temple, formerly known as Xilei Nunnery. It was the place where Master Xinming of Puji Temple built for his master Chaoyin in 1693 to live after retirement. In 1699, the Imperial Minister Wang Hongxu and the maester Gao Shiqi donated money to build the Wanshou Shrine here, which was covered with yellow tiles on the top, and the plaque "Longevity canopy" adorned by the emperor was hung inside. In the forty-nine years of Kangxi, the crown prince Cheng wrote the plaque of the "Fragrant Vihara" for the temple, and the emperor then wrote the plaque of the "Moonlight can always shine" for the temple in 58 years. In 1919, the temple was managed by Master Yingzhao of Baoben Temple because of its disrepair. Master Yingzhao built Abbot's Quarters and guest rooms here, and also rebuilt the Wanshou Hall, and the architectural structure of the temple was completely new. 

At the beginning of the founding of New China, Wudao, Yuanheng and other five masters lived there. In 1998, the Zhoushan Municipal People's Government approved the return of Xilei Temple to the Buddhist Association of Mount Putuo for management and use. In December 2013, the Buddhist Association of Mount Putuo planned to restore the temple according to its original appearance. This temple is now the Precept Learning Hall of Puji Temple and is not open to the public.