Retrospective information about the state of the urbanized environment can be obtained by analyzing images located in the Internet. Methods based on artificial intelligence allow recognizing objects of a given class in images, moreover models based on neural networks demonstrate high recognition accuracy. In this paper, a neural network model is proposed based on its own dataset for recognizing objects of an urbanized environment on the example of a classes of construction works, asphalt damage and gravel to get retrospective data by analyzing panoramas with different time capture. The pixellib library was used for image segmentation. Photos found on the Internet were used to create the dataset. During the testing of the model, 9 versions of the neural network were trained. The best results were demonstrated by the model mask_rcnn_model.124-1.11.h5 (version 9) with F-score 97% for asphalt damage class, 94 % for tower crane class and 75% for gravel class. The main reason for increasing the model performance of this model was more accurate outline of objects and the addition of self-captured images to the dataset.