In today’s surveillance systems, there is a need for enhancing the situation awareness of an operator. Supporting the situation assessment process can be done by extending the system with a module for automatic interpretation of the observed environment. In this article the information flow in an intelligent surveillance system is described and the separation of the real world and the world model, which is used for the representation of the real world in the system, is clarified. The focus of this article is on modeling situations of interest in surveillance applications and inferring them from sensor observations. For the representation in the system, concepts of objects, scenes, relations, and situations are introduced. Situations are modeled as nodes in a dynamic Bayesian network, in which the evidences are based on the content of the world model. Several methods for inferring situations of interest are suggested, which make use of the underlying network modeling. Due to this modeling, we get a probability of all the situations in the network in every time step. By collecting more evidences over time, the probability of a specific situation is either increasing or decreasing. Finally, we give an example of a situation of interest in the maritime domain and show how the probability of the situation of interest evolves over time.