A Framework for Content Based Semantic Information Extraction from Multimedia Contents

One of the main characteristics of the new digital era is the media  “big bang”, where images (still images or moving pictures) are one of the main type of data. Moreover, this is an increasing trend mainly pushed by the easy of capturing given by all new mobile devices that include one or more cameras. From a professional perspective, most content related sectors are facing two main problems in order to operate efficient content management systems: a) need of new technologies to store, process and retrieve huge and continuously increasing datasets and b) lack of effective methods for automatic analysis and characterization of unannotated media. More specifically, the audiovisual and broadcasting sector which is experiencing a radical transformation towards a fully Internet convergent ecosystem, requires content based search and retrieval systems to browse in huge distributed datasets and include content from different and heterogeneous sources. On the other hand, earth observation technologies are improving the quantity and quality of the sensors installed in new satellites. This fact implies a much higher input data flow that must be stored and processed. In general terms, the aforementioned sectors and many other media related activities are good examples of the Big Data phenomenon where one of the main problem relies on the semantic gap; the inability to transform mathematical descriptors obtained by image processing algorithms into concepts that humans can naturally understand. This dissertation work presents an applied research activity overview along different R&D projects related with computer vision and multimedia content management. One of the main outcomes of this research activities is theMandragora framework where the main goal is to reduce the semantic gap and create automatic annotation based on a previously defined ontology. As on of the main problems of theMandragora framework is the initial characterization process where no prior knowledge is available, a novel domain characterization method (DITEC) has been designed in order to meet this requirements. The good results obtained in the experimental tests have lead the research to analyze if the feature extraction method based on the trace transformcan also be adapted to its use as a local descriptor. The local DITEC method is also presented in this dissertation and even if its implementation is still in a preliminary status, experimental results show that its performance is really competitive when compared with the most popular local descriptors available on the literature.