Data Systems and Preprocessing

Data systems are computerized systems that store information about students, schools, and educators. They allow users to access the data, manage it and analyze it. These systems are known by many names like student information system (SIS), learning management system and decision support system and data warehouse.

Data system design aims to improve the way information is stored, collected and retrieved within an company. It involves determining the methods for storage and retrieval that are most efficient, designing data models and schemas as well as constructing robust security. Data system design involves determining which tools and technologies are best for storing, delivering and processing data.

Big sensor data systems rely on a collection of different data sources, sourced from various sensors that are physical and not, such as mobile and wireless devices, wearables, telecommunication networks, and public databases. Each of these sources generates the same set of sensor readings, each with their specific metric value. The primary challenge is to identify a suitable time resolution for the data as well as the process of aggregation that allows the sensor data to be presented as a single representation using an identical metric.

To ensure that data analysis is efficient, it is essential to ensure that data can be understood correctly. This requires preprocessing the process of preparing all activities that prepare data for analysis and transformations, such as formatting, combing, and replication. Preprocessing can be batch-based, or stream-based.

read