IOT ANALYTICS
Fig: IoT Analytics process flow on Data Lakehouse
The Internet of Things (IOT) is a network of physical devices. These devices are characterized by a Unique Identifier (UDI) and a sensor. They are essentially self-reporting devices that send signals (data) to each other or a centralized computer server. Home security devices, wearable computing like smart watches, personal medical devices like pace maker, industrial sensors are some examples of IOT devices. IOT Analytics is about analysing the data collected from the IOT devices.
The data transmitted by an IOT device may be a continuous or intermittent stream of measures or voice or images. The data may be real time or may be sent intermittently, depending on the connectivity and use case.
Benefits of IOT Analytics
- Industrial use cases like preventive maintenance and automated stock refills lead to reduction in operational costs and better resource utilization
- Healthcare sector can utilize state of art IOT technology to enable remote monitoring of patients on a continuous basis
IOT Analytics process flow
- Gather data from different IOT devices.
- Store data in a central store and time sequence the input messages (signals). Depending on the use case, input is forwarded for subsequent processing or held until all required data arrives. Some use cases require forwarding input data after a pre-defined time interval.
- The data is then cleaned, homogenized (e.g. units could be in oz or lb) and prepared for further processing.
- The data is consumed by pre-trained ML models or BI analytics pipelines to generate insights.
- The results could be used to keep a dashboard up to date so that humans are provided with actionable insights. In some use cases, there can be an automatic action like reloading stocks.
AI in IOT Analytics
Artificial Intelligence can be of immense support in processing IOT data. Here we list some common techniques used:
- Sensor data may be in form of images or some other media. This data can be converted into numerical values. For example, Google Whisper can convert speech to text. Large Learning Models, neural networks and other NLP techniques can convert text to numerical vectors which are further fed to ML pipelines to generate insights.
- Lot of sensor data is often considered as noise because it does not contain any processible input. For example, a home security camera has to keep collecting data 24x7 but only the few seconds when some movement happened in front of the camera is of significance. Rest of the data is noise from the use case perspective. Such anomalies that really matter can be detected using anomaly detection methods (e.g. deviation beyond 3 standard deviations in the signal is an anomaly).
- Identification of patterns or co-occuring events is possible using apriori algorithms. For example, temperature beyond 60°C and coolant sensor tripping events are happening together. Identification of such patterns can help preventing the tripping by making sure temperature does not go beyond 60°C.
- Computer Vision algorithms can be used to detect a lesion in medical imagery.
- Time series analysis is a common part of IOT data analysis.
- Clustering algorithms like K-means can be used to identify similarly behaving devices.
- Software libraries other than ready to use tools include, but not limited to, the following:
- pandas, scikit-learn, matplotlib, seaborn, apyori, spacy, TensorFlow, PyTorch, pyspark, keras, pykafka
Challenges
- Lots of data to be collected from different devices, often using different versions.
- Connectivity may not be continuous.
- Sequence of processing may matter depending on the use case. So, data must be collected, sequenced and perhaps kept on hold until all the desired data has arrived before sending it to further processing.
- Designing the data pipelines can be challenging due to varied rates of incoming data and consumption of data. There has to be enough storage capacity so that data is not run over by fresh batch before it is consumed. Edge computing is one solution where some data processing like noise reduction and aggregation happens at the data source itself. This reduces bandwidth requirements and processing queue requirements at the central server. However, edge computing is still evolving.
IOT Analytics at Y Point Analytics
As evident from the discussion so far, IOT analytics is at the intersection of several disciplines. It requires expertise in Big Data processing, Streaming data processing, Data integration, Data quality, Security, Data Wrangling, Data Analysis, Cloud Data Warehouse, dashboards, Machine Learning, Natural Language Processing, predictive modeling, prescriptive modeling and pre-emptive modeling.
Y Point analytics has expertise in all these inter-disciplinary skills and also has proven experience in rolling out high quality solutions as per plan backed by its time tested project management and delivery practices.