How do water managers handle Big Data?

The managers of water resources constantly strive to achieve ‘water security’, which means sufficient quantity and quality of water to sustain livelihoods, health, socio-economic development and ecosystems. Timely access to reliable, insightful, defensible data is therefore necessary, but with the volume and speed of data now available, will existing data infrastructure be able to cope? 

Recent developments in sensors, dataloggers and telemetry, have delivered intelligent monitoring systems that are able to operate remotely, running on very low levels of power and delivering reliable data in real-time. This has prompted dramatic growth in the implementation of high-intensity monitoring systems generating huge volumes of data. These systems also possess advanced alarm capability; issuing alert messages by SMS, email etc. when user-set conditions arise.  

It is important to ensure compatibility between sensors and a datalogger. In practice, most modern dataloggers are able to accommodate all of the more popular communication protocols. However, it will be necessary to check that the datalogger can provide power to those sensors that need it, and that the datalogger is able to translate the signal from the sensors (which may not be linear) into engineering units. 

Dataloggers generally feature a fixed number of inputs for analogue, digital and counter sensors, but protocols such as SDI12 and Modbus facilitate a master and slave configuration that can expand input capacity considerably. SDI-12 sensors are frequently employed in environmental applications because they are low-power and feature integrated microprocessors which can perform complex internal correction, compensation and averaging algorithms that enhance the quality of the data. 

There are three main telemetry options: 

1. Environmental Satellites - high Earth orbiting geosynchronous satellites (such as GOES, METEOSAT, HIMAWARI, FY, and INSAT) so receiving antennas only need to point at one location. Publically owned, these satellites are ideal (often free of charge) for government agencies that require highly reliable, low-cost, one-way scheduled data and alarm transmissions. 
2. Iridium Telemetry - uses a network of 66 active low-earth orbiting satellites, providing two-way communication and almost immediate alerts. The cost of data transmission is priced according to the volume of data.  
3. Cellular Telemetry - with the growth of digital cellular technologies, such as GPRS, CDMA, 4G / LTE, 5G and IoT connectivity, the use of remote data collection and high-speed data transmission services using IP technology is growing in popularity. Cellular networks are typically distributed over land areas, so transmission may not work for very remote locations, and signal quality is prone to variability.  

In some circumstances it may be necessary to combine two or more telemetry options (hybrid) for redundancy or to offset the disadvantages of one method with the advantages of another.  For example, some users employ an environmental satellite for regular transmissions, but automatically switch to cellular for larger, more frequent transmissions during an alarm state. 

Water management policies must be sustainable and resilient, so it is vital that both strategic and operational decisions are based on sound data. However, if data are not managed correctly, users may become ‘data-rich but information-poor’, so some of the most important decisions in the organisation of a monitoring programme relate to the ways in which data are managed. 

Working at every stage of the data collection process, OTT HydroMet has developed sensors that can operate remotely for extended periods; loggers that run on very low power but provide enormous data processing capability; telemetry solutions to transmit data from and to anywhere, and software that can assimilate vast quantities of data. The latest software (as a service) packages, such as Hydromet Cloud and AQUARIUS, are able to provide insights and alerts, while also providing easy access to live or historical data. 

Environmental monitoring data is of no value until it is in the possession of those that need it. Furthermore, data is at its most valuable when first created, so it is usually important for data to be transferred as soon as possible – ideally as ‘live’ data. Data management programmes such as AQUARIUS can be deployed with many off-the-shelf telemetry systems as well as AQUARIUS EnviroSCADA. Alternatively, AQUARIUS can be integrated with a data acquisition system through the use of a comprehensive web services API. 

With more monitoring points producing more data, the challenge for data management systems (software) is to be able to handle the volume of data whilst also providing insightful, actionable information for users and other stakeholders. Organisations may have sufficient internal IT resources to achieve this, but commercial cloud-based solutions are now readily available, offering the backend infrastructure to receive, process, display, and store measurement data from almost any number of monitoring stations via a secure data hosting platform. 

In combination, the OTT HydroMet solutions, from sensor to screen, enable water managers to meet the needs of disparate stakeholders and enhance water security. So, when it comes to the management of water, Big Data represents a fantastic opportunity.