HgIS

Správa a analýza dat o životním prostředí
Environmental data management and analysis

User Tools

Site Tools


en:documentation

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
en:documentation [2017-11-11]
Kamil Nešetřil [Further development]
en:documentation [2019-03-19] (current)
Kamil Nešetřil
Line 1: Line 1:
 ====== Documentation ====== ====== Documentation ======
-This is simplified technical documentation. User's documentation is available for costumers ​only.+This is simplified technical documentation. User's documentation is available for customers ​only.
  
 HgIS is mostly built upon the Pentaho platform. It uses ETL tool Pentaho Data Integration to load data to the database. The database PostgreSQL uses our data model. To formulate conceptual models, data can be explored in 3D hydrogeological visualization software EnviroInsite. <wrap hide>The approach is simplified due to the compatibility of our data model with that of EnviroInsite.</​wrap>​ We have designed multiple reports and developed multiple analyses and models (identifying redox processes, hydrochemical type of water or average hydraulic gradient). HgIS is mostly built upon the Pentaho platform. It uses ETL tool Pentaho Data Integration to load data to the database. The database PostgreSQL uses our data model. To formulate conceptual models, data can be explored in 3D hydrogeological visualization software EnviroInsite. <wrap hide>The approach is simplified due to the compatibility of our data model with that of EnviroInsite.</​wrap>​ We have designed multiple reports and developed multiple analyses and models (identifying redox processes, hydrochemical type of water or average hydraulic gradient).
  
 ===== ETL ===== ===== ETL =====
-The data are loaded to the database (data warehouse) by the ETL tool PDI. Data transformations in PDI can be implemented without coding through an intuitive graphical user interface and run also in command-line interface or on the ETL server. We implemented the loading of following data: +The data are loaded to the database (data warehouse) by the ETL tool PDI. Data transformations in <​nowiki>​PDI</​nowiki> ​can be implemented without coding through an intuitive graphical user interface and run also in the command-line interface or on the <​nowiki>​ETL</​nowiki> ​server. We implemented the loading of the following data: 
-  * {{:​xbase.png?​nolink&​18|}} ​analyses ​from LIMS “[[http://​www.labsys.cz|Labsystém]]” (xBase files),+  * {{:​xbase.png?​nolink&​18|}} ​assays ​from LIMS “[[http://​www.labsys.cz|Labsystém]]” (xBase files),
   * {{:​access.png?​nolink&​18|}} {{:​xml.png?​nolink&​18|}} geologic description and water quality from Czech Geological Survey (MS Access and XML files),   * {{:​access.png?​nolink&​18|}} {{:​xml.png?​nolink&​18|}} geologic description and water quality from Czech Geological Survey (MS Access and XML files),
   * {{:​access.png?​nolink&​18|}} [[https://​www.rockware.com/​product/​overview.php?​id=165|{{:​rockworks_logo.jpg?​nolink&​18|}}]] RockWorks   * {{:​access.png?​nolink&​18|}} [[https://​www.rockware.com/​product/​overview.php?​id=165|{{:​rockworks_logo.jpg?​nolink&​18|}}]] RockWorks
Line 13: Line 13:
   * {{:​excel.png?​nolink&​18|}} general cross-table and    * {{:​excel.png?​nolink&​18|}} general cross-table and 
   * formats from some other data vendors (groundwater pumping, river discharges etc.).   * formats from some other data vendors (groundwater pumping, river discharges etc.).
-Subsequent transformations provide data cleaning, anti-aliasing,​ validation and loading to the database. Coordinate conversion and loading of data to PostGIS is performed ​by GeoKettle – a spatially enabled fork of PDI or simply ​by [[https://​github.com/​atolcd/​pentaho-gis-plugins|PDI GIS Plugins]]. Ad-hoc loading is performed by common GIS software (e.g. [[http://​www.qgis.org|QGIS]]). PDI is also used to export data to third-party simulation tools.+Subsequent transformations provide data cleaning, anti-aliasing,​ validation and loading to the database. Coordinate conversion and loading of data to PostGIS is performed by [[https://​github.com/​atolcd/​pentaho-gis-plugins|PDI GIS Plugins]]. Ad-hoc loading is performed by common GIS software (e.g. [[https://​www.qgis.org|QGIS]]). PDI is also used to export data to third-party simulation tools.
  
 ===== Visualization ===== ===== Visualization =====
Line 28: Line 28:
  
 Some analyses and a model are utilizing PDI and formulas in Pentaho Reporting (OpenFormula),​ results are depicted in reports: Some analyses and a model are utilizing PDI and formulas in Pentaho Reporting (OpenFormula),​ results are depicted in reports:
-  * Identifying redox processes in ground water from chemical composition (dissolved O<​sub>​2</​sub>,​ NO<​sub>​3</​sub>,​ Mn<​sup>​2+</​sup>,​ Fe<​sup>​2+</​sup>,​ SO<​sub>​4</​sub><​sup>​2-</​sup>​ and sulfides) without measured Eh and pH((CHAPELLE,​ Francis H., Paul M. BRADLEY, Mary Ann THOMAS a Peter B. MCMAHON, 2009. Distinguishing iron-reducing from sulfate-reducing conditions. //Ground Water//. **47**(2), 300–305. ISSN\_1745-6584. DOI:[[http://dx.doi.org/​10.1111/​j.1745-6584.2008.00536.x|10.1111/​j.1745-6584.2008.00536.x]])).+  * Identifying redox processes in ground water from chemical composition (dissolved O<​sub>​2</​sub>,​ NO<​sub>​3</​sub><​sup>​−</​sup>, Mn<​sup>​2+</​sup>,​ Fe<​sup>​2+</​sup>,​ SO<​sub>​4</​sub><​sup>​2</​sup>​ and sulfides) without measured Eh and pH((CHAPELLE,​ Francis H., Paul M. BRADLEY, Mary Ann THOMAS a Peter B. MCMAHON, 2009. Distinguishing iron-reducing from sulfate-reducing conditions. //Ground Water//. **47**(2), 300–305. ISSN\_1745-6584. DOI:[[https://​doi.org/​10.1111/​j.1745-6584.2008.00536.x|10.1111/​j.1745-6584.2008.00536.x]])).
   * Multicriterial analysis assessing water quality trends in correspondence to eutrophication. Aggregated values of quantities (nitrogen/​phosphorus ratio, saturation of oxygen, pH etc.) were compared to estimated limits. The trend of the sum of the weighted logical values (overall score) indicates the trend in water quality.   * Multicriterial analysis assessing water quality trends in correspondence to eutrophication. Aggregated values of quantities (nitrogen/​phosphorus ratio, saturation of oxygen, pH etc.) were compared to estimated limits. The trend of the sum of the weighted logical values (overall score) indicates the trend in water quality.
  
 Some analyses were performed in external tools (data were exported with PDI): Some analyses were performed in external tools (data were exported with PDI):
-  * Average ​hydraulic gradient was calculated from the hydraulic heads of selected boreholes. Consecutively seepage velocity and retention time were computed. This calculation is performed in MS Excel spreadsheet ((DEVLIN, J.F., 2003. A spreadsheet method of estimating best-fit hydraulic gradients using head data from multiple wells. //Ground Water//. **41**(3), 316–320. ISSN\_1745-6584. DOI:[[http://dx.doi.org/​10.1111/​j.1745-6584.2003.tb02600.x|10.1111/​j.1745-6584.2003.tb02600.x]])) by matrix formulas. +  * The average ​hydraulic gradient was calculated from the hydraulic heads of selected boreholes. Consecutively seepage velocity and retention time were computed. This calculation is performed in MS Excel spreadsheet ((DEVLIN, J.F., 2003. A spreadsheet method of estimating best-fit hydraulic gradients using head data from multiple wells. //Ground Water//. **41**(3), 316–320. ISSN\_1745-6584. DOI:[[https://​doi.org/​10.1111/​j.1745-6584.2003.tb02600.x|10.1111/​j.1745-6584.2003.tb02600.x]])) by matrix formulas. 
-  * To support geological interpretation of thousands of exploratory boreholes from former mining area we tested an automated classification of the detailed text characterizations of strata. We used [[https://​rapidminer.com|RapidMiner]] software.+  * To support ​the geological interpretation of thousands of exploratory boreholes from former mining area we tested an automated classification of the detailed text characterizations of strata. We used [[https://​rapidminer.com|RapidMiner]] software.
  
 The above-mentioned analyses support conceptual model developments. The same techniques can be used to implement simple procedural site-specific groundwater models. ​ The above-mentioned analyses support conceptual model developments. The same techniques can be used to implement simple procedural site-specific groundwater models. ​
  
 ===== Use cases ===== ===== Use cases =====
-Components of HgIS are being used in a state enterprise (that carries out recultivations) for data management and for the water balance model of a lake. HgIS is deployed and used within research projects at the Technical University of Liberec.+Components of HgIS are being used in a state enterprise (that carries out recultivation) for data management and for the water balance model of a lake. HgIS is deployed and used within research projects at the Technical University of Liberec.
  
-===== Further development ===== 
-Currently we are working to 
-  * Integrate all components (Pentaho platform and online map application and unstructured data) to a single GUI. 
-  * Simplify design of reports and dashboards for business users by creating an abstract business layer (Pentaho Metadata) including localization. 
  
 ===== Further reading ===== ===== Further reading =====
-  * **[[en:resources|]]**+  * **[[en:simple|]]**
 Pages only in Czech: ​ Pages only in Czech: ​
   * **[[cs:​cheatsheet|PDI cheat sheet]]**   * **[[cs:​cheatsheet|PDI cheat sheet]]**
-  ​* [[cs:​resources|Links to interesting software]] +  * [[cs:​development|Development]].\\ Howeverthe whole Czech version of this site is more elaborate than the English version.
-  ​* [[cs:​development|Development]].\\ However the whole Czech version of this site is more elaborate than the English version.+
en/documentation.txt · Last modified: 2019-03-19