Kamil Nešetřil

Environmental data management and analytics

User Tools

Site Tools

Differences

This shows you the differences between two versions of the page.


Previous revision
—en:tools [2024-03-21] (current) – Kamil Nešetřil
Line 1: Line 1:
 +====== Tools =======
 +|[[https://docs.hitachivantara.com/p/pentaho-dia|{{:pentaho-logo.svg?nolink&150|Pentaho Platform – BI}}]]   [[cheatsheet|{{:spoon.png?nolink|Pentaho Data Integration (Kettle)}}]]   [[https://www.postgresql.org|{{:logo_postgres.svg?nolink&36|PostgreSQL – database}}]] [[https://dbeaver.io|{{:logo_dbeaver.svg?nolink&40|DBeaver – database manager}}]] + ([[https://www.r-project.org|{{:r_logo.svg?nolink&45|R – programming language}}]]  [[https://www.qgis.org|{{:qgis-icon32.svg?nolink&36|QGIS – geographical information system}}]] [[https://nextcloud.com/|{{:nextcloud.svg?nolink&60|Nextcloud – alternative to Dropbox, Google Drive or OneDrive}}]])|
  
 +HgIS applies BI tools ([[https://docs.hitachivantara.com/p/pentaho-dia|Pentaho platform]]) and GIS for the groundwater domain. Technologies and tools used in HgIS are listed here.
 +
 +> [[https://hop.apache.org/|{{ :hop_logo.svg?nolink&40|Apache Hop}}]] :!: Since Hitachi acquired Pentaho, the development of the Pentaho platform has stagnated, and the platform's website is confusing. The original author of PDI (Matt Casters) created [[https://hop.apache.org/|Apache Hop]], a fork of PDI that finally has elegant architecture and is being intensively developed. PDI transformations can be imported into it.
 +
 +[[https://docs.hitachivantara.com/p/pentaho-dia|{{  :pentaho-logo.svg?nolink&150|Pentaho platform}}]]
 +===== Pentaho platform =====
 +[[pentaho>https://docs.hitachivantara.com/p/pentaho-dia|Pentaho platform]] is a set of business intelligence tools.\\
 +
 +{{:pba-architecture.jpg?nolink |Pentaho Platform components and typical deployment}}
 +
 +It contains the following components ([[download>https://www.hitachivantara.com/pentaho/pentaho-plus-platform/data-integration-analytics/pentaho-community-edition.html|download]]):
 +
 +  * [[cheatsheet|{{ :spoon.png?nolink|PDI cheat sheet}}]][[https://www.hitachivantara.com/en-us/products/dataops-software/data-integration-analytics/pentaho-community-edition.html|Pentaho Data Integration]] (<nowiki>PDI</nowiki>, also known as Kettle)((Acronym means: //Kettle Extraction, Transformation, Transportation and Loading Environment//.)) It is an [[wp>Extract,_transform,_load|ETL]] tool. It can read data from diverse databases, resources (e-mail, local computer, FTP, HTTP), and files (MS Excel, MS Access, [[wp>Comma-separated values|CSV]], TXT, [[wp>Shapefile|ESRI SHP]], [[wp>XML]], [[wp>JSON]], [[wp>YAML]], [[wp>RSS]], [[wp>XBase]], or ZIP).  Official resource lists all steps of [[https://help.hitachivantara.com/Documentation/Pentaho/9.4/Products/Transformation_step_reference|transformations]] and [[https://help.hitachivantara.com/Documentation/Pentaho/9.4/Products/Job_entry_reference|jobs]]. Geospatial data can be handled with [[git>atolcd/pentaho-gis-plugins|PDI GIS Plugins]]. See more info in [[cheatsheet|]].
 +
 +  * Pentaho Report Designer ({{:profil_prd.png?linkonly|screenshot}}). \\ [[https://www.metabase.com/|{{ :metabase.png?nolink&50|}}]] Development of Pentaho Reporting does not move forward, so it makes sense to use an alternative as [[https://www.metabase.com/|Metabase]] – easy-to-use data analytics. FOSS alternatives in Python are [[git>apache/superset/blob/master/README.md|Superset]] or even [[https://redash.io/|Redash]]. \\ \\
 +  * Pentaho Server (<nowiki>PBA</nowiki>, formerly BI Server, {{:prubeh_server.png?linkonly| screenshot}}) is a key component of the platform. Dashboards can be executed here. Dashboards can be designed by CDE – more: [[https://help.hitachivantara.com/Documentation/Pentaho/9.4/Products/CTools|Community Tools (CTools)]].\\ \\ 
 +  * [[https://help.hitachivantara.com/Documentation/Pentaho/9.4/Products/Pentaho_Metadata_Editor|Pentaho Metadata Editor]] – creates an abstract data layer. See a figure: \\ {{ :01_metadata_domain.png?nolink |}} \\ \\ 
 +
 +[[https://www.hitachivantara.com/en-us/pdfd/datasheet/pentaho-machine-learning-orchestration-datasheet.pdf|Machine Intelligence]] utilizes engines: R, Python, Weka, or [[https://deeplearning4j.konduit.ai/|Deeplearning4j]] (Java with GPU Nvidia).
 +
 +Adaptive Execution Layer ([[https://help.hitachivantara.com/Documentation/Pentaho/9.3/Products/Adaptive_Execution_Layer|AEL]]) runs transformations not only locally or on the Pentaho Server, but also on  Spark. Pentaho also supports cloud platforms like Amazon Web Services, Google Cloud Platform, or Microsoft Azure (Cloud agnostic analytic pipeline). Alternative to AEL is [[foss>https://diethardsteiner.github.io/pdi/streaming/2018/12/01/Kettle-Beam.html|Apache Beam]].
 +
 +
 +=== Pentaho platform is built on===
 +  * [[wp>Java (programming language)|Java]] – programming language
 +  * [[wp>Apache Tomcat|Apache Tomcat]] – web server
 +  * [[wp>Java Database Connectivity|JDBC]] – API for access to relational databases
 +  * [[wp>Java Naming and Directory Interface|JNDI]]
 +  * [[https://jackrabbit.apache.org/jcr/index.html|Jackrabbit]]: implementation of the [[wp>Content repository API for Java]]
 +  * [[wp>Common Warehouse Metamodel]] – used in  Pentaho Metadata – [[https://www.omg.org/spec/CWM|specification]]
 +
 +===== PostgreSQL / PostGIS =====
 +  * [[https://www.postgresql.org|PostgreSQL]] is a database server we use. ([[wp>PostgreSQL]]).
 +  * We use [[https://dbeaver.io|DBeaver]]<wrap hide> and [[https://www.pgadmin.org|pgAdmin]]</wrap>  for database administration. It can also [[https://dbeaver.com/docs/dbeaver/Working-with-Spatial-GIS-data/|display]] geospatial data.
 +  * [[http://postgis.net|PostGIS]] is a spatial extension to PostgreSQL database ([[wp>PostGIS]]).
 +
 +===== Other GIS tools =====
 +  * [[qgis>https://www.qgis.org|QGIS]] – desktop GIS
 +  * [[qgis>https://docs.qgis.org/2.14/en/docs/user_manual/working_with_ogc/ogc_server_support.html|QGIS Server]] – map server
 +  * [[https://earthsoft.com/enviroinsite/|EnviroInsite]] – advanced hydrogeological visualization. The company developing  EI was acquired by the corporation developing EQuIS (EarthSoft). It does not [[https://earthsoft.com/2017/10/10/earthsoft-announces-enviroinsite-acquisition-and-contract-with-dr-bruce-jacobs/|plan]] to develop the standalone EI.(("EarthSoft will release new versions of the desktop EnviroInsite in EQuIS Professional, but does not currently plan to release new versions of non-EQuIS versions of EnviroInsite.")) Therefore, we plan to replace EI probably by QGIS with the [[git>jkall/qgis-midvatten-plugin|Midvatten]] or [[http://www.freewat.eu/software-0|FREEWAT]]((The AkvaGIS tools enhance FREEWAT with hydrochemical and hydrogeological data processing and analysis.)) plugin.
 +
 +For detailed information, go to the [[cs:start|Czech version]] of this site.