4th EMODnet Technical Working Group Meeting

Date: Monday 01/10/2018 - 13h00 to Tuesday 02/10/2018 - 16h00
Meeting Venue: DG MARE, rue Joseph II 99, Brussels, Belgium

Attendees

Summary of action points TWG-4

Meeting Minutes

Expected outcomes of this meeting (Pascal Derycke)

Two main goals:

• To improve coordination and make collaborative work easier, DG-Mare has set up a JIRA (bug tracking, issue tracking, and project management service) for EMODnet. We all share similar systems and face similar challenges and bottlenecks. With JIRA everybody can view and contribute to the different issues & tasks even if it is not directly related with their lot; everybody can be up to date with developments.

• To work towards harmonization of the portals and improve m2m communication, keeping the OSLII deadline in mind with the objective to have a hackathon package ready.

Status EMODnet web services (Pascal Derycke)

Web services documentation

Full documentation needs to be in place for the OSLII (September 2019).

Pascal D. has prepared a template document with the core information about Web services that each lot should adopt and customize according its own services (server URL, requests and examples, etc.). The document will be distributed and it should be amended and restated by each of the thematic lots before to be published.

1. AP. Pascal D. to upload the Web services documentation template in JIRA.
2. AP. All lots to review, amend and restate the Web services documentation template according to their specific lot.

Where the documentation will be published, is not decided. Pascal D. thinks that, from a user-point of view, the information should be included on every portal since not all users realise that there are several EMODnet portals. Pascal D. thinks each thematic portal should publish this information – by keeping the same structure, but adapting the content. It is a mean to link the portals between them (Web ring approach).

Web services harmonization

All the data portals and products are different but the way to disseminate has to be standardized to a certain degree, taking into account the particularities of each lot and the technical implementations (e.g. geoserver vs. oceanbrowser). To work towards harmonization on different levels:

• Server URL naming & SSL protocol
  • Naming: good practice to include the name of emodnet + lot in the URL fo the Web services (e.g. https://ows.emodnet-bathymetry.eu/ows or https://ows.emodnet-seabedhabitats.eu/ows ).
  • Implementation of the SSL protocol: move towards https for all lots not having implemented it yet. It is also required for the central Matomo instance.
  • Seabed habitats has recently changed the naming of the URL and adopted https. It can help other portals to do so.
• Metadata harmonization:
  • Pascal D. presents the experience with the EU Open Data Portal. Some Data products from the EMODnet lots were manually ingested so EMODnet could be visible for the hackathon organized by the EU Open Data Portal. All the data products should be ingested to increase the visibility of EMODnet work. This official portal maintained by the Publications Office promotes the EU open data Policy (The European Union Open Data Portal (EU ODP) gives you access to open data published by EU institutions and bodies. All the data you can find via this catalogue are free to use and reuse for commercial or non-commercial purposes.)
  • EU Open Data Portal enables m2m communication, documenting data with the following 3x2 structure:
    • Documentation: metadata record + website URL of the data provider
    • Visualization: WMS service + map viewer URL
    • Resources: WFS service + download URL
  • Manual ingestion not desired, so we will explore the possibility to do a batch import from central portal Geonetwork. To do so, all the lots have first to provide the 3x2 accesses. Menashè E. and Pascal D. to do a test to with Sextant / Geonetwork and check if a batch import can be successfully performed.
  • Besides the 3x2 entries, Pascal D. to share the Web services assessment checklist of the European Atlas of the Seas in JIRA and all lots to review it and give feedback. Once the checklist is met we can say that m2m is fully enabled. Pascal D. says that an assessment tool based on the checklist needs to be developed in order to assess if a Web service meets the requirements. The tool is expected to produce an assessment report.
    • The already available records in the EU Open Data Portal are combined products, satellite imagery. Some lots (Dick S., Alexander B.) think that the ingestion exercise should be limited for the moment to the data products.

Addendum: end of October, the Director of DG MARE asked EMODnet to participate to the next hackathon organised by the EU OPEN DATA portal.

• GetCapabilities:
  • For INSPIRE compliancy, GetCapabilites of all the web services should contain the INSPIRE metadata URL field (ref: TWG3).
• Supporting time and elevation dimensions.
  • Pascal D. puts emphasis on the importance to distribute historical data and thus to have the Web services supporting the time dimension. This should be done following ISO standards.

1. AP: All lots to provide six entry points following schema of the EU Open Data Portal.
2. AP: Menashè E. and Pascal D. to do a test to with Sextant / Geonetwork and check if a batch import can be performed.
3. AP: Pascal D. to share the European Atlas of the Seas Web services assessment checklist and all lots to review it through JIRA.
4. AP: All lots to include the INSPIRE metadata URL field in the services GetCapabilites.
5. AP: Pascal D. will explore use cases on supporting time dimension in Geoserver, documenting them in JIRA, using Human Activities vessel density as test case.

36. AP: Participation of EMODnet to the next EU Open Data portal hackathon

37. AP: Implementation of ssl protocole for all lots and CP Matomo and harmonization of the web services URL

Monitoring Web services (Update Spatineo trial)

Pascal D. ran a Spatineo trial to check its suitability to monitor EMODnet web services. Spatineo checks the status and compliance of the server, and gives metrics on WMS and WFS services. However, the results of the test are not considered positive, main reasons being: a) monitoring of WCS services is not implemented and, b) server logs need to be uploaded in the Spatineo server to monitor WFS services.

Checking the performance and INSPIRE compliance of the service can be done in-house (e.g. Pascal D. knowcean tool), but a solution to monitor WMS and specifically WFS and WCS is needed. One possible solution is to test server log analytics in Matomo. Bart V. states that the solutions should distinguish between WMS, WFS and layer names.

Seabed Habitats and Bathymetry have in-house solutions. These existing solutions can be shared and documented in JIRA, to test whether they can be applicable to all lots.
Pascal D. stressed that ideally a common solution which would allow to report on metrics the same way for all lots is sought.

1. AP: Seabed habitats (GD, JP) and Bathymetry (DS) to share their Web services monitoring solutions in JIRA for the rest of the lots to test.

EMODnet Open Sea Lab II Hackathon (Pascal Derycke)

The OSLII will take place in September 2019, together with Copernicus and ICES. This will help to widen the profile of the participants. We expect to target more students by running the hackathon in the month of September.

1. AP: Secretariat to consider inviting Copernicus to the next TWG meeting.

Packages, documentation, toolbox

A hackathon package (user documentation, portfolio, toolbox, …) needs to be finalised sometime before the OSL II. This will allow to engage with some potential participants that will be able to evaluate in advance what can be done during the hackathon. The typical hackathon participant is a data scientist, but we expect (marine) scientists too. The hackathon gives us a deadline. We need to elaborate a set of documents that are valuable for other types of users beyond hackathon participants. The documentary background has to be user-oriented: different types of documentation for users of different areas and levels of expertise. All documentation should be managed via JIRA and then published in the Central portal.

The hackathon package should be a combination of libraries, APIs, documentation for different users. This will make us to be prepared for the hackathon but also to be stronger within the community of marine scientists.

• Portfolio to discover what is available and video tutorials (TrustIT will help with the elaboration) on where to find data access.
• QGIS plugin and/or tutorial to use EMODnet web services in QGIS for basic GIS users.

• OGC web services documentation based on Pascal D. template, for GIS experts and developers.
• APIs & documentation (data scientists, developers).
• Libraries / tutorials on how to use EMODnet data and services in different environments (R, Python,…), for scientists and data scientists.
• Libraries to combine datasets, produce indicators, etc.

Goal: to test this documentation and have it ready by next summer (before the OSL II).

Standardization menu items

Feedback from users collected by Secretariat (Nathalie T.) shows that users have difficulties finding what they are looking for in the portals, and that in fact the same information is structured differently across the lots.

The menus of all the thematic portals should be standardized so pages with a similar content (map viewer, services documentation) can be accessed in a similar way (UX improvement in the context of a Web ring). Nathalie T. presents new proposal to organize content:

https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-1

The lots discussed the proposed structure and made few comments:

* Some lots would prefer to have separate menu for Data access and another for data products. This are different things and often offered through different platforms. It helps the users to recognise immediately that both types of data are offered (i.e. there is a (work) flow from data to data products). Another possibility is to keep it under the same menu with a different title (e.g. data and data products, data services).

** To be compliant with GDPR, a separate link needs to be provided for the privacy policy. This link will be given at the footer (separate – as is already done in most Portals), but can at the same time be given under the data policy menu.

1. AP: Nathalie T. to distribute proposal for menu standardization, and to already include the agreed changes (e.g. inclusion of “Associated Partners” under the Get involved!-menu).
2. AP: All lots to give feedback on the menu standardization proposal (specifically on separating or not data vs. data product access).
3. AP: All lots to re-organize their menus according to the agreed items.
4. AP: All lots to update the privacy policy, providing separate link in the footer. Example of reference document - privacy statement given by TrustIT and updated by Nathalie T. according to privacy statement VLIZ in the meantime (http://www.emodnet.eu/privacy-statement-protection-personal-data). Note: Chemistry has adapted it to mention that they share their information with Central Portal (third party).

Status EMODnet

EU Open Data portal / European Atlas of the Seas (Pascal D.)

Eu Open Data portal: see Web services harmonisation / m2m communication.

The European Atlas of the Seas works as an aggregator of Web services with a system in place (Content management system) which allows to operate a catalogue of more than 200 Web services supplied by various data providers as EMODnet portals. A tool is daily monitoring the status of all services.

In order to improve the consistency and performance of the Atlas, an assessment protocol of the Web services was written (see Web services harmonisation).

EMODnet progress indicators

Chemistry (Menashè E.) has made a comparison study for two months between their metrics system and Matomo. They get very different results (related to Indicator 6 in the quarterly progress reports: one table on web pages and one on web sections). Bart V. suggests the differing results could be an issue produced by their server (Post-meeting update: web-server is acting as bot, generating ~500 extra visits daily). However, many of the lots have observed a decrease in the numbers compared to previous systems. This should be highlighted in the progress reports.

Pascal D. stated that Matomo is one of the most community accepted metrics system (after Google Analytics) and should be considered a fully reliable analytic tool. He lighted that Matomo collects visit hits when AWSTAT analyses server log files which are two different conceptual approaches to deliver metrics.

1. AP: Chemistry to look into Bart V. suggestion.
2. AP: Secretariat/TrustIT to include a disclaimer/caveat in the reports to indicate that numbers have to be taken with caution and to look at the relative trends over time rather than the absolute numbers. Also think about the reports that have been published already on every Portal (upon request from DG MARE).
3. AP: TrustIT to create one page per use case to monitor individual use cases.

Menashè E. thinks that the web pages-indicator table can be removed or moved to a less prominent part of the report, because the pages and sections are the same in case of Chemistry (prove the same metrics). This is not the case for the other portals, so Alessio C (Trust-IT) suggests to keep both tables because they do provide different information, and to identify new web pages for Chemistry worth monitoring.

Alessio C. mentions that after this reporting period there will be for the first time the opportunity to do some analysis and present trends of the indicators over time (3 full reporting periods considered).

INSPIRE updates

Dick S. gives feedback on the Chemistry / SDC presentation during the INSPIRE 2018 conference, a use case of implementing INSPIRE data models to map EMODnet nutrients data. By testing the implementation rules for EMODnet data, shortcomings were found and change requests were formulated. New scheme is more rigid and INSPIRE validator has errors; Chemistry metadata is compliant but some data is not. In general, requesting changes is a slow, arduous process. Simon C., in collaboration with Seabed Habitats, has requested INSPIRE to adopt the DarwinCore (DwC) schema. Initial feedback was that they were open to it but there was no follow-up on the INSPIRE side. There is frustration because there is a lot of effort done in making EMODnet INSPIRE compliant but the feedback is limited. DG-Env has to support the member states but no obligation to support EMODnet (which is thus not a priority). It is suggested that it could be a project on its own, a pilot to implement INSPIRE in the marine data community. It is agreed that there should be a response to move things forward at the level of DG-Mare. There was a meeting on how to link EMODnet and INSPIRE. It was said that input can be provided to INSPIRE (new phase), by means of a form (e.g. Biology and Chemistry did). The INSPIRE JRC team gave follow-up feedback to the EMODnet comments:

https://webgate.ec.europa.eu/fpfis/wikis/display/MarINSP/feedback+and+solutions

1. AP: to elaborate official response from all lots, (with special contribution from Simon C. and Dick S.) and convey the response via DG-Mare. Elaborate response by mid-October and send response by the end of the month to Iain Shepherd.

Data products categorization

Giuseppe M. presents the physics approach to categorize data & data products (“downstream products”). To elaborate this proposal, they looked at existing best practice initiatives (SCOR, QUASIMEME, QARTOD temperature-salinity). Nathalie T. and Paula O. present the current categories suggested for EMODnet data and data products. It differs from the physics approach mainly by the addition of two extra levels (Level 4 data and Level 6 data product). See: https://docs.google.com/document/d/1MERpY1bui67tTTM6vMkZdCoBnrsHWZos7icBy2zh-2E/edit?usp=sharing

Level 4 is in general accepted by the lots (many portals have data that flow from level 3 to 4 (individual dataset to harmonized collections), but Skype connection with Giuseppe M. is lost.

The final goal is to publish these categories together with the portfolio, assigning the showcased data and products with their correspondent categories. Paula and Nathalie indicated the definitions should be ready before 20 October. A final version of the portfolio has to be ready to be printed for the EOOS Conference (21-23 November 2018).

1. AP: Nathalie T. to re-distribute the google document with the categorization of data and data products. Establish deadline within email.
2. AP: All lots to review, make corrections and suggestions to the categories document, adding and re-locating their data and products accordingly.
3. AP: Nathalie T. and Paula O. to update the portfolio with the final categories.
4. AP: Once these categories are agreed, all lots to add the categories to the metadata (lineage).

Central Portal data services update

• CP data services are organized with a user-targeted approach (different services for different users).
• Updates related to Geoviewer and Geonetwork catalogue are presented.
• The concept of the query tool is presented.
  • The tool is though for a preliminary assessment and early phase planning of a particular activity, with the practitioner or marine planner user in mind. It retrieves information from multiple thematic data products via one single interface, and produces a summary report of the area selected for the layers selected.
  • The technical set up is also presented. APIs documentation will be provided.
  • Timeline: to implement two more layer types (WCS, WFS point), with layers from different thematic portals by the end of the year 2018.
• VLIZ has also elaborated a tutorial (html page) for using EMODnet data and services in R. Examples of layers from Bathymetry, Seabed Habitats plus Biology API. The tutorial covers:
  • How to download and use data in R.
  • How to use EMODnet products in R using WFS service (SH) and WCS (Bathymetry).
  • How to visualize WMS services in R.

1. AP: VLIZ to include also the OSPAR and Habitat areas layers in the Query tool.
2. AP: VLIZ to elaborate API documentation of the Query tool processing service.
3. AP: VLIZ to launch first version by the end of the year.
4. AP: VLIZ to set up work team for QT and define milestones (JIRA).
5. AP: VLIZ to publish the EMODnet data in R tutorial. This to be shared with lots, so they can contribute with similar examples for their services (To publish Web services documentation too once is completed by lots).
6. Pascal D. asked the query tool to be conceived and designed as a portable application.

Pascal D. asked information about the technology supporting the implementation of the query tool. Filip W. said that the query tool is being developed in JAVA. Pascal D. asked the reasons for choosing JAVA.

Overview of new data products & developments by thematic portals

EMODnet Ingestion (Dick S.)

• Over 240 submissions, mostly in the first phase (ingestion), with about 25 submissions in the second phase (digestion, integration in EMODnet portals).
• There are wide differences in the number of submissions per country. It could be to various reasons, including the degree of existing organization that is already in place.
• In some cases, they are also working into m2m communication (e.g. BODC), establishing specific data exchange procedures.
• Main challenge remains involving industry.
• SH is not formally represented at Ingestion but the communication is fluent.

Bathymetry (Dick S.)

• New DTM with 1/16 * 1/16 arc (approx. 115m) and greater spatial coverage. Derived data products updated (e.g. source reference layer).
• 3D visualization tool -> documented and will be shared in GitHub.
• Download registration form.

Geology (Henrik K.)

• GeoNetwork CSW for OGC access.
• Data entity indexes are ready.
• Restructured portal content.
• New data products and updates presented.
• Upcoming products: multiscale substrate map, seafloor quaternary geology, seafloor geomorphology, etc…
• Working on stability of servers and webGIS download (possibility to draw polygon).

Biology (Simon C.)

• Second week of October there will be a data product workshop.
• Deliverables will be compiled in the Atlas of Marine Life which will contain different sections: a combination of tools, models, Geoviewer with mapped products. The Web services and workflows will be documented (R-markdown).
• Data products on the Geoviewer of the Atlas of Marine Life will be basic data maps (e.g. gridded abundances), indicators (e.g. non-indigenous species) and story maps, distribution and trait-based models (e.g. low resilience benthic communities). Products will be structured around EOV (some of the EOV categories are covered by Seabed Habitats).
• It is desired to work only with publically available data so that the user can replicate the analysis by using documented workflows.
• Main technical challenge is to upload DIVA output – NetCDF into Geoserver.

1. AP: VLIZ to report on experience with NetCDF in Geoserver (extension not stable?). If output of DIVA model needs adaptations, Alexander B. confirms this could be looked at.

Human Activities (Nick E.)

• Visual guidelines implemented, common terminology adopted, responsive design, blog updates.
• Live chat implemented, managed by Cogea (9-17h), plus contact form.
• User needs to fill in form to access to Web service URL. This should be changed.
• Updated datasets: protected areas, bathing waters, fish catches, ports, cables, waste disposal. New: waste at ports.
• Coming soon: vessel density maps. Output unit: hours/sqkm per month. Currently improving processing speed. Initial launch will be one-year data -> update map viewer to allow time sliding. Challenge: to implement WCS for vessel density (no technical experience with WCS).

Seabed habitats (Graeme D. and Jordan P.)

• WP4 survey point data update (ground-truthing data).
  • Current status: beta testing. UK data transformed and uploaded (already in Geoserver but testing mode).
  • More European data to do further ingestion and testing.
  • Format DwC OBIS-ENV based. It allows to store different classification systems (EUNIS, Habitats Directive) for a single point, together with measurement device (BODC vocab), metatada and dataset identifications and different contact points.
  • The table accessible via Geoserver is a flattened version with only public data.
  • Bboxes will be displayed for areas where there is restricted data to let user know the data exists.
  • Working on developing BODC/NERC vocabulary for classification systems.
  • Next steps: finish testing with wider datasets and publish, and work with VLIZ towards OBIS integration.
• Other updates:
  • 180 new maps uploaded, and more third party will come in the next year.
  • EUNIS MSFD for the Black Sea and recent data from OSPAR uploaded.
  • Looking into revamp of the map viewer, to move forward to Geoserver based implementation.
  • Working towards automatization of updates and INSPIRE compliancy.

Chemistry (Dick S.)

• Harvested nutrient data and now working on the product (DIVA), which will replace the previous products. We learn from previous runs.
• Harvesting contaminants (data from new areas like Norway) à data sent over to research coordinators for harmonization, to be published the in future.
• Further work on marine litter and micro-litter, trying to access data from more areas (Mediterranean is difficult, but in contact with the area). We are now enriching the maps. Both datasets progressively being populated.
• DG-Env has accepted EMODnet Chemistry as a source for the MSFD implementation (as an official reporter).
• Plots: P36 layers per aggregated parameters have been created (action point from TWG2). For each parameter group you get position of CDIs, with a link to the CDI service (you get a list of all relevant datasets for a specific region).

1. AP: VLIZ to integrate Chemistry parameter aggregated layers in the Central Portal Geoviewer.
2. AP: VLIZ to add beach litter sampling effort layer to CP.
3. AP: Nathalie T. and Paula O. to add Chemistry new layers to portfolio.
4. AP: All lots to communicate to Secretariat and VLIZ when a new layer/service is about to be published.

Physics (Marco A.)

• New products on underwater noise. Two products:
  • Pulse base, starting from data from ICES and expanded to Mediterranean Sea with other sources. Already available.
  • Noise map, currently being developed/calculations ongoing. Time consuming process. First test area around Barcelona. For other areas is difficult because lack of accessible data and 3-4 weeks processing to make a one-month layer.
  • Spatial coverage of the product: Europe except Black Sea, Biscay Bay. It’s point and polygon data.
  • HF radar services: published on request of DG MARE, and now nearest time in EAS (European Atlas of the Seas).

Technical support & other updates

JIRA

JIRA and Confluence will be used to improve coordination and make collaborative work easier. Pascal D. will email with credentials and skype calls can be made with the different lots to explain how it works.

1. Pascal D. to distribute JIRA credentials with TWG members.

MarineID

Technical details and advantages of implementing MarineID where discussed during TWG3 and SC in Mallorca. It will be implemented in all portals (part of new contract), step-wise approach. Information on usage of each portal will be stored by the specific lot (separate from the MarineID user information at Ifremer).

MarineID grants access (is a gateway), and it will be possible for the user to switch from one to another portal within one session. It does not register when user downloads; this is one through a download application which makes a track record.

1. Pascal D. to generate the JIRA issues related to the implementation of the MarineID. Dick S. to attach the documentation to JIRA. Seabed Habitats will be the test case with assistance from Pascal D., Menashè E. and Dick S.

Background layer

All the lots agree in the convenience to have a common background layer for all the viewers. Background layer should have global coverage, be empty on the marine side and support other projections (e.g. polar for Physics). A proposal by Dick S. was presented at the 3TWG. The initiative was well received but there is no budget at the moment.

1. Dick S. to consider the development and sharing of a base layer for all the lots.

The lots agree on the new structure with the exception of the distinction between data and data product access, to be further discussed.

Revision of action points from previous meetings

Annex 1 - Agenda

Monday 1 October 2018

13:00-13:15 - Welcome & introduction (Jan-Bart Calewaert)

• Welcome to participants 5’
• Practical arrangements 5’
• Tour de Table 5’

13:15-13:45 - Progress & objectives (Jan-Bart Calewaert)

• Updates since last meetings (revision of action points from previous meetings) (Paula Oset) 15’
• Expected outcomes of this meeting (Pascal Derycke) 15’

13:45-14:45 – Status EMODnet web services

• Web services documentation 20’
• Web services harmonization 20’
• Monitoring Web services (Update Spatineo trial) 20’

14:45-15:00 – Coffee Break

15:00-16:00 - EMODnet Open Sea Lab II Hackathon (Pascal Derycke)

• Packages 20’
• Toolbox 20’
• Required actions/contributions 20’

Tuesday 2 October 2018

09:30-11:30 - Status EMODnet

• EU Open Data portal / GEOSS (geoportal.org)/ European Atlas of the Seas (Pascal Derycke) 30’
• INSPIRE updates 30’
• Data services (VLIZ) 1h
  • Data and data products categorization 30’
  • Query tool update 30’

11:30-13:30 - Overview of new data products & developments by thematic portals

Thematic lot representatives present (15’ max) an overview of existing/upcoming data products, functionalities and Web services & links to the Central portal. Particular attention should be given to technical challenges of common concern.

• Data Ingestion 15’

13:30-14:30 – Lunch Break

14:30-15:45 – Technical support & other updates

• JIRA 15’
• Marine ID 15’
• Background layer and other reference layers 15’
• Specific questions & technical issues 15’
• Other updates 15’

15:45-16:00 - Wrap up, actions, next meeting (Secretariat)