3rd EMODnet Technical Working Meeting

Tuesday 20/03/2018 - 14:00 to Wednesday 21/03/2018 - 12:30

Meeting Venue: Pollentia Club, Alcudia, Mallorca

Attendees

Summary of action points TWG-3

Meeting minutes

Updates since last meeting (Jan-Bart Calewaert)

• Coherence between data portals: there have been significant improvements since last meeting in terms of the convergence between the different thematic data portals has greatly improved. This is partly due to the efforts to make available data products from different thematic assembly group via the EMODnet Central Portal and more recently also via the European Atlas of the Seas. This progress can be seen very clearly in the EMODnet data and products catalogue, which provides a good overview of the resources available via EMODnet and how they can be obtained. We could then maybe put the report on the forum.

• Open Sea Lab: general positive results, many of lasts years’ efforts dedicated to coordinate the OSL (VLIZ, Secretariat, Commission, imec). Web services exposed, weaknesses detected. Feedback from participants and lessons learned key to define priorities for next period. A new network developed where EMODnet brand is now present.

• New contract for the Secretariat since September: two new elements were added a) support the emerging European network on ocean observing (beyond data assembling and sharing) and b) development European Atlas of the Seas: initiative from DG Mare, to provide marine information to non-experts produced by EU funds. EEA also involved. Currently 80 layers: https://ec.europa.eu/maritimeaffairs/atlas/maritime_atlas/ but not very well promoted and interface to be improved. EMODnet branding will be visible. EMODnet Secretariat will also be responsible for organizing the Technical Working Group meetings in close collaboration with VLIZ.

Expected outcomes of the Meeting (Jan-Bart Calewaert)

Jan-Bart Calewaert clarified the expectations of the Technical Working Group, its function and relationship vis-à-vis with the Steering Committee, the roles and responsibilities of the different partners and members as well as its modus operandi as recorded in the Terms of Reference of the Technical Working Group (TWG) which exists as a living document circulated before meetings and available upon request from the EMODnet Secretariat. The list of actions of TWG meetings together with the action plans of specific priority developments (e.g. query tool, thematic portal improvement plans) serve as work plan for the technical working group.

• Thematic data portals: identify priority issues and bottlenecks and develop an action plan, especially with regards to web services, data accessibility, harmonization of documentation.

• Central Portal: current status and developments of data services, Query tool 2.0, map viewer, catalogue.

Action point: Each thematic data portal to develop an action plan to resolve identified bottlenecks/issues with regards to web services, data accessibility, harmonization of documentation.

Action point: develop work plan/action plan for the development of the query tool and other central portal data services.

EMODnet Open Sea Lab Hackathon (Pascal Derycke)

Pascal Derycke gives a short summary of the Open Sea Lab competition. It is agreed to keep in touch with the coaches of the OSL and potentially invite them to future TWG meetings.

Action point: Paula Oset to circulate email list of both TWG and OSL coaches to be updated.

OSL feedback from participants and lessons learned (Pascal Derycke).

- Questionnaire divided in four big blocks: discovering data, accessing data, processing data and general evaluation of the challenges. Variety in responses but majority found difficult or impossible to discover and/or access data; only a few found this easy depending of the data/portal. Many participants accessed the data from the CP map viewer or thematic map viewers instead of catalogue services. Different services used: WMS, WFS, WCS, etc. but very often the direct download from the map viewers was used.

Required action/response (Pascal Derycke).

a) A comprehensive documentation on data access, existing APIs and use-cases needs to be published. This should be done by each thematic portal and should be agreed on where to store it (e.g. specific page on thematic portal, Github). The CP should have low expertise guidelines (e.g. how to use web services in QGIS) and link to the documentation per lot. The documentation per thematic lot would assume some more advanced knowledge on how to use web services, access data, process data, …

The field “Online resource” in INSPIRE metadata xml record (i.e. catalogue csw entry point) should be used to provide a link to the documentation on “accessing EMODnet data”.

Action point: Pascal Derycke to provide guidelines on how to provided harmonized documentation.

b) Develop an operational machine to machine service: need to improve INSPIRE compliance of web services. Pascal Derycke has developed a tool to monitor web services (server status and time response and metadata compliance with INSPIRE http://www.knowcean.eu/har). The tool helps to control that metadata are not missing in WMS GetCapabilities response (e.g. abstract, keywords, metadata URL). When metadata are missing, it hampers data discovery and machine to machine communication, i.e. the possibility of EMODnet products being automatically harvested and shared in other catalogues like EU Open Data Portal (reduced visibility).

There are 2 possibilities to fulfil INSPIRE metadata requirements in WMS GetCapabilities response:

• you can embed all INSPIRE metadata in WMS GetCapabilites response

• or you can provide a link to the metadata record of a catalogue service with the field “metadata URL”.

The second option is commonly applied and a solution to implement it in the case of GeoServer is to have the GeoServer INSPIRE extension installed.

Action point: Pascal Derycke to report the list of issues and actions needed from the different partners to harmonize web services and facilitate machine to machine communication.

Status EMODnet Central Portal

Website (Pascal Derycke, Jan-Bart Calewaert)

- Website redesigned, new visual identity, communication kit expanding. Statistics show an increase in the number of visitors since end of summer 2017 (figures are with Piwik, which is usually conservative).

- Internal communication: the doc manager is not being used at the moment. Function is to store documents, presentations, etc.; news item is added when new documents are uploaded. Some partners didn’t know about it but find it useful.

Action point: Secretariat to communicate again and remind login details for the doc manager to partners.

- Calendar: a calendar will be produced so that every partner knows which events are interesting, whether partners want to join or send material, etc. Partners can communicate to Secretariat to add high level events (not internal meetings) to calendar.

- Use cases: the need to document more uses cases to highlight the value and potential of EMODnet is stressed. New use cases can be communicated by partners and these will be added in the website. Alessandra Giorgetti asks why not all the use cases that were given for the annual report have been added. (Note: Sara Garavelli explained during Steering Committee that the use cases are being added one every week).

Action point: Secretariat to define who has to be contacted in the future whenever partners want to add new use cases.

Product catalogue (GeoNetwork)

Progress made and issues with the product catalogue are presented.

Action point: VLIZ to set up Search page as default and possibly hide the INSPIRE page.

Action point: VLIZ to check with Erwin (Ifremer), responsible for Sextant, what are the issues with Chemistry harvests. Keep Pascal Derycke in the loop.

Action point: VLIZ to provide deep link from metadata record to geoviewer display of the layer.

Action point: VLIZ and Pascal Derycke to work towards the harmonization of metadata.

Geoviewer

Chemistry combined products added. Alessandra Giorgetti requests to make visible data only after year 2000.

Action point: Paula Oset to filter out Chemistry combined layers before 2000.

Action point: VLIZ to follow up and integrate Physics climatologies in the CP geoviewer. Definition of legend in URL needed? Fix issue with parameter in URL and not in the get capabilities.

Action point: Antonio Novellino to provide aggregated (combined) climatology products for European scale (not per sea basin, similar to Chemistry).

Action point: VLIZ to integrate Physics Sea Level product in the CP geoviewer.

Action point: VLIZ, Gerrit Hendriksen, Dick Schaap (ongoing from TWG-2), GetFeatureInfo to actually retrieve bathymetry values (not RGB).

Web services documentation

Partly discussed during OSL agenda item, where an action point will be to give guidelines to provide harmonized documentation.

Action point: Paula Oset and Pascal Derycke to review web services documentation page to remove direct links to data (custom package datasets to use during OSL) and other not relevant links.

Query tool 2.0

Filip Waumans presents a proposed new approach for the Query tool. Main issues of the current QT are: a) It is not possible to programmatically reach the functionality behind the tool; b) No indication of availability of data layers: user should know in advance. Two improvements to be made: a) use of polygons instead of points to define the geographical extent of query; b) a central processing web service.

The output has to be defined depending on who should be the final user of the query tool. E.g. it could return a report with summary statistics for an input area + data availability.

Jan-Bart Calewaert proposes to set up a user group + task force group (TWG + OSL coaches?). The users group should be a small group (~5) to provide feedback during development. Antonio Novellino suggests to set up a first definition of the purpose of the query tool based on the existing use cases. (Note: During the Steering Committee meeting, Charlotte Herman – Policy Officer, Maritime Innovation, Marine Knowledge and Investment, DGMARE – suggests to present preliminary ideas and get feedback at Marine Knowledge Expert Group meeting to take place May/June).

Action point: VLIZ to develop a proposal of the Query tool 2.0 main functionalities with input from Secretariat; to be presented at Marine Knowledge Expert Group meeting.

Action point: Set up a user group + task force group (selected members of TWG and OSL Coaches/participants). The users group should be a small group (~5) to provide feedback during development

Increasing the performance and access to the central portal data services: mirroring EMODnet Data Products

In order to increase performance of the new version of Query tool, it is discussed whether to access the data on remote servers, mirror them at VLIZ server or having them pre-cached, under agreed terms for that purpose. To be further discussed during development (e.g. only cache the layers used in the Query tool). Graeme Duncan points that it will be difficult to make the tool performant if it relies on remote servers (e.g. clipping operations for complex polygons and/or large areas).

Overview of new data products & developments by thematic portals

Seabed Habitats (Graeme Duncan)

- Site redesigned, new look, responsive, Help Desk, aim to be live by 1^st May. Move from MapServer to Geoserver ongoing. It will include Service monitoring, INSPIRE plugin, WFS, WCS as standard. WPS for CP? Testing new method for delivering complex layers. Aim: server running in April. Reference terms will be updated as part of the website re-design.

- New products: Article 17 grids (Habitats Directive), more habitat maps from Surveys (EUNIS, Annex 1), a new multi-resolution EUSeaMap update. Currently everything averaged to 250m resolution, moving to multi-resolution. New data based on points, working with EMODnet Biology to get habitat data in OBIS Event core, first tests datasets sent, working on standardization of NERC vocabulary.

Chemistry (Dick Schaap)

- New products: eutrophication, marine litter. Work being done to develop marine litter central database (marine beach litter, seafloor litter, etc.). System ready, data hosted, producing first maps. Micro litter data is populated into SDN CDI service (ODV format). Jan-Bart Calewaert states that there is pressure from EC to have that material.

- Launch of new portal, restyled service and ocean browser. DOIs now for all products.

- Work on progress: upgrading visualization and access to validated, aggregated and harmonized data collections. Setting API to give regulated access to validated data to more advance users.

Physics (Antonio Novellino)

- Reorganizing of the catalogue, shortcuts to datasets download (reorganize data and data products access and download).

- New and coming products: e.g. absolute sea level trends, sea level anomalies, etc. Some Copernicus products are exposed in Physics and used more via Physics than via Copernicus.

- Updated landing page web to make it easier for the user to access all the services. Created github (OSL). Cross reference to all the services (from parameter to platform, etc).

- Interoperability: exposing more layers in geoserver, erdapp, THREDD.

Biology (Simon Claus)

- Simon Claus first briefly shows presentation was prepared for OSL with overview of all the data services of the different lots.

Action point: Simon Claus to share presentation of overview of EMODnet data services in intranet.

- Developments: implementation of the new data schema (OBIS Event core); implications for the download toolbox (currently under development) and geoviewer (under development).

- Update on new data products: Atlas of Marine Life (M18, within 6 months), species distribution models, trait based analyses, Essential Biodiversity Variables. R-shiny apps (evolution of abundance), vulnerability maps: resilient vs. vulnerable communities.

Action point: Simon Claus (and all lots) to channel material that can demonstrate the value generated by EMODnet in relation to MSFD (similar to discussion of Chemistry).

Human Activities (Nick Earwaker)

- WMS & WFS they have been moved from MapServer to GeoServer, the live portal is still served by Map Server but will be changed. Web services documentation page became more visible for user (OSL), continuous work with feeback from OSL.

- Products: most of the data layers have been updated to some extent, new dataset (pipelines). Vessel density map to become available in the next few months (main focus at the moment). Alessandro Pititto to give more details about this dataset in Steering Committee.

- Harmonization of portal design nearly complete, to be launched ASAP. Pascal Derycke recommends to choose Drupal because TrustIT has designed the website based on Drupal. Antonio Novellino: wants to keep updated about vessel density, Physics wants to use it for other products (e.g. underwater noise).

Action point: Nick Earwaker to let Secretariat and TWG know when launch of new portal is done.

Geology

To be discussed at the Steering Committee.

Bathymetry (Dick Schaap)

To be discussed at the Steering Committee.

Common base layer (Dick Schaap)

- To support a common identity. Four services proposed: combinations of water and/or land, unprojected and projected Web Mercator. Developed to 10 zoom levels. Source for European Bathymetry: HRSM; rest of the world: downsampled GEBCO, using harmonized symbology. Source for land data: combination of SRTM and ASTER, and OpenStreetMap, leaving symbology for ocean away (no ferry lines, no names, avoid disputes).

- Financial proposal: total budget is 37180 euro, 1550 per thematic portal per year. World layer interesting product for external uses. Jan-Bart Calewaert asks coordinators of thematic lots to think about it.

Action Point: Secretariat to check with individual portals what is their opinion about the proposal for a new common base layer.

Monitoring use

Progress Indicators & monitoring system (Alessio Corongiu)

- Alessio Corongiu states the importance of a common approach to all the portals (monitoring performance, quality of services, etc.) to facilitate annual reporting, every three months.

- Two types of indicators: a) indicators related to visibility, user friendliness, harmonization (mainly Group 3 & 4) will be reported by TrustIT, who will send it to the partners for review. b) Indicators related to data, data product development, etc. (mainly Group 1 & 2) have to be reported by each partner. Group 4: not all portals collect information on the users and the purpose. Only Human Activities implemented suggested user form; Seabed Habitats, Geology and Biology have similar forms. Discussion MarineID: next agenda item.

- The reporting of the new indicators from January - March 2018 has been covered by the pilot project. Partners have to report to Secretariat with the new schema from April onwards. Partners can keep the old indicators for their internal reports. Secretariat asks that, if possible, there could be a period of overlap with reporting old & new indicators (not mandatory). The final report will have a year of a mixed situation (old & new indicators); the Secretariat will communicate on this to DGMARE.

Open points: collection of user information & tracking web services requests. Discussed under following agenda item.

Action point: TrustIT to send email with clear guidance on the new list of indicators.

Action point: All thematic lots to report using new indicators from April onwards. If possible, overlap old and new indicators for a period. When not possible to report on a particular indicator (e.g. WMS), give an indication about the progress in setting up procedure to report on that indicator.

Single Sign-on - Marine_ID (Dick Schaap)

- How to monitor users and purpose of data usage: two possible solutions. a) web form every time user downloads; b) Make use of authentication, authorization and accounting (AAA) system.

- MarineID is an AAA for marine data users (SeaDataNet, EMODnet, Coriolis). The user can stay in session and the service allows to get info on downloads per user (from transaction database).

- Sara Garavelli asks who owns data. Dick Schaap: LDAP is Ifremer, user data will be at the portal levels (own transaction database). MarineID AAA per se does not give information on the download purpose, which is enabled by the form. Graeme Duncan: If their personal info will be used further (e.g. to ask them about use cases), it has to be actively consented.

- Simon Claus suggests a combination of both options. For non-regular users a form is enough (no registration burden). For regular users of various portals, we can offer a marineID.

Monitoring of the web services & PIWIK/Matomo status (Paula Oset & Pascal Derycke)

- Paula Oset presents about Matomo (formerly Piwik) and the different features and analytics provided by the tool. Bart Vanhoorne has installed a solution to track the use of the layers in the CP mav viewer (user interaction).

Action point: Bart Vanhoorne to share the solution to track user interaction in map viewer with individual portals.

- Monitoring web services (server logs): Pascal Derycke has not found yet a solution that is satisfactory. It is suggested that EMODnet tries to find a bespoke and harmonized solution based on Matomo with collaboration from all technicians from the different portals. Dick Schaap and team have developed a script to track WMS and WFS requests.

Action point: Dick Schaap to report and share with the rest of the partners about the solution implemented to track services.

Action point: VLIZ to lead on implementing a common solution to monitor web services.

Other items agenda:

To be covered in the Steering Committee.

Date next meeting:

Previously decided to have 2 TWG per year. Next meeting will be remote.

Action point: Secretariat to circulate doodle with dates for TWG4 (end September – beginning of October).

Annex 1 - Agenda

Tuesday 20 March 2018

14:00-14:15 - Welcome & introduction (Jan-Bart Calewaert / Pascal Derycke)

• Welcome to participants

• Practical arrangements

• Tour de Table

14:15-14:30 - Progress & objectives (Jan-Bart Calewaert)

• Updates since last meeting

• Expected outcomes of this meeting

14:30-15:45 - EMODnet Open Sea Lab Hackathon (Pascal Derycke)

• Short summary

• Feedback from participants & lessons learned

• Implications in terms of required action/response

15:45-16:15 - Break

16:15-18:00 - Status EMODnet Central Portal

• Website (Pascal Derycke)

• Data services (VLIZ)

  • Product Catalogue

  • Geoviewer 2.0

  • Overview services & API’s

  • Recap discussion on querying multidimensional data products (2nd EMODnet technical meeting) – Solutions for EMODnet Service query tool

  • Increasing the performance and access to the central portal data services: mirroring EMODnet Data Products

Wednesday 21 March 2018

9:00-10:30 - Overview of new data products & developments by thematic portals

Thematic lot representatives present (15’ max) an overview of existing/upcoming data products, functionalities and webservices & links to the central portal. Particular attention should be given to technical challenges of common concern

10:30-11:00 – Break

11:00-12:00 - Monitoring use

• Progress indicators & monitoring system (Trust-IT)

• Single Sign-on - Marine_ID

• Monitoring of the web services & PIWIK/Matomo status

12:00-12:45 – Other updates

• Harmonisation guidelines (Trust-IT)

• User form & GDPR (Trust-IT)

• European Atlas of the Seas (Pascal Derycke)

• EMODnet overview of data policies

• RSS News feeds

• Search Engine Optimization (SEO)

12:45-13:00 - Wrap up, actions, next meeting (Secretariat)

Added item (Dick Schaap): common basemap layer

13:00-14:00 – Lunch

Annex 2 - Summary of action points from TWG-2