10^th EMODnet Technical Working Group

Autumn 2021

Meeting Venue: InnovOcean, Oostende and remote meeting – Zoom

Dates: 8^th September 2021 (14:00-17:00*) + 9^th September 2021 (14:00-17:00) + 10^th September 2021 (09:00-13:00) (All times are in Central European Summer Time, CEST)
*Joint session with EMODnet Steering Committee (SC), Technical Working Group (TWG) and EC Marine Knowledge Expert Group (MKEG)

Meeting Chairs: Conor Delaney (EMODnet Secretariat) and Zoi Konstantinou (EC, DG MARE)

List of Participants: See Annex I below

Go to meeting agenda and documents here

Actions list – To be tracked in JIRA

Meeting Minutes

Wednesday 8th September 2021

14:00 – 17:00: EMODnet SC Joint Session*

*This Session was open for Members of the EMODnet Technical Working Group (E-TWG) and the EC Marine Knowledge Expert Group (MKEG).

The minutes of the joint session are contained in the separate minutes of the 15th EMODnet Steering Committee. The numbering of topics in the closed session (reported below) is aligned with topics as they appear in the 10^th Technical Working Group (TWG) Agenda.

Thursday 9th September 2021

14:00 – 17:00: EMODnet TWG

Welcome & introduction (Zoi Konstantinou, EC DG MARE; Conor Delaney, EMODnet Secretariat)

Conor Delaney (EMODnet Secretariat) introduced the agenda and indicated the main goal of the meeting was to bring everyone up to speed on the technical developments.

Updates and progress since previous Technical Working Group Meeting (Tim Collart, Conor Delaney, Francis Strobbe, EMODnet Secretariat; Joana Beja, VLIZ)

Action List from 9th TWG

The EMODnet Secretariat ran through the previous (9^th) TWG meeting Action List. There were no further comments on the actions.

Jira: Overview of usage (EMODnet Secretariat) - JIRA overview of usage

• Francis Strobbe (EMODnet Secretariat) presented the JIRA interactions, reporting an increase in the creation of tickets, as a consequence of the increase in JIRA usage. Regarding the distribution of tickets across the portal, most tickets have been assigned to the central portal, as a result of repatriation and centralisation actions.

OGC service monitoring Update & INSPIRE compliance and updates INSPIRE compliance and updates

• Tim Collart (EMODnet Secretariat) gave an update of the metadata and data URL's in Service GetCapabilities:
  • Some lots have finished this task (service metadata URL's) with others still to complete it;
  • For the dataset metadata and data URL's, most lots have improved and this ticket is nearing its completion. Monitoring will be continuous as new data layers are made available.
• INSPIRE Quality of Service (QoS) monitoring (performance and availability) is being done via the GeoHealthCheck monitoring tool implemented and maintained by VLIZ (monitor.emodnet.eu);
  • Most portals met INSPIRE requirements for performance, as evaluated since the last meeting, except the Physics' WMS. Issues were recently resolved and the next update (in the next TWG) should reflect that;
  • In terms of service availability most lots are compliant, with a few exceptions that are being followed up via JIRA (could have been due to a previous issue that resolved itself);
  • The CP team is planning to further implement the full INSPIRE QoS testing procedures as plugins to GeoHealthCheck, but this development is currently on hold (resources prioritised for the centralisation);
  • Need to prioritise the test-suites of the INSPIRE validator tool that are fit-for-purpose to EMODnet metadata and services, but this development is currently on hold (resources prioritised for the centralisation).

Review and discussion on the collection of metrics for reporting (EMODnet Secretariat/CINEA)

• Conor Delaney (EMODnet Secretariat): The Secretariat is aware that there are changes due to centralisation;
• Dick Schaap (MARIS, EMODnet Bathymetry and Chemistry): Every quarterly report we're asked to provide user information and we're being asked to stop that. We would rather know who's using the products, so is it a good decision to not ask for the details through a form?
  • Conor Delaney (EMODnet Secretariat): Repatriation brings new activities and things to think about. Going forward, especially with the products and catalogue, there will be less ability to include forms and tracking users due to the CP being on the Europa domain.
  • Bart Vanhoorne (VLIZ): we can track the users without any kind of form, but we won't have the purpose of data/product usage.
  • Conor Delaney (EMODnet Secretariat): another idea is to have a voluntary form that allows to track this usage.
  • Iain Shepherd (EC, DG MARE): Yes - if only 50% reply it is already very useful. But maybe a cookie could store this so the second time they connect, they don't have to put the same information again.
    • Bart Vanhoorne (VLIZ): Yes, that is technically certainly possible. We are allowed to store the country of origin, which can be derived from the IP address (which is not stored) i.e. via Europa Analytics. This is in line with GDPR regulations.
  • Iain Shepherd (DG MARE): we discussed this before and it's very useful for us to know what sector they are from. Across all the lots we know that 25% of users come from a private/commercial background. Can we not implement a means to track what their background is?
  • Conor Delaney (EMODnet Secretariat): if we are requesting the tracking through cookies, we will have to go through DG-COMM process again.

• Alessandra Giorgetti (EMODnet Chemistry): There is interest to collect this information and the common decision was to not bother the users to provide basic information.
  • Jan-Bart Calewaert (EMODnet Secretariat): the suggestion is to map the different technical options without taking a decision today, but to have an informed discussion and decision based on their feasibility, benefits and disadvantages.
  • Conor Delaney (EMODnet Secretariat): we have restrictions on the type of login we can use in the repatriated website and MarineID is not an approved system. There is a technical limitation introduced by the repatriation (we cannot have a login other than ECAS).
• From CHAT: Helen Lillis (EMODnet Sea Bed Habitats): On the Seabed Habitats download page, we ask for download reason and originating region and country - they are optional and about 50% of people fill it in. These are the 'download reason' categories we use, which I think were agreed across all lots: Academia/Research Government/Public administration Business and Private company NGOs/Civil society Other.
  • Bart Vanhoorne (VLIZ): This looks good. Country can be derived from the IP address, so you basically end up with 1 question: the download reason/user type.
  • Joana Beja (EMODnet Biology): Within Biology, there is a form that collects the data. Contrary to Seabed Habitats (SBH), they have mandatory fields, which Biology uses on all the Quarterly Reports they produce. MarineID is not used on all thematic lots’ portals.
• CINEA emphasizes that open access to the data is paramount, and has priority over the collection of statistics. Of course, the indicators are important to evaluate the progress (i.e. we cannot change them every year) but there is some flexibility. It is useful that the portals have given their approaches so we can evaluate which Key Performance Indicators (KPIs) we will select. They also indicate that the centralisation onto the Europa domain introduces some restriction on how these KPI's can be collected. But together we should reach an agreement on the set of indicators we will collect (within those restrictions).

EMODnet Harmonisation

Overview of the Repatriated website, News Room and Europa Analytics (EA) Status repatriation

• Conor Delaney (EMODnet Secretariat) indicated that most of the coordinators already have access to Europa Analytics and some still have to click on the link sent by the Secretariat. The Secretariat is also working with TRUST-IT to update the Grafana Dashboard to fetch the CP statistics from EA (for the Central Portal).
• Francis Strobbe (EMODnet Secretariat) gave a demonstration of the repatriated website, explaining there is a limitation with the depth of the menu items, but we will have to work around this. He noted:
  • In the reports section, asked the Thematic lots to notify if there are important reports from their portal that we need to add here;
  • On the “Use cases” section, there is now a free text search field, as well as number of keywords and portal filters which make it more user friendly to find the use cases;
  • The thematic portals pages. Currently the information is a bit distributed. Francis Strobbe (EMODnet Secretariat) suggests improving page navigation with a menu on the left-hand side (e.g. as done for https://emodnet.ec.europa.eu/en/conference2021). And use this for the thematic lots’ pages;
  • Design needs to follow the European Commission Style Guide. However, there is a lot of flexibility to make it work for EMODnet and it is nice that it is more harmonised with other EC pages (e.g. DG MARE pages);
  • Important to pay attention when using images subject to copyright. Also important to not link to remote images, but save them on the Central Portal;
  • Relevant re-occurring sections (use cases and news) may also be displayed on the thematic lots’ pages;
  • We have moved away from MailChimp and to Newsroom (EC tool) for the EMODnet newsletters. Coordinators (and other thematic partners) will have to subscribe to the newsroom mailing list as, by the end of the year, we will drop using the mailing list originally compiled in MailChimp. It allows for a good integration in the CP. After the centralisation, the newsletter will be automatically compiled from news stories posted on the CP by the thematic lots.
• Helen Lillis (JNCC, EMODnet Seabed Habitats): SBH is about to enter a new phase, should we keep both websites up to date in terms of use cases/reports or should we stop updating our website?
  • Francis Strobbe (EMODnet Secretariat): use cases should first be published in the CP, point of access for reports should be the repatriated website but these can be added to the thematic lots’ website.
  • Juan Carlos Fernandez (CINEA): advised to maintain the website and reinforce the CP but keep doing the same work as before.
  • Helen Lillis (JNCC, EMODnet Seabed Habitats): is there a new template to submit new use cases?
  • Francis Strobbe (EMODnet Secretariat): we're working on it and the tagging. Would be good to have a session later in the year to explain how this will work.
  • Helen Lillis (JNCC, EMODnet Seabed Habitats): You can filter reports by Theme, but they are all grouped together and should be ordered to have the active ones at the top, while leaving the older ones (e.g. checkpoints) lower at the bottom

[Action Item 1] Francis Strobbe (EMODnet Secretariat): to create a JIRA ticket and follow up with Helen Lillis (JNCC, EMODnet Seabed Habitats) on making a better ordering of the reports

• Dick Schaap (Maris, EMODnet Bathymetry, Chemistry and Data Ingestion): the future will we be using the EMODnet logo? It's part of our identity.
  • Jan-Bart Calewaert (EMODnet Secretariat): For the time being we keep using the logo, but it has to have the European flag in it. Until further notice that is the logo we should use. There are ongoing discussions between the different DGs in order to preserve our brand and visual identity.
  • Francis Strobbe (EMODnet Secretariat): shared that official logos are available through the CP https://emodnet.ec.europa.eu/en/logos .
• Jan-Bart Calewaert (EMODnet Secretariat) also asked for the lots to establish a dialogue with the Central Portal team to work on the static content presented. For now, the CP and thematic portals will exist in parallel. They can be updated but don't invest too much effort (especially structural changes) as it will in the future fade out and live on as a micro site in the Central Portal.
  • Cécile Nys (EMODnet Secretariat), a new colleague of the EMODnet Secretariat, is doing a mapping of the pages in the thematic portals to the site structure of the CP.

• Alessandra Giorgetti (OGS, EMODnet Chemistry): all our portals have a section with information on data, products. Can these things remain/be considered in the CP.
  • Conor Delaney (EMODnet Secretariat): products will appear in the GeoNetwork, so metadata should be well described.
• Conor Delaney (EMODnet Secretariat): We don't have control over the menu structure, but we are mapping as much of the content as we can into this fixed structure.
  • Joana Beja (VLIZ, EMODnet Biology): mentions there is an EMODnet GitHub where Biology has a lot of information on how the products are derived. Perhaps this is also a solution for the other thematic portals. Biology can share their expertise.
  • Tim Collart (EMODnet Secretariat): In the Central Portal, we are currently forking and keeping the Thematic Portal GitHub repositories updated.

New CP GeoNetwork and its impact on product sharing

• Bart Vanhoorne (VLIZ) gave an overview of the updated GeoNetwork (VLIZ) - Geonetwork updates
  • Upgraded GeoNetwork to facilitate a number of functionalities. We will be reviewing the product and data catalogues as part of centralisation so we have more complete metadata and compliant with our requirements but also external parties;
  • Metadata has always been harvested automatically from all lots via CSW in the CP GeoNetwork. Lots’ remain responsible for managing and updating their metadata, and exposing it through CSW;
  • Currently it only includes the metadata for products, but we plan to expand this to metadata on the datasets as well (as well as the service metadata);
  • Link to EU Open Data Portal is done via a DCAT-AP protocol which will be implemented with the GeoNetwork update. This is not a standard/default GeoNetwork functionality;
  • Current version is 3.0.4 as we were looking into implementing the DCAT-AP protocol we noticed that version 4.0 did not include all the functionalities included in version 3;
  • Contacted gim.be (https://www.gim.be/en) as they have implemented this protocol to request advice on what the best GeoNetwork version to use;
  • The metadata updates that are need to leverage the features of the new GeoNetwork (EMODnet Secretariat);
  • Demo version available on: https://emodnet.development.ec.europa.eu/geonetwork/.

A discussion followed on the Product Catalogue, as follows:

• Dick Schaap (EMODnet Bathymetry and Chemistry): there are download links in the catalogue, listing the original sources. Will this distribution info be updated to the sources from the Central Portal Data catalogue?
  • Bart Vanhoorne (VLIZ): yes indeed.
  • Conor Delaney (EMODnet Secretariat): for now, we will keep the Portal Web Services live. We are assessing this on a case by case scenario as for some products it might be quicker to keep these services alive for downloads/query purposes.
  • Conor Delaney (EMODnet Secretariat): with the new developments and GeoNetwork and the open to the EU Open Data, any mistakes will be reproduced, so thematic lots should be looking at the product metadata.

• Dick Schaap (EMODnet Bathymetry and Chemistry): Is it a problem if the harvest is done from a previous version?
  • Bart Vanhoorne (VLIZ): We have tested and it is working properly. We 're not expecting any problems.
  • Tim Collart (EMODnet Secretariat): We use the CSW standard and it should be agnostic to any version.

Discussion on the Product Catalogue (EMODnet Chemistry)

• Alessandra Giorgetti (EMODnet Chemistry) provided a very short presentation of their catalogue. She analysed how the products appear in the central portal catalogue.
  • It gathers many datasets, making the searching criteria very important so that users can still efficiently find the data;
  • We may need to evaluate if there are duplicates (e.g. from Chemistry and Physics). There may be a need for some harmonisation of the keywords to make this searching more efficient;
  • Some portals implement "STATUS" field which are not present across all portals. There is a need for some instructions from the technical group, so that some of these fields can be harmonised across the portals;
  • Bug in the catalogue identified. no multiple choice in the same fields;
  • Request to have an extended list of keywords (similar to how contacts are presented).

A discussion followed on this Product Catalogue presentation.

• Conor Delaney (EMODnet Secretariat): some aspects can be solved on the CP level but some of the descriptions have to be resolved on the thematic lots' side.
• Bart Vanhoorne (VLIZ): focus was on search options that are currently there. These are the default options. Maybe these are not all relevant and we can remove a few of them. First an agreement is needed on the search criteria that are important to each lot and then work towards standardising those. Doing that for all search criteria is possibly not easily achievable. Will check if the bug has been fixed with the new version.
• Conor Delaney (EMODnet Secretariat): there's a lot of search options that are not relevant.

[Action Item 2] Conor Delaney (EMODnet Secretariat): to ask for the presentation from Alessandra Giorgetti (Chemistry).

[Action Item 3] Bart Vanhoorne (VLIZ): to look into the bug that multiple selections in a field are not possible (as reported by Alessandra Giorgetti (EMODnet Chemistry)) and if it is still present in the new version of the GeoNetwork.

[Action Item 4] CP team action: propose which metadata fields are useful for filtering in the CP GeoNetwork, so that thematic portals can focus their efforts to make that metadata useful.

Discussion on how to improve the reporting of geospatial indicators Progress monitoring: geographic coverage

From Phase III to Phase IV there are changes on the geographical coverage. Alerting and reminding of these changes and these will be analysed in due time.

• Alessandra Giorgetti (EMODnet Chemistry) presented what indicators Chemistry are collecting and reporting. They indicate that for Indicator 5, they have a form to collect downloads from their own catalogue, but have no data from data downloaded through the CP GeoNetwork. For the direct download, they are using the Marine ID instead of a download form. AG indicates that if we want to continue collecting data for indicator 5, a form needs to be implemented for the CP products catalogue.
  • Conor Delaney (EMODnet Secretariat): indicates there are some problems with implementing forms in the new repatriated website.
  • Bart Vanhoorne (VLIZ): We may be able to implement it for the Map Viewer, but not (easily) within the Geonetwork catalogue.
  • Jan-Bart Calewaert (EMODnet Secretariat): suggests that the central portal team (Secretariat & VLIZ) has a lot of information on how the stats are collected and this is already available and the CP team should summarise this in a document, so that we can make a decision of which indicators we will carry on to the future.

[Action Item 5] CP team to create a mapping of the different ways the indicators are collected across the portals as basis for a discussion for the future indicators.

Gifts were presented to Iain Shepherd (senior advisor to EC DG MARE) to recognise his long-standing work, commitment and vision to EMODnet and marine knowledge activities over the past decade and beyond.

Friday 10th September 2021

09:00 – 12:00 EMODnet TWG

EMODnet Centralisation. Progress performance monitoring, reporting and harmonisation (Secretariat, VLIZ, & All)

Centralisation technical overview: A deep dive into the technologies behind the new central portal map viewer

• Bilbomatica presented the workflow for the development of the Central Portal Visualisation Tool.
  • Natalia Orio Moreno (Bilbomatica): Showed the workflow of the development in a schematic image. First step was a feasibility study. The best option was to start a new viewer from scratch, but building on the EMODnet Bathymetry map viewer functionalities (listed in the presentation). An image of the technical architecture was shown. The viewer reads the configuration from a JSON file, generated by the MetaGIS CMS maintained by VLIZ. This workflow includes development and testing on the Bilbomatica test environment until a stable version is prepared. Once a version is released it is deployed on the VLIZ infrastructure (the staging environment).
  • Clara Becares (Bilbomatica): Gave an overview of the features/functionalities, prioritised in three batches (1,2,3). The first version was released in June, having all the Priority 1 functionalities of which Clara gave a live demo.
    • Dick Schaap (EMODnet Bathymetry and Chemistry): Please have active coastline as default when using the EMODnet World Base Map as otherwise several low-lying areas such as part of the Netherlands seem no longer land, but sea

[Action Item 6] Conor Delaney (EMODnet Secretariat): to make a Jira Ticket displaying the Bathymetry provided coastline on top of the EMODnet Bathymetry world base layer, as requested by Dick Schaap (EMODnet Bathymetry and Chemistry).

• Clara Becares (Bilbomatica): The second version is now released, implementing the Priority 2 functionalities, which were also presented in a live demo
  • Graeme Duncan (Sea Bed Habitats): Is the measurement a geodesic calculation? Just wondering regarding its "accuracy".
    • Bart Vanhoorne (VLIZ): yes, follow link to see specification (https://openlayers.org/en/latest/apidoc/module-ol_sphere.html).
  • Graeme Duncan (Sea Bed Habitats): One additional piece of functionality that the Seabed Habitats map has that we've received feedback of its usefulness is the ability to create a URL that you can share that will return the current "state" of the portal. Is this something that is being considered for future development of the central portal? Our users like being able to load layers and move to a region of interest, and then email the link across to colleagues to bring context to emails. (Maybe this is already a priority 3?).

[Action Item 7] Bart Vanhoorne (VLIZ): To look into URL sharing options for the new viewer. A system is already in place for the current/legacy map viewer.

• Thierry Schmitt (SHOM, EMODnet Bathymetry): Nice work! Concerning the Getinfo feature on the Bathymetry reference map, it was not clear to me if it is possible to reach the metadata of the corresponding contribution/DTM source (XML of the CDI or CPRD).
• Marco Alba (ETT, EMODnet Physics): The share link is also a functionality implemented in Physics that users like.
• Bart Vanhoorne (VLIZ) presented some slides on the back end for the download functionality:
  • The Download functionality is implemented in the viewer for an integrated user experience.
  • Requirement: all products need to be in a central ERDDAP server (for raster data/products), and a central GeoServer/PostGIS (for vector data/products)
  • Alessandra Giorgetti (OGS, EMODnet Chemistry): Can I have a clarification? Is it possible to go from the layer information (or from metadata) to the Central catalogue, containing the detailed description of the layer? And is it possible to make multiple choices from different thematic lots?
    • Bart Vanhoorne (VLIZ): This is exactly what the "more info" button does, as explained by Natalia.
    • Clara Becares (Bilbomatica): Yes, it is possible. We just need to change the config.json file and add the right configuration.
• Different functionalities based on requirements for lots
  • Bathymetry:
    • Merged 58 tiles in 1 ERDDAP layers (further explained by Fred);
    • Separate tiles downloadable, but we plan to make them available through the catalogue (GeoNetwork);
    • the Access to CDI's will be provided by linking to the SDN portal;
  • Human Activities:
    • Multiple Geotiffs for vessel density will be turned in the netCDF file format that can be queried by time dimensions;
    • Vector data will go into the central GeoServer;
    • Extra filters need to be applied;
  • Biology (see presentations):
    • Schematic overview of central portal cache architecture. Still have to define the synchronisation with portal PostGIS servers.
• Marco Alba (ETT, EMODnet Physics): If gridded data in ERDDAP need WMS functionality (the WMS in ERDDAP is very basic), ncWMS is a very good tool to provide them (is the WMS server used also in THREDDS)
• Frederic Leclercq (VLIZ) presented the ERDDAP developments. It is possible to merge the tiles from Bathymetry, allowing users to query across tile boundaries. There are however limitations to the bounding box that can be queried.
  • Dick Schaap (MARIS, EMODnet Bathymetry and Chemistry): Good that you managed to merge the files into one DTM, and allowing users to query from the map based on a bounding box. Great achievement! However, what formats can be exported? ERDDAP can export a number of file formats already, with exception of the EDMO format. While this could be written in ERDDAP, it may be a better solution to just offer this format from the GeoNetwork Catalogue?

• Graeme Duncan (JNCC, Sea Bed Habitats): Very nice infrastructure diagram. I see there is a direct PostgreSQL to PostgreSQL connection involved for harvesting vector data from the lots, is this being considered over harvesting via WFS for technical reasons? I will warn our digital and data team of the possibilities as classically our PostgreSQL database is pretty strictly locked down, so it will need some liaison to alter the security to allow the central server to access.
  • Joana Beja (VLIZ, EMODnet Biology): Might be useful to have those colleagues in a meeting with the CP technical team so as to explain and discuss the best way forward.
  • Graeme Duncan (JNCC, Sea Bed Habitats): Sounds good.
  • Joana Beja (VLIZ, EMODnet Biology): - I will invite them in when we have our first meeting on the topic.
  • Tim Collart (EMODnet Secretariat): I think we may be able to do an ETL approach to harvest the WFS in our PostGIS. We can also consider to use the OGR Foreign data wrapper, see link (https://github.com/pramsey/pgsql-ogr-fdw).
• Graeme Duncan (Sea Bed Habitats): At Seabed Habitats we're moving towards raster storage in S3 (Object Storage on the Internet) using the COG raster type (COG = Cloud optimised GeoTiff), is this an extra option that could be considered? It seems to be relatively "up and coming" as it were. We don't directly expose the COG GeoTiffs yet, but it could be a direction of travel. It's relatively simple to create a COG from a standard GeoTiff through ogr2ogr, it hasn't added much overhead to our pipelines.
• Dick Schaap (EMODnet Bathymetry and Chemistry): asks when the thematic coordinators will be able to be tested.
  • Fred Leclercq (EMODnet Central Portal): Email was sent the Monday before the TWG meeting with a link and login details.
  • Joana Beja (VLIZ, EMODnet Biology): As you're "playing" with the viewer, please do send us any issues you might find so these can be addressed. As the email mentions, please focus on existing functionalities, not what is not yet there.
• Erik Geletti (OGS, EMODnet Chemistry) Will the map viewer be open source: Needs to be discussed with Bathymetry and VLIZ/Bilbo team but they believe this is a good strategy.

Next Technical Working Group Meeting: format, date and location (Secretariat & all).

The next TWG meeting will be in six months from now. Date and location still to be decided

AOB - NONE

END OF MEETING

Annex I: List of Participants