Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Noteinfo

Work EWP in progressGitHub

GitHub is a platform and cloud-based service for software development and version control using Git, allowing developers to store and manage their code and documentation. It provides the distributed version control of Git plus access control, bug tracking, software feature requests, task management, continuous integration, and wikis. It is commonly used to host open source software development projects.

...

All specifications and code in GitHub are licensed under MIT License, see e.g. https://github.com/erasmus-without-paper/ewp-specs-architecture/blob/stable-v1/LICENSE.

...

This document contains the general information about GitHub repositories of the EWP Network, information about repositories with the description of the network architecture and security protocols, repositories related to the EWP Registry, and repositories which contain a source code.

All active GitHub repositories related to the EWP are documented on a separate page: Documents and specifications.

...

Table of Contents
minLevel1
maxLevel6
outlinefalse
typelist
printablefalse

...

Reporitories Repositories with APIs are listed in a separate section, with a short description of their content.

Some supplementary repositories are listed and explained below.

...

Introduction to EWP Network

Architecture and common datatypes

Description of EWP’s Architecture and Security

This document describes EWP vocabulary, network components, common features and data types, and explains basic rules about how partners communicate with each other and handle errors.

Authentication and security

Authentication and security in the context of the EWP Network is described in this GitHub repository. The security of the communication is split into four security aspects, for which the following methods are available:

Each endpoint of each API must support at least one method of covering each of these security aspects. Before making the request, clients verify security protocols supported by the server, and construct their request appropriately.

...

Registry

The following repositories in GitHub are related to the EWP registry:

...

The Registry Service is described in GitHub. It is the only centralized part of the EWP architecture. It allows all EWP hosts to access the list of other EWP hosts and to list APIs implemented by these hosts, along with the URLs these APIs are implemented at.

...

The major advantage of such automatic updating is that the partners do not need to contact the registry maintainer when they want to change some of their Registry entries. Most changes in the registry can be performed simply by updating the manifest on the partner's server (and the Registry will fetch these changes automatically). This process is both more convenient and more secure than updating the registry manually.

Working with EWP technical documentation

Workibg with EWP Tchnical Documentation

This document explains how to work with the technical documentation within the EWP project (where to store it, how to submit change proposals, etc.).

Introduction to EWP Network

Architecture and common datatypes

Description of EWP’s Architecture and Security

This document describes EWP vocabulary, network components, common features and data types, and explains basic rules about how partners communicate with each other and handle errors.

Authentication and security

Authentication and security in the context of the EWP Network is described in the GitHub repository. The security of the communication is split into four security aspects, for which the following methods are available:

...

Catalogue file

The Registry Service catalogue file endpoint takes no parameters and returns the response at a specified URL. The response format is described in the catalogue.xsd file.

Security is based on server certificate validation. The registry does not validate the client (all clients are allowed to access the catalogue anonymously).

EWP Registry Catalogue which enables replicating the EWP registry service on multiple servers pulls the most recent version of the catalogue file from ewp-registry-log-prod and ewp-registry-log-dev, respectively.

...

Github source code

Some EWP-related GitHub repositories contain the source code and binaries.

...

The EWP registry can be installed in a local environment for testing local development installation. There is a package ready for download with embedded readme README and HOWTO files.

...

Issues

Each EWP repository has its own, separate issue tracker. If a developer plans to submit a new issue, and the topic of this issue applies to the contents of a single repository only, then it should be submitted in the issue tracker of this repository. There is also one special repository, general-issues, for tracking issues which do not apply to any single project.

...

Info

Open issues in GitHub (if you get a 404 error, then sign in to GitHub first)

Changes in API specifications

The API management cycle in the EWP Network is described in detail in [1]. Here we list the most important stages and activities of the process.

Plan

The business analysts from the ESCI team identify all documents which create a legal and operational framework for the Erasmus+ mobility processes. Based on this, documents like Mandatory Business Requirements, Semantic Interoperability and  User Guide for the designed processes are elaborated. Their role is to define the business goals and set up the business processes in the context of EWP. A roadmap is developed with a detailed plan of further steps, which sets a timeframe for development, testing & validation, and finally deployment in the production network.

Design and develop

The EWP technical team responsible for the API design maps new requirements and the involved processes into the sequence of API requests and responses, API query parameters, data formats of exchanged objects, error responses. They also define the scope of each API, i.e. plan out the capabilities and functionality that the APIs will provide, as well as any constraints or limitations.

First draft ideas are consulted with the internal technical groups from the ESCI team and then with the mobility software providers.

Draft specifications are posted in GitHub repositories where they can be consulted and reviewed by the technical teams. Also change proposals may be submitted in GitHub.

Documentation of the EWP APIs is hosted in GitHub. It contains general description of the API, formal XSD schemas, example XML responses, elaborated test scenarios, and other supporting documents (e.g. official templates, Mandatory Business Requirements).

When the detailed specifications are posted in GitHub, the mobility software providers may start implementing the APIs and the API design team may start developing API validators.

Test

The developed APIs are first tested locally by means of unit testing. Then APIs can be posted in the DEV EWP network. Here the API validators can be used for basic validation that APIs are conformant with the specification. Then teams of providers may test bilaterally. All aspects of the API should be tested: business logic, security, scalability, and compatibility.

Deploy

Once the APIs are developed and tested, in compliance with the regulations of the EWP network (see EWP entry procedure and Technical testing and validation site), they can be deployed in the production network. This deployment stage involves coordinating the process, setting up monitoring and logging, and checking the APIs in the target environment to ensure everything is running smoothly.

Maintain

Maintenance is an ongoing activity and involves monitoring API’s performance, fixing bugs, and releasing updates to keep an API running smoothly. The EWP Monitoring system allows to track behaviour of the nodes in real time in the production network, identify misbehaving nodes and counteract.

Retire

Business goals change, new priorities arise, requirements get reformulated or change priority, technical needs may require change in the network architecture. For such reasons, an API may reach the end of its useful life and needs to be retired. There is a well-defined process in place to ensure a smooth and orderly transition, which includes decommissioning an API, removing all dependencies, and notifying transition plans to API users.

According to the currently agreed rule, upgrades of the existing APIs taking into account new requirements should not happen more often than once a year.

Rules of API design and versioning

In the EWP project we use Semantic Versioning for releases of EWP technical documentation:

  • We assign documents their version numbers only after they are officially approved.

  • Semantic versioning scheme allows developers to easily determine when a breaking change occurs in a particular API.

  • It follows a set of explicit rules.

Here is the summary of the numbering schema. Given a version number MAJOR.MINOR.PATCH, increment the:

  1. MAJOR version when you make incompatible API changes.

  2. MINOR version when you add functionality in a backward compatible manner.

  3. PATCH version when you make backward compatible bug fixes.

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in specifications are to be interpreted as described in RFC 2119 when, and only when, they appear in all capitals, as shown here.

Once a document is released, every subsequent release of such document MUST be accompanied by a changelog. All changes SHOULD be noted in this changelog (and backward-incompatible changes should be additionally highlighted).

Development of the EWP Registry

Process of development of the EWP Registry is described in detail in [2] (Section 2.2 Continuous deployment). The main steps are listed here:

  1. Programmers develop code locally and commit to their local Git repositories.

  2. The code is then committed into the Git/Gerrit stack. Commits trigger Jenkins job which calls Maven for tests followed by a build of the JAR package.

  3. When the code passes all tests and builds with no errors it is presented in Gerrit for peer review.

  4. Once the code is accepted, the change is merged into the repository creating a release.

  5. That triggers another Jenkins job. The job pushes the code to GitHub repository, builds a jar with Maven, requests Docker build and deposits the image in the local Docker registry and in the remote GitHub Docker registry, and then restarts the test registry. The restart of the registry deploys a new Docker image.

The changes at the PROD and DEV catalogue and manifests are git-pushed to the respective repositories for logging and backup purposes:

Source code of other applications (EWP Stats portal, EWP Monitoring system, Registration Portal) is not yet committed to GitHub as it undergoes active development. Not all steps of the programming cycle are yet automated.

Catalogue file

The Registry Service catalogue file endpoint takes no parameters and returns the response at a specified URL. The response format is described in the catalogue.xsd file.

Security is based on server certificate validation. The registry does not validate the client (all clients are allowed to access the catalogue anonymously).

...

...