Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 9 Next »

Work in progress

GitHub is a platform and cloud-based service for software development and version control using Git, allowing developers to store and manage their code and documentation. It provides the distributed version control of Git plus access control, bug tracking, software feature requests, task management, continuous integration, and wikis. It is commonly used to host open source software development projects.

Technical documentation of the EWP Network and the public code are stored in GitHub. There is an “organization” named Erasmus Without Paper which owns a set of repositories in Github located at https://github.com/erasmus-without-paper.

Documentation and source code are publicly available and open-sourced. This means that anyone can read, comment on and contribute to its development. Rules for accepting change proposals are also explained in GitHub.

All specifications and code in GitHub are licensed under MIT License, see e.g. https://github.com/erasmus-without-paper/ewp-specs-architecture/blob/stable-v1/LICENSE.

EWP in GitHub

GitHub repositories

In September 2023 there are 76 repositories in GitHub, 18 of which were archived.

There are 26 public repositories for APIs – their names start with the prefix ewp-specs-api. Typically they follow the same structure and contain (among others) the following elements:

Reporitories with APIs are listed in a separate section, with a short description of their content.

Some supplementary repositories are listed and explained below.

Registry

The following repositories in GitHub are related to the EWP registry:

  1. EWP Registry API. It contains a specification of the EWP Registry API. The Registry API is implemented by the Registry Service.

  2. EWP Registry Service. It contains the Java implementation of the registry server part. There are two public installations of the EWP registry:

    1. PROD Network,

    2. DEV Network.

  3. EWP Registry Client. It contains Java implementation of the EWP Registry Service client.

    1. Developers, who implement a client, can use it to discover institutions in the network and APIs implemented by them.

    2. Developers, who implement a server, can use it to verify incoming requests (prove that they come from within the EWP network and prove which institutions the requester covers).

  4. EWP Registry Catalogue. It contains the implementation of the application that enables replicating the EWP registry service on multiple servers.

  5. EWP Registry log. It contains a record of changes made to the Registry’s catalogue in the production network (for logging purposes).

  6. EWP DEV Registry log. It contains a record of changes made to the DEV Registry’s catalogue (for logging purposes).

The Registry Service is described in GitHub. It is the only centralized part of the EWP architecture. It allows all EWP hosts to access the list of other EWP hosts and to list APIs implemented by these hosts, along with the URLs these APIs are implemented at.

The registry is being updated automatically. It periodically reads all the information that all EWP Hosts provide within their Discovery Manifest files, and these changes are instantly reflected in the Registry Service responses.

The major advantage of such automatic updating is that the partners do not need to contact the registry maintainer when they want to change some of their Registry entries. Most changes in the registry can be performed simply by updating the manifest on the partner's server (and the Registry will fetch these changes automatically). This process is both more convenient and more secure than updating the registry manually.

Working with EWP technical documentation

Workibg with EWP Tchnical Documentation

This document explains how to work with the technical documentation within the EWP project (where to store it, how to submit change proposals, etc.).

Introduction to EWP Network

Architecture and common datatypes

Description of EWP’s Architecture and Security

This document describes EWP vocabulary, network components, common features and data types, and explains basic rules about how partners communicate with each other and handle errors.

Authentication and security

Authentication and security in the context of the EWP Network is described in the GitHub repository. The security of the communication is split into four security aspects, for which the following methods are available:

Each endpoint of each API must support at least one method of covering each of these security aspects. Before making the request, clients verify security protocols supported by the server, and construct their request appropriately.

Github source code

Some EWP-related GitHub repositories contain the source code and binaries.

  1. EWP Registry Service contains Java implementation of the registry server part. It is distributed as a Docker image.

  2. EWP Registry Client contains Java implementation of the EWP Registry Service client library:

    1. Developers, who implement a client, can use it to discover institutions in the network and APIs implemented by them.

    2. Developers, who implement a server, can use it to verify incoming requests (prove that they come from within the EWP network and prove which institutions the requester covers).

  3. IIAs API repository contains

    1. a folder xslKit with XSLT templates and Java class example;

    2. a code in Go language for hash calculation.

There are two Maven libraries: EWP registry client and Java support library for aes128gcm encryption format, which can be included in the external code. Because GitHub cannot be used for sharing Maven libraries, they are distributed from the sonatype.org public repository.

The EWP registry can be installed in a local environment for testing local development installation. There is a package ready for download with embedded readme and HOWTO files.

Issues

Each EWP repository has its own, separate issue tracker. If a developer plans to submit a new issue, and the topic of this issue applies to the contents of a single repository only, then it should be submitted in the issue tracker of this repository. There is also one special repository, general-issues, for tracking issues which do not apply to any single project.

Issues contain suggestions, ideas, which may be inspiring, some are used actively by developers discussing details of their implementations, and others contain change proposals. Some issues are kept open for a long time as discussions are ongoing.

Open issues in GitHub (if you get a 404 error, then sign in to GitHub first)

Changes in API specifications

The API management cycle in the EWP Network is described in detail in [1]. Here we list the most important stages and activities of the process.

Plan

The business analysts from the ESCI team identify all documents which create a legal and operational framework for the Erasmus+ mobility processes. Based on this, documents like Mandatory Business Requirements, Semantic Interoperability and  User Guide for the designed processes are elaborated. Their role is to define the business goals and set up the business processes in the context of EWP. A roadmap is developed with a detailed plan of further steps, which sets a timeframe for development, testing & validation, and finally deployment in the production network.

Design and develop

The EWP technical team responsible for the API design maps new requirements and the involved processes into the sequence of API requests and responses, API query parameters, data formats of exchanged objects, error responses. They also define the scope of each API, i.e. plan out the capabilities and functionality that the APIs will provide, as well as any constraints or limitations.

First draft ideas are consulted with the internal technical groups from the ESCI team and then with the mobility software providers.

Draft specifications are posted in GitHub repositories where they can be consulted and reviewed by the technical teams. Also change proposals may be submitted in GitHub.

Documentation of the EWP APIs is hosted in GitHub. It contains general description of the API, formal XSD schemas, example XML responses, elaborated test scenarios, and other supporting documents (e.g. official templates, Mandatory Business Requirements).

When the detailed specifications are posted in GitHub, the mobility software providers may start implementing the APIs and the API design team may start developing API validators.

Test

The developed APIs are first tested locally by means of unit testing. Then APIs can be posted in the DEV EWP network. Here the API validators can be used for basic validation that APIs are conformant with the specification. Then teams of providers may test bilaterally. All aspects of the API should be tested: business logic, security, scalability, and compatibility.

Deploy

Once the APIs are developed and tested, in compliance with the regulations of the EWP network (see EWP entry procedure and Technical testing and validation site), they can be deployed in the production network. This deployment stage involves coordinating the process, setting up monitoring and logging, and checking the APIs in the target environment to ensure everything is running smoothly.

Maintain

Maintenance is an ongoing activity and involves monitoring API’s performance, fixing bugs, and releasing updates to keep an API running smoothly. The EWP Monitoring system allows to track behaviour of the nodes in real time in the production network, identify misbehaving nodes and counteract.

Retire

Business goals change, new priorities arise, requirements get reformulated or change priority, technical needs may require change in the network architecture. For such reasons, an API may reach the end of its useful life and needs to be retired. There is a well-defined process in place to ensure a smooth and orderly transition, which includes decommissioning an API, removing all dependencies, and notifying transition plans to API users.

According to the currently agreed rule, upgrades of the existing APIs taking into account new requirements should not happen more often than once a year.

Rules of API design and versioning

In the EWP project we use Semantic Versioning for releases of EWP technical documentation:

  • We assign documents their version numbers only after they are officially approved.

  • Semantic versioning scheme allows developers to easily determine when a breaking change occurs in a particular API.

  • It follows a set of explicit rules.

Here is the summary of the numbering schema. Given a version number MAJOR.MINOR.PATCH, increment the:

  1. MAJOR version when you make incompatible API changes.

  2. MINOR version when you add functionality in a backward compatible manner.

  3. PATCH version when you make backward compatible bug fixes.

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in specifications are to be interpreted as described in RFC 2119 when, and only when, they appear in all capitals, as shown here.

Once a document is released, every subsequent release of such document MUST be accompanied by a changelog. All changes SHOULD be noted in this changelog (and backward-incompatible changes should be additionally highlighted).

Development of the EWP Registry

Process of development of the EWP Registry is described in detail in [2] (Section 2.2 Continuous deployment). The main steps are listed here:

  1. Programmers develop code locally and commit to their local Git repositories.

  2. The code is then committed into the Git/Gerrit stack. Commits trigger Jenkins job which calls Maven for tests followed by a build of the JAR package.

  3. When the code passes all tests and builds with no errors it is presented in Gerrit for peer review.

  4. Once the code is accepted, the change is merged into the repository creating a release.

  5. That triggers another Jenkins job. The job pushes the code to GitHub repository, builds a jar with Maven, requests Docker build and deposits the image in the local Docker registry and in the remote GitHub Docker registry, and then restarts the test registry. The restart of the registry deploys a new Docker image.

The changes at the PROD and DEV catalogue and manifests are git-pushed to the respective repositories for logging and backup purposes:

Source code of other applications (EWP Stats portal, EWP Monitoring system, Registration Portal) is not yet committed to GitHub as it undergoes active development. Not all steps of the programming cycle are yet automated.

Catalogue file

The Registry Service catalogue file endpoint takes no parameters and returns the response at a specified URL. The response format is described in the catalogue.xsd file.

Security is based on server certificate validation. The registry does not validate the client (all clients are allowed to access the catalogue anonymously).

EWP Registry Catalogue which enables replicating the EWP registry service on multiple servers pulls the most recent version of the catalogue file from ewp-registry-log-prod and ewp-registry-log-dev, respectively.

  • No labels