SciForge API: Getting Started

Development takes place in a dockerized environment that is nearly identical to the production system. While it is possible to develop without docker, training covers the recommended approach. LabLynx developers generally must develop within the dockerized environment.

Scripts are used to start and stop the SciForge development environments. Starting the development environment makes the GraphQL sandbox available. Changes may be made to code while the environment is running, and these changes will auto-load in the running environment.

Video: Sciforge API_ Getting Started.mp4

Linux Users

The use of docker permits the use of a shared development server, where multiple users may each be working independently. Development normally occurs under the user account of the logged in user. However, docker commands need to be executed under “sudo” (or equivalent).

Docker Initialization

Docker will normally be installed by an administrative user, such as root.

The scripts that start and stop the development API utilize swarm mode. Prior to the first usage, swarm mode should be initialized, and several docker networks created or edited. Commands used for this are below.

sudo docker swarm init;
sudo docker network ls
sudo docker network rm ingress
sudo docker network create --ingress --driver overlay --opt encrypted 
  --subnet 10.10.0.0/16 ingress
sudo docker network create --subnet 10.11.0.0/16 --driver overlay 
  --scope swarm --opt encrypted --attachable cloud-edge
sudo docker network create --subnet 10.12.0.0/16 --driver overlay 
  --scope swarm --opt encrypted --attachable cloud-socket-proxy
sudo docker network create --subnet 10.13.0.0/16 --driver overlay 
  --scope swarm --opt encrypted --attachable swarm-web
sudo docker network ls will list the configured docker networks, such that the above networks can be verified as being present or not.

Getting the code

Login to your dev box
Create a project directory (up to you how you like to organize) that development will occur in.
Clone the SciForge COTS repository into your project dir.
https://bitbucket.org/team_lablynx/elab-api/src/master/
Navigate to the ./elab-api/src/ directory, and clone the plugin repository.
Rename the cloned plugin directory to “plugin”
Verify the shell scripts have the executable file attribute

Example commands:

git clone https://[someuser]@bitbucket.org/team_lablynx/elab-api.git
cd elab-api/src
git clone https://[someuser]@bitbucket.org/team_lablynx/training-api-plugin.git
mv training-api-plugin plugin
cd .. #back to the elab-api directory
chmod +x *.sh

Replace the “training-api-plugin” with your plugin

Replace [someuser] with your username. This is most easily obtained by copying the clone command from Bitbucket via the “Clone” button on the “Source” page of the repository.

Https cloning is not required, just used as an example. The setup of keys for SSH cloning is beyond the scope of these training sections.

You will be prompted for your Bitbucket password, if using https to clone. Note that this is not your account password. Bitbucket requires you to setup an “app password.” Your account password authenticates against the Bitbucket site. Your app password authenticates against repositories.

You can being to familiarize yourself with code at this point. A later section will get into starting and stopping the development environment. First, there remains some additional setup.

LabLynx uses VS Code and enables X11 forwarding for coding. With this application, typing “code” on the command line brings up VS Code with the last project you had loaded. “code -n” will start VS code without loading projects, allowing you to choose which directory to work from.

Traefik & Let’s Encrypt

Production environments do not directly expose the API to the internet. Instead, an application called Traefik is used as a reverse proxy to the API. SciForge is one of several application hosted on our SciCloud server platform. Applications are enabled as needed. Traefik performs several functions, to help secure the traffic:

Only https traffic is permitted. Http traffic is captured and redirected to https for a seamless user experience.
Only ports 80 and 443 require public exposure; all other ports may be closed. Applications, such as SciForge, that internally work on other ports are proxied via port 443.
Traefik ensures that the Web certificates that https traffic requires are automatically issued and kept up to date.

A graphical representation of the public and docker network configuration for a production SciCloud server instance is shown below.

It is best practice that the development environment be as similar to the production environment as possible. Nearly every developer, at some point, has experienced an unanticipated problem that arises because of differences in the development and production environments. Maintaining this best practice becomes increasingly relevant for customized software, where only a single entity may be using a particular codebase and may request frequent changes (ie. the SciForge plugin, in this context).

This best practice is the primary reason that development occurs in a dockerized environment. It is also the reason that the use of Traefik is included in the development environment.

At the time of writing, Traefik is using version 1.7; there is work in progress that will update this to version 2. When these changes are pulled from the main SciForge repository, some updates to the Traefik configuration discussed below will likely be necessary.

Let’s Encrypt has made a variety of good documentation available. In the context of SciForge development and the use of this via Traefik, the HTTP challenge is used (see Challenge Types ). For this challenge to work:

Traefik will be configured with a web address that it operates from.
Let’s Encrypt will try to retrieve a token from Traefik, via web requests, on notification from Traefik that the token is ready. It will use the web address that Traefik operates under. Since Let’s Encrypt will try to reach Treafik over the internet, this requires that
- The Traefik web address has a DNS pointing at a publicly available address.
- Traffic for port 80/443 is allowed from the internet to the development machine.

Traefik Setup

In the event that the development environment is shared, or multiple dockerized projects exist, only one Traefik instance should exist. It only needs to be setup once. Multiple SciForge (or other dockerized apps) may be run simultaneously, as long as they have distinct, unique addresses.

In the SciForge COTS repository, there is a directory labeled “traefik,” with two files:

docker-compose.yml
traefik.toml

Steps:

Use an administrative account such as sudo or root
Create the directory Traefik will be setup in.
- This directory can be named and located as you decide is appropriate.
- This directory should not be located within the code repository
- Ideally, it will not be located under a user’s home directory, unless that user is the only individual that will be using the computer.
- As an example, LabLynx typically uses a directory of /opt/scicloud/traefik.
Copy the two Traefik files noted just above into this directory
Create an “acme.json” file in the same directory, and set permissions to 600

Example:

sudo mkdir -p "/opt/scicloud/traefik/"
#Copy the two files noted above to the new directory, using sudo
sudo touch "/opt/scicloud/traefik/acme.json"
sudo chmod 600 /opt/scicloud/traefik/acme.json

Edit the traefik.toml file. There are two settings to pay special attention to
- Domain: This is the address that Traefik will talk to Let’s Encrypt from, and which Let’s Encrypt will request validation tokens from. This does not need to be under the scicloud domain, however, if desired, a request can be made to LabLynx to add an A or CNAME record, provided the desired subdomain is available for use.
- Email: Adjust this to an appropriate email address

Verify file permissions. All files in the treafik folder should (typically) be owned by root, and owner should have read/write access. Read access may be provided to other groups at your discretion.
Deploy the Traefik Stack into Docker Swarm. For example:

cd /opt/scicloud/traefik/
sudo docker stack deploy -c docker-compose.yml dev
#dev is the name of the service in this example. Can be any desired value.

Traefik should now be running. This can be verified with commands such as:

sudo docker service ls
sudo docker ps