Sciforge API: Development Pipelines

The Sciforge API has two code bases that it can pull from for any given client. This module provides an overview of the software pipeline and automated aspects of code implementation.

The first is our commercial-off-the-shelf (COTS) repository “elab_api”. This is required for any development, though will be read-only when developing plugins. This repository contains all of the out of the box API calls that are necessary for our various services and apps to communicate with our databases successfully.

The second repository is a client specific plugin API. Each client can have a plugin API that will allow them to add handling for custom tables, views, queries, that may be needed in order to access data. The plugin repository may be edited by clients or by LabLynx personnel.

The deployment process is largely automated, and is designed to accommodate changes made by our clients. The process is diagrammed below. Commits to the plugin repository will be automatically pulled by git, depending on the branch, on the test and production SciForge server instances, when the application is next cycled.

Production and Test environments

Production and test are dockerized web sites that are hosted within the SciCloud platform. SciCloud is the name of the hosted services platform. Production and test environments reside on a server called the “SciCloud server.”

Docker images for SciForge are maintained at Docker Hub. There are two images that are maintained; one for production, and the other for test. These images are created using the Bitbucket pipelines technology from code commits into the COTS SciForge repository.

For clients that have plugins, a docker volume is mapped between the plugin repository on the host machine and the plugins directory in the docker container.

The SciCloud platform includes automation that executes when the dockerized stacks are cycled. The cycling must (currently) be initiated by someone with direct access to the server shell. This automation will

Pull the most recent docker image appropriate to that environment.
Pull the most recent plugin code from the configured branch. (per configuration discussion in “Code Architecture” module.

Pipeline for COTS

As noted above, the code repository is named elab-api, and the official, centralized remote is Bitbucket. This codebase is maintained by LabLynx personnel. Pull requests may be made by those with read only access.

As changes are made, the most recent code is cloned or pulled into a development environment. Changes are made in a development branch.

Changes are grouped into a release, which is, for the LabLynx team, equivalent to a sprint.

When changes are ready to be tested, these are committed to the branch. Another developer can then pull these changes to their development environment to review and test.

The changes are then merged into a release branch, with a “first come first serve” policy. The last person to commit has to resolve any conflicts in a manner consistent with all requirements for that release.

A developer then logs into BitBucket, finds the latest commit in the release branch and runs the “UAT” pipeline (user acceptance testing). This will update the docker image used by the test instance. The test instance must then be cycled, which will automatically pull that latest UAT docker image.

Additional testing can then be completed by other team members.

On successful testing, the release branch is merged into the master/main branch. This will automatically, via Bitbucket pipelines, update the production docker image. The next time the SciCloud production stack is cycled, the latest docker image will be pulled and used in production.

Pipeline for plugin

The pipeline for changes to the plugin is very similar to the COTS, but utilizes git instead of docker.

Each client that requires one has a dedicated plugin code repository. The naming convention for these repositories are sciforge_<client ID>.

As changes are made, the most recent code is cloned or pulled into a development environment. Changes are made in a development branch.

When changes are ready to be tested, these are committed to the branch. Another developer can then pull these changes to their development environment to review and test.

A release branch for testing purposes will be created and reused. This may then be referenced in the SciCloud configuration for the test instance. Since both client developers and LabLynx teams may be making changes, it is important to review git logs. Some coordination may be required.

The changes are then merged into the dedicated release branch, with a “first come first serve” policy. The last person to commit has to resolve any conflicts in a manner consistent with all requirements for that release. The test instance must then be cycled, which will automatically pull that latest plugin repository code from the release branch.

Additional testing can then be completed by other team members.

Rollback

It is incredibly important that suitable testing occurs before promotion to the master branch. However, in the anticipation that something at some point goes wrong, both COTS and plugin aspects of production are directly and completely a result of changes made and committed in the repository. Therefore:

A problem with COTS can be rolled back by reverting to the prior commit in the master branch, ensuring the latest docker image is created from that rolled back commit and cycling the SciCloud stack.
A problem with the plugin can be be solved in a similar fashion. Simply revert the last commit, and cycle the SciCloud stack.

As a reminder, the ability to cycle the SciCloud stack currently requires direct access to the SciCloud server shell; LabLynx may need to be alerted to make this happen. There are plans to provide a feature in a management studio in the future that will allow this to be managed by authorized individuals.