Development

Install the development environment with Docker

If you want to develop new Agilepy features or try the newest yet unreleased ones, a development Docker image called agilepy-recipe is availabe on DockerHub. It contains all the dependencies but Agilepy, which must be manually installed by cloning the repository.

Agilepy’s development containers can be found at dockerhub agilescience/agilepy-recipe page, please check it for the latest tag.

Instructions

1. Pull the development Docker image, replace <LATEST-TAG> with the latest tag available at agilepy-recipe/tags.

export LATEST_TAG=BUILD25b6-v3
docker pull agilescience/agilepy-recipe:$LATEST_TAG

2. Prepare your workspace by creating a directory agilepy_development and store its path in a variable $PATH_TO_AGILE. The agilepy_development directory is going to be shared between your local file system tree and the developement container’s one.

mkdir agilepy_development && cd agilepy_development

3. Clone the GitHub Agilepy repository, switch to the development branch you are interested to work on (e.g. develop or any other branch with a given feature).

git clone https://github.com/AGILESCIENCE/Agilepy.git && cd Agilepy && git switch develop && cd ..

4.a For Linux users. Execute the boostrap_dev.sh script to change the user inside the container to your local user.

./Agilepy/agilepy/scripts/bootstrap_dev.sh $LATEST_TAG
export IMAGE_NAME="${LATEST_TAG}_$(whoami)"
export CONTAINER_NAME="agilepy_dev_$(whoami)"
export CONTAINER_JUPYTER_PORT=8090

4.b. For Mac users.

export IMAGE_NAME="${LATEST_TAG}"
export CONTAINER_NAME="agilepy_dev_$(whoami)"
export CONTAINER_JUPYTER_PORT=8090

  1. Create a Docker container with name $CONTAINER_NAME from the Docker image.

docker run --rm -t -d -p $CONTAINER_JUPYTER_PORT:8888 --name $CONTAINER_NAME -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix:rw -v $(pwd)/Agilepy:/home/flareadvocate/Agilepy agilescience/agilepy-recipe:$IMAGE_NAME

  1. The command above shares the agilepy_development directory between host and container.

  2. Mount any additional volumes you need to share with the container with the -v option.

  3. The command above binds port 8888 of the container to port $CONTAINER_JUPYTER_PORT of your local host, change it if already in use.

  4. If you have problem with the network connection, add the --network=host option.

  1. Enter the container with:

docker exec -it $CONTAINER_NAME bash -l

  1. Inside the container move to the repository location and install Agilepy’s Python dependencies and Agilepy in editable mode:

python3 -m pip install -r Agilepy/requirements.lock
python3 -m pip install -e Agilepy

Now you have the Agilepy’s latest development version installed in your environment. You can also edit it to implement your own agilepy features!

  1. The documentation can be built with:

cd Agilepy/docs
make html
cd $HOME

  1. If you need to start a Jupyter server run the following command:

nohup jupyter-lab --ip="*" --port 8888 --no-browser --notebook-dir=$HOME/Agilepy/agilepy/notebooks > jupyterlab_start.log 2>&1 &

  1. Use --NotebookApp.token='CUSTOM TOKEN' or --NotebookApp.password='CUSTOM PASSWORD' to set a custom authentication token or password.

  2. The notebook will be available at localhost:8090

  3. If the remote machine needs authentication you can set an ssh tunnel with: ssh -N -f -L localhost:8090:localhost:8090 <user>@<remote_machine>

  4. To obtain the Jupter access token from outside the container: docker exec $CONTAINER_NAME /home/flareadvocate/.local/bin/jupyter server list

  1. The unit tests can be started with the following command:

start_coverage.sh

  1. When you need to exit the container just enter exit.

  2. To stop the container use

docker stop $CONTAINER_NAME

Example of development deployment

This document describes an example of development deployment of Agilepy on agilehost3.

Docker images

To build the docker image of Agilepy clone the [Agilepy-recipe repository](https://github.com/AGILESCIENCE/Agilepy-recipe)

Git flow

Branches

Two main branches:

  • master: contains only production releases.

  • develop: contains commits that will be included in the next production release.

Two support branches:

  • feature branch: each new feature (Trello’s card) should be developed in its own feature branch, branching from develop and merged back into it. The feature branch are not pushed into the remote.

  • hotfix branch: if an hotfix is needed it should be develop in its own branch, branching from master and merged back to it.

Git flow

Versioning

The master branch contains only production releases: when the develop branch (or hotfix branch) is merged to master a new release tag must be created. Its name follows the semantic versioning.

x.y.z

Incrementing:

  • x version when you make incompatible API changes,

  • y version when you add functionality in a backwards compatible manner, and

  • z version when you make backwards compatible bug fixes.

Branches names

The master and the develop branch have an infinite lifetime, hence their name is fixed.

The feature branch takes the following format:

feature-#<card-number>-<short-description>

e.g. feature-#61-new-cool-feature

The hotfix branch name takes the following format:

hotfix-#<card-number>-<release-number>

e.g. hotfix-#57-1.0.0

The release number is the one of the production release from which it originates from.

Getting started

Development of a new feature

Create a new feature branch:

git checkout develop
git pull origin develop
git checkout -b feature-#61-new-cool-feature develop

Development and testing of the new feature.

When you have finished, update the CHANGELOG.md and commit your changes.

vim CHANGELOG
git commit -m "feature-#61-new-cool-feature done"

In the meantime it is possible that someone else have pushed his work into the develop branch. In this case you have to merge the changes in your feature branch.

git pull **origin** develop

Finally, you can open a merge request to merge your feature branch back to the develop branch.

Add configuration parameters

Let’s say we want to add the following configuration section to the AGAnalysis’ configuration file.

ap:
    radius: 0.25
    timeslot: 3600

  • Add the new section to the AGAnalysis.getConfiguration() method.

  • Add the type of the configuration parameters within the AGAnalysisConfig.checkOptionsType() method (in the corresponding lists).

  • If the parameters need some kind of validation (this is not the case), add a new method in ValidationStrategies and call it within the AGAnalysisConfig.validateConfiguration() (check examples).

  • If the parameters need some kind of transformation (this is not the case), add a new method in CompletionStrategies and call it within the AGAnalysisConfig.completeConfiguration() (check examples).

  • Add the new configuration section to all the unit test configuration files.

  • Document the new configuration parameters within the manual/configuration_file.rst file.

Add a new science tool

Let’s say we want to add a new (c++) science tool: AG_ap.

  • Add a new class within the api/ScienceTools.py script. You need to implement some abstract methods.

  • You can use the new class as follows:

apTool = AP("AG_ap", self.logger)
apTool.configureTool(self.config)
if not apTool.allRequiredOptionsSet(self.config):
    raise ScienceToolInputArgMissing("Some options have not been set.")
products = apTool.call()

Release of a new version

Change the version of the software in setup.py. The version increment must be take in account all the commits of the develop branch. You can check the CHANGELOG.md to facilitate this process. Please, add the new tag within the CHANGELOG.md file.

git checkout master
git merge --no-ff develop
git tag -a <new-tag>
git push origin <new-tag>

DevOps

A high level description of agilepy’s devops is in the image below:

Git flow

This scheme workflow produces three images:

  • base_image: It’s an image with all the dependencies except Agilepy python library, it’s used for developing purposes only by developers. Base image is built after a new commit in agilepy-recipe repository.

  • latest code image: It’s the base_image with Agilepy’s develop branch at latest commit, useful for using or testing agilepy’s updates not officially released. This image is not supported nor stable and is built by dockerhub after github’s testing pipelines are successful.

  • released image: The base_image with Agilepy’s release tag. By default the community shall be download this image. It’s built when a new tag is created.