Developer Guide

Note

If you have any questions during set up or development post on our discussion board and we’ll answer them.

This project uses the GitHub Flow (more detail below) for collaboration and consists primarily of Python code and Javascript code. A variety of tools are used to aid in its development. Below is a brief list of the most commonly used tools:

Tool

Used For

Git

version control

Nox

automating development tasks.

PyTest

executing the Python-based test suite

pre-commit

helping impose basic style guidelines

NPM

managing and installing Javascript packages

Selenium and ChromeDriver

to control the browser while testing

GitHub Actions

hosting and running our CI/CD suite

Docker and Heroku

containerizing and hosting this documentation

Making a Pull Request

To make your first code contribution to IDOM, you’ll need to install Git (or Git Bash on Windows). Thankfully there are many helpful tutorials about how to get started. To make a change to IDOM you’ll do the following:

Fork IDOM:

Go to this URL and click the “Fork” button.

Clone your fork:

You use a git clone command to copy the code from GitHub to your computer.

Create a new branch:

You’ll git checkout -b your-first-branch to create a new space to start your work

Prepare your Development Environment:

We explain in more detail below how to install all IDOM’s dependencies

Push your changes:

Once you’ve made changes to IDOM, you’ll git push them to your fork.

Create a Pull Request:

We’ll review your changes, run some tests and equality checks and, with any luck, accept your request. At that point your contribution will be merged into the main codebase!

Development Environment

In order to develop IDOM locally you’ll first need to install the following:

Note

NodeJS distributes a version of NPM, but you’ll want to get the latest

Once done, you can clone a local copy of this repository:

git clone https://github.com/idom-team/idom.git
cd idom

Then, you should be able to run the command below to:

  • Install an editable version of the Python code

  • Download, build, and install Javascript dependencies

  • Install some pre-commit hooks for Git

pip install -e . -r requirements.txt && pre-commit install

If you modify any Javascript, you’ll need to re-install IDOM:

pip install -e .

However you may also cd to the src/idom/client/app directory which contains a package.json that you can use to run standard npm commands from.

Running The Tests

The test suite for IDOM uses Nox and NPM in order to validate:

  1. Server-side Python code with PyTest

  2. The end-to-end application using Selenium in Python

  3. Client-side Javascript code with UVU

Running Python Tests

To run the full suite of Python tests you’ll need to install:

Warning

Be sure the version of Google Chrome and ChromeDriver you install are compatible.

Once you’ve installed the aforementioned browser and web driver you should be able to run:

nox -s test

If you prefer to run the tests using a headless browser:

nox -s test -- --headless

You can pass other options to pytest in a similar manner:

nox -s test -- arg --flag --key=value

Running Javascript Tests

If you’ve already run npm install inside the src/idom/client/app directory, you can execute the suite of workspace tests under packages/* with:

npm test

As a final check, you might want to run npm run build. This command is run in the top-level setup.py installation script for the Python package, so if this command fails, the installation of the Python package with pip will too.

Code Quality Checks

Several tools are run on the Python codebase to help validate its quality. For the most part, if you set up your Development Environment with pre-commit to check your work before you commit it, then you’ll be notified when changes need to be made or, in the best case, changes will be made automatically for you.

The following are currently being used:

  • MyPy - a static type checker

  • Black - an opinionated code formatter

  • Flake8 - a style guide enforcement tool

  • ISort - a utility for alphabetically sorting imports

The most strict measure of quality enforced on the codebase is 100% coverage. This means that every line of coded added to IDOM requires a test case that exercises it. This doesn’t prevent all bugs, but it should ensure that we catch the most common ones.

If you need help understanding why code you’ve submitted does not pass these checks, then be sure to ask, either in the Community Forum or in your Pull Request.

Note

You can manually run nox -s format to auto format your code without having to do so via pre-commit. However, many IDEs have ways to automatically format upon saving a file (e.g.`VSCode <https://code.visualstudio.com/docs/python/editing#_formatting>`__)

Building The Documentation

To build and display the documentation simply run:

nox -s docs

This will compile the documentation from its source files into HTML, start a web server, and open a browser to display the now generated documentation. Whenever you change any source files the web server will automatically rebuild the documentation and refresh the page. Under the hood this is using sphinx-autobuild.

To run some of the examples in the documentation as if they were tests run:

nox -s test_docs

Building the documentation as it’s deployed in production requires Docker. Once you’ve installed, you can run:

nox -s docs_in_docker

You should then navigate to to see the documentation.

Release Process

  1. Update version

  2. Add changelog entry

    • Include merged pull requests

    • Include closed issues

  3. Commit final release changes

  4. Create a release tag

  5. Manually author a release in GitHub

Update Release Version

To update the version for all core Javascript and Python packages in IDOM run:

nox -s update_version -- <new-version>

Note

The new version must adhere to SemVer. Once IDOM becomes stable we will shift to using CalVer.

Create Changelog Entry

Immediately after updating the version you’ll need to create a changelog entry for the release. This should always include a human authored summary of the changes it includes. For example, new or deprecated features, performance improvements, and bug fixes (whatever is relevant). To help with this, there are some useful tools for gathering the Pull Requests and Issues that have been merged and resolved since the last release. While reviewing these items can help in writing a human authored release summary, you must not resort to a bullet list of Pull Request and Issue descriptions. Putting these two together, the format of a changelog entry should look a bit like this:

:release:`X.Y.Z`
----------------

The release summary...

**Closed Issues**

- Some issue - :issue:`123`
- Another issue - :issue:`456`

**Pull Requests**

- Some pull request - :pull:`123`
- Another pull request - :pull:`456`

**Deprecated Features**

- Description one
- Description two

To create the list of pull requests and closed issues you can copy the output of the following commands using the <format> of your choosing (rst, md, text):

Note

You should currate the list - not everything needs to be included.

nox -s latest_closed_issues -- <format>
nox -s latest_pull_requests -- <format>

Once the version has been updated and the changelog entry completed, you should commit the changes.

Creating The Release

The final release process involves two steps:

  1. Creating a tag for the release

  2. Authoring a release in GitHub

To create the release tag you can run the following command:

Note

To just create the tag without pushing, omit the trailing push argument

nox -s tag -- push

Last, you must create a “Release” in GitHub. Because we pushed a tag using the command above, there should already be a saved draft which needs a title and desription. The title should simply be the version (same as the tag), and the description should, at minimum, be a markdown version of the already authored Changelog summary.

Other Core Repositories

IDOM depends on, or is used by several other core projects. For documentation on them you should refer to their respective documentation in the links below: