Contributing¶

This page provides a guide for developers wishing to contribute to Sphinx-Needs.

Bugs, Features and PRs¶

For bug reports and well-described technical feature requests, please use our issue tracker:
https://github.com/useblocks/sphinx-needs/issues

For feature ideas and questions, please use our discussion board:
https://github.com/useblocks/sphinx-needs/discussions

If you have already created a PR, you can send it in. Our CI workflow will check (test and code styles) and a maintainer will perform a review, before we can merge it. Your PR should conform with the following rules:

A meaningful description or link, which describes the change
The changed code (for sure :) )
Test cases for the change (important!)
Updated documentation, if behavior gets changed or new options/directives are introduced.
Update of docs/changelog.rst.
If this is your first PR, feel free to add your name in the AUTHORS file.

Installing Dependencies¶

Sphinx-Needs requires only Poetry to be installed as a system dependency, the rest of the dependencies are ‘bootstrapped’ and installed in an isolated environment by Poetry.

Install Poetry
Install project dependencies
```
poetry install
```
Install Pre-Commit
Install the Pre-Commit hooks
```
pre-commit install
```
For running tests, install the dependencies of our official documentation:
```
pip install -r docs/requirements.txt
```

List make targets¶

Sphinx-Needs uses make to invoke most development related actions.

Use make list to get a list of available targets.

benchmark-memory
benchmark-time
docs-html
docs-html-fast
docs-linkcheck
docs-pdf
format
lint
test
test-matrix
test-short

Build docs¶

To build the Sphinx-Needs documentation stored under /docs, run:

# Build HTML pages
make docs-html

# Build PDF pages
make docs-pdf

It will always perform a clean build (calls make clean before the build). If you want to avoid this, run the related sphinx-commands directly under /docs (e.g. make docs).

Check links in docs¶

To check if all used links in the documentation are still valid, run:

make docs-linkcheck

Running Tests¶

Hint

Please be sure to have the dependencies of the official documentation installed:

pip install -r docs/requirements.txt
make test

Linting & Formatting¶

Sphinx-Needs uses black and isort to format its source code.

make lint

Benchmarks¶

Sphinx-Needs own documentation is used for creating a benchmark for each PR. If the runtime takes 10% longer as the previous ones, the benchmark test will fail.

Benchmark test cases are available under tests/benchmarks. And they can be locally executed via make benchmark.

The results for each PR/commit get added to a chart, which is available under http://useblocks.com/sphinx-needs/bench/index.html.

The benchmark data is stored on the benchmarks branch, which is also used by github-pages as source.

Running Test Matrix¶

This project provides a test matrix for running the tests across a range of Python and Sphinx versions. This is used primarily for continuous integration.

Nox is used as a test runner.

Running the matrix tests requires additional system-wide dependencies

Install Nox
Install Nox-Poetry
You will also need multiple Python versions available. You can manage these using Pyenv

You can run the test matrix by using the nox command

nox

or using the provided Makefile

make test-matrix

For a full list of available options, refer to the Nox documentation, and the local noxfile.

Our noxfile.py

import nox
from nox_poetry import session

# The versions here must be in sync with the github-workflows.
# Or at least support all version from there.
# This list can contain more versions as used by the github workflows to support
# custom local tests

PYTHON_VERSIONS = ["3.8", "3.10"]
SPHINX_VERSIONS = ["5.2.1", "4.5.0"]
DOCUTILS_VERSIONS = ["0.19", "0.17", "0.16", "0.15"]


TEST_DEPENDENCIES = [
    "pytest",
    "pytest-xdist",
    "pytest_lsp",
    "myst-parser",
    "responses",
    "lxml",
    "pyparsing!=3.0.4",
    "requests-mock",
]

# Some pytest-extension do not work well with pytest-benchmark, so we create our
# own list for benchmarking
BENCHMARK_DEPENDENCIES = [e for e in TEST_DEPENDENCIES if e not in ['"pytest-xdist"']]
BENCHMARK_DEPENDENCIES.append("py")
BENCHMARK_DEPENDENCIES.append("memray")
BENCHMARK_DEPENDENCIES.append("pytest-benchmark")


def is_supported(python: str, sphinx: str, docutils: str) -> bool:
    return not (sphinx in ["4.5.0"] and docutils in ["0.19", "0.18"])


def run_tests(session, sphinx, docutils):
    session.install(".")
    session.install(*TEST_DEPENDENCIES)
    session.run("pip", "install", "-r", "docs/requirements.txt", silent=True)
    session.run("pip", "install", f"sphinx=={sphinx}", silent=True)
    session.run("pip", "install", f"docutils=={docutils}", silent=True)
    session.run("echo", "TEST FINAL PACKAGE LIST")
    session.run("pip", "freeze")
    session.run("make", "test", external=True)


@session(python=PYTHON_VERSIONS)
@nox.parametrize("docutils", DOCUTILS_VERSIONS)  # order is important, last options needs to be given first ...
@nox.parametrize("sphinx", SPHINX_VERSIONS)  # ... by GH workflow
def tests(session, sphinx, docutils):
    if is_supported(session.python, sphinx, docutils):
        run_tests(session, sphinx, docutils)
    else:
        session.skip("unsupported combination")


@session(python="3.10")
def linkcheck(session):
    session.install(".")
    # LinkCheck can handle rate limits since Sphinx 3.4, which is needed as
    # our doc has to many links to GitHub.
    session.run("pip", "install", "sphinx==3.5.4", silent=True)

    session.run("pip", "install", "-r", "docs/requirements.txt", silent=True)
    session.run("make", "docs-linkcheck", external=True)


@session(python="3.10")
def benchmark_time(session):
    session.install(".")
    session.install(*BENCHMARK_DEPENDENCIES)
    session.run("pip", "install", "-r", "docs/requirements.txt", silent=True)
    session.run(
        "pytest",
        "tests/benchmarks",
        "-k",
        "_time",
        "--benchmark-json",
        "output.json",
        external=True,
        env={"ON_CI": "true", "FAST_BUILD": "true"},
    )


@session(python="3.10")
def benchmark_memory(session):
    session.install(".")
    session.install(*BENCHMARK_DEPENDENCIES)
    session.run("pip", "install", "-r", "docs/requirements.txt", silent=True)
    session.run(
        "pytest",
        "tests/benchmarks",
        "-k",
        "_memory",
        "--benchmark-json",
        "output.json",
        external=True,
        env={"ON_CI": "true", "FAST_BUILD": "true"},
    )
    session.run("memray", "flamegraph", "-o", "mem_out.html", "mem_out.bin")

Running Commands¶

See the Poetry documentation for a list of commands.

In order to run custom commands inside the isolated environment, they should be prefixed with poetry run (ie. poetry run <command>).

Publishing a new release¶

There is a release pipeline installed for the CI.

This gets triggered automatically, if a tag is created and pushed. The tag must follow the format: [0-9].[0-9]+.[0-9]. Otherwise the release jobs won’t trigger.

The release jobs will build the source and wheel distribution and try to upload them to test.pypi.org and pypy.org.

Debugging Sphinx-Needs Language Server features¶

Sphinx-Needs provides some language server functions for the Esbonio Language Server.

The complete functionality can used in VsCode by using the extension vscode-restructuredtext. The whole configuration is done automatically and Sphinx-Needs features gets loaded, if the Sphinx-Needs extension is part of ´extensions` variable inside conf.py.

Debugging¶

Most information is coming from https://docs.restructuredtext.net/articles/development.html.

Check out the source code of all the following projects:
- vscode-restructuredtext: links…
- esbonio
Follow https://docs.restructuredtext.net/articles/development.html to install all dependencies, compile it and get the Development host running in VsCode.
Create a test folder inside the project with a Sphinx projects using Sphinx-Needs, for example under /docs by using sphinx-quickstart.

Add the following to docs/.vscode/settings.json:

{
  "esbonio.server.sourceFolder": "/Path/to/checked_out/esbonio/lib/esbonio",  # absolute path
  "esbonio.server.debugLaunch": true,
  "esbonio.server.logLevel": "debug",
}

Add the args ${workspaceFolder}/docs to configuration Launch Extension in .vscode/launch.json like this:

{
  "name": "Launch Extension",
  "type": "extensionHost",
  "request": "launch",
  "runtimeExecutable": "${execPath}",
  "args": [
      "--extensionDevelopmentPath=${workspaceRoot}",
      "${workspaceFolder}/docs",
  ],
  "sourceMaps": true,
  "outFiles": ["${workspaceRoot}/out/extension.js"],
  "preLaunchTask": "watch"
},

Test it by pressing F5 (running the preconfigured tasks Launch Extension)
- In the opened extensionDevelopmentHost instance, select the correct Python interpreter. e.g. vscode-restructuredtext/.venv/bin/python
Open another instance of VsCode for the checked out esbonio folder.

Add this to .vscode/launch.json under configurations:

{
  "name": "Python: Remote Attach",
  "type": "python",
  "request": "attach",
  "connect": {
      "host": "localhost",
      "port": 5678
  },
  "pathMappings": [
      {
          "localRoot": "${workspaceFolder}/lib/esbonio",
          "remoteRoot": "."
      }
  ]
},

Test it by running the new task Python: Remote Attach. For this the task Launch Extension from VsCode-restructuredText Extension must be already running, as this one starts a python debug server.
Now you set set breakpoints anywhere in the esbonio code.

Debugging Sphinx-Needs functions¶

To debugging Sphinx-Needs Language Server functions, you can repeat the steps 7-10 from above with the Sphinx-Needs repository.

Note:

For step 8: adapt the localRoot path accordingly, e.g. “${workspaceFolder}/../esbonio/lib/esbonio”
If it doesn’t stop at breakpoints, set a breakpoint at sphinx_needs/__init__.py, where you import esbonio_setup. When debugger stops there, choose step into to continue debug.