Contributing

This page provides a guide for developers wishing to contribute to Sphinx-Needs.

Bugs, Features and PRs

For bug reports and well-described technical feature requests, please use our issue tracker:
https://github.com/useblocks/sphinx-needs/issues

For feature ideas and questions, please use our discussion board:
https://github.com/useblocks/sphinx-needs/discussions

If you have already created a PR, you can send it in. Our CI workflow will check (test and code styles) and a maintainer will perform a review, before we can merge it. Your PR should conform with the following rules:

  • A meaningful description or link, which describes the change

  • The changed code (for sure :) )

  • Test cases for the change (important!)

  • Updated documentation, if behavior gets changed or new options/directives are introduced.

  • Update of docs/changelog.rst.

  • If this is your first PR, feel free to add your name in the AUTHORS file.

Installing Dependencies

Sphinx-Needs requires only Poetry to be installed as a system dependency, the rest of the dependencies are ‘bootstrapped’ and installed in an isolated environment by Poetry.

  1. Install Poetry

  2. Install project dependencies

    poetry install
    
  3. Install Pre-Commit

  4. Install the Pre-Commit hooks

    pre-commit install
    
  5. For running tests, install the dependencies of our official documentation:

    pip install -r docs/requirements.txt
    

List make targets

Sphinx-Needs uses make to invoke most development related actions.

Use make list to get a list of available targets.

benchmark-memory
benchmark-time
docs-html
docs-html-fast
docs-linkcheck
docs-pdf
format
lint
test
test-matrix
test-short

Build docs

To build the Sphinx-Needs documentation stored under /docs, run:

# Build HTML pages
make docs-html

or

# Build PDF pages
make docs-pdf

It will always perform a clean build (calls make clean before the build). If you want to avoid this, run the related sphinx-commands directly under /docs (e.g. make docs).

To check if all used links in the documentation are still valid, run:

make docs-linkcheck

Running Tests

Hint

Please be sure to have the dependencies of the official documentation installed:

pip install -r docs/requirements.txt
make test

Linting & Formatting

Sphinx-Needs uses black and isort to format its source code.

make lint

Benchmarks

Sphinx-Needs own documentation is used for creating a benchmark for each PR. If the runtime takes 10% longer as the previous ones, the benchmark test will fail.

Benchmark test cases are available under tests/benchmarks. And they can be locally executed via make benchmark.

The results for each PR/commit get added to a chart, which is available under http://useblocks.com/sphinx-needs/bench/index.html.

The benchmark data is stored on the benchmarks branch, which is also used by github-pages as source.

Running Test Matrix

This project provides a test matrix for running the tests across a range of Python and Sphinx versions. This is used primarily for continuous integration.

Nox is used as a test runner.

Running the matrix tests requires additional system-wide dependencies

  1. Install Nox

  2. Install Nox-Poetry

  3. You will also need multiple Python versions available. You can manage these using Pyenv

You can run the test matrix by using the nox command

nox

or using the provided Makefile

make test-matrix

For a full list of available options, refer to the Nox documentation, and the local noxfile.

Our noxfile.py
import nox
from nox_poetry import session

# The versions here must be in sync with the github-workflows.
# Or at least support all version from there.
# This list can contain more versions as used by the github workflows to support
# custom local tests

PYTHON_VERSIONS = ["3.8", "3.10"]
SPHINX_VERSIONS = ["5.2.1", "4.5.0"]
DOCUTILS_VERSIONS = ["0.19", "0.17", "0.16", "0.15"]


TEST_DEPENDENCIES = [
    "pytest",
    "pytest-xdist",
    "pytest_lsp",
    "myst-parser",
    "responses",
    "lxml",
    "pyparsing!=3.0.4",
    "requests-mock",
]

# Some pytest-extension do not work well with pytest-benchmark, so we create our
# own list for benchmarking
BENCHMARK_DEPENDENCIES = [e for e in TEST_DEPENDENCIES if e not in ['"pytest-xdist"']]
BENCHMARK_DEPENDENCIES.append("py")
BENCHMARK_DEPENDENCIES.append("memray")
BENCHMARK_DEPENDENCIES.append("pytest-benchmark")


def is_supported(python: str, sphinx: str, docutils: str) -> bool:
    return not (sphinx in ["4.5.0"] and docutils in ["0.19", "0.18"])


def run_tests(session, sphinx, docutils):
    session.install(".")
    session.install(*TEST_DEPENDENCIES)
    session.run("pip", "install", "-r", "docs/requirements.txt", silent=True)
    session.run("pip", "install", f"sphinx=={sphinx}", silent=True)
    session.run("pip", "install", f"docutils=={docutils}", silent=True)
    session.run("echo", "TEST FINAL PACKAGE LIST")
    session.run("pip", "freeze")
    session.run("make", "test", external=True)


@session(python=PYTHON_VERSIONS)
@nox.parametrize("docutils", DOCUTILS_VERSIONS)  # order is important, last options needs to be given first ...
@nox.parametrize("sphinx", SPHINX_VERSIONS)  # ... by GH workflow
def tests(session, sphinx, docutils):
    if is_supported(session.python, sphinx, docutils):
        run_tests(session, sphinx, docutils)
    else:
        session.skip("unsupported combination")


@session(python="3.10")
def linkcheck(session):
    session.install(".")
    # LinkCheck can handle rate limits since Sphinx 3.4, which is needed as
    # our doc has to many links to GitHub.
    session.run("pip", "install", "sphinx==3.5.4", silent=True)

    session.run("pip", "install", "-r", "docs/requirements.txt", silent=True)
    session.run("make", "docs-linkcheck", external=True)


@session(python="3.10")
def benchmark_time(session):
    session.install(".")
    session.install(*BENCHMARK_DEPENDENCIES)
    session.run("pip", "install", "-r", "docs/requirements.txt", silent=True)
    session.run(
        "pytest",
        "tests/benchmarks",
        "-k",
        "_time",
        "--benchmark-json",
        "output.json",
        external=True,
        env={"ON_CI": "true", "FAST_BUILD": "true"},
    )


@session(python="3.10")
def benchmark_memory(session):
    session.install(".")
    session.install(*BENCHMARK_DEPENDENCIES)
    session.run("pip", "install", "-r", "docs/requirements.txt", silent=True)
    session.run(
        "pytest",
        "tests/benchmarks",
        "-k",
        "_memory",
        "--benchmark-json",
        "output.json",
        external=True,
        env={"ON_CI": "true", "FAST_BUILD": "true"},
    )
    session.run("memray", "flamegraph", "-o", "mem_out.html", "mem_out.bin")

Running Commands

See the Poetry documentation for a list of commands.

In order to run custom commands inside the isolated environment, they should be prefixed with poetry run (ie. poetry run <command>).

Publishing a new release

There is a release pipeline installed for the CI.

This gets triggered automatically, if a tag is created and pushed. The tag must follow the format: [0-9].[0-9]+.[0-9]. Otherwise the release jobs won’t trigger.

The release jobs will build the source and wheel distribution and try to upload them to test.pypi.org and pypy.org.

Debugging Sphinx-Needs Language Server features

Sphinx-Needs provides some language server functions for the Esbonio Language Server.

The complete functionality can used in VsCode by using the extension vscode-restructuredtext. The whole configuration is done automatically and Sphinx-Needs features gets loaded, if the Sphinx-Needs extension is part of ´extensions` variable inside conf.py.

Debugging

Most information is coming from https://docs.restructuredtext.net/articles/development.html.

  1. Check out the source code of all the following projects:

    • vscode-restructuredtext: links…

    • esbonio

  2. Follow https://docs.restructuredtext.net/articles/development.html to install all dependencies, compile it and get the Development host running in VsCode.

  3. Create a test folder inside the project with a Sphinx projects using Sphinx-Needs, for example under /docs by using sphinx-quickstart.

  4. Add the following to docs/.vscode/settings.json:

    {
      "esbonio.server.sourceFolder": "/Path/to/checked_out/esbonio/lib/esbonio",  # absolute path
      "esbonio.server.debugLaunch": true,
      "esbonio.server.logLevel": "debug",
    }
    
  5. Add the args ${workspaceFolder}/docs to configuration Launch Extension in .vscode/launch.json like this:

    {
      "name": "Launch Extension",
      "type": "extensionHost",
      "request": "launch",
      "runtimeExecutable": "${execPath}",
      "args": [
          "--extensionDevelopmentPath=${workspaceRoot}",
          "${workspaceFolder}/docs",
      ],
      "sourceMaps": true,
      "outFiles": ["${workspaceRoot}/out/extension.js"],
      "preLaunchTask": "watch"
    },
    
  6. Test it by pressing F5 (running the preconfigured tasks Launch Extension)

    • In the opened extensionDevelopmentHost instance, select the correct Python interpreter. e.g. vscode-restructuredtext/.venv/bin/python

  7. Open another instance of VsCode for the checked out esbonio folder.

  8. Add this to .vscode/launch.json under configurations:

    {
      "name": "Python: Remote Attach",
      "type": "python",
      "request": "attach",
      "connect": {
          "host": "localhost",
          "port": 5678
      },
      "pathMappings": [
          {
              "localRoot": "${workspaceFolder}/lib/esbonio",
              "remoteRoot": "."
          }
      ]
    },
    
  9. Test it by running the new task Python: Remote Attach. For this the task Launch Extension from VsCode-restructuredText Extension must be already running, as this one starts a python debug server.

  10. Now you set set breakpoints anywhere in the esbonio code.

Debugging Sphinx-Needs functions

To debugging Sphinx-Needs Language Server functions, you can repeat the steps 7-10 from above with the Sphinx-Needs repository.

Note:

  • For step 8: adapt the localRoot path accordingly, e.g. “${workspaceFolder}/../esbonio/lib/esbonio”

  • If it doesn’t stop at breakpoints, set a breakpoint at sphinx_needs/__init__.py, where you import esbonio_setup. When debugger stops there, choose step into to continue debug.

Maintainers

Daniel Woste <daniel@useblocks.com>

Contributors

Marco Heinemann <marco@useblocks.com>

Trevor Lovett <trevor.lovett@gmail.com>

Magnus Lööf <magnus.loof@gmail.com>

Harri Kaimio

Anders Thuné

Daniel Eades <danieleades@hotmail.com>

Philip Partsch <philip.partsch@googlemail.com>

David Le Nir <david.lenir.e@thalesdigital.io>

Baran Barış Yıldızlı <arisbbyil@gmail.com>

Roberto Rötting <roberto.roetting@gmail.com>

Nirmal Sasidharan <nirmal.sasidharan@de.bosch.com>

Jacob Allen <jacob.allen@etas.com>

Jörg Kreuzberger <j.kreuzberger@procitec.de>

Duodu Randy <duodurandy19@gmail.com>