expand readme and make various fixes

Author: felixgwilliams

.markdownlint-cli2.yaml

@@ -0,0 +1,2 @@
+config:
+  MD013: false

.pre-commit-config.yaml

@@ -15,7 +15,3 @@ repos:
     rev: v0.17.2
     hooks:
       - id: markdownlint-cli2
-  - repo: https://github.com/KyleKing/copier-template-tester
-    rev: "2.1.4"
-    hooks:
-      - id: copier-template-tester

README.md

@@ -1,5 +1,104 @@
-# felixgwilliams python-copier-template
+# Python Copier Template for Data Science
 
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 [![Copier](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-inverted-border-orange.json)](https://github.com/copier-org/copier)
+[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
+
+This is a template built with [Copier](https://github.com/copier-org/copier) to generate a data science focused python project.
+
+Get started with the following command:
+
+```shell
+copier copy gh:felixgwilliams/python-copier-template-ds path/to/destination
+```
+
+## Features
+
+### Project structure
+
+It is assumed that most of the work will be done in Jupyter Notebooks.
+However, the template also includes a python project, in which you can put functions and classes shared across notebooks.
+The repository is set up to use [Pytest](https://docs.pytest.org/en/stable/) for unit testing this module code.
+
+The template also includes a `data` directory whose contents will be ignored by git.
+You can use this folder to store data that you do not commit.
+You may also put a readme file in which you can document the source datasets you use and how to acquire them.
+
+### [just](https://github.com/casey/just)
+
+`just` is a command runner that allows you to easily to run project-specific commands.
+In fact, you can use `just` to run all the setup commands listed below:
+
+```shell
+just setup
+```
+
+### [uv](https://github.com/astral-sh/uv)
+
+The repository is set up to use [uv](https://github.com/astral-sh/uv) for package or project management.
+You may set up your python environment with
+
+```shell
+uv sync --all-groups --all-extras
+```
+
+### [Ruff](https://github.com/astral-sh/ruff)
+
+The repository is configured to use [Ruff](https://github.com/astral-sh/ruff) for linting and formatting.
+By default, all lints are enabled except
+
+- [`COM`](https://docs.astral.sh/ruff/rules/#flake8-commas-com) Enforces trailing commas
+- [`ERA`](https://docs.astral.sh/ruff/rules/#eradicate-era) Disallows commented-out code
+- [`ISC001`](https://docs.astral.sh/ruff/rules/single-line-implicit-string-concatenation/#flake8-executable-exe) (conflicts with the formatter).
+
+In addition, the following rules are only enforced for module code as they are inappropriate or too strict for unit tests and notebooks:
+
+- [`D`](https://docs.astral.sh/ruff/rules/#pydocstyle-d) Requires docstrings on functions, classes and modules
+- [`ANN`](https://docs.astral.sh/ruff/rules/#flake8-annotations-ann) Requires type annotations on functions and methods
+- [`S101`](https://docs.astral.sh/ruff/rules/assert/) Disallows use of `assert`
+- [`PLR2004`](https://docs.astral.sh/ruff/rules/magic-value-comparison/) Disallows "magic" values in comparisons
+- [`T20`](https://docs.astral.sh/ruff/rules/#flake8-print-t20) Disallows print statements
+
+The target line length is 120 and the docstring convention is google.
+
+### [pre-commit](https://github.com/pre-commit/pre-commit)
+
+pre-commit is a tool that runs checks on your files before you commit them with git, thereby helping ensure code quality.
+Enable it with the following command:
+
+```shell
+pre-commit install --install-hooks
+```
+
+The configuration is stored in `.pre-commit-config.yaml`.
+
+### [nbwipers](https://github.com/felixgwilliams/nbwipers)
+
+`nbwipers` is a tool written in rust to ensure Jupyter notebooks are clean.
+Committing notebooks that are not clean makes diffs more confusing, can degrade performance and increases the risk of leaking sensitive information.
+You can set it up as a git filter with the following command.
+
+```shell
+nbwipers install local
+```
+
+### [pytest](https://docs.pytest.org/en/stable/)
+
+The repository comes configured to use `pytest` for unit testing the module code.
+Feel free to ignore it if you do not write module code.
+
+### Github Actions
+
+You may optionally add a github workflow file which checks the following:
+
+- uses ruff to check files are formatted and linted
+- Runs unit tests and checks coverage
+- Checks any markdown files are formatted with [markdownlint-cli2](https://github.com/DavidAnson/markdownlint-cli2)
+- Checks that all jupyter notebooks are clean
+
+### [Typos](https://github.com/crate-ci/typos)
+
+Typos checks for common typos in code, aiming for a low false positive rate.
+The repository is configured not to use it for Jupyter notebook files, as it tends to find errors in cell outputs.
 
 Test with [Copier](https://github.com/copier-org/copier) and [copier-template-tester](https://github.com/KyleKing/copier-template-tester).

copier.yml

@@ -15,5 +15,5 @@ python_version:
 
 github_actions:
   type: bool
-  help: Include github directory
+  help: Add github actions
   default: false

template/.markdownlint-cli2.yaml

@@ -0,0 +1,2 @@
+config:
+  MD013: false

template/.pre-commit-config.yaml

@@ -5,11 +5,10 @@ repos:
       - id: trailing-whitespace
       - id: end-of-file-fixer
       - id: check-yaml
-      # - id: check-added-large-files
       - id: check-merge-conflict
   - repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.
-    rev: v0.9.2
+    rev: v0.9.3
     hooks:
       # Run the linter.
       - id: ruff

template/README.md.jinja

@@ -1 +1,11 @@
 # {{project_name}}
+
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+[![Copier](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-inverted-border-orange.json)](https://github.com/copier-org/copier)
+[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
+
+Get started by running the following with [just](https://github.com/casey/just):
+
+```shell
+just setup
+```

template/justfile.jinja

@@ -1,5 +1,6 @@
 module_name := "{{module_name}}"
 
+# list commands
 default:
   @just --list
 
@@ -8,7 +9,7 @@ setup: install git-setup
 
 # install the packages
 install:
-  {% raw %}{{if path_exists("uv.lock") != "true" {"uv sync --dev --all-extras --inexact"} else {"uv sync --dev --all-extras --locked --inexact"} }}{% endraw %}
+  {% raw %}{{if path_exists("uv.lock") != "true" {"uv sync --all-groups --all-extras --inexact"} else {"uv sync --dev --all-extras --locked --inexact"} }}{% endraw %}
 
 # ensure code quality before git commit via pre-commit and nbwipers
 git-setup: install
@@ -34,8 +35,12 @@ lint:
 
 # update packages and uv lock file
 update:
-  uv sync -U --dev --all-extras --inexact
+  uv sync -U --all-groups --all-extras --inexact
 
 # update pre-commit file
 pc-update:
   uvx pre-commit-update
+
+# check for typos
+typos:
+  uv run typos .

template/pyproject.toml.jinja

@@ -29,10 +29,16 @@ convention = "google"
 files.extend-exclude = ["*.ipynb"]
 
 [tool.uv]
-dev-dependencies = [
+default-groups = ["dev", "lint"]
+
+[dependency-groups]
+dev = [
     "nbwipers>=0.6.0",
-    "ruff>=0.9.2",
     "pytest>=8",
     "pytest-cov>=6.0.0",
     "pre-commit>=4.1.0",
+    "typos>=1.29.4",
+]
+lint = [
+    "ruff>=0.9.3",
 ]

template/{% if github_actions %}.github{% endif %}/workflows/ci.yml.jinja

@@ -27,7 +27,7 @@ jobs:
         with:
           python-version-file: ".python-version"
       - name: Install the project
-        run: uv sync --all-extras --dev
+        run: uv sync --only-group lint
       - name: check format
         run: uv run ruff format --check --diff .
       - name: check lints
@@ -43,7 +43,7 @@ jobs:
         with:
           python-version-file: ".python-version"
       - name: Install the project
-        run: uv sync --all-extras --dev
+        run: uv sync --all-extras --all-groups
       - name: unit tests
         run: |
           set -o pipefail && \