The Contributor Guide
Mailman 3 consists of a collection of separate-but-linked projects, each of which has its own development setup guide. This makes sense when you want to focus on a single piece of Mailman, but can make setting up the entire Mailman Suite in one go somewhat confusing. This guide attempts to move all the information currently in the wiki and various package documentation into a single “definitive” guide.
- Main package documentation on Readthedocs.io:
Getting prerequisites
For the most part, setup for each project will download any needed packages. However, you will need a few system packages to be sure you’ve got the necessary version of Python and its tools, git (to get the source code), postfix (a mail server), and a few other tools that are used during setup.
On Fedora, you probably want to run:
$ sudo yum install python3-setuptools python3-virtualenv python3-devel git gcc nodejs-less postfix python3-tox
On Debian and Ubuntu, this may be something like:
$ sudo apt install python3-setuptools python3-virtualenv python3-dev git gcc node-less nodejs postfix tox
On macOS, if you have Homebrew installed, you may run:
$ brew install tox python node gcc git
$ npm install --global less
$ # Note: postfix is pre-installed on macOS.
If you prefer, you can substitute Exim4 for Postfix. Postfix is the MTA used by most Mailman developers, but we do support Exim 4. (Sendmail support is very much desired, but the Mailman core developers need contributors with Sendmail expertise to help.) For development purposes it doesn’t matter, since we will mock all interactions to external MTA.
You will need tox to run tests.
HyperKitty also needs sassc
. You can install sassc
using your OS
package manager
For Fedora/CentOS:
$ sudo dnf install sassc
sassc
is available in the newer versions of Debian(9)/Ubuntu(18.04):
$ sudo apt install sassc
On macOS with Homebrew, you may run:
$ brew install sassc
You can also install sassc from source as per their build documentation
Gitlab Setup
We use Gitlab for source code hosting and our CI. You can fork any of the projects you want and start working on it. If you don’t already have an account on Gitlab, please create one, you will need that for contributing code or participating in any other way.
We also use Gitlab for code reviews. Our workflow looks very similar to the official Gitlab Workflow. Please remember to enable shared runners on your fork, it will be used to build your code and run unittests on pull requests that you will make. It is mandatory that you have runners enabled before you send any pull requests.
Set up a directory
Setting up the whole Mailman suite means you have to pull code from a bunch of different related repositories. You can put all the code anywhere you want, but you might want to set up a directory to keep all the pieces of mailman together. For example:
$ mkdir ~/mailman
# cd ~/mailman
For the rest of this development guide, we are going to assume you’re using
~/mailman
as your directory, but you can use whatever you want.
Set up virtual environments
All parts of Mailman support only Python 3.6+. For your development, it is advised that you create a virtualenv so that the packages you install don’t break any of the system packages using Python.
To create the virtualenv run the following command:
$ python3 -m venv venv3
To activate a virtualenv, you need to run the appropriate activate script:
$ source venv3/bin/activate
You must use source
(or .
if your shell is a pure POSIX shell)
everytime you want to activate your development environment. To make your life
easier when managing virtualenvs, see virtualenvwrapper .
Set up and run Mailman Core
First, get the code:
$ cd ~/mailman
$ git clone https://gitlab.com/mailman/mailman.git
To set up Mailman Core, you’ll need to switch to your Python 3 virtualenv:
$ source venv3/bin/activate
Then, go into the mailman directory, run setup, and then run mailman info
to be sure everything is set up correctly, and that the right settings are in
place:
$ cd mailman
$ pip install -e .
$ mailman info
You can edit your mailman.cfg
file to make any necessary changes. By
default, during development, it is located at var/etc/mailman.cfg
. Then
start things up:
$ mailman start
$ mailman status # Check if Mailman started correctly.
$ cd ..
Note that mailman just makes a var/
directory wherever you start it and uses
that to store your data. This is great for the purposes of testing so you can
easily make fresh installs, but might be confusing if you restart your instance
later from a different location and don’t have your original mailman.db file, or
if you start looking around and finding var/ directories everywhere.
Later on, if you need to restart Mailman (i.e. if you get the error “Mailman
REST API not available. Please start Mailman core.”) then you can also do that
by calling the mailman
executable from the venv as follows:
$ ~/mailman/venv3/bin/mailman -C ~/mailman/mailman/var/etc/mailman.cfg start
If you are using different virtualenv paths, your path will be different than
one listed above. You can find it using which mailman
command.
Note that the mailman
executable has several sub-commands. One
that is particularly useful for debugging is mailman shell
.
Note
If you like IPython shell (like I do!), you
add the following to your mailman.cfg
:
[shell]
use_ipython: yes
Also, remember to install ipython using pip::
$ pip install ipython
Testing Mailman Core
You can run tests for Mailman Core (or any Mailman project) using tox <https://tox.readthedocs.io/en/latest/>
$ tox -e py311-nocov
This requires that you have Python3.11 installed. You can change it to py3XX depending on the latest version supported by Core.
Note
Take a look at the tox.ini
for the latest
version of Python supported by Mailman as this documentation could
be out of date.
For contributing to Mailman it is incredibly helpful, time saving and resource-saving,
running a specific test case, which only tests your contributions
without testing every other component of Mailman.
To run test cases from a specific file, for example the Mailman Core’s
src/mailman/commands/tests/test_cli_members.py
file,
run a command like this:
$ tox -e py311 -- mailman.commands.tests.test_cli_members
Note
Please notice that /
are replaced with .
and the .py
extension is cut off
You can omit the -e py311
(Or any environment, not just ‘py311’) option and tox will test your
files with every environment specified in tox.ini
.
$ tox – mailman.commands.tests.test_cli_members
The option -p auto
parallelizes the tests:
$ tox -p auto
Of course, you can mix options to run all environments in parallel on one file:
$ tox -p auto -- mailman.commands.tests.test_cli_members
Set up Mailman Client
Get the code:
$ cd ~/mailman
$ git clone https://gitlab.com/mailman/mailmanclient.git
Then set up mailmanclient:
$ cd mailmanclient
$ pip install -e .
$ cd ..
To run the tests:
$ tox -e py311
Set up Django-mailman3
This package holds the Django libraries and templates used by Postorius and HyperKitty.
Get the code and set it up:
$ cd ~/mailman
$ git clone https://gitlab.com/mailman/django-mailman3.git
$ cd django-mailman3
$ pip install -e .
$ cd ..
To run the tests:
$ tox -e py311-django41
Note
It is possible for the version of Python and Django supported at a later time will be different. To get the right environments to run the tests, please run `tox `
Set up and run Postorius
The Postorius documentation, including a more extensive setup guide, can be found here: http://postorius.readthedocs.org/
Make sure to install mailmanclient and django-mailman3 before setting up Postorius. (If you’re following this guide in order, you’ve just done that.)
Get the code and run setup. Make sure you’re in venv which has Python 3.5+ for Postorius:
$ cd ~/mailman
$ git clone https://gitlab.com/mailman/postorius.git
$ cd postorius
$ pip install -e .
$ cd ..
Postorius and HyperKitty both come with example_project
directories with
basic configuration so you can try them out. For this tutorial, however,
we’ll be using a project that combines both instead.
You can run tests using:
$ tox -e py311-django41
Set up a Fake mail server
To be able to actually receive emails, you need to setup a mail server. Mailman core receives emails over LMTP Protocol, which most of the modern MTAs support. However, setup instructions are provided only for Postfix, Exim4 and qmail. Please refer to the MTA documentation at Mailman Core for the details.
You will also have to add some settings to your django configuration. The setup instructions are provided in django’s email documentation.
For development setup, you don’t _have_ to install a working MTA. You can add
the following to your mailman.cfg
to make sure that it doesn’t try to send
emails out:
[devmode]
enabled: yes
recipient: you@yourdomain.com
[mta]
smtp_port: 9025
lmtp_port: 9024
incoming: mailman.testing.mta.FakeMTA
Also, in Django you can add the following configuration to your
settings.py
:
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
This writes everything to stdout
. There are other email backends
available to use for testing like
django.core.mail.backends.filebased.EmailBackend
that one can use
to write outgoing emails to a file on disk. Please see the docs for
other options.
Set up and run HyperKitty
Complete guide here: https://hyperkitty.readthedocs.org/en/latest/development.html
Make sure to install mailmanclient and django-mailman3 before setting up Hyperkitty. (If you’re following this guide in order, you’ve just done that.)
HyperKitty’s default configuration uses the Whoosh search engine in the backend. Install Whoosh
using:
$ pip install whoosh
Get the code and run setup:
$ cd ~/mailman
$ git clone https://gitlab.com/mailman/hyperkitty.git
$ cd hyperkitty
$ pip install -e .
$ cd ..
Postorius and HyperKitty both come with example_project
directories with
basic configuration so you can try them out. By default, they both use port
8000, so if you do want to run both example projects at the same time, do
remember that you’ll need to specify a different port on the command line for
one of them.
You can run tests using:
$ tox -e py311-django41
However, we’re going to run them both in a single Django instance at the end of this guide, so don’t worry about ports right now.
Set up mailman-hyperkitty
mailman-hyperkitty
is the package that actually sends the incoming emails
to HyperKitty for archiving. Note that this is one of the components that uses
Python 3.
Setting it up:
$ cd ~/mailman
$ git clone https://gitlab.com/mailman/mailman-hyperkitty.git
$ cd mailman-hyperkitty
$ pip install -e .
$ cd ..
You’ll need to fix the default mailman-hyperkitty.cfg
file to use the
correct url for HyperKitty. If you’re running it on http://localhost:8002
then you need to change base_url
to match that.
You can run tests using:
$ tox -e py311-coverage
Link Mailman to HyperKitty
Now you have to enable HyperKitty in Mailman. To do that, edit the
mailman.cfg
(in ~/mailman/mailman/var/etc
, or wherever the output of
mailman info
says it is) and add the following config. Note that you need
to fill in the absolute path to your mailman-hyperkitty.cfg
in the
configuration below:
# mailman.cfg
[archiver.hyperkitty]
class: mailman_hyperkitty.Archiver
enable: yes
configuration: <absolute path to mailman-hyperkitty.cfg>
Run the Mailman Suite (combined hyperkitty+postorius)
You can run HyperKitty and Postorius as separate applications, but many developers are going to want to run them on a single server. The configuration files for this are in a repository called mailman-suite.
The first time you run the suite, you will want to set up a superuser account. This is the account you will use in the web interface to set up your first domains. Please enter an email address otherwise the database won’t be setup correctly and you will run into errors later:
$ cd ~/mailman
$ git clone https://gitlab.com/mailman/mailman-web.git
$ cd mailman-web
$ pip install -e .
$ export DJANGO_SETTINGS_MODULE='mailman_web.settings.dev'
$ mailman-web migrate
$ mailman-web createsuperuser
You’ll want to run the following commands in a window where you can leave them running, since it dumps all the django logs to the console:
$ mailman-web runserver
At this point, you should be able to see Mailman Suite running! In the default setup, you can go to http://127.0.0.1:8000 and start poking around. You should be able to use the superuser account you created to log in and create a domain and then some lists.
The default config file uses a dummy email backend created by this line in settings.py:
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
Using this backend, all emails will be printed to the console (rather than sent as email) so you can get the url to verify your email from the console. You can also use FileBackend to write emails to a file on disk.
Don’t leave the console email backend configured and running once you get to the point where you want to send real emails, though!