===================== The Contributor Guide ===================== Mailman 3 consists of a collection of separate-but-linked projects, each of which has its own development setup guide. This makes sense when you want to focus on a single piece of Mailman, but can make setting up the entire Mailman Suite in one go somewhat confusing. This guide attempts to move all the information currently in the wiki and various package documentation into a single "definitive" guide. Main package documentation on Readthedocs.io: * `Mailman core start guide `_ * `Postorius dev guide `_ * `Hyperkitty dev guide `_ Getting prerequisites --------------------- For the most part, setup for each project will download any needed packages. However, you will need a few system packages to be sure you've got the necessary version of Python and its tools, git (to get the source code), postfix (a mail server), and a few other tools that are used during setup. On Fedora, you probably want to run:: $ sudo yum install python3-setuptools python3-virtualenv python3-devel git gcc nodejs-less postfix python3-tox On Debian and Ubuntu, this may be something like:: $ sudo apt install python3-setuptools python3-virtualenv python3-dev git gcc node-less nodejs postfix tox On macOS, if you have `Homebrew`_ installed, you may run:: $ brew install tox python node gcc git $ npm install --global less $ # Note: postfix is pre-installed on macOS. If you prefer, you can substitute Exim4 for Postfix. Postfix is the MTA used by most Mailman developers, but we do support Exim 4. (`Sendmail support is very much desired`_, but the Mailman core developers need contributors with Sendmail expertise to help.) For development purposes it doesn't matter, since we will mock all interactions to external MTA. You will need `tox `_ to run tests. HyperKitty also needs ``sassc``. You can install ``sassc`` using your OS package manager For Fedora/CentOS:: $ sudo dnf install sassc ``sassc`` is available in the newer versions of Debian(9)/Ubuntu(18.04):: $ sudo apt install sassc On macOS with Homebrew, you may run:: $ brew install sassc You can also install sassc from source as per their `build documentation `_ Gitlab Setup ------------ We use `Gitlab `_ for source code hosting and our CI. You can `fork `_ any of the projects you want and start working on it. If you don't already have an account on `Gitlab `_, please create one, you will need that for contributing code or participating in any other way. We also use Gitlab for code reviews. Our workflow looks very similar to the official `Gitlab Workflow `_. Please remember to `enable shared runners `_ on your fork, it will be used to build your code and run unittests on pull requests that you will make. It is mandatory that you have runners enabled before you send any pull requests. Set up a directory ------------------ Setting up the whole Mailman suite means you have to pull code from a bunch of different related repositories. You can put all the code anywhere you want, but you might want to set up a directory to keep all the pieces of mailman together. For example:: $ mkdir ~/mailman # cd ~/mailman For the rest of this development guide, we are going to assume you're using ``~/mailman`` as your directory, but you can use whatever you want. Set up virtual environments --------------------------- All parts of Mailman support only Python 3.6+. For your development, it is advised that you create a virtualenv so that the packages you install don't break any of the system packages using Python. To create the virtualenv run the following command:: $ python3 -m venv venv3 To activate a virtualenv, you need to run the appropriate activate script:: $ source venv3/bin/activate You *must* use ``source`` (or ``.`` if your shell is a pure POSIX shell) everytime you want to activate your development environment. To make your life easier when managing virtualenvs, see `virtualenvwrapper `_ . Set up and run Mailman Core --------------------------- First, get the code:: $ cd ~/mailman $ git clone https://gitlab.com/mailman/mailman.git To set up Mailman Core, you'll need to switch to your Python 3 virtualenv:: $ source venv3/bin/activate Then, go into the mailman directory, run setup, and then run ``mailman info`` to be sure everything is set up correctly, and that the right settings are in place:: $ cd mailman $ pip install -e . $ mailman info You can edit your ``mailman.cfg`` file to make any necessary changes. By default, during development, it is located at ``var/etc/mailman.cfg``. Then start things up:: $ mailman start $ mailman status # Check if Mailman started correctly. $ cd .. Note that mailman just makes a ``var/`` directory wherever you start it and uses that to store your data. This is great for the purposes of testing so you can easily make fresh installs, but might be confusing if you restart your instance later from a different location and don't have your original mailman.db file, or if you start looking around and finding var/ directories everywhere. Later on, if you need to restart Mailman (i.e. if you get the error "Mailman REST API not available. Please start Mailman core.") then you can also do that by calling the ``mailman`` executable from the venv as follows:: $ ~/mailman/venv3/bin/mailman -C ~/mailman/mailman/var/etc/mailman.cfg start If you are using different virtualenv paths, your path will be different than one listed above. You can find it using ``which mailman`` command. Note that the ``mailman`` executable has several sub-commands. One that is particularly useful for debugging is ``mailman shell``. .. note:: If you like `IPython `_ shell (like I do!), you add the following to your ``mailman.cfg``:: [shell] use_ipython: yes Also, remember to install ipython using pip:: $ pip install ipython Testing Mailman Core -------------------- You can run tests for Mailman Core (or any Mailman project) using `tox ` :: $ tox -e py311-nocov This requires that you have Python3.11 installed. You can change it to py3XX depending on the latest version supported by Core. .. note:: Take a look at the ``tox.ini`` for the latest version of Python supported by Mailman as this documentation could be out of date. For contributing to Mailman it is incredibly helpful, time saving and resource-saving, running a specific test case, which only tests your contributions without testing every other component of Mailman. To run test cases from a specific file, for example the Mailman Core's ``src/mailman/commands/tests/test_cli_members.py`` file, run a command like this:: $ tox -e py311 -- mailman.commands.tests.test_cli_members .. note:: Please notice that ``/`` are replaced with ``.`` and the ``.py`` extension is cut off You can omit the ``-e py311`` (Or any environment, not just 'py311') option and tox will test your files with every environment specified in ``tox.ini``. $ tox -- mailman.commands.tests.test_cli_members The option ``-p auto`` parallelizes the tests:: $ tox -p auto Of course, you can mix options to run all environments in parallel on one file:: $ tox -p auto -- mailman.commands.tests.test_cli_members Set up Mailman Client --------------------- Get the code:: $ cd ~/mailman $ git clone https://gitlab.com/mailman/mailmanclient.git Then set up mailmanclient:: $ cd mailmanclient $ pip install -e . $ cd .. To run the tests:: $ tox -e py311 Set up Django-mailman3 ---------------------- This package holds the Django libraries and templates used by Postorius and HyperKitty. Get the code and set it up:: $ cd ~/mailman $ git clone https://gitlab.com/mailman/django-mailman3.git $ cd django-mailman3 $ pip install -e . $ cd .. To run the tests:: $ tox -e py311-django41 .. note:: It is possible for the version of Python and Django supported at a later time will be different. To get the right environments to run the tests, please run `tox ` Set up and run Postorius ------------------------ The Postorius documentation, including a more extensive setup guide, can be found here: http://postorius.readthedocs.org/ Make sure to install mailmanclient and django-mailman3 before setting up Postorius. (If you're following this guide in order, you've just done that.) Get the code and run setup. Make sure you're in venv which has Python 3.5+ for Postorius:: $ cd ~/mailman $ git clone https://gitlab.com/mailman/postorius.git $ cd postorius $ pip install -e . $ cd .. Postorius and HyperKitty both come with ``example_project`` directories with basic configuration so you can try them out. For this tutorial, however, we'll be using a project that combines both instead. You can run tests using:: $ tox -e py311-django41 Set up a Fake mail server ------------------------- To be able to actually receive emails, you need to setup a mail server. Mailman core receives emails over LMTP Protocol, which most of the modern MTAs support. However, setup instructions are provided only for Postfix, Exim4 and qmail. Please refer to the `MTA documentation`_ at Mailman Core for the details. You will also have to add some settings to your django configuration. The setup instructions are provided in `django's email documentation`_. For development setup, you don't _have_ to install a working MTA. You can add the following to your ``mailman.cfg`` to make sure that it doesn't try to send emails out:: [devmode] enabled: yes recipient: you@yourdomain.com [mta] smtp_port: 9025 lmtp_port: 9024 incoming: mailman.testing.mta.FakeMTA Also, in Django you can add the following configuration to your ``settings.py``:: EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend' This writes everything to ``stdout``. There are `other email backends `_ available to use for testing like ``django.core.mail.backends.filebased.EmailBackend`` that one can use to write outgoing emails to a file on disk. Please see the docs for other options. Set up and run HyperKitty ------------------------- Complete guide here: https://hyperkitty.readthedocs.org/en/latest/development.html Make sure to install mailmanclient and django-mailman3 before setting up Hyperkitty. (If you're following this guide in order, you've just done that.) HyperKitty’s default configuration uses the `Whoosh `_ search engine in the backend. Install ``Whoosh`` using:: $ pip install whoosh Get the code and run setup:: $ cd ~/mailman $ git clone https://gitlab.com/mailman/hyperkitty.git $ cd hyperkitty $ pip install -e . $ cd .. Postorius and HyperKitty both come with ``example_project`` directories with basic configuration so you can try them out. By default, they both use port 8000, so if you do want to run both example projects at the same time, do remember that you'll need to specify a different port on the command line for one of them. You can run tests using:: $ tox -e py311-django41 However, we're going to run them both in a single Django instance at the end of this guide, so don't worry about ports right now. Set up mailman-hyperkitty ------------------------- ``mailman-hyperkitty`` is the package that actually sends the incoming emails to HyperKitty for archiving. Note that this is one of the components that uses Python 3. Setting it up:: $ cd ~/mailman $ git clone https://gitlab.com/mailman/mailman-hyperkitty.git $ cd mailman-hyperkitty $ pip install -e . $ cd .. You'll need to fix the default ``mailman-hyperkitty.cfg`` file to use the correct url for HyperKitty. If you're running it on http://localhost:8002 then you need to change ``base_url`` to match that. You can run tests using:: $ tox -e py311-coverage Link Mailman to HyperKitty -------------------------- Now you have to enable HyperKitty in Mailman. To do that, edit the ``mailman.cfg`` (in ``~/mailman/mailman/var/etc``, or wherever the output of ``mailman info`` says it is) and add the following config. Note that you need to fill in the absolute path to your ``mailman-hyperkitty.cfg`` in the configuration below:: # mailman.cfg [archiver.hyperkitty] class: mailman_hyperkitty.Archiver enable: yes configuration: Run the Mailman Suite (combined hyperkitty+postorius) ----------------------------------------------------- You can run HyperKitty and Postorius as separate applications, but many developers are going to want to run them on a single server. The configuration files for this are in a repository called mailman-suite. The first time you run the suite, you will want to set up a superuser account. This is the account you will use in the web interface to set up your first domains. Please enter an email address otherwise the database won't be setup correctly and you will run into errors later:: $ cd ~/mailman $ git clone https://gitlab.com/mailman/mailman-web.git $ cd mailman-web $ pip install -e . $ export DJANGO_SETTINGS_MODULE='mailman_web.settings.dev' $ mailman-web migrate $ mailman-web createsuperuser You'll want to run the following commands in a window where you can leave them running, since it dumps all the django logs to the console:: $ mailman-web runserver At this point, you should be able to see Mailman Suite running! In the default setup, you can go to http://127.0.0.1:8000 and start poking around. You should be able to use the superuser account you created to log in and create a domain and then some lists. The default config file uses a dummy email backend created by this line in settings.py:: EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend' Using this backend, all emails will be printed to the console (rather than sent as email) so you can get the url to verify your email from the console. You can also use `FileBackend `_ to write emails to a file on disk. Don't leave the console email backend configured and running once you get to the point where you want to send real emails, though! .. _`Sendmail support is very much desired`: https://gitlab.com/mailman/mailman/issues/307 .. _`MTA documentation`: https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html .. _`django's email documentation`: https://docs.djangoproject.com/en/1.10/topics/email/#topic-email-backends .. _`Homebrew`: https://brew.sh/