Message decoration

Message decoration is the process of adding headers and footers to the original message. A handler module takes care of this based on the settings of the mailing list and the type of message being processed.

>>> from mailman.app.lifecycle import create_list
>>> mlist = create_list('ant@example.com')
>>> msg_text = """\
... From: aperson@example.org
...
... Here is a message.
... """
>>> from mailman.testing.helpers import (specialized_message_from_string
...   as message_from_string)
>>> msg = message_from_string(msg_text)

Short circuiting

Digest messages get decorated during the digest creation phase so no extra decorations are added for digest messages.

>>> from mailman.handlers.decorate import process
>>> process(mlist, msg, dict(isdigest=True))
>>> print(msg.as_string())
From: aperson@example.org

Here is a message.

>>> process(mlist, msg, dict(nodecorate=True))
>>> print(msg.as_string())
From: aperson@example.org

Here is a message.

Simple decorations

Message decorations are specified by URI and can be specialized by the mailing list and language. Internal Mailman decorations can be referenced by using the mailman:/// URL scheme. Here we create a simple English header and footer for all mailing lists in our site.

>>> import os, tempfile
>>> from mailman.config import config
>>> template_dir = tempfile.mkdtemp()
>>> site_dir = os.path.join(template_dir, 'site', 'en')
>>> os.makedirs(site_dir)
>>> config.push('templates', """
... [paths.testing]
... template_dir: {}
... """.format(template_dir))

>>> myheader_path = os.path.join(site_dir, 'myheader.txt')
>>> with open(myheader_path, 'w') as fp:
...     print('header', file=fp)
>>> myfooter_path = os.path.join(site_dir, 'myfooter.txt')
>>> with open(myfooter_path, 'w') as fp:
...     print('footer', file=fp)

Adding these template URIs to the template manager sets the mailing list up to use these templates. Since these are site-global templates, we can use a shorter path.

>>> from mailman.interfaces.template import ITemplateManager
>>> from zope.component import getUtility
>>> manager = getUtility(ITemplateManager)
>>> manager.set('list:member:regular:header',
...             mlist.list_id, 'mailman:///myheader.txt')
>>> manager.set('list:member:regular:footer',
...             mlist.list_id, 'mailman:///myfooter.txt')

Text messages that have no declared content type are, by default encoded in ASCII. When the mailing list’s preferred language is en (i.e. English), the character set of the mailing list and of the message will match, allowing Mailman to simply prepend the header and append the footer verbatim.

>>> mlist.preferred_language = 'en'
>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.org
...

header
Here is a message.
footer

Mailman supports a number of interpolation variables, placeholders in the header and footer for information to be filled in with mailing list specific data. An example of such information is the mailing list’s real name (a short descriptive name for the mailing list).

>>> with open(myheader_path, 'w') as fp:
...     print('$display_name header', file=fp)
>>> with open(myfooter_path, 'w') as fp:
...     print('$display_name footer', file=fp)

>>> msg = message_from_string(msg_text)
>>> mlist.display_name = 'Ant'
>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.org
...
Ant header
Here is a message.
Ant footer

You can’t just pick any interpolation variable though; if you do, the variable will remain in the header or footer unchanged.

>>> with open(myheader_path, 'w') as fp:
...     print('$dummy header', file=fp)
>>> with open(myfooter_path, 'w') as fp:
...     print('$dummy footer', file=fp)

>>> msg = message_from_string(msg_text)
>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.org
...
$dummy header
Here is a message.
$dummy footer

Adding archiver permalink URLs in the message footer

You can add links to archived messages in the footer using special placeholder variables. For all available and enabled archiver for the mailing list, use a placeholder of the format $<archiver_name>_url. For example, if you have HyperKitty enabled you can add ${hyperkitty_url} to point to the message in HyperKitty.

Handling RFC 3676 ‘format=flowed’ parameters

RFC 3676 describes a standard by which text/plain messages can marked by generating MUAs for better readability in compatible receiving MUAs. The format parameter on the text/plain Content-Type header gives hints as to how the receiving MUA may flow and delete trailing whitespace for better display in a proportional font.

When Mailman sees text/plain messages with such RFC 3676 parameters, it preserves these parameters when it concatenates headers and footers to the message payload.

>>> with open(myheader_path, 'w') as fp:
...     print('header', file=fp)
>>> with open(myfooter_path, 'w') as fp:
...     print('footer', file=fp)

>>> mlist.preferred_language = 'en'
>>> msg = message_from_string("""\
... From: aperson@example.org
... Content-Type: text/plain; format=flowed; delsp=no
...
... Here is a message\x20
... with soft line breaks.
... """)
>>> process(mlist, msg, {})
>>> # Don't use 'print' here as above because it won't be obvious from the
>>> # output that the soft-line break space at the end of the 'Here is a
>>> # message' line will be retained in the output.
>>> print(msg['content-type'])
text/plain; format="flowed"; delsp="no"; charset="us-ascii"
>>> for line in msg.get_payload().splitlines():
...     print('>{0}<'.format(line))
>header<
>Here is a message <
>with soft line breaks.<
>footer<

Decorating mixed-charset messages

When a message has no explicit character set, it is assumed to be ASCII. However, if the mailing list’s preferred language has a different character set, Mailman will still try to concatenate the header and footer, but it will convert the text to utf-8 and base-64 encode the message payload.

# 'ja' = Japanese; charset = 'euc-jp'
>>> mlist.preferred_language = 'ja'

>>> with open(myheader_path, 'w') as fp:
...     print('$description header', file=fp)
>>> with open(myfooter_path, 'w') as fp:
...     print('$description footer', file=fp)
>>> mlist.description = '\u65e5\u672c\u8a9e'

>>> from email.message import Message
>>> msg = Message()
>>> msg.set_payload('Fran\xe7aise', 'iso-8859-1')
>>> print(msg.as_string())
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<BLANKLINE>
Fran=E7aise
>>> process(mlist, msg, {})
>>> print(msg.as_string())
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
<BLANKLINE>
5pel5pys6KqeIGhlYWRlcgpGcmFuw6dhaXNlCuaXpeacrOiqniBmb290ZXIK

Sometimes the message even has an unknown character set. In this case, Mailman has no choice but to decorate the original message with MIME attachments.

>>> mlist.preferred_language = 'en'
>>> with open(myheader_path, 'w') as fp:
...     print('header', file=fp)
>>> with open(myfooter_path, 'w') as fp:
...     print('footer', file=fp)

>>> msg = message_from_string("""\
... From: aperson@example.org
... Content-Type: text/plain; charset=unknown
... Content-Transfer-Encoding: 7bit
...
... Here is a message.
... """)

>>> process(mlist, msg, {})
>>> msg.set_boundary('BOUNDARY')
>>> print(msg.as_string())
From: aperson@example.org
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

header
--BOUNDARY
Content-Type: text/plain; charset=unknown
Content-Transfer-Encoding: 7bit

Here is a message.

--BOUNDARY
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

footer
--BOUNDARY--

Decorating multipart messages

Multipart messages have to be decorated differently. The header and footer cannot be simply concatenated into the payload because that will break the MIME structure of the message. Instead, the header and footer are attached as separate MIME subparts.

When the outer part is multipart/mixed, the header and footer can have a Content-Disposition of inline so that MUAs can display these headers as if they were simply concatenated.

>>> part_1 = message_from_string("""\
... From: aperson@example.org
...
... Here is the first message.
... """)
>>> part_2 = message_from_string("""\
... From: bperson@example.com
...
... Here is the second message.
... """)
>>> from email.mime.multipart import MIMEMultipart
>>> msg = MIMEMultipart('mixed', boundary='BOUNDARY',
...                     _subparts=(part_1, part_2))
>>> process(mlist, msg, {})
>>> print(msg.as_string())
Content-Type: multipart/mixed; boundary="BOUNDARY"
MIME-Version: 1.0

--BOUNDARY
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

header
--BOUNDARY
From: aperson@example.org

Here is the first message.

--BOUNDARY
From: bperson@example.com

Here is the second message.

--BOUNDARY
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

footer
--BOUNDARY--

Decorating other content types

Non-multipart non-text content types will get wrapped in a multipart/mixed so that the header and footer can be added as attachments.

>>> msg = message_from_string("""\
... From: aperson@example.org
... Content-Type: image/x-beautiful
...
... IMAGEDATAIMAGEDATAIMAGEDATA
... """)
>>> process(mlist, msg, {})
>>> msg.set_boundary('BOUNDARY')
>>> print(msg.as_string())
From: aperson@example.org
...
--BOUNDARY
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

header
--BOUNDARY
Content-Type: image/x-beautiful

IMAGEDATAIMAGEDATAIMAGEDATA

--BOUNDARY
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

footer
--BOUNDARY--