mozreview: Fix pygments options to not strip \n from files (
bug 1281712) r=smacleod
`pygments` lexer accepts a Boolean `stripnl` option to [not] remove \n from files. We've got it spelled as "stripln" in `overridable_chunk_generator.py` file.
If `stripnl` is set to True (which is a default option) `pygments` removes "\n" characters. This behaviour causes issues with displaying FileDiffs. In a very special case where a second to last line has been changed and the last line is empty (contains only the "\n" character), length of the file changes, but diff markers remain the same. In `chunk_generator.py#843` we assume `lines` is a non-empty list. All empty lines are removed by `pygments` entirely. As a result, `markup_a` and `markup_b` - lists returned from lexer are shorter by that empty lines. Chunk's `lines` are created by slicing markup files with `i1`, `i2`, `j1`, `j2` - opcodes generated with `diff_opcode_generatos` which are operating on files before modified by lexer. Because for the last chunkk `i2 > len(markup_a)` and `j2 > len(markup_b)`, `markup_a[i1:i2]` and `markup_b[j1:j2]` are empty lists for the last chunk. `IndexError: list index out of range` is raised in `chunk_generator.py", line 843`.
MozReview-Commit-ID: ESyugDhKYJD
--- a/pylib/pygments_override/pygments_override/overridable_chunk_generator.py
+++ b/pylib/pygments_override/pygments_override/overridable_chunk_generator.py
@@ -24,17 +24,17 @@ class OverridableDiffChunkGenerator(Diff
"""Applies Pygments syntax-highlighting to a file's contents.
Syntax highlight obeys a explicitly provided list of preferences by
extension or it is derived from the contents of the file.
The resulting HTML will be returned as a list of lines.
"""
lexer = self._get_preferred_lexer(
- filename, stripln=False, encoding='utf-8')
+ filename, stripnl=False, encoding='utf-8')
logger.debug('preferred lexer for %s: %s' % (filename, lexer))
if not lexer:
lexer = guess_lexer_for_filename(
filename, data, stripnl=False,
encoding='utf-8')
lexer.add_filter('codetagify')
