raw2md home
GitHubPyPI
View as Markdown

Cleaning

On this page
  1. How the stage edits the text
  2. What cleaning repairs
    1. 1. Headings
    2. 2. Formulas
    3. 3. Lists
    4. 4. Tables
    5. 5. Text
    6. 6. Repeats
    7. 7. Recognition errors
    8. 8. Images and links
  3. What cleaning does not change
  4. When a repair does not apply
    1. Findings
  5. Settings
  6. How to see the result

Cleaning is the first stage after conversion, and it always runs. It repairs the typical defects of the converters without an LLM. Therefore it runs on every input, md included, and in every mode. The stage has no flag of its own.

How the stage edits the text

Cleaning repairs defects with deterministic rules. The same input gives the same result, and a second cleaning pass changes nothing. A rule repairs a defect only when the defect is unambiguous. An ambiguous defect stays in the text and goes into the findings, which quality evaluation uses.

Constants in the code set the thresholds of the rules. A table under each group of rules lists them. The module paths start at src/raw2md/.

What cleaning repairs

1. Headings

Defect Before After
Bold or italics over the whole heading ## **2.3 Load Tables** ## 2.3 Load Tables
A symbol before the number; № and § stay ## ● 4 Maintenance ## 4 Maintenance
A letter-spaced heading; repaired only when the word also occurs in the document without spaces ## I n t r o d u c t i o n ## Introduction
A numbered heading merged with a paragraph; the title is 60 characters at most, and each word starts with a capital letter ### 3.1 Problem Statement. The pump runs… ### 3.1 Problem Statement, with the paragraph on its own
A heading longer than 200 characters ## <long line> the same line as a plain paragraph
A running head recognized as a heading: the number and the title repeat while the section is still open a second ## 5.2 Filters deleted; a repeat without the number stays
The heading Contents repeated on each page of a table of contents, at any level; only a running head can stand between the copies, and a copy with more text after the word stays ## Contents, the rows, #### CONTENTS., the rows ## Contents, with all the rows under it
The level does not match the depth of the number ## 4, ## 4.1, #### 4.3.1 # 4, ## 4.1, ### 4.3.1
A lost heading: a separate paragraph with a multi-level number that continues the numbering of the adjacent headings 4.2 Valves between ## 4.1 and ## 4.3 ## 4.2 Valves
An unnumbered heading at the level of its section or higher ## Notes under ## 4.1 Pumps ### Notes
A heading series (Chapter 1, Chapter 2, and so on) or a heading with 3 copies or more at different levels ## Chapter 1, ### Chapter 2, ## Chapter 3 all at the level of the majority
A skipped level of the structure #, then ### #, then ##
A docx without heading styles: bold numbered paragraphs, when 3 numbers or more follow in sequence under a section number **1.1 Pumps**, **1.2 Valves**, **1.3 Seals** ## 1.1 Pumps, and so on
A docx without heading styles: paragraphs in capital letters, when the document has 3 such paragraphs or more PUMPS ## PUMPS

The level rules change only the number of # signs in the markup. The text of the heading does not change.

Cleaning takes the level of a heading from four sources. They are, from the most reliable: the PDF bookmarks, the table of contents in the text, the numbering, and the nesting of sections. A less reliable source does not overwrite a level that a more reliable source set. Some sources and rules have extra conditions:

  • PDF bookmarks set the levels of the whole document when the text holds at least half of the bookmarks, and 3 at least.
  • The table of contents in the text counts only when the PDF has no bookmarks. The lines of the table of contents stay in place.
  • A docx without styles: the rules in the last two rows of the table work only on a docx or md input. The document must also have no heading at all.
Threshold Constant Module
1 occurrence: the word of a letter-spaced heading occurs in the text without spaces _SPACED_TITLE_WITNESS_MIN cleaning/headings.py
60 characters: the title of a merged heading _RUN_IN_TITLE_MAX_LENGTH cleaning/headings.py
200 characters: the length of a heading _HEADING_MAX_LENGTH cleaning/headings.py
3 copies: a heading series _HEADING_CLASS_MIN_SIZE cleaning/headings.py
3 copies: a repeated heading _REPEATED_TITLE_MIN_COPIES cleaning/headings.py
3 numbers in sequence: a docx without styles _LADDER_MIN cleaning/headings.py
3 paragraphs in capital letters: a docx without styles _CAPS_SEED_MIN cleaning/headings.py
half of the PDF bookmarks in the text _OUTLINE_SKELETON_MIN_SHARE cleaning/headings.py
3 PDF bookmarks in the text _OUTLINE_SKELETON_MIN_TITLES cleaning/headings.py

2. Formulas

Cleaning repairs the markup of a formula, but not its content. A misrecognized symbol stays: its repair is the task of inspection.

Defect Before After
A backslash joined to the closing $$; an escaped sign such as 5\$ stays $$ x = a + b \$$ $$ x = a + b$$
A number directly before a formula, without a space; the notation $5-$6 stays 3$S$ 3 $S$
An inline formula longer than 16 characters, with a sign such as = or <, as a separate paragraph $E = m c^2 + a b c d$ $$E = m c^2 + a b c d$$
$ signs around a text environment $\begin{tabular}…\end{tabular}$ \begin{tabular}…\end{tabular}
A formula of one symbol; a letter formula such as $R$ stays $\varnothing$, ${^\circ}$ ∅, °
A formula of spaces only gap $\,\,\,$ here gap here
A formula cut into parts $$ y = a x^2 + b x $$, $$ + c $$ $$ y = a x^2 + b x + c$$
An equation number under the formula or after it $$ F = m a $$, (12) $$ F = m a \tag{12}$$
A punctuation mark under the formula $$ s = v t $$, . $$ s = v t.$$
A condition under the formula; a reference such as [See 45.] stays $$ x = y $$, [n \neq 1] $$ x = y \quad [n \neq 1]$$
Commands that KaTeX does not support \mbox{max}, \textsc{max}, \eqno(3) \text{max}, \text{max}, \tag{3}
The \label command x = 1 \label{eq:1} x = 1
A double subscript x_{a}_{b} x_{a}{}_{b}
A quotation mark instead of a double prime t" t''
An HTML tag inside a formula $<sup>^{\ast}</sup>$ $^{\ast}$
\left without a matching \right $\left( a + b$ $\left( a + b \right.$
\left or \right without a bracket $a + b \right$ $a + b$
The tail of a cut environment a & = b \end{split} a = b
A line of the symbol legend inside a formula $$v - \text{flow velocity, m/s}$$ $v$ - flow velocity, m/s
A display formula in a table row $$…\left| y \right|$$ in a cell the formula on one line, with | escaped
A pandoc formula in a cell of an HTML table <td><span class="math inline">$x$</span></td> blank lines around the cell content; the formula renders

Cleaning applies a \left or \right repair only when the formula renders after it. Otherwise the formula stays as it was.

Threshold Constant Module
16 characters: an inline formula as a separate paragraph _PARAGRAPH_FORMULA_MIN cleaning/math.py

3. Lists

Defect Before After
A numbered list recognized as a bulleted list; the numbers run in sequence - 1. Drain, a nested - 2. Remove, - 3. Fit 1. Drain, 2. Remove, 3. Fit
A single item one level deeper after two items or more; two such items or more count as a real nested list - Oil, - Water, a nested - Air - Oil, - Water, - Air
A bold number without a space in a list item - **10.**Check - **10.** Check
The • symbol at the start of a line • Stainless steel - Stainless steel
The ❚ glyph at the end of an item or on a line of its own - Oil ❚ - Oil

Numbers with a gap (2023., 2025.) stay as they are. They are the text of the author, not a parsed list.

4. Tables

Defect Before After
Records laid out in several column groups, with a repeated header row | No. | Size | No. | Size | | No. | Size |, with the records one under another
A table cut by a page break: the header row repeats, and no text separates the parts two tables with the same header row one table
A table without data rows | Size | Weight |, |---|---| Size Weight
All values written into one cell of the header row; each column holds values of one kind | Size | Weight | M6<br>10<br>M8<br>12… | the rows | M6 | 10 |, | M8 | 12 |, and so on
Text attached to the header row The sizes are: | Size | Weight | the paragraph The sizes are: and the table
Rows with | and no delimiter row; at least 2 rows and 2 columns | Size | Weight |, | M6 | 10 | |---|---| under the first row
The delimiter row as the first line of the block |---|---| above the rows the delimiter row under the first row
An empty column | Size | | Weight | | Size | Weight |
The values of two adjacent cells written into one cell, and the other cell is empty | 1,000.0 1,027.3 | | | 1,000.0 | 1,027.3 |
The values of a row shifted across the columns; repaired only when the kinds of values allow exactly one layout a part number and the sizes in one cell each value in its own column
A table of contents in one column | 1 Pumps ........ 5 | 1 Pumps 5
<br> in a cell of the table of contents 1 Pumps and<br>valves 1 Pumps and valves
An entry of the table of contents split across two table rows the title and the page on different rows one row
Cells padded with spaces to the column width cells of different lengths aligned with spaces a table up to 120 characters wide stays aligned; a wider table gets one space around each value

The rules for a table of contents work only between the heading Contents and the next heading. A table of contents that runs over several pages counts as one part: cleaning deletes the repeated heading Contents first, so the rules reach the rows of every page. Another heading between the pages still ends the part. The table rules do not change the values in the cells.

Threshold Constant Module
2 rows: rows with | and no delimiter row _DELIMITERLESS_MIN_ROWS cleaning/tables.py
2 columns: rows with | and no delimiter row _DELIMITERLESS_MIN_WIDTH cleaning/tables.py
120 characters: the width of an aligned table _TABLE_WIDTH_CEILING cleaning/tables.py

5. Text

Defect Before After
Navigation markup of raw2md-engine and marker See Table [2,](#page-49-0) See Table 2,
A file path instead of the image caption ![C:\Users\me\pump.png](media/image1.png) ![](media/image1.png)
A line of punctuation marks only; --- and a single mark stay .,;: deleted
Invisible characters: the soft hyphen (U+00AD), the zero-width characters (U+200B to U+200D), and the byte order mark (U+FEFF) pre + U+00AD + ssure, which looks like pressure on screen pressure
Bold over a whole paragraph longer than 120 characters, with two sentences or more **The pump must be stopped… housing.** no **
A bold letter attached to a word **I**nterface Interface
A letter of another alphabet in a word; a word whose letters all exist in both alphabets does not change Cоntrol (a Cyrillic о) Control
A space inside a Roman numeral Fig. V II.11 Fig. VII.11
A space inside a clause number; the sequence 1. 2. 3. stays 82. 1. Scope 82.1. Scope
No space after the # signs of a heading ##Text ## Text
A running head: a title and a page number repeat 4 times or more, one page apart Hydraulic Pump Manual, Page 14 deleted
A word break with a hyphen; the join needs a confirmation from the PDF text layer, or a joined word that occurs in the text 2 times or more; S7-200 is not joined actu-, ator actuator
A hyphen in a compound word; repaired when the other spelling occurs 3 times more often costeffective, load-ing cost-effective, loading
A letter-spaced word; repaired when the spelling without spaces occurs in the text 2 times or more i m p o r t a n t important
A paragraph broken into lines: 6 short paragraphs or more in a row, where a phrase breaks off and continues with a lowercase letter The pump is started, only after the tank, and so on The pump is started only after the tank …
Consecutive blank lines and spaces at the end of a line one blank line; the spaces deleted
Threshold Constant Module
120 characters: a bold paragraph _WHOLE_PARAGRAPH_EMPHASIS_MIN_CHARS cleaning/prose.py
2 sentences: a bold paragraph _WHOLE_PARAGRAPH_EMPHASIS_MIN_SENTENCES cleaning/prose.py
4 repeats: a running head _FURNITURE_MIN_PAIRS cleaning/prose.py
2 times: the joined word of a word break in the text _MERGED_FORM_MIN_COUNT cleaning/prose.py
3 times more often: the spelling of a compound word _COMPOUND_MAJORITY_RATIO cleaning/prose.py
2 times: a letter-spaced word without spaces in the text _SPACED_WITNESS_MIN cleaning/prose.py
6 paragraphs: a paragraph broken into lines FLATTENED_PARAGRAPH_MIN_RUN cleaning/segments.py

6. Repeats

Recognition sometimes falls into a loop and prints the same piece many times in a row. Cleaning cuts such a repeat down to one copy.

Repeat Threshold Before After
A piece of 3 to 80 characters in a formula 6 times \sum_{i}\sum_{i}…\sum_{i} \sum_{i}
A piece of 3 to 80 characters in the text 10 times abc abc … abc abc
The same inline formula in a row 6 times if the formula holds one of \ { } _ ^, otherwise 20 $v_1$ $v_1$ … $v_1$ $v_1$
A line or a block in a row 3 copies three identical lines one line

Two copies in a row do not count as a repeat. Table rows and the lines of a multi-line formula do not count either. If the cut leaves only a fragment of a formula, cleaning deletes the whole formula. The repeat then goes into the findings as broken-formula.

Threshold Constant Module
3 characters: the shortest piece _LOOP_MIN_PERIOD mdtext/loops.py
80 characters: the longest piece _LOOP_MAX_PERIOD mdtext/loops.py
6 times: a piece in a formula; an inline formula with \ { } _ ^ _LOOP_MIN_REPEATS mdtext/loops.py
10 times: a piece in the text _PROSE_LOOP_MIN_REPEATS mdtext/loops.py
20 times: an inline formula without \ { } _ ^ _SPAN_RUN_PLAIN_MIN_REPEATS mdtext/loops.py
3 copies: a line or a block _REPEATED_COPY_MIN_RUN cleaning/findings.py

7. Recognition errors

Both repairs rely on a correct spelling that occurs in the text at least as often as the threshold. The cleaning.witness_min field in settings.json sets the threshold: 20 by default, and 2 at least. A lower threshold repairs more words. However, it can replace a word that the document really uses only once.

Defect Before After Condition
A code lost its case; a code is a word with a digit and 2 letters or more ru9101100 RU9101100 RU9101100 occurs in the text at least as often as the threshold
A word of 6 letters or more occurs once and differs by one letter from a frequent word pressuse pressure pressure occurs in the text at least as often as the threshold, and no other word is that close
Threshold Constant Module
20 times: the default threshold DEFAULT_WITNESS_MIN cleaning/ocr.py
2 times: the lowest threshold _WITNESS_MIN_FLOOR settings.py
2 letters: a code _MIN_CODE_LETTERS cleaning/ocr.py
6 letters: a word that cleaning can repair _MIN_REPAIRABLE_WORD cleaning/ocr.py

The repairs of images and links depend on the run mode. Without image extraction, an image link points to a file that does not exist. In LLM-OCR mode, the model invents images and links that the source does not have.

Mode Before After
--disable-image-extraction ![Pump housing](missing.png), <img src="missing.png" alt="Pump"> the caption as a separate paragraph; the link deleted
LLM-OCR: an image ![Pump](img.png) deleted with its caption
LLM-OCR: an external link [site](https://example.com) site
LLM-OCR: a model placeholder in place of an image [Image of a pump] deleted; if it repeats the caption, the caption stays without the brackets

The model markers [unreadable] and [?] stay. Outside LLM-OCR mode, external links do not change.

What cleaning does not change

Cleaning edits the whole document, except for:

  • the YAML header;
  • code blocks;
  • HTML tables. Cleaning does not change the markup of the table. In the cells, it makes three repairs only. It deletes a file path from alt. In LLM-OCR mode (for example, -e gemini_api), it deletes the images and links that the model invented. It also puts blank lines around the content of a cell with a formula, so that the formula renders.

When a repair does not apply

Findings

Findings are defects that cleaning detects but does not repair, because the repair would be ambiguous. Quality evaluation puts each finding into its check. It counts some defects from the text itself.

Finding What it is Quality evaluation
hyphenation a word break that nothing confirms as a join hyphenation
hyphenation-gap the same word break, with the second half after a blank line, an image, or a page break hyphenation
flattened-block 6 lines or more in a row, each up to 25 characters long; an unnumbered heading longer than 200 characters flattened_blocks
flattened-paragraphs a paragraph broken into lines that the join rule did not repair flattened_blocks
run-in-heading a heading merged with a paragraph, with a function word in its title (Design for Assembly) flattened_blocks
broken-formula a formula cut because of a repeat; an unclosed environment lost_formulas
broken-table a row with a different number of cells; the values of a row in one cell; a table without a header row counted from the text
broken-image a link to an image file that does not exist counted from the text
letter-spacing a letter-spaced word whose spelling without spaces does not occur in the text not counted
Threshold Constant Module
6 lines: flattened-block FLATTEN_MIN_RUN cleaning/segments.py
25 characters: flattened-block FLATTEN_MAX_CHARS cleaning/segments.py
200 characters: flattened-block _HEADING_MAX_LENGTH cleaning/headings.py

Settings

Some rules find the parts of a document by keywords, for example Contents, Page, Appendix, and Fig.. The keywords.yaml file holds the word list. To add the words of your language, run raw2md keywords edit.

The cleaning.witness_min threshold in settings.json controls the repair of recognition errors. Section 7, Recognition errors, describes it.

How to see the result

Three places show the result of the stage:

  • the difference between conversion.md and cleaning.md in the <name>.debug/ folder, after a run with --debug (-d): everything that cleaning did;
  • the log ~/.raw2md/logs/raw2md.log: the number of findings of each kind, and each finding with its line number;
  • the header of the result: status and issues, where quality evaluation sums up the findings.