My 'Math God' Project: What I Learned from the Boostcourse Math OCR Course
Loading comments...
Loading comments...
I've been working on a personal project called 'Math God': the idea is to build an AI that can read and solve handwritten math problems. Sounds simple, but the very first step hit a wall: how do you even get a machine to _read_ a formula image in the first place?
Beyond simple text extraction, I needed technology to convert 2D structured formulas like superscripts, fractions, and roots (√) into a form that machines can understand. To find an answer to this concern, I enrolled in the advanced course 'Deep Learning Model for Understanding Mathematical Formulas (OCR)' from Naver Connect Foundation's Boostcourse.
General OCR (Optical Character Recognition) reads text in images sequentially from left to right. But mathematical formulas are different.
2D Spatial Structure: Fractions are arranged vertically, and superscripts/subscripts are arranged diagonally up and down.
Contextual Meaning: The same symbol can have different meanings depending on its position (e.g., x² vs. x₂).
Structured Output: It must be converted to a format like LaTeX that can perfectly express the structure of formulas, not just simple text.
In practice, formula OCR needed two parts to work together: understanding 2D spatial structure in images (Computer Vision) and generating valid sequential representations (NLP).
The course deeply covers the standard model architecture for solving this problem. I understood this structure by dividing it into three parts: 'eyes', 'brain', and 'connection'.
The 'eyes' role is handled by a CNN (Convolutional Neural Network)-based encoder. Using powerful image recognition models like ResNet or VGG, it extracts visual features from formula images. The result is a high-dimensional feature map containing information about each symbol's shape and relative positions.
The 'brain' role is performed by a Transformer-based decoder. It receives the image feature map extracted by the encoder and generates LaTeX string tokens one by one. Rather than simply listing symbols, it learns grammatical and structural relationships like after a fraction symbol, the numerator must come to logically reconstruct the entire formula.
What effectively connects 'eyes' and 'brain' is the Attention mechanism. Each time the decoder generates a LaTeX token, attention tells it which part of the image to focus on. For example, when generating the numerator, it pays higher attention to the upper region of the image, and when generating the denominator, it focuses on the lower region. This is the key that enables the model to accurately unravel complex 2D structures into sequential language.
The knowledge gained from this course greatly helped concretize the direction of my project.
Synthetic Data Generation (Data Augmentation): Since real formula image data is hard to obtain, I plan to generate various LaTeX formulas randomly and render them as images to build large-scale training data. I will apply various transformations like fonts, noise, and rotation to increase the model's robustness.
Custom Tokenizer: I realized that I need to build a custom LaTeX tokenizer centered on symbols frequently used in the domain, tailored to the category of math problems my project will handle (e.g., high school calculus). This will directly impact the model's learning efficiency and performance.
Beyond Recognition to Interpretation: The OCR model outputting LaTeX strings is just the first step of the 'Math God' project. The next step is to develop a module that parses this LaTeX and converts it into symbolic expressions or computation graphs that can actually perform operations.
This Boostcourse lecture was useful because it gave me a concrete system design frame, not just model code. I could map that frame directly to my own 'Math God' pipeline and identify what to build next.