Google ML Bootcamp: Building a Financial Advisor Chatbot with Gemma-2B

From the summer of 2024 I spent five months in Google ML Bootcamp, starting with used-car price prediction and ending with a financial advice chatbot built on Gemma. This post is about that last project, and in particular getting LLM training to finish on a small GPU.

Enter a user profile and it returns advice tailored to it.

Fundamentals: used-car prices

The early weeks were a competition predicting used-car prices with scikit-learn, XGBoost and LightGBM. More than calling models, it meant studying how variables relate, building derived features and searching hyperparameters to push the score as far as it would go. The habit of breaking a problem down carried straight into the LLM project.

Final sprint: a profile-based finance advisor

The last task was a sprint with Google's Gemma. The goal was to adapt the general gemma-2-2b-it model to financial advice, and the real challenge was doing it inside a Kaggle notebook's limited resources.

Deciding what to teach first

Since the model is instruction-tuned, the data came as instruction and model-answer pairs: a user profile in a fixed text format as input, suitable advice as output, so it learned to answer in context.

Running out of memory

Training a 2B model in 16-bit on Kaggle's T4 hit out-of-memory errors again and again. Two techniques together got past it:

• 4-bit quantization. Weights stored as NF4: about 5.0 GB in 16-bit becomes about 1.25 GB, roughly 75% less.
• LoRA. Instead of the whole model, small adapters beside the original weights are trained: about 12.8 million parameters, around 0.5% of the model.

1from peft import LoraConfig
2
3lora_config = LoraConfig(
4    r=64,
5    lora_alpha=16,
6    target_modules=["q_proj", "v_proj"],
7    lora_dropout=0.05,
8    bias="none",
9    task_type="CAUSAL_LM",
10)

Result

After only three epochs with Hugging Face's SFTTrainer the model started giving plausible advice in the profile format. I merged the trained adapter into the base model so it loads anywhere as a single model.

What I learned

When the memory errors kept coming I honestly wanted to quit, but that pressure is what made me dig into how quantization and LoRA work inside. I learned fastest when stuck.

Data design mattered more than model size: small changes to the input format visibly changed answer quality. And on a personal GPU, LoRA is the realistic place to start.

From used-car prices to fine-tuning Gemma was a long way for five months. Next time I would like to do it again with a bigger model and more data.