KRAFTON AI R&D Hackathon Prelims: Teaching Multiplication with 7,000 Parameters, Recovering a Rule from Noise
See also
Loading comments...
I did not advance to the finals of the KRAFTON AI R&D Hackathon, but the online preliminary round on March 28–29, 2026 was among the most concentrated two-day blocks I have worked through recently. The first task was a compact Transformer for 6-bit binary multiplication under fixed training rules (MultiplierBoard); the second was noisy LFSR / LPN-style recovery (SparseTap). This section summarizes the constraints, approaches, and why I believe I missed the cut for the next round.
Per the poster: applications closed 2026-03-27 12:00, round 1 was Mar 28–29 (online), finals Apr 4–5 (offline).
Day 1: MultiplierBoard
Problem
- Autoregressively predict the 12-bit product of two 6-bit integers A and B (as bit sequences).
- Fixed training protocol (AdamW, cosine schedule, 200 epochs, etc.). You need validation accuracy Acc2 above 99% while pushing trainable parameter count P2 as low as possible.
- Also submit a written P1 argument: a hand-crafted lower bound on how small a solution could be.
I initially treated the problem like a small language model: label smoothing, scheduled sampling, and hand-tuned ReLU shapes meant to mimic AND gates. Neither path led to stable training or acceptable validation accuracy.
Problem 1: exposure bias
Padding targets with zeros taught a shortcut: if the prefix looks like padding, emit zeros instead of doing real multiplication. Teacher forcing looked fine on paper; the model was really fitting padding artifacts.
Problem 2: vanishing gradients (“dead” activations)
Aggressive thresholding cut off gradients; accuracy remained near 3%. The model had effectively removed differentiable paths while chasing a logic-gate metaphor.
What worked
I removed NLP-oriented heuristics and relied on input design and architecture only.
- Aligned inputs
Rebuilt the pipeline so training sees something like A + B + P[:-1] + [0], and evaluation autoregressively fills from A + B + [0]*12 one bit at a time.
- Weight tying
With a binary vocab, share embedding and classifier weights: fc.weight = embed.weight.
- Learned positions
nn.Parameter positional encodings let the model pick up multiplication geometry (e.g. diagonal index patterns) without hand-coded sin/cos.
- Small capacity
d_model=16, nhead=2, layers=2, FFN 64, dropout 0, norm_first, GELU.
Causal mask: upper triangle set to -inf so only past tokens attend.
python
1class TinyTransformer(nn.Module):
2 def __init__(self):
3 super().__init__()
4 self.d_model = 16
5 self.nhead = 2
6 self.num_layers = 2
7 self.embed = nn.Embedding(2, self.d_model)
8 self.pos_encoding = nn.Parameter(torch.randn(1, 24, self.d_model) * 0.02)
9 encoder_layer = nn.TransformerEncoderLayer(
10 d_model=self.d_model,
11 nhead=self.nhead,
12 dim_feedforward=self.d_model * 4,
13 batch_first=True,
14 norm_first=True,
15 activation="gelu",
16 dropout=0.0,
17 )
18 self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=self.num_layers)
19 self.fc = nn.Linear(self.d_model, 2, bias=True)
20 self.fc.weight = self.embed.weight # weight tying
21
22 def forward(self, x):
23 seq_len = x.size(1)
24 mask = torch.triu(
25 torch.full((seq_len, seq_len), float("-inf"), device=x.device),
26 diagonal=1,
27 )
28 x = self.embed(x) + self.pos_encoding[:, :seq_len, :]
29 x = self.transformer(x, mask=mask, is_causal=True)
30 return self.fc(x)My Day 1 numbers
| Metric | Value |
|---|---|
| P1 (theoretical lower bound I argued) | 36 |
| P2 (trained params) | 6,978 |
| Acc2 | 99.98% |
P1 was a rough counting-style argument; P2 is the actual parameter count of the model above.
Day 2: SparseTap
Problem
- Many length-256 binary sequences with ~20% Bernoulli noise.
- Hidden rule: LFSR-like XOR with at most S ≤ 16 taps.
- Recover taps and predict 192 clean bits continuing from a given 64-bit seed.
Pure deep learning fights flat objectives and discrete XOR, classic LPN territory.
Attempt 1: continuous relaxation + SIREN
Rules asked for a numerical method, so I mapped {0,1} → {-1,1}, treated XOR like multiplication, and ran a wide two-layer SIREN net (sin(30·Wx)) with Sign-SGD.
For a while ~49.8% accuracy and plausible-looking offsets seemed promising. Under 20% label noise, however, a solution that truly recovers the parity rule should score near 80% on held-out checks, not near 50%. The network had likely fit noise, not the underlying rule.
Attempt 2: brute-force correlation
GPU correlation-style search worked until S ≈ 6; ground truth was closer to S = 8, and combinatorics at C(64,8) blew memory.
Final approach: ISD + bit packing + Numba
Randomly sample 64 equations, Gaussian elimination over GF(2), drop solutions with Hamming weight > 16, then verify candidates with packed uint64 inner products and popcount on a 10k subset.
Clean-system probability per trial scales like 0.8^64, so expected trials sit in the millions. Elimination is branchy: Numba JIT on one CPU core beat a GPU for my implementation.
Skeleton matches run_isd_ultra_fast in my solution.py:
python
1@njit
2def run_isd_ultra_fast(X_packed, Y, max_trials=5000000):
3 N = len(Y)
4 N_sub = 10000
5 X_sub_packed = X_packed[:N_sub]
6 Y_sub = Y[:N_sub]
7 for trial in range(1, max_trials + 1):
8 A_packed = np.zeros(64, dtype=np.uint64)
9 B = np.zeros(64, dtype=np.int8)
10 for i in range(64):
11 idx = np.random.randint(0, N)
12 A_packed[i], B[i] = X_packed[idx], Y[idx]
13 # GF(2) Gaussian elimination on A_packed, B
14 # reject if Hamming weight > 16
15 # pack solution, verify vs X_sub_packed / Y_sub; if correct/N_sub > 0.75 return
16 return np.zeros(0, dtype=np.int8)My Day 2 numbers
- Converged around trial 3,612,183, about 95 seconds on one Numba’d CPU core.
- Recovered taps (1-based):
[5, 14, 21, 29, 36, 42, 50, 57] - Check accuracy ~79.99%, matching the ~80% ceiling you expect under 20% label noise.
- Submitted the 192-bit continuation in the required format.
Takeaways
- Don’t let models lean on padding and noise quirks: force the real operation in the inputs.
- GPUs for fat matmuls; CPU + JIT + bitwise for GF(2) elimination.
- The noise rate is a calibration signal: if validation is not near 80%, the candidate is unlikely to be the true parity solution.
Why I did not reach the finals
Core pipelines ran on both days; Day 2 also matched the theoretical accuracy ceiling. I still did not pass the cut. My assessment:
- Selection isn’t code-only: reports, P1 writeup quality, tie-breakers on params or clarity probably mattered. I cut P2 hard; maybe P1 prose or packaging didn’t.
- Time on Day 2: I paid hours to SIREN for compliance; ISD was what actually scored. Next time: lock a deterministic path first, treat neural stuff as extra.
- Opaque scoring: undisclosed weights or tie rules can shave you without feedback.
- Repro: one lucky random seed isn’t enough. I should have logged multiple seeds and worst-case runtime in a README.
The weekend was instructive. A correct local solution does not guarantee advancement when selection blends reports, theory, and opaque tie-breakers. For the next event I plan to lock a deterministic approach first, draft the write-up in parallel with code, and timebox each sub-task explicitly.