The first version of my mental health text classifier had what I thought was a clever design. BERT is accurate but expensive, so I'd use it only as a feature extractor: run each post through BERT once to get a 768-dimensional embedding, then hand that embedding to a Random Forest to make the actual prediction. A Random Forest is cheap. So the whole thing would be cheaper than running BERT end to end. That was the hypothesis.
It was wrong, and the reason is kind of obvious once you see it.
Where the time actually goes
To get an embedding, every input still has to pass through the entire BERT encoder: twelve transformer layers, every time. The part I replaced, BERT's own classification head, is a single linear layer on top. It was never the expensive part. Swapping that one layer for 300 decision trees didn't remove cost. It added some.
And it didn't buy anything back. Accuracy went from 0.8312 to 0.8300, which is basically a tie, while recall on the Suicidal class dropped 1.67 points. For this task, that's the number I least want to drop.
Measuring one thing at a time
So I started over and treated deployment as its own problem, with four separate places I could change something:
- the classification head (linear layer or Random Forest),
- the encoder (BERT or the smaller, distilled DistilBERT),
- the runtime (PyTorch or ONNX Runtime),
- the numeric precision of the weights (32-bit floats or 8-bit integers).
Changing them one at a time meant I could say exactly what each one contributed. The head, as above, hurt. Distillation roughly halved latency for a small accuracy cost. Quantizing to int8 roughly halved it again. ONNX Runtime was numerically lossless but only accounted for 5 to 9 percent of the speedup. Two of the four axes did nearly all the work, and the one I'd originally bet on was the one that made things worse.
Deciding what counts as good enough
Faster is easy if you're willing to be worse, so I needed a rule before looking at results. I borrowed the one MLPerf Inference uses: an optimized model has to keep at least 99% of the reference model's accuracy and 99% of its Suicidal recall. Of everything that clears both, deploy the fastest.
Fig. 1. Speed against what you give up
Show the numbers
| Configuration | Accuracy | Macro-F1 | Suicidal recall | ms / request | Size (MB) |
|---|---|---|---|---|---|
| BoW + logistic regression | 0.7608 | 0.6841 | 0.6711 | 0.84 | 4.2 |
| TF-IDF + logistic regression | 0.7758 | 0.6883 | 0.6788 | 0.98 | 4.6 |
| BERT fp32, PyTorch (reference) | 0.8312 | 0.7964 | 0.7132 | 186.4 | 438.7 |
| BERT int8, ONNX Runtime | 0.8007 | 0.7406 | 0.7787 | 84.4 | 111.4 |
| DistilBERT fp32, PyTorch | 0.8206 | 0.7839 | 0.7424 | 93.2 | 268.8 |
| DistilBERT int8, ONNX Runtime (deployed) | 0.8231 | 0.7876 | 0.7103 | 43.8 | 68.3 |
That turned out to be DistilBERT, quantized to int8, on ONNX Runtime: 43.8 milliseconds per request instead of 186.4, 68 MB instead of 439, at 82.3% accuracy. It clears both bars, but not by much, and it isn't the safest model on the chart. Unquantized DistilBERT and quantized BERT both catch more suicidal posts. I picked it because it meets the bar at less than half the cost of anything else that does, and I wrote that tradeoff down in the paper instead of hoping nobody would ask.
What I'd tell myself at the start
Measure where the time goes before designing around it. My original architecture was built on an assumption I never checked, and a profiler would have killed it in an afternoon. The second thing: pick the rule for "good enough" before you see the numbers. Otherwise it's very easy to draw the line wherever your favorite model happens to land.
The full write-up, including the data audit that turned up train/test leakage, is on the project page, and the paper itself is published in IJSCAR.