The problem
People increasingly write about distress online, on Reddit, on Twitter, in support forums, and there is far more of it than any moderator or clinician can read. An automated classifier could help sort it. The catch is that the accurate models are big and slow, and the fast ones miss things. In mental health, "misses things" can mean missing someone who is suicidal.
Where I started, and why it was wrong
My first version used BERT as a frozen feature extractor. BERT turned each post into a 768-dimensional embedding, and a Random Forest made the final call. The theory was that a lightweight classifier on top would cut inference cost.
It didn't. Every input still has to go through the full BERT encoder to get its embedding, no matter what sits on top. Swapping one linear layer for 300 decision trees added cost instead of removing it. Accuracy barely moved (0.8300 against 0.8312) and suicidal recall dropped 1.67 points. The encoder was where the time went, so that is where I had to look. I wrote more about this in a separate post.
Four axes, measured one at a time
I ended up treating deployment as an optimization problem along four separate axes and changing one at a time, so I could say exactly how much each one contributed, including the ones that didn't help:
- Classification head. The Random Forest made things slower, as above.
- Encoder. Distilling BERT down to DistilBERT roughly halves latency for a small accuracy cost.
- Numeric precision. Quantizing the encoder to 8-bit integers roughly halves it again.
- Runtime. Exporting to ONNX Runtime is numerically lossless, but only accounts for 5 to 9 percent of the total speedup.
Picking a model honestly
I borrowed the selection rule from MLPerf Inference: an optimized model has to keep at least 99% of the reference model's accuracy and 99% of its recall on the Suicidal class. Among the configurations that cleared both bars, the fastest one gets deployed.
Fig. 1. Speed against what you give up
Show the numbers
| Configuration | Accuracy | Macro-F1 | Suicidal recall | ms / request | Size (MB) |
|---|---|---|---|---|---|
| BoW + logistic regression | 0.7608 | 0.6841 | 0.6711 | 0.84 | 4.2 |
| TF-IDF + logistic regression | 0.7758 | 0.6883 | 0.6788 | 0.98 | 4.6 |
| BERT fp32, PyTorch (reference) | 0.8312 | 0.7964 | 0.7132 | 186.4 | 438.7 |
| BERT int8, ONNX Runtime | 0.8007 | 0.7406 | 0.7787 | 84.4 | 111.4 |
| DistilBERT fp32, PyTorch | 0.8206 | 0.7839 | 0.7424 | 93.2 | 268.8 |
| DistilBERT int8, ONNX Runtime (deployed) | 0.8231 | 0.7876 | 0.7103 | 43.8 | 68.3 |
That was DistilBERT, quantized to int8, served through ONNX Runtime: 82.3% accuracy across seven classes, 43.8 ms per request instead of 186.4, and 68 MB instead of 439. It clears both bars, but narrowly. A paired bootstrap puts the 95% confidence interval on its accuracy ratio at 98.31% to 99.74%, so I don't treat its edge over unquantized DistilBERT as settled.
It also isn't the safest option on the table. Unquantized DistilBERT and quantized BERT both miss fewer suicidal statements. I chose the deployed model because it meets the bar at less than half the cost of anything else that does, which is a reasonable call for a prototype on limited compute. Anyone relying on it should know that, so it's written down in the paper and the README instead of left out.
Checking the data before trusting it
The dataset is 53,043 posts compiled on Kaggle from nine separately collected and separately labeled sources, so I audited it before training anything:
- 3.27% of rows were exact duplicates after normalization.
- 30 statements showed up with conflicting labels, and 57% of those conflicts were Depression versus Suicidal.
- 5.87% of the test set overlapped with the training data, concentrated in the smallest classes.
Every number I report uses a leakage-filtered test set of 9,919 posts. Removing the leaked rows lowered accuracy by only 0.3 to 0.5 points, but lowered macro-F1 by 1.5 to 2 points, because the leakage sat in the minority classes. If I had only looked at accuracy, I would have missed it completely.
Where it gets things wrong
The most common confusion is Depression versus Suicidal. That matches the clinical overlap between the two, and it matches the label conflicts in the data itself, so some of that "error" is probably inconsistent supervision rather than the model. When I tested the earlier version on hand-written edge cases, it handled hyperbole ("this math test is going to end me") and sarcasm well, but read "I am absolutely killing it at my new job" as negative, and read quiet, apologetic posts that minimize their own pain as stress instead of depression. That last kind of miss matters most, because understatement is how a lot of people actually talk about feeling bad.
Read it, try it
The paper was published in IJSCAR in September 2026. The demo runs the deployed model. If it assigns at least 25% probability to the Suicidal class, it shows crisis resources alongside the result. It is a research prototype, not a diagnostic tool.
Ayaan Goswami and Clayton Greenberg. Precision versus Efficiency: A Quantized DistilBERT Framework for Automated Mental Health Text Triage. International Journal of Secondary Computing and Applications Research, vol. 3, no. 3, pp. 21-35, 2026. doi:10.67149/yhjs2024.5/5wcbbc4m