From Failure to Top 12%: Three Kaggle Lessons for Data Scientists
Participating in the international Kaggle competition ASHRAE on predicting building energy consumption was a turning point for me. Starting from 2431st place out of 3614, over five months of work with 158 late submissions, I climbed to 558th, entering the top 12%. This journey taught me three key lessons, which I'm sharing with fellow data scientists.
Lesson 1: Only Winners Can Be Your Teachers
After the competition ended in December 2019, I faced harsh reality: my private score of 1.415 (2431st place) was far from victory. In January 2020, the winners published their solutions—and that became my salvation. Most teams openly shared model architectures, code, and data preprocessing strategies. I dedicated a whole day to studying the key solutions and realized: to achieve results, you need to copy the methods of top teams, not reinvent the wheel.
Despite 12 years of experience developing mathematical models for the Russian wholesale electricity market, I felt like a complete newbie. Initial attempts to implement complex pipelines ended in failure: expectations didn't match reality, metrics worsened. In an email to a colleague, I described it as a "feeling of horror." But that's where Kaggle's power lies: the platform forces you out of your comfort zone. Each late submission is a step toward understanding state-of-the-art solutions. After 50 iterations, a sense of "it's working" began to form, and by the 158th upload, I entered the top 12%.
Lesson 2: An Effective Data Scientist Operates with Multiple Models
ASHRAE opened up a world of libraries I hadn't worked with before: LightGBM, CatBoost, Prophet. Winners used combinations of thousands of models, blended together. For example, the second-place team created 20,000 models, mixing results from XGBoost, LightGBM, CatBoost, and neural networks. Key insight: for effective ensembling, two conditions are needed:
- Models must demonstrate comparable accuracy
- Algorithms must introduce diversity in predictions
My solution was built on averaging three sets:
- 56 LightGBM models by territories
- 24 LightGBM models by energy types
- 36 XGBoost models by territories
Preprocessing included linear interpolation of weather data and removal of constant energy consumption values. Lack of post-processing and limited model diversity explain my ~12% result versus top-1% for winners. Conclusion: single-algorithm solutions are doomed on Kaggle. Efficiency is achieved through hybrid pipelines with controlled diversity.
Lesson 3: Extensive Experience Is an Inevitable Stage of Mastery
Working with hyperparameters requires systematic grid search. For LightGBM and XGBoost, I tested dozens of combinations, changing only one parameter at a time. The total number of computed predictions exceeded 16 billion: 158 uploads × 20M records for training + 158 × 42M for prediction. Most experiments were discarded before submission, but they formed the intuition.
Now I instantly see how changing num_leaves in LightGBM affects overfitting, or why regularization reg_lambda is critical for XGBoost on noisy data. This isn't theoretical knowledge—it's muscle memory developed through thousands of iterations. Kaggle is ideal for such training: you can experiment without risk to commercial projects. In work tasks, I can't spend 4 hours a day on hyperparameter tuning, but on Kaggle, this "time loss" becomes an investment in professional growth.
Key Takeaways
- Learn from Winners: Their solutions are the best textbook. Analyze their approaches, even if it bruises your ego.
- Use Ensembles: Blend results from multiple models (LightGBM, XGBoost, CatBoost) to boost accuracy. Diversity is key.
- Build Extensive Experience: Repeated iterations with hyperparameters build intuition. Kaggle is the perfect playground for this outside commercial projects.
— Editorial Team
No comments yet.