The Challenge
Customer churn is the silent revenue killer for SaaS businesses. Our client, a mid-market fintech platform with 120,000 active users, was losing 8% of their subscriber base every quarter — and they had no way to predict who was about to leave.
The traditional approach of surveying users after they've already cancelled is like performing an autopsy to prevent death. We needed a system that could identify at-risk users before they made the decision to leave.
Our Approach
We built a three-stage pipeline that combined behavioral analytics, NLP-driven sentiment analysis, and gradient-boosted decision trees.
Stage 1: Feature Engineering
The most critical step wasn't choosing the right model — it was choosing the right features. We identified 47 behavioral signals across three categories:
- •Usage patterns: Login frequency decay, feature adoption breadth, session duration trends
- •Support interactions: Ticket sentiment scores, resolution satisfaction, escalation frequency
- •Financial signals: Payment failure rates, plan downgrade inquiries, billing page visits
# Feature decay calculation
def compute_usage_decay(user_sessions, window=30):
recent = user_sessions[-window:]
previous = user_sessions[-2*window:-window]
if len(previous) == 0:
return 0
return (np.mean(recent) - np.mean(previous)) / np.mean(previous)Stage 2: Model Architecture
We evaluated Random Forest, XGBoost, and LightGBM classifiers. LightGBM outperformed on our imbalanced dataset after applying SMOTE oversampling.
"The best model is the one your team can maintain, monitor, and improve. Accuracy without operability is a science project, not a product."
Stage 3: Real-Time Scoring
The model runs nightly on a Kubernetes-scheduled job, scoring every active user and pushing high-risk flags into the CRM. Customer success teams receive prioritized outreach lists every morning.
Results
After 90 days in production:
- •34% reduction in quarterly churn rate
- •$2.1M in preserved annual recurring revenue
- •12-day average early warning before cancellation
Key Takeaways
The technical solution was only half the story. The real impact came from embedding the predictions into existing workflows. A model that sits in a notebook is worthless — a model that changes behavior is priceless.
Building for production means building for trust. We invested heavily in explainability features so that customer success managers could understand why a user was flagged, not just that they were flagged.
Don't miss the next architectural breakdown.
Join thousands of engineers who receive our weekly deep-dives on system design, AI/ML, and product engineering.
