Language and classification models built with fastai and deployed on Render.
When a new customer contacts our office to set up a sales appointment, a customer service representative asks some basic questions about how we can best serve them. Some common questions include how the customer heard about us, what problems or needs they have, where they are in the buying process, etc. These answers are typed into our CRM software and are relayed to our sales professionals prior to the appointment date. Here is an example:
Cust Interested in ductless mini split for part of home that doesn’t get cold enough or hot enough with the existing heat & air. Cust in a hurry over the phone, wasn’t able to get more info. New customer, found us by Googling “ductless split system” in Woodland CA. GB1/3/18
The purpose of this research project was to classify the probability that we could meet the customer’s needs. In other words, the sales appointment would culminate in work sold. This information was used to help the office triage customers during times of excessive demand, decide whether to send a salesperson or service technician, and to help sales professionals calibrate their approach to improve each customer’s experience.
The text for roughly 3000 calls was joined with a binary variable indicating whether a we had had gone into business with the customer within 60 days of the initial sales appointment.
Given that the text data contained abbreviated words, mis-spellings and other aspects of the written language that would be difficult for a model to parse, I knew that I would need a state of the art model to analyze this text.
For that reason, I used fast.ai’s universal language model fine-tuning (ULMFiT). The ULMFiT model is a deep learning neural network designed to be trained on text. What makes the ULMFiT model unique is that it has been pre-trained on a lot of text- 10 million Wikipedia articles to be precise. So by the time it sees my data, it has basically learned English.
The model was built using the fastai v1 for PyTorch library in Python 3. The IDE was a Jupyter Notebook that you can view here. The IDE was hosted on Google Colab, which was chosen for its built-in libraries, remote capabilities, free GPUs and direct access to my Google Drive storage.
The model building proceeds in roughly two steps: first, building a language model, and second, building a classification model.
The purpose of constructing the language model is to expose the pre-trained neural network to the specific vocabulary my dataset. The objective function of the language model is to predict the next word in a sentence. After tuning the model parameters, the model achieved about 40% accuracy and could even imitate intelligence by completing sentences:
The classification model determines, given some text input, the probability that the job will culminate in a sale.
The model was deployed on Render and can be seen in the video below:
After fine-tuning, the model achieved about 79% accuracy on the test data compared to a naive baseline of 61%. There were relatively few false negatives (86% specificity) while still performing having better than naive classification for positive results (61% sensitivity). Given that there are many other variables that determine whether a sale is a success, perfect prediction is not possible based solely on the data used. The model accuracy could be improved by outputting the classification probabilities into an ensemble machine-learning model that accounts for other customer characteristics.