GenAI: Improving Customer Communication Using Generative AI and DataRobot

This notebook aims to provide an example of how generative AI models like GPT-3 can be used to augment predictions and provide customer friendly subject matter expert responses.

Build with Free Trial

A crucial part of the machine learning life cycle is the effective consumption of the prediction results by the end users. A good machine learning model provides not only the prediction, but also auxillary information like prediction explanations and the prediction threshold. This additional information is crucial for interpreting and recommending subsequent actions for the user consuming the model predictions. However, all this information is technical in nature and an end user not familiar with this information might not be able to utilize its full potential.

This notebook aims to provide an example of how generative AI models like GPT-3 can be used to augment predictions and provide customer friendly subject matter expert responses. The example chosen for this notebook shows how a generative AI model can provide positive user communication to adverse events like loan rejection that is predicted by the model. The generative AI model provides expert advice that is tailored for each individual loan applicant based on the prediction explanations provided by DataRobot.

Positive and engaging customer communication is a key factor for customer success for organizations and DataRobot along with Large Language Model can provide highly tailored, expert level customer communication.

• The dataset used in this notebook is the Lending Club dataset. This dataset can be used to build machine learning models that ingest various features of a loan application and infer if the applicant will default on the loan if approved.
• This notebook will use the prediction explanations from the loan default model and use generative AI to provide positive and domain-expert-level responses to loan applicants whose loans applications have been ejected by the model.
• The notebook assumes data is uploaded and available in DataRobot’s AI Catalog.
• The notebook assumes that you have an API key for OpenAI systems. This method can be used for generative text AI models similar to GPT-3.

Setup

Import libraries

In [1]:

import datarobot as dr
import pandas as pd
import yaml

Configure connections to DataRobot and OpenAI

In [2]:

with open("./settings.yaml", "r") as stream:
    config = yaml.safe_load(stream)

In [3]:

dr.Client(endpoint=config["endpoint"], token=config["token"])

Out [3]:

<datarobot.rest.RESTClientObject at 0x7fb8a3162520>

In [4]:

import openai

openai.api_key = config["openai_key"]

Retrieve a loan default project

A loan default model is already built and a deployment has been created for making predictions on the recommended DataRobot model. Please use this tutorial to create the project and deployment. The dataset used for this project is available here.

In [5]:

projectID = "64a63925cdbc0e8191b96bb0"
project = dr.Project.get(projectID)
project

Out [5]:

Project(Project Big Query)

In [6]:

DEPLOYMENT_ID = "64a63eaaccaae422aae17bbf"
deployment = dr.Deployment.get(DEPLOYMENT_ID)
deployment

Out [6]:

Deployment(is_bad Predictions)

In [7]:

df_inference_id = "64954b1d2ec1de1758d5bb07"
df_inference_dataset = dr.Dataset.get(df_inference_id)
df_inference_dataset

Out [7]:

Dataset(name='gcp-demo-390701-Demo-10K_Lending_club-2023-06-23T07:34:52.472Z', id='64954b1d2ec1de1758d5bb07')

Make predictions from inference data

The following cells illustrate the process of making predictions from inference data and filtering the negative class predictions which have to be communicated to the loan applicants.

In [8]:

# Set the number of top prediction explanations to extract from DataRobot
# This will also be used as the number of bullet points in the prompt response from the LLM
n_explanations = 3

In [9]:

job = dr.BatchPredictionJob.score(
    deployment=DEPLOYMENT_ID,
    intake_settings={"type": "dataset", "dataset": df_inference_dataset},
    output_settings={
        "type": "localFile",
        "path": "./prediction.csv",
    },
    max_explanations=n_explanations,
)

In [10]:

predictions = pd.read_csv("./prediction.csv")
predictions.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 18 columns):
 #   Column                              Non-Null Count  Dtype  
---  ------                              --------------  -----  
 0   is_bad_1_PREDICTION                 10000 non-null  float64
 1   is_bad_0_PREDICTION                 10000 non-null  float64
 2   is_bad_PREDICTION                   10000 non-null  int64  
 3   THRESHOLD                           10000 non-null  float64
 4   POSITIVE_CLASS                      10000 non-null  int64  
 5   EXPLANATION_1_FEATURE_NAME          10000 non-null  object 
 6   EXPLANATION_1_STRENGTH              10000 non-null  float64
 7   EXPLANATION_1_ACTUAL_VALUE          9762 non-null   object 
 8   EXPLANATION_1_QUALITATIVE_STRENGTH  10000 non-null  object 
 9   EXPLANATION_2_FEATURE_NAME          10000 non-null  object 
 10  EXPLANATION_2_STRENGTH              10000 non-null  float64
 11  EXPLANATION_2_ACTUAL_VALUE          9680 non-null   object 
 12  EXPLANATION_2_QUALITATIVE_STRENGTH  10000 non-null  object 
 13  EXPLANATION_3_FEATURE_NAME          10000 non-null  object 
 14  EXPLANATION_3_STRENGTH              10000 non-null  float64
 15  EXPLANATION_3_ACTUAL_VALUE          9798 non-null   object 
 16  EXPLANATION_3_QUALITATIVE_STRENGTH  10000 non-null  object 
 17  DEPLOYMENT_APPROVAL_STATUS          10000 non-null  object 
dtypes: float64(6), int64(2), object(10)
memory usage: 1.4+ MB

In [11]:

rejections = predictions[predictions.is_bad_PREDICTION == 1]
rejections.shape

Out [11]:

(34, 18)

Response generation

Once the negative outcome records are available, use Generative AI models like GPT-3 to consume prediction explanations and generate responses for communication. This demo uses OpenAI’s ChatGPT, but the approach can be used on similar LLM models. The prompt structure and completion functions are inspired from Andrew Ng’s course on Prompt Engineering.

In [12]:

max_token_size = 4097


def get_completion(prompt, model="gpt-3.5-turbo"):
    messages = [{"role": "user", "content": prompt}]
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages,
        temperature=0,
    )
    return response.choices[0].message["content"]

The function below takes the prediction explanations from DataRobot and applies domain knowledge to convert highly technical information into user communication.

In [46]:

def provide_rejection_advice(sample_1, n_explanations):
    sample_1.fillna("not available", inplace=True)
    explanation_string = (
        sample_1.EXPLANATION_1_FEATURE_NAME.iloc[0]
        + " is "
        + str(sample_1.EXPLANATION_1_ACTUAL_VALUE.iloc[0])
        + ", "
        + sample_1.EXPLANATION_2_FEATURE_NAME.iloc[0]
        + " is "
        + str(sample_1.EXPLANATION_2_ACTUAL_VALUE.iloc[0])
        + ", "
        + sample_1.EXPLANATION_3_FEATURE_NAME.iloc[0]
        + " is "
        + str(sample_1.EXPLANATION_3_ACTUAL_VALUE.iloc[0])
        + ", "
    )
    explanation_string = (
        explanation_string.replace("loan_amnt", "loan amount")
        .replace("emp_length", "employment tenure")
        .replace(
            "inq_last_6mths", "number of customer inquiries for loan in last 6 months"
        )
        .replace("emp_title", "employee designation")
    )

    prompt = (
        'You are a telephonic loan sales representative. Based on the model prediction of loan rejection for a customer due to the following reasons "'
        + explanation_string
        + '", please provide a positive sentiment reply to the customer with '
        + str(n_explanations)
        + " of the most urgent steps to improve the chances of loan approval. Do not mention about any models or predictions in the response."
    )
    response = get_completion(prompt)
    return prompt, response

Outcome

In the below examples, it is evident that marrying DataRobot’s prediction explanations with LLM’s like GPT-3/4 provides a superior customer experience. This also reduces the effort on SMEs and Domain experts in an organization and improves their productivity.

In[47]:

sample_1 = rejections.sample(1)
sample_1

Out [47]:

	3869
is_bad_1_PREDICTION	0.540126
is_bad_0_PREDICTION	0.459874
is_bad_PREDICTION	1
THRESHOLD	0.5
POSITIVE_CLASS	1
EXPLANATION_1_FEATURE_NAME	int_rate
EXPLANATION_1_STRENGTH	1.300985
EXPLANATION_1_ACTUAL_VALUE	0.2248
EXPLANATION_1_QUALITATIVE_STRENGTH	+++
EXPLANATION_2_FEATURE_NAME	term
EXPLANATION_2_STRENGTH	0.36881
EXPLANATION_2_ACTUAL_VALUE	60 months
EXPLANATION_2_QUALITATIVE_STRENGTH	++
EXPLANATION_3_FEATURE_NAME	sub_grade
EXPLANATION_3_STRENGTH	0.272515
EXPLANATION_3_ACTUAL_VALUE	G2
EXPLANATION_3_QUALITATIVE_STRENGTH	++
DEPLOYMENT_APPROVAL_STATUS	APPROVED

In [48]:

# please replace n_explanations with a lower number if you want to reduce the amount of
# text in the response.
prompt, loan_rejection_advice = provide_rejection_advice(sample_1, n_explanations)
print(prompt)
print("=====================")
print(loan_rejection_advice)

You are a telephonic loan sales representative. Based on the model prediction of loan rejection for a customer due to the following reasons "int_rate is 0.2248, term is  60 months, sub_grade is G2, ", please provide a positive sentiment reply to the customer with 3 of the most urgent steps to improve the chances of loan approval. Do not mention about any models or predictions in the response.
=====================
Dear Customer,

Thank you for considering our loan services. We understand that you may have concerns about the loan approval process, and we appreciate the opportunity to address them. While we cannot guarantee approval, we can certainly provide you with some steps that may improve your chances:

1. Improve your credit score: Lenders often consider credit history as a crucial factor in loan approval. To enhance your chances, focus on paying your bills on time, reducing outstanding debts, and maintaining a low credit utilization ratio. This will demonstrate your financial responsibility and increase your creditworthiness.

2. Consider a shorter loan term: In your case, the loan term is currently set at 60 months. Shortening the term may positively impact your chances of approval. A shorter term reduces the overall risk for the lender and shows your commitment to repaying the loan in a timely manner.

3. Explore alternative loan options: While our loan may not be suitable for your current circumstances, there might be other loan products available that better align with your financial situation. We encourage you to discuss your needs with our loan specialists, who can guide you towards alternative options that may increase your chances of approval.

Remember, each loan application is unique, and our decision is based on various factors. By taking these steps, you can demonstrate your commitment to responsible financial management and potentially improve your chances of loan approval.

If you have any further questions or require assistance, please do not hesitate to reach out to our team. We are here to help you navigate the loan process and find the best solution for your needs.

Best regards,
[Your Name]
Telephonic Loan Sales Representative

In [17]:

sample_2 = rejections.sample(1)
sample_2

Out [17]:

	2105
is_bad_1_PREDICTION	0.543579
is_bad_0_PREDICTION	0.456421
is_bad_PREDICTION	1
THRESHOLD	0.5
POSITIVE_CLASS	1
EXPLANATION_1_FEATURE_NAME	int_rate
EXPLANATION_1_STRENGTH	0.692471
EXPLANATION_1_ACTUAL_VALUE	0.1991
EXPLANATION_1_QUALITATIVE_STRENGTH	+++
EXPLANATION_2_FEATURE_NAME	emp_title
EXPLANATION_2_STRENGTH	0.276402
EXPLANATION_2_ACTUAL_VALUE	NaN
EXPLANATION_2_QUALITATIVE_STRENGTH	++
EXPLANATION_3_FEATURE_NAME	term
EXPLANATION_3_STRENGTH	0.276129
EXPLANATION_3_ACTUAL_VALUE	60 months
EXPLANATION_3_QUALITATIVE_STRENGTH	++
DEPLOYMENT_APPROVAL_STATUS	APPROVED

In [49]:

prompt, loan_rejection_advice = provide_rejection_advice(sample_2, n_explanations)
print(prompt)
print("=====================")
print(loan_rejection_advice)

You are a telephonic loan sales representative. Based on the model prediction of loan rejection for a customer due to the following reasons "int_rate is 0.1991, employee designation is not available, term is  60 months, ", please provide a positive sentiment reply to the customer with 3 of the most urgent steps to improve the chances of loan approval. Do not mention about any models or predictions in the response.
=====================
Dear Customer,

Thank you for considering our loan services. We appreciate your interest in obtaining a loan from us. While we understand that your recent loan application was not approved, we would like to provide you with some steps that can help improve your chances of loan approval in the future.

1. Improve your credit score: Lenders often consider credit scores as an important factor in loan approval. Maintaining a good credit score by making timely payments, reducing outstanding debts, and avoiding new credit applications can significantly enhance your chances of loan approval.

2. Provide complete employment details: It is crucial to provide accurate and detailed information about your employment, including your job title, employer's name, and duration of employment. This information helps lenders assess your stability and ability to repay the loan.

3. Consider a shorter loan term: While a 60-month term may seem appealing due to lower monthly payments, it can also increase the risk for lenders. Opting for a shorter loan term, if feasible, can demonstrate your commitment to repaying the loan quickly and may increase your chances of approval.

We hope you find these suggestions helpful in improving your chances of loan approval. If you have any further questions or require assistance, please do not hesitate to reach out to us. We are here to support you throughout the loan application process.

Best regards,
[Your Name]
Telephonic Loan Sales Representative

In [39]:

sample_3 = rejections[rejections.index == 9918].head()
sample_3

Out [39]:

	9918
is_bad_1_PREDICTION	0.502272
is_bad_0_PREDICTION	0.497728
is_bad_PREDICTION	1
THRESHOLD	0.5
POSITIVE_CLASS	1
EXPLANATION_1_FEATURE_NAME	loan_amnt
EXPLANATION_1_STRENGTH	0.524048
EXPLANATION_1_ACTUAL_VALUE	1000
EXPLANATION_1_QUALITATIVE_STRENGTH	+++
EXPLANATION_2_FEATURE_NAME	int_rate
EXPLANATION_2_STRENGTH	0.353008
EXPLANATION_2_ACTUAL_VALUE	0.1629
EXPLANATION_2_QUALITATIVE_STRENGTH	++
EXPLANATION_3_FEATURE_NAME	inq_last_6mths
EXPLANATION_3_STRENGTH	0.262294
EXPLANATION_3_ACTUAL_VALUE	3
EXPLANATION_3_QUALITATIVE_STRENGTH	++
DEPLOYMENT_APPROVAL_STATUS	APPROVED

In [50]:

prompt, loan_rejection_advice = provide_rejection_advice(sample_3, n_explanations)
print(prompt)
print("=====================")
print(loan_rejection_advice)

You are a telephonic loan sales representative. Based on the model prediction of loan rejection for a customer due to the following reasons "loan amount is 1000, int_rate is 0.1629, number of customer inquiries for loan in last 6 months is 3, ", please provide a positive sentiment reply to the customer with 3 of the most urgent steps to improve the chances of loan approval. Do not mention about any models or predictions in the response.
=====================
Dear Customer,

Thank you for considering our loan services. We appreciate your interest in obtaining financial assistance. We understand that loan approval is important to you, and we are here to help you improve your chances. 

To increase the likelihood of loan approval, we recommend focusing on the following three steps:

1. Strengthen your credit history: Maintaining a good credit score is crucial for loan approval. We suggest reviewing your credit report and ensuring that all information is accurate. Paying bills on time, reducing credit card balances, and avoiding new credit inquiries can positively impact your creditworthiness.

2. Adjust the loan amount: In some cases, a smaller loan amount can increase the chances of approval. Consider adjusting the loan amount to a more manageable level that aligns with your financial situation. This can demonstrate responsible borrowing and improve your chances of approval.

3. Minimize recent loan inquiries: Lenders often consider the number of recent loan inquiries when evaluating loan applications. While we understand your need for financial assistance, it may be beneficial to limit new loan inquiries for the time being. This will show lenders that you are focused on responsible borrowing and reduce the perceived risk.

Remember, our team is here to guide you through the loan application process and provide any additional assistance you may need. We believe that by following these steps, you can improve your chances of loan approval. We appreciate your trust in our services and look forward to helping you achieve your financial goals.

Best regards,
[Your Name]
Telephonic Loan Sales Representative

Conclusion

In this notebook, you can see how you can use Generative AI with DataRobot’s prediction explanations to augment predictions and provide customer friendly and subject matter expert-level communication.