{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "cs480_fall20_asst2_logistic_regression_skeleton.ipynb",
"provenance": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "JFXZug066bJX"
},
"source": [
"# Upload files in Google Colab\n",
"If you are running this Jupyter Notebook on Google Colab, run this cell to upload the data files (train_inputs.csv, train_targets.csv, test_inputs.csv, test_targets.csv) in the colab virtual machine. You will be prompted to select files that you would like to upload. \n",
"\n",
"If you are running this Jupyter Notebook on your computer, you do not need to run this cell."
]
},
{
"cell_type": "code",
"metadata": {
"id": "dqBJV_Br4XeI"
},
"source": [
"from google.colab import files\n",
"uploaded = files.upload()\n",
"%ls"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "LZDpxE4jmFwA"
},
"source": [
"# Import libraries \n",
"Do not use any other Python library."
]
},
{
"cell_type": "code",
"metadata": {
"id": "m_1d0BPfmacB"
},
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt"
],
"execution_count": 13,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "6keYhcgi7nbf"
},
"source": [
"# Function: load_logistic_regression_data\n",
"\n",
"This function loads the data for Logistic Regression from a local drive into RAM\n",
"\n",
"Outputs:\n",
"\n",
"* **train_inputs**: numpy array of N training data points x M features\n",
"* **train_labels**: numpy array of N training labels\n",
"* **test_inputs**: numpy array of N' test data points x M features\n",
"* **test_labels**: numpy array of N' test labels"
]
},
{
"cell_type": "code",
"metadata": {
"id": "vcG5U2lR7utt"
},
"source": [
"def load_logistic_regression_data():\n",
" test_inputs = np.genfromtxt('test_inputs.csv', delimiter=',')\n",
" test_labels = np.genfromtxt('test_labels.csv', delimiter=',')\n",
" train_inputs = np.genfromtxt('train_inputs.csv', delimiter=',')\n",
" train_labels = np.genfromtxt('train_labels.csv', delimiter=',')\n",
" return train_inputs, train_labels, test_inputs, test_labels"
],
"execution_count": 14,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "NZTxUMDM2PDx"
},
"source": [
"# Function: sigmoid\n",
"\n",
"This function implements the logistic sigmoid.\n",
"\n",
"Input:\n",
"* **input**: vector of inputs (numpy array of floats)\n",
"\n",
"Output:\n",
"* **output**: vector of outputs (numpy array of floats)"
]
},
{
"cell_type": "code",
"metadata": {
"id": "evR9GYnV3FmM"
},
"source": [
"def sigmoid(input):\n",
"\n",
" # dummy assignment until the function is filled in\n",
" output = np.zeros(len(input))\n",
" return output"
],
"execution_count": 15,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "GwLo3p4f8bTa"
},
"source": [
"# Function: predict_logistic_regression\n",
"\n",
"This function uses a vector of weights to make predictions for a set of inputs. The prediction for each data point is a distribution over the labels. Assume that there are only two possible labels {0,1}.\n",
"\n",
"Inputs:\n",
"* **inputs**: matrix of input data points for which we want to make a prediction (numpy array of N data points x M+1 features)\n",
"* **weights**: vector of weights (numpy array of M+1 weights)\n",
"\n",
"Output:\n",
"* **predicted_probabilities**: matrix of predicted probabilities (numpy array of N data points x 2 labels)"
]
},
{
"cell_type": "code",
"metadata": {
"id": "iX04_wClRqkV"
},
"source": [
"def predict_logistic_regression(inputs, weights):\n",
"\n",
" # dummy assignment until the function is filled in\n",
" predicted_probabilities = np.zeros((inputs.shape[0],2))\n",
" return predicted_probabilities"
],
"execution_count": 16,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "fmfPN7K0RtQ5"
},
"source": [
"# Function eval_logistic_regression\n",
"\n",
"This function evaluates a set of predictions by computing the negative log probabilities of the labels and the accuracy (percentage of correctly predicted labels). Assume that there are only two possible labels {0,1}. A data point is correctly labeled when the probability of the target label is >= 0.5.\n",
"\n",
"Inputs:\n",
"* **inputs**: matrix of input data points for which we will evaluate the predictions (numpy array of N data points x M+1 features)\n",
"* **weights**: vector of weights (numpy array of M+1 weights)\n",
"* **labels**: vector of target labels associated with the inputs (numpy array of N labels)\n",
"\n",
"Outputs:\n",
"* **neg_log_prob**: negative log probability of the set of predictions (float)\n",
"* **accuracy**: percentage of correctly labeled data points (float)"
]
},
{
"cell_type": "code",
"metadata": {
"id": "wC14LEsvTxbu"
},
"source": [
"def eval_logistic_regression(inputs, weights, labels):\n",
" \n",
" # dummy assignment until the function is filled in\n",
" accuracy = 0\n",
" neg_log_prob = 0\n",
" return neg_log_prob, accuracy"
],
"execution_count": 17,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "R7hZ4XVP4U4y"
},
"source": [
"Function: initialize_weights\n",
"\n",
"This function initializes the weights uniformly at random in the interval [-0.05,0.05]\n",
"\n",
"Input:\n",
"* **n_weights**: # of weights to be initialized (integer)\n",
"\n",
"Output:\n",
"* **random_weights**: vector of weights (numpy array of floats)"
]
},
{
"cell_type": "code",
"metadata": {
"id": "9OjhevpV5FBg"
},
"source": [
"def initialize_weights(n_weights):\n",
"\n",
" # dummy assignment until the function is filled in\n",
" random_weights = np.zeros(n_weights)\n",
" return random_weights"
],
"execution_count": 18,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "RMAzC5xXT0H-"
},
"source": [
"# Function train_logistic_regression\n",
"\n",
"This function optimizes a set of weights for logistic regression based on a training set. Initialize the weights with the function initialize_weights. Implement Newton's algorithm to optimize the weights. Stop Newton's algorithm when the maximum change for all weights is less than 0.001 in two consecutive iterations. Assume that there are only two labels {0,1}.\n",
"\n",
"Inputs:\n",
"* **train_inputs**: matrix of input training points (numpy array of N data points x M+1 features)\n",
"* **train_labels**: vector of labels associated with the inputs (numpy array of N labels)\n",
"* **lambda_hyperparam**: lambda hyperparameter used to adjust the importance of the regularizer (scalar)\n",
"\n",
"Output:\n",
"* **weights**: vector of weights that have been optimized (numpy array of M+1 weights)\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "_DkzoT5QVy41"
},
"source": [
"def train_logistic_regression(train_inputs, train_labels, lambda_hyperparam):\n",
"\n",
" # dummy assignment until the function is filled in\n",
" weights = np.zeros(train_inputs.shape[1])\n",
" return weights"
],
"execution_count": 19,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "VYIbLxX7V2DW"
},
"source": [
"# Function cross_validation_logistic_regression\n",
"\n",
"This function performs k-fold cross validation to determine the best lambda hyperparameter in logistic regression\n",
"\n",
"Inputs:\n",
"* **k_folds**: # of folds in cross-validation (integer)\n",
"* **hyperparameters**: list of hyperparameters where each hyperparameter is a different lambda value (list of floats)\n",
"* **inputs**: matrix of input points (numpy array of N data points by M+1 features)\n",
"* **labels**: vector of labels associated with the inputs (numpy array of N labels)\n",
"\n",
"Outputs:\n",
"* **best_hyperparam**: best lambda value for logistic regression (float)\n",
"* **best_neg_log_prob**: negative log probabilty achieved with best_hyperparam (float)\n",
"* **neg_log_probabilities**: vector of negative log probabilities for the corresponding hyperparameters (numpy array of floats)\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "_ZzoiZxLZMcV"
},
"source": [
"def cross_validation_logistic_regression(k_folds, hyperparameters, inputs, labels):\n",
" \n",
" # dummy assignments until the function is filled in\n",
" best_hyperparam = 0\n",
" best_neg_log_prob = 0\n",
" neg_log_probabilities = np.zeros(len(hyperparameters))\n",
" return best_hyperparam, best_neg_log_prob, neg_log_probabilities"
],
"execution_count": 20,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "_ah5AAayZfVU"
},
"source": [
"# Function: plot_logistic_regression_neg_log_probabilities\n",
"\n",
"Function that plots the negative log probabilities for different lambda values (hyperparameters) in logistic regression based on cross validation\n",
"\n",
"Inputs:\n",
"* **neg_log_probabilities**: vector of negative log probabilities for the corresponding hyperparameters (numpy array of floats)\n",
"* **hyperparams**: list of hyperparameters where each hyperparameter is a different lambda value (list of floats)"
]
},
{
"cell_type": "code",
"metadata": {
"id": "dh9qZuzMatsZ"
},
"source": [
"def plot_logistic_regression_neg_log_probabilities(neg_log_probabilities,hyperparams):\n",
" plt.plot(hyperparams,neg_log_probabilities)\n",
" plt.ylabel('negative log probability')\n",
" plt.xlabel('lambda')\n",
" plt.show()"
],
"execution_count": 21,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "s21LRP5Qb3m8"
},
"source": [
"# Main Logistic Regression code\n",
"\n",
"Load data (rescale the inputs to be in the [-1,1] range, add 1 at the end of each datapoint and rename the labels 5,6 to 0,1).\n",
"Use k-fold cross validation to find the best lambda value for logistic regression.\n",
"Plot the negative log probabilities for different lambda values.\n",
"Test logistic regression with the best lambda value."
]
},
{
"cell_type": "code",
"metadata": {
"id": "njlK2bf7sycQ"
},
"source": [
"# load data\n",
"train_inputs, train_labels, test_inputs, test_labels = load_logistic_regression_data()\n",
"\n",
"# rescale inputs in the [-1,1] range\n",
"train_inputs = (train_inputs - 8)/8\n",
"test_inputs = (test_inputs - 8)/8\n",
"\n",
"# add 1 at the end of each data point\n",
"train_inputs = np.concatenate((train_inputs,np.ones((train_inputs.shape[0],1))),1)\n",
"test_inputs = np.concatenate((test_inputs,np.ones((test_inputs.shape[0],1))),1)\n",
"\n",
"# rename the classes 5,6 to 0,1\n",
"train_labels = train_labels.astype(int) - 5\n",
"test_labels = test_labels.astype(int) - 5\n",
"\n",
"# lambda values to be evaluated by cross validation\n",
"hyperparams = range(201)\n",
"k_folds = 10\n",
"best_lambda, best_neg_log_prob, neg_log_probabilities = cross_validation_logistic_regression(k_folds,hyperparams,train_inputs,train_labels)\n",
"\n",
"# plot results\n",
"plot_logistic_regression_neg_log_probabilities(neg_log_probabilities,hyperparams)\n",
"print('best lambda: ' + str (best_lambda))\n",
"print('best cross validation negative log probability: ' + str(best_neg_log_prob))\n",
"\n",
"# train and evaluate with best lambda\n",
"weights = train_logistic_regression(train_inputs,train_labels,best_lambda)\n",
"neg_log_prob, accuracy = eval_logistic_regression(test_inputs, weights, test_labels)\n",
"print('test accuracy: ' + str(accuracy))\n",
"print('test negative log probability: ' + str(neg_log_prob))\n",
"\n"
],
"execution_count": null,
"outputs": []
}
]
}