#Predict the price of the Uber ride from a given pickup point to the agreed drop-off location.

Perform following
tasks:
1. Pre-process the dataset.
2. Identify outliers.
3. Check the correlation.
4. Implement linear regression and random forest regression models.
5. Evaluate the models and compare their respective scores like R2, RMSE, etc. Dataset link: https://www.kaggle.com/datasets/yasserh/uber-fares-dataset

In [1]: #Importing the required libraries

import pandas as pd

import numpy as np

import seaborn as sns

import matplotlib.pyplot as plt

In [2]: #importing the dataset

df = pd.read_csv("uber.csv")

1. Pre-process the dataset.

In [3]: df.head()

Out[3]: Unnamed: 0 key fare_amount pickup_datetime pickup_longitude pickup_latitude dropoff_longitude dropoff_latitude passenger_count

0 24238194 2015-05-07 19:52:06.0000003 7.5 2015-05-07 19:52:06 UTC -73.999817 40.738354 -73.999512 40.723217 1

1 27835199 2009-07-17 20:04:56.0000002 7.7 2009-07-17 20:04:56 UTC -73.994355 40.728225 -73.994710 40.750325 1

2 44984355 2009-08-24 21:45:00.00000061 12.9 2009-08-24 21:45:00 UTC -74.005043 40.740770 -73.962565 40.772647 1

3 25894730 2009-06-26 08:22:21.0000001 5.3 2009-06-26 08:22:21 UTC -73.976124 40.790844 -73.965316 40.803349 3

4 17610152 2014-08-28 17:47:00.000000188 16.0 2014-08-28 17:47:00 UTC -73.925023 40.744085 -73.973082 40.761247 5

In [4]: df.info() #To get the required information of the dataset

<class 'pandas.core.frame.DataFrame'>

RangeIndex: 200000 entries, 0 to 199999

Data columns (total 9 columns):

Unnamed: 0 200000 non-null int64

key 200000 non-null object

fare_amount 200000 non-null float64

pickup_datetime 200000 non-null object

pickup_longitude 200000 non-null float64

pickup_latitude 200000 non-null float64

dropoff_longitude 199999 non-null float64

dropoff_latitude 199999 non-null float64

passenger_count 200000 non-null int64

dtypes: float64(5), int64(2), object(2)

memory usage: 13.7+ MB

In [5]: df.columns #TO get number of columns in the dataset

Index(['Unnamed: 0', 'key', 'fare_amount', 'pickup_datetime',

Out[5]:
'pickup_longitude', 'pickup_latitude', 'dropoff_longitude',

'dropoff_latitude', 'passenger_count'],

dtype='object')

In [6]: df = df.drop(['Unnamed: 0', 'key'], axis= 1) #To drop unnamed column as it isn't required

In [7]: df.head()

Out[7]: fare_amount pickup_datetime pickup_longitude pickup_latitude dropoff_longitude dropoff_latitude passenger_count

0 7.5 2015-05-07 19:52:06 UTC -73.999817 40.738354 -73.999512 40.723217 1

1 7.7 2009-07-17 20:04:56 UTC -73.994355 40.728225 -73.994710 40.750325 1

2 12.9 2009-08-24 21:45:00 UTC -74.005043 40.740770 -73.962565 40.772647 1

3 5.3 2009-06-26 08:22:21 UTC -73.976124 40.790844 -73.965316 40.803349 3

4 16.0 2014-08-28 17:47:00 UTC -73.925023 40.744085 -73.973082 40.761247 5

In [8]: df.shape #To get the total (Rows,Columns)

(200000, 7)
Out[8]:

In [9]: df.dtypes #To get the type of each column

fare_amount float64

Out[9]:
pickup_datetime object

pickup_longitude float64

pickup_latitude float64

dropoff_longitude float64

dropoff_latitude float64

passenger_count int64

dtype: object

Column pickup_datetime is in wrong format (Object). Convert it to DateTime Format

In [10]: df.pickup_datetime = pd.to_datetime(df.pickup_datetime)

In [11]: df.dtypes

fare_amount float64

Out[11]:
pickup_datetime datetime64[ns, UTC]

pickup_longitude float64

pickup_latitude float64

dropoff_longitude float64

dropoff_latitude float64

passenger_count int64

dtype: object

Filling Missing values

In [12]: df.isnull().sum()

fare_amount 0

Out[12]:
pickup_datetime 0

pickup_longitude 0

pickup_latitude 0

dropoff_longitude 1

dropoff_latitude 1

passenger_count 0

dtype: int64

In [13]: df['dropoff_latitude'].fillna(value=df['dropoff_latitude'].mean(),inplace = True)

df['dropoff_longitude'].fillna(value=df['dropoff_longitude'].median(),inplace = True)

In [14]: df.isnull().sum()

fare_amount 0

Out[14]:
pickup_datetime 0

pickup_longitude 0

pickup_latitude 0

dropoff_longitude 0

dropoff_latitude 0

passenger_count 0

dtype: int64

To segregate each time of date and time

In [15]: df= df.assign(hour = df.pickup_datetime.dt.hour,

day= df.pickup_datetime.dt.day,

month = df.pickup_datetime.dt.month,

year = df.pickup_datetime.dt.year,

dayofweek = df.pickup_datetime.dt.dayofweek)

In [16]: df.head()

Out[16]: fare_amount pickup_datetime pickup_longitude pickup_latitude dropoff_longitude dropoff_latitude passenger_count hour day month year dayofweek

0 7.5 2015-05-07 19:52:06+00:00 -73.999817 40.738354 -73.999512 40.723217 1 19 7 5 2015 3

1 7.7 2009-07-17 20:04:56+00:00 -73.994355 40.728225 -73.994710 40.750325 1 20 17 7 2009 4

2 12.9 2009-08-24 21:45:00+00:00 -74.005043 40.740770 -73.962565 40.772647 1 21 24 8 2009 0

3 5.3 2009-06-26 08:22:21+00:00 -73.976124 40.790844 -73.965316 40.803349 3 8 26 6 2009 4

4 16.0 2014-08-28 17:47:00+00:00 -73.925023 40.744085 -73.973082 40.761247 5 17 28 8 2014 3

Here we are going to use Heversine formula to calculate the distance between two points and journey, using the
longitude and latitude values.
Heversine formula
hav(θ) = sin**2(θ/2).

In [19]: from math import *

# function to calculate the travel distance from the longitudes and latitudes

def distance_transform(longitude1, latitude1, longitude2, latitude2):

travel_dist = []

for pos in range(len(longitude1)):

long1,lati1,long2,lati2 = map(radians,[longitude1[pos],latitude1[pos],longitude2[pos],latitude2[pos]])

dist_long = long2 - long1

dist_lati = lati2 - lati1

a = sin(dist_lati/2)**2 + cos(lati1) * cos(lati2) * sin(dist_long/2)**2

c = 2 * asin(sqrt(a))*6371

travel_dist.append(c)

return travel_dist

In [20]: df['dist_travel_km'] = distance_transform(df['pickup_longitude'].to_numpy(),

df['pickup_latitude'].to_numpy(),

df['dropoff_longitude'].to_numpy(),

df['dropoff_latitude'].to_numpy()

In [21]: df.head()

Out[21]: fare_amount pickup_datetime pickup_longitude pickup_latitude dropoff_longitude dropoff_latitude passenger_count hour day month year dayofweek dist_travel_km

0 7.5 2015-05-07 19:52:06+00:00 -73.999817 40.738354 -73.999512 40.723217 1 19 7 5 2015 3 1.683323

1 7.7 2009-07-17 20:04:56+00:00 -73.994355 40.728225 -73.994710 40.750325 1 20 17 7 2009 4 2.457590

2 12.9 2009-08-24 21:45:00+00:00 -74.005043 40.740770 -73.962565 40.772647 1 21 24 8 2009 0 5.036377

3 5.3 2009-06-26 08:22:21+00:00 -73.976124 40.790844 -73.965316 40.803349 3 8 26 6 2009 4 1.661683

4 16.0 2014-08-28 17:47:00+00:00 -73.925023 40.744085 -73.973082 40.761247 5 17 28 8 2014 3 4.475450

In [22]: # drop the column 'pickup_daetime' using drop()

# 'axis = 1' drops the specified column

df = df.drop('pickup_datetime',axis=1)

In [23]: df.head()

Out[23]: fare_amount pickup_longitude pickup_latitude dropoff_longitude dropoff_latitude passenger_count hour day month year dayofweek dist_travel_km

0 7.5 -73.999817 40.738354 -73.999512 40.723217 1 19 7 5 2015 3 1.683323

1 7.7 -73.994355 40.728225 -73.994710 40.750325 1 20 17 7 2009 4 2.457590

2 12.9 -74.005043 40.740770 -73.962565 40.772647 1 21 24 8 2009 0 5.036377

3 5.3 -73.976124 40.790844 -73.965316 40.803349 3 8 26 6 2009 4 1.661683

4 16.0 -73.925023 40.744085 -73.973082 40.761247 5 17 28 8 2014 3 4.475450

Checking outliers and filling them

In [24]: df.plot(kind = "box",subplots = True,layout = (7,2),figsize=(15,20)) #Boxplot to check the outliers

fare_amount AxesSubplot(0.125,0.787927;0.352273x0.0920732)

Out[24]:
pickup_longitude AxesSubplot(0.547727,0.787927;0.352273x0.0920732)

pickup_latitude AxesSubplot(0.125,0.677439;0.352273x0.0920732)

dropoff_longitude AxesSubplot(0.547727,0.677439;0.352273x0.0920732)

dropoff_latitude AxesSubplot(0.125,0.566951;0.352273x0.0920732)

passenger_count AxesSubplot(0.547727,0.566951;0.352273x0.0920732)

hour AxesSubplot(0.125,0.456463;0.352273x0.0920732)

day AxesSubplot(0.547727,0.456463;0.352273x0.0920732)

month AxesSubplot(0.125,0.345976;0.352273x0.0920732)

year AxesSubplot(0.547727,0.345976;0.352273x0.0920732)

dayofweek AxesSubplot(0.125,0.235488;0.352273x0.0920732)

dist_travel_km AxesSubplot(0.547727,0.235488;0.352273x0.0920732)

dtype: object

In [25]: #Using the InterQuartile Range to fill the values

def remove_outlier(df1 , col):

Q1 = df1[col].quantile(0.25)

Q3 = df1[col].quantile(0.75)

IQR = Q3 - Q1

lower_whisker = Q1-1.5*IQR

upper_whisker = Q3+1.5*IQR

df[col] = np.clip(df1[col] , lower_whisker , upper_whisker)

return df1

def treat_outliers_all(df1 , col_list):

for c in col_list:

df1 = remove_outlier(df , c)

return df1

In [26]: df = treat_outliers_all(df , df.iloc[: , 0::])

In [27]: df.plot(kind = "box",subplots = True,layout = (7,2),figsize=(15,20)) #Boxplot shows that dataset is free from outliers

fare_amount AxesSubplot(0.125,0.787927;0.352273x0.0920732)

Out[27]:
pickup_longitude AxesSubplot(0.547727,0.787927;0.352273x0.0920732)

pickup_latitude AxesSubplot(0.125,0.677439;0.352273x0.0920732)

dropoff_longitude AxesSubplot(0.547727,0.677439;0.352273x0.0920732)

dropoff_latitude AxesSubplot(0.125,0.566951;0.352273x0.0920732)

passenger_count AxesSubplot(0.547727,0.566951;0.352273x0.0920732)

hour AxesSubplot(0.125,0.456463;0.352273x0.0920732)

day AxesSubplot(0.547727,0.456463;0.352273x0.0920732)

month AxesSubplot(0.125,0.345976;0.352273x0.0920732)

year AxesSubplot(0.547727,0.345976;0.352273x0.0920732)

dayofweek AxesSubplot(0.125,0.235488;0.352273x0.0920732)

dist_travel_km AxesSubplot(0.547727,0.235488;0.352273x0.0920732)

dtype: object

In [28]: #Uber doesn't travel over 130 kms so minimize the distance

df= df.loc[(df.dist_travel_km >= 1) | (df.dist_travel_km <= 130)]

print("Remaining observastions in the dataset:", df.shape)

Remaining observastions in the dataset: (200000, 12)

In [29]: #Finding inccorect latitude (Less than or greater than 90) and longitude (greater than or less than 180)

incorrect_coordinates = df.loc[(df.pickup_latitude > 90) |(df.pickup_latitude < -90) |

(df.dropoff_latitude > 90) |(df.dropoff_latitude < -90) |

(df.pickup_longitude > 180) |(df.pickup_longitude < -180) |

(df.dropoff_longitude > 90) |(df.dropoff_longitude < -90)

In [30]: df.drop(incorrect_coordinates, inplace = True, errors = 'ignore')

In [31]: df.head()

Out[31]: fare_amount pickup_longitude pickup_latitude dropoff_longitude dropoff_latitude passenger_count hour day month year dayofweek dist_travel_km

0 7.5 -73.999817 40.738354 -73.999512 40.723217 1.0 19 7 5 2015 3 1.683323

1 7.7 -73.994355 40.728225 -73.994710 40.750325 1.0 20 17 7 2009 4 2.457590

2 12.9 -74.005043 40.740770 -73.962565 40.772647 1.0 21 24 8 2009 0 5.036377

3 5.3 -73.976124 40.790844 -73.965316 40.803349 3.0 8 26 6 2009 4 1.661683

4 16.0 -73.929786 40.744085 -73.973082 40.761247 3.5 17 28 8 2014 3 4.475450

In [32]: df.isnull().sum()

fare_amount 0

Out[32]:
pickup_longitude 0

pickup_latitude 0

dropoff_longitude 0

dropoff_latitude 0

passenger_count 0

hour 0

day 0

month 0

year 0

dayofweek 0

dist_travel_km 0

dtype: int64

In [33]: sns.heatmap(df.isnull()) #Free for null values

<matplotlib.axes._subplots.AxesSubplot at 0x8d8af2a080>
Out[33]:

In [34]: corr = df.corr() #Function to find the correlation

In [35]: corr

Out[35]: fare_amount pickup_longitude pickup_latitude dropoff_longitude dropoff_latitude passenger_count hour day month year dayofweek dist_travel_km

fare_amount 1.000000 0.154069 -0.110842 0.218675 -0.125898 0.015778 -0.023623 0.004534 0.030817 0.141277 0.013652 0.844374

pickup_longitude 0.154069 1.000000 0.259497 0.425619 0.073290 -0.013213 0.011579 -0.003204 0.001169 0.010198 -0.024652 0.098094

pickup_latitude -0.110842 0.259497 1.000000 0.048889 0.515714 -0.012889 0.029681 -0.001553 0.001562 -0.014243 -0.042310 -0.046812

dropoff_longitude 0.218675 0.425619 0.048889 1.000000 0.245667 -0.009303 -0.046558 -0.004007 0.002391 0.011346 -0.003336 0.186531

dropoff_latitude -0.125898 0.073290 0.515714 0.245667 1.000000 -0.006308 0.019783 -0.003479 -0.001193 -0.009603 -0.031919 -0.038900

passenger_count 0.015778 -0.013213 -0.012889 -0.009303 -0.006308 1.000000 0.020274 0.002712 0.010351 -0.009749 0.048550 0.009709

hour -0.023623 0.011579 0.029681 -0.046558 0.019783 0.020274 1.000000 0.004677 -0.003926 0.002156 -0.086947 -0.038366

day 0.004534 -0.003204 -0.001553 -0.004007 -0.003479 0.002712 0.004677 1.000000 -0.017360 -0.012170 0.005617 0.003062

month 0.030817 0.001169 0.001562 0.002391 -0.001193 0.010351 -0.003926 -0.017360 1.000000 -0.115859 -0.008786 0.011628

year 0.141277 0.010198 -0.014243 0.011346 -0.009603 -0.009749 0.002156 -0.012170 -0.115859 1.000000 0.006113 0.024278

dayofweek 0.013652 -0.024652 -0.042310 -0.003336 -0.031919 0.048550 -0.086947 0.005617 -0.008786 0.006113 1.000000 0.027053

dist_travel_km 0.844374 0.098094 -0.046812 0.186531 -0.038900 0.009709 -0.038366 0.003062 0.011628 0.024278 0.027053 1.000000

In [36]: fig,axis = plt.subplots(figsize = (10,6))

sns.heatmap(df.corr(),annot = True) #Correlation Heatmap (Light values means highly correlated)

<matplotlib.axes._subplots.AxesSubplot at 0x8d8affc588>
Out[36]:

Dividing the dataset into feature and target values

In [37]: x = df[['pickup_longitude','pickup_latitude','dropoff_longitude','dropoff_latitude','passenger_count','hour','day','month','year','dayofweek','dist_travel_km']]

In [38]: y = df['fare_amount']

Dividing the dataset into training and testing dataset

In [39]: from sklearn.model_selection import train_test_split

X_train,X_test,y_train,y_test = train_test_split(x,y,test_size = 0.33)

Linear Regression
In [40]: from sklearn.linear_model import LinearRegression

regression = LinearRegression()

In [41]: regression.fit(X_train,y_train)

LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)

Out[41]:

In [42]: regression.intercept_ #To find the linear intercept

2809.192377415925
Out[42]:

In [43]: regression.coef_ #To find the linear coeeficient

array([ 1.75328304e+01, -9.83172673e+00, 1.54611809e+01, -1.69707270e+01,

Out[43]:
5.40456388e-02, 9.46950748e-03, 1.66720620e-03, 5.40917698e-02,

3.61743634e-01, -3.69474342e-02, 2.00077959e+00])

In [44]: prediction = regression.predict(X_test) #To predict the target values

In [45]: print(prediction)

[10.80422002 4.74707896 9.95283165 ... 5.89597937 17.00144322

5.38487972]

In [46]: y_test

16850 8.50

Out[46]:
181076 4.10

70798 9.30

87421 12.90

169443 22.25

18976 11.00

50921 13.70

199564 14.50

125215 5.30

67510 8.50

85217 22.25

156903 21.50

116795 4.10

112179 16.90

124459 3.70

173299 22.25

51448 19.70

99502 22.25

174467 10.90

78880 20.50

26798 22.25

38501 4.50

63091 12.90

171207 22.25

142238 8.50

101106 7.30

120177 4.50

154585 14.50

75840 5.50

85918 14.00

...

104227 10.10

14172 19.70

49985 3.70

183045 6.50

11927 12.90

93684 4.50

101795 13.70

21444 6.10

85147 8.50

81311 8.00

157686 11.70

194074 6.50

132558 10.50

132616 11.70

188536 5.70

179629 8.90

11277 3.70

147880 7.30

116553 5.70

157394 6.50

103519 13.30

41348 12.90

12608 4.50

6820 5.50

84612 5.00

168836 3.70

39719 21.00

124536 4.90

90432 22.10

12543 4.90

Name: fare_amount, Length: 66000, dtype: float64

Metrics Evaluation using R2, Mean Squared Error, Root Mean Sqared Error
In [47]: from sklearn.metrics import r2_score

In [48]: r2_score(y_test,prediction)

0.7471032194200018
Out[48]:

In [49]: from sklearn.metrics import mean_squared_error

In [50]: MSE = mean_squared_error(y_test,prediction)

In [51]: MSE

7.464818887848474
Out[51]:

In [52]: RMSE = np.sqrt(MSE)

In [53]: RMSE

2.7321820744321696
Out[53]:

Random Forest Regression

In [54]: from sklearn.ensemble import RandomForestRegressor

In [55]: rf = RandomForestRegressor(n_estimators=100) #Here n_estimators means number of trees you want to build before making the prediction

In [56]: rf.fit(X_train,y_train)

RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,

Out[56]:
max_features='auto', max_leaf_nodes=None,

min_impurity_decrease=0.0, min_impurity_split=None,

min_samples_leaf=1, min_samples_split=2,

min_weight_fraction_leaf=0.0, n_estimators=100,

n_jobs=None, oob_score=False, random_state=None,

verbose=0, warm_start=False)

In [57]: y_pred = rf.predict(X_test)

In [58]: y_pred

array([ 9.7025, 4.744 , 9.202 , ..., 6.468 , 16.2802, 4.47 ])

Out[58]:

Metrics evaluatin for Random Forest

In [59]: R2_Random = r2_score(y_test,y_pred)

In [60]: R2_Random

0.8024361566950065
Out[60]:

In [64]: MSE_Random = mean_squared_error(y_test,y_pred)

MSE_Random

5.831542440662031
Out[64]:

In [65]: RMSE_Random = np.sqrt(MSE_Random)

RMSE_Random

2.4148586792319815
Out[65]: