Data Cleaning Process

Cleaning data is an essential process in any data science project. It ensures data quality and improves the reliability of my analysis. Here's a look at how I cleaned my dataset for Yillow.

First and foremost, I made sure there were no duplicate entries in our dataset. Duplicate records can skew the analysis and provide inaccurate results, but thankfully there were actually 0 duplicates!

Next, I dealt with missing values in the secondary features. Wherever data was missing, I assigned a 'NoValue' tag. This would ensure that, during the feature engineering process, features with missing values would get their own category/assignment. By doing this, I retained the record while acknowledging the absence of certain information.

Lastly, I ensured data consistency. It's vital that the same type of data is represented consistently across the dataset. For example, if living area is represented in square feet, I made sure it's not accidentally entered as square meters in some instances.

Outlier Detection and Removal

Outliers can be killers when it comes to predictive models because they can really skew predictions. The real Zestimate has to take into account outliers, but because of the small dataset of Yillow and also because the Yestimate is supposed to be a predictive model to help people (me, in particular) buy a house, I am going to assume that I'm not very interested in houses that are super overpriced. Because of this, I decided to remove outliers from the dataset. To do this, I used box-and-whisker plots to identify outliers and then removed them. Although this makes the Yestimate less effective on outliers, it makes it more effective on the majority of houses.