Data Mining · CSC213

Data Cleaning: Noise, Outliers, Missing Values and Inconsistent Data

Data Mining Study guides

Data cleaning removes noise and inconsistent data and fills in missing values so mining works on reliable input.

Real data is dirty. Typical quality problems are noise (random error in a measured variable), outliers (objects that do not comply with the general model or behaviour of the data), missing values, duplicate data and inconsistent data (for example a zip code that does not match the city).

Missing values can be handled by ignoring the record (when many attributes are missing), filling in manually, using a global constant, using the attribute mean or median (or the class mean), filling with the most probable value from a model such as regression or a decision tree, or estimating by ignoring the value during analysis. Inconsistent data can be detected by comparing against rules, domain constraints and other sources.

Noisy data can be handled by binning (sort the values and smooth using their neighbours), regression (fit a function and smooth by it), clustering (values outside the clusters are outliers) and combined computer and human inspection. An outlier is not always an error: for fraud detection, the outlier is the thing you are looking for.

Key points

  • Noise is random error; outliers are objects that deviate from the general behaviour.
  • Missing values: ignore the record, fill with mean or median, or predict the value.
  • Noise smoothing: binning, regression, clustering.
  • Inconsistent data is found by rules and cross-checking sources.
  • Outliers can be errors or the actual signal.

Worked example

Ages in a table: 23, 25, 24, 26, 250. The 250 is an outlier and probably an entry error. Filling a missing age with the median (25) is more robust than the mean (about 62 with the 250 included).

Common questions

What is the main use of data cleaning?

To remove noise and inconsistent data and handle missing values so the data is reliable for mining.

Is every outlier an error?

No. In fraud or fault detection the outliers are the interesting objects.

Practice questions on this topic

Try these, then open the app to check your answers with explanations.

  1. Which of the following is true about outliers
    • A. Data points that deviate a lot from normal observations
    • B. Can reduce the accuracy of the model
    • C. Both A and B
    • D. None
  2. The primary use of data cleaning is
    • A. Transformations for correcting the wrong data
    • B. Correction of the data inconsistencies
    • C. Removing the noisy data
    • D. All of the above
  3. Data cleaning means removing the inconsistent data or noise and collecting necessary information of a collection of interrelated data.
    • A. True
    • B. False
  4. What are some examples of data quality problems:
    • A. Noise and outliers
    • B. Duplicate data
    • C. Missing values
    • D. All of the Above
  5. What is the way to detect inconsistent data?
    • A. numerical computing
    • B. commercial tools
    • C. global constant
    • D. none
  6. Missing values can be handled by:
    • A. Fill in the missing data
    • B. Ignore the tuple
    • C. Binning
    • D. Both A, and B