Data Quality and Summary API PNG PDF

Validating the data quality and generate summary statistics reports.

  1. Rows and Columns : The questions to be answered serve as guides in EDA
    1. Rows
      1. Descriptions

        What does the row mean?

      2. Count
    2. Columns
      1. Descriptions

        What does the column mean?

      2. Count

        How many columns?

      3. Possible values or ranges

        List the theoretical limits on the values and validate against the data.

  2. Types and Formats :
    1. Data Types

      What is each column consists of?

      1. Types of data

        Ordinal, Nominal, Interval, Generative, etc

      2. Is the type of the data correct
    2. Data Formats
      1. Are the dates loaded as dates?
      2. Are the numbers loaded as numbers?

        Are they strings?

      3. Are the financial values correct?

        Are they strings or numbers? EU format, US format?

  3. Missing Values : Are there missing values in each column
    1. Different types of missing values

      Notations of missing values are different in different datasets. Read the documentation of the dataset to find out.

      1. Standard missing values

        nan, nat, None, na, null...

      2. Represented with a specific value

        -1, 0, MISSING, ...

    2. Percentage of missing values in each column
    3. Visualizations

      e.g., missingno python package

  4. Duplications : Are there duplications of rows/columns?
    1. Validate by yourself

      Do not trust the metadata and documentation of the dataset. Duplications of fields may occur when the documentation says they are unique.

  5. Distributions :
    1. What is the generation process?
      1. Is it a histogram analysis of another row?
      2. Is it a linear combination of other rows?
    2. Visualize the distributions of the values

      Know all the values

      1. Value count bar plot

        For descrete data, list all possible values and counts

      2. Histogram and KDE

        for continuous data, use histograms or KDE.

      3. Boxplot

        Boxplot is easier to understand for business people

      4. Scatter plot

        Gut feeling of where the data points are located

      5. Contour plot
    3. Dispersion of the target value

      Is the dispersion of the target value small enough for the algorithm to perform a good prediction?

    4. Numerical Summarization

      Use summary statistics to find out the moments.

      1. Locations

        Mean, median, quartiles, mode...

      2. Spreads

        range, variance, standard deviation, IQR

      3. Skewness


      4. Kurtosis
  6. Correlations, Similarities :
    1. Pairplot
    2. Correlations

      Pearson, Kendall Tau Correlation

    3. Distances

      Calculate the distance between features or rows to understand the relations between them; Euclidean distance, Mahalanobis distance, Minkowski distance, Jaccard distance, ...

  7. Size : How much space will the data take on our storage device?
    1. Memory usage

      To estimate the hardware requirements when deploying the model

    2. Storage on Hard Drive in Different Formats

      How much space will the dataset take in different formats?

  8. Combining Data Files : One dataset may come in different files, combine them carefully.
    1. Concat

      The files should be concated with caution.

      1. Validate overlap

        Check if there is an overlap between the files.

Exploratory Data Analysis API PNG PDF

EDA is one of the very first steps of my data science projects.

  1. Objectives of EDA : The questions to be answered serve as guides in EDA
    1. Polish the Questions

      Check if the questions to be answered are valid or well stated; If not, modify them or come up with new ones

    2. Validate Data I/O Methods

      Check and validate the methods to load and save the datasets

    3. Is the Dataset Good Enough for the Problem?
      1. Are the features/variables required for the project included?

        If not, what other data should be included.

      2. What is the General Quality of the Dataset
      3. Can one answer the questions semi-quantitatively using the data?

        Is the dispersion of the target value small enough?

    4. Retrieve Domain Knowledge and Anomalies

      Determine the ranges, outliers of the dataset; Talk to domain experts and validate with domain experts.

    5. Propose the Next Steps
  2. Communicate with Domain Experts :
    1. What are the features?

      Pay attention to the units

    2. Do the results from EDA make sense to the experts?
    3. What do the experts want to know from the data?
  3. Workflow :
    1. Polish the Questions
      1. Will there be any new restrictions to the solutions?

        related to the dataset

    2. Data Quality and Summary
      1. Data Quality and Summary Statistics
      2. Report Data Quality and Summary
        1. Does the result make sense?

          This is a crucial step in EDA. Use techniques such as Fermi estimates to evaluate the summary.

        2. Consistencies between the summary and expert expectations