Data Analysis and Visualization Assignment And Project Help | Realcode4you

Data Visualisation

What is it and why is it important?

• Data visualization is the graphical representation of data.

• Essential for large data.

• Visualising you data is probably the best way to understand your data, in particular trend or patterns in the data.

• Allows one to make data-driven decisions.

• Useful data cleaning, i.e. duplication, data entry error.

• Plots are a great way to detect abnormalities in your data (outliers).

Bar Charts

• Does not work well for continuous features.

  

  
  

Histograms

• Work well for continuous features

• Similar to bar charts, but the observations are group by bins

• Bins are equal-sized intervals along the observation scale

• Choice of bin size and/or the number of bins is important

Histograms

• Effective in visualising data distribution

Boxplots

• Also useful for continuous features.

• Graphical illustration of the 5-number summary

• Outliers are noted, but be careful not to over-react.

Excellent for comparing distributions across different groups.

-Potential pitfalls

• Outliers that are not actual outliers

• Distribution of observations is not necessary clear.

-Do not rely on boxplots alone.

Normal Distribution

The 68-95-99.7 rule states that for a normal distribution with mean μ and σ:

1. Approximately 68 % of the observations fall within 1 σ from the mean μ.  

2. Approximately 95 % of the observations fall within 2 σ from the mean μ.  

3. Approximately 99.7 % of the observations fall within 3 σ from the mean μ.

   
   

Assumption of Normality

• The assumption of normality is important for many parametric methods.

• Can be checked graphically using quantile-quantile (Q-Q) plot (requiring a straight line with a positive slope at 450 angle)

• Can be formally tested (requiring a p>0.05)

-Shapiro-Wilk test (better for small-moderate datasets, n<100)

-Pearson Chi-square test (better for larger datasets)

Example

Calcium intake (mg/day) of 111 women (Calcium Intake.csv)

Data Transformation

• Data transformation is often performed when the original data do not satisfy the assumption of normality

• Data transformation can also stabilise the sample variance between groups.

• A rough guideline:

-Logarithmic transformations: log (x) or ln(x) – scale data that are positively skewed (and all data is positive),

-Square- (√x) or fourth-root (∜x) transformation: – count data;

-Arcsine(x) – proportions.

• Box-Cox transformation is another popular approach.

Example – Log-transformed Calcium Intake

Scatter plots

• 2-D plots using Cartesian coordinates to display the values of two continuous variables, say x and y.

• Each point in a scatter plot represent the (x,y) pairing.

• Great for identifying bivariate trends and patterns (if any)

Covariance and Correlation

• Once a relationship is established visually, we can formally quantify it by using covariance and correlation.

• Covariance is a measure of joint variability between two features.

• Given two continuous features, x and y, and each with n samples, the sample covariance is given by :

Covariance

• A covariance value ranges between [-∞,∞]

• A positive covariance value implies a positive relationship

• A negative covariance value implies a negative relationship

• A covariance value near zero implies a neutral relationship

• The magnitude of a covariance value is difficult to interpret as it depends on the scale and unit of the features.

Iris Setosa data

Sepal length vs Sepal width

• In R, we use the cov(.) command

• The result is a covariance matrix

• Covariance is given as the off-diagonal element

• The diagonal entries are the variances