top of page
realcode4you

Case Study Assignment Help In Data Mining

Requirement Details

An organization wanted to check association among employee experience, skills, traits etc. to better manage human resources.


As a data scientist, you are required to recognize patterns from the available data and evaluate efficacy of methods to obtain patterns. Your activities should include - performing various activities pertaining to the data such as, preparing the dataset for analysis; investigating the relationships in the data set with visualization; identify frequent patterns; formulate association rules and evaluate quality of rules.


Demonstrate KDD process with following activities:

  • Problem statement

  • ·Perform exploratory data analysis

  • Preprocess the data

  • Propose parameters such as support, confidence etc.

  • Discover frequent patterns

  • Iterate previous steps by varying parameters

  • Formulate association rules

  • Compare association rules

  • Briefly explain importance of discovered rules


Following are some points for you to take note of, while doing the assignment:

  • The data in some of the rows in the data set may be noisy

  • Some of the attributes have large number of values – you can consider merging them into 2 or 3 values to simplify the solution

  • State all your assumptions clearly

  • Provide clear explanations to explain your stand


Solution:

Import Libraries

#import libraires
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.cluster import KMeans
from sklearn.metrics import mean_absolute_error, make_scorer
import sklearn.metrics as metrics
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
import seaborn as sns
import itertools 
import numpy as np
import matplotlib.pyplot as plt

Read Data

#Read Dataset
df_train = pd.read_csv("Employee_skills_traits.csv")
df_train

Output:


Replace column attribute space

#Replace Column attribute space with '-'
df_train.columns = df_train.columns.str.replace(' ','_')

Now Check columns

#Show all dataset Columns
df_train.columns

Output:

Index(['ID', 'Employment_period_', 'Time_in_current_department_', 'Gender_',
       'Team_leader_', 'Age_', 'Member_of_professional_organizations_',
       '.Net_', 'SQL_Server_', 'HTML_CSS_Java_Script_', 'PHP_mySQL_',
       'Fast_working', 'Awards', 'Communicative_'],
      dtype='object')

Check Data After Rename Attribute

df_train


Here we see that all column name are renames and space is removed(Here we do this steps because space raise issue to read columns)



Checking Dataset Null Values

#Checking Null Value
#Visualize for check null value
check_null_value = df_train.isnull()
sns.heatmap(check_null_value,yticklabels=False,cbar=False,cmap='viridis')

Output:














As per above heat map we can say that it has no null value.



Checking Shape of dataset

#– the shape of the dataset
df_train.shape

Output:

(998, 14)

Dataset has 998 rows and 14 columns



Checking Data Type

#– info of the dataset
df_train.info()

Output:














Summary of the dataset

#– summary of the dataset
df_train.describe()

Output:



Finding Features and target variable and Split dataset


#Devided dataset into with target attribute "Awards"
x=df_train.drop(['Awards'],axis=1)
target=df_train.Awards
#split dataset with 25 percent test sample
X_train, X_test, y_train, y_test = train_test_split(x, target, test_size=0.25, random_state=0)

Use K means Clustering Algorithm And Fit into Model

#Use Data mining Clustering and Find the Mean Square Error
kmeans = KMeans(n_clusters=2) 
kmeans.fit(X_train, y_train)
y_pred = kmeans.predict(X_test)
test_mae = mean_absolute_error(y_test, y_pred)
print('Validation MAE: %0.3f' % test_mae)

Output:

Validation MAE: 0.548

Accuracy

#Find Accuracy of Model
score = metrics.accuracy_score(y_test,y_pred)
print('Accuracy: %0.2f' % score)

Output:

Accuracy: 0.45


Fine Tune the Model To increase the accuracy

#Use Data mining Clustering and Find the Mean Square Error
#Fine Tune The Model to minimize the MSE and increase Accuracy
kmeans = KMeans(n_clusters=2, init='k-means++', n_init=5, max_iter=100, tol=0.0001) 
kmeans.fit(X_train, y_train)
y_pred = kmeans.predict(X_test)
test_mae = mean_absolute_error(y_test, y_pred)
print('Validation MAE: %0.3f' % test_mae)

Output:

Validation MAE: 0.452

Accuracy

#Find Accuracy of Model
#Find Accuracy After Fine tune Parameter
score = metrics.accuracy_score(y_test,y_pred)
print('Accuracy: %0.2f' % score)

Here we have see increase the model Accuracy when we have tune the model using extra K means parameters



Plot the Confusion Matrix

#Evaluation of Model - Confusion Matrix Plot

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    plt.tight_layout()


# Compute confusion matrix
cnf_matrix = confusion_matrix(y_test, y_pred)
np.set_printoptions(precision=2)

# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=['0','1'],
                      title='Confusion matrix, without normalization')

Output:













Find Recall And Precision

#Find Recall and Precision
cnf_matrix = confusion_matrix(y_test, y_pred)
recall = np.diag(cnf_matrix) / np.sum(cnf_matrix, axis = 1)
precision = np.diag(cnf_matrix) / np.sum(cnf_matrix, axis = 0)

Confusion Matrix:

cnf_matrix

Output:

array([[63, 61],
       [52, 74]], dtype=int64)

Precision:

#Precision Score
precision

Output:

array([0.55, 0.55])

Recall Score:

#Recall Score
recall

Output:

array([0.51, 0.59])


Exploratory Data Analysis

#EDA(Exploratory Data Analysis)
#ploting Graph of co-rrelation
fig = plt.figure(figsize=(10,5))
sns.heatmap(df_train.corr())

Output:











Box Plot

#Box Plot between ID & Employment_Period
#Here Both attribute(ID & Employment_period_) is associated to each other
sns.boxplot(x=df_train['Employment_period_'],y=df_train['ID'] )

Output:












# visualize frequency distribution of `Gender` variable
f, ax = plt.subplots(figsize=(9, 7))
ax = sns.countplot(x="Gender_", data=df_train, palette="Set1")
ax.set_title("Frequency distribution of Gender variable")
ax.set_xticklabels(df_train.Gender_.value_counts().index, rotation=30)
plt.show()

Output:















Line Chart

#Bar Plot(Here we choose some records of daset due to show visible plots)
df_train_plot = df_train.iloc[:18]
df_train_plot.plot(x= "ID", y= "Employment_period_", kind="bar")

Output:















If you need any other help related to machine learning then send your requirement details at realcode4you@gmail.com and get instant help.

Comments


bottom of page