Pandas Complete Pandas Tutorial


📌 1️⃣ What is Pandas?

Pandas is a Python library used for:

  • Data Analysis

  • Data Cleaning

  • Data Manipulation

  • Handling CSV, Excel, SQL data

It works mainly with:

  • Series → 1D data

  • DataFrame → 2D tabular data (like Excel table)

📌 2️⃣ Installation

pip install pandas

import pandas as pd

📌 3️⃣ Pandas Core Data Structures

🔹 3.1 Series (1D)

import pandas as pd

data = [10, 20, 30, 40]
s = pd.Series(data)

print(s)

With custom index:

s = pd.Series(data, index=["a", "b", "c", "d"])

🔹 3.2 DataFrame (2D Table)

data = {
    "Name": ["Hari", "Rahul", "Amit"],
    "Age": [25, 30, 35],
    "Salary": [50000, 60000, 70000]
}

df = pd.DataFrame(data)
print(df)

📌 4️⃣ Reading Data

🔹 CSV

df = pd.read_csv("data.csv")

🔹 Excel

df = pd.read_excel("data.xlsx")

🔹 JSON

df = pd.read_json("data.json")

📌 5️⃣ Basic Data Inspection

df.head()      # First 5 rows
df.tail()      # Last 5 rows
df.info()      # Structure
df.describe()  # Statistics
df.shape       # Rows & columns
df.columns     # Column names
df.dtypes      # Data types

📌 6️⃣ Selecting Data

🔹 Column Selection

df["Age"]
df[["Name", "Salary"]]

🔹 Row Selection

df.loc[0]        # By label
df.iloc[0]       # By position

🔹 Slicing

df.iloc[0:2]

📌 7️⃣ Filtering Data

df[df["Age"] > 30]

Multiple conditions:

df[(df["Age"] > 25) & (df["Salary"] > 60000)]

📌 8️⃣ Adding & Modifying Columns

df["Bonus"] = df["Salary"] * 0.1

Modify values:

df["Age"] = df["Age"] + 1

📌 9️⃣ Handling Missing Data

🔹 Check missing

df.isnull()
df.isnull().sum()

🔹 Fill missing

df.fillna(0)
df.fillna(df["Age"].mean())

🔹 Drop missing

df.dropna()

📌 🔟 Sorting

df.sort_values("Age")
df.sort_values("Salary", ascending=False)

📌 1️⃣1️⃣ GroupBy (VERY IMPORTANT 🔥)

df.groupby("Department")["Salary"].mean()

Multiple aggregations:

df.groupby("Department").agg({
    "Salary": "mean",
    "Age": "max"
})

📌 1️⃣2️⃣ Merging DataFrames

pd.merge(df1, df2, on="EmployeeID")

Join types:

  • inner

  • left

  • right

  • outer

📌 1️⃣3️⃣ Apply Function

df["Salary"].apply(lambda x: x * 1.1)

📌 1️⃣4️⃣ Pivot Tables

pd.pivot_table(df, 
               values="Salary",
               index="Department",
               aggfunc="mean")

📌 1️⃣5️⃣ Handling Dates

df["Date"] = pd.to_datetime(df["Date"])
df["Year"] = df["Date"].dt.year

📌 1️⃣6️⃣ Performance Tips 🚀

✔ Use vectorized operations
✔ Avoid loops
✔ Use .loc instead of chained indexing
✔ Use categorical dtype for repeated strings

📌 1️⃣7️⃣ Real ML Workflow Example

import pandas as pd

df = pd.read_csv("data.csv")

# Clean
df = df.dropna()

# Feature Engineering
df["AgeGroup"] = df["Age"] // 10

# Encoding
df = pd.get_dummies(df, columns=["Gender"])

# Split
X = df.drop("Target", axis=1)
y = df["Target"]

📌 1️⃣8️⃣ Most Asked Interview Questions

  1. Difference between loc and iloc?

  2. What is groupby?

  3. How does merge work?

  4. How to handle missing data?

  5. Difference between apply and map?

  6. What is pivot table?

  7. How to improve Pandas performance?

📌 1️⃣9️⃣ Common Mistakes

❌ Using loops instead of vectorization
❌ Not handling missing values
❌ Wrong merge keys
❌ Forgetting inplace behavior

📌 2️⃣0️⃣ Advanced Topics (For Senior ML Role)

  • MultiIndex

  • Window functions

  • Rolling mean

  • Resampling time-series

  • Chunk processing

  • Memory optimization

  • Query optimization

Perfect 👍
Let’s do a complete real-world Pandas example step-by-step.

Since you're preparing for ML roles, I’ll make a Salary Prediction Data Cleaning Example like a real interview / ML project.

🐼 Real-World Pandas Example

📊 Employee Salary Analysis & Cleaning

🔹 Step 1: Create Sample Dataset

import pandas as pd
import numpy as np

data = {
    "Name": ["Hari", "Rahul", "Amit", "Neha", "Priya", "Karan"],
    "Age": [25, 30, np.nan, 28, 35, 40],
    "Department": ["IT", "HR", "IT", "Finance", "HR", "IT"],
    "Salary": [50000, 60000, 55000, np.nan, 65000, 70000],
    "Experience": [2, 5, 3, 4, np.nan, 10]
}

df = pd.DataFrame(data)
print(df)

🔹 Step 2: Inspect Data

df.info()
df.describe()
df.isnull().sum()

Output insight:

  • Missing values in Age, Salary, Experience

  • Salary and Age are numeric

🔹 Step 3: Handle Missing Values

Fill numeric columns with mean

df["Age"].fillna(df["Age"].mean(), inplace=True)
df["Salary"].fillna(df["Salary"].mean(), inplace=True)
df["Experience"].fillna(df["Experience"].mean(), inplace=True)

Now check again:

df.isnull().sum()

✅ All missing handled

🔹 Step 4: Feature Engineering

1️⃣ Add Bonus column

df["Bonus"] = df["Salary"] * 0.10

2️⃣ Create Age Group

df["AgeGroup"] = df["Age"] // 10

3️⃣ Salary per Experience

df["SalaryPerYearExp"] = df["Salary"] / df["Experience"]

🔹 Step 5: Filtering Data

Employees with Salary > 60000

high_salary = df[df["Salary"] > 60000]
print(high_salary)

🔹 Step 6: GroupBy Analysis (VERY IMPORTANT 🔥)

Average Salary per Department

dept_salary = df.groupby("Department")["Salary"].mean()
print(dept_salary)

Multiple aggregations

df.groupby("Department").agg({
    "Salary": "mean",
    "Experience": "max"
})

🔹 Step 7: Sorting

df.sort_values("Salary", ascending=False)

🔹 Step 8: Encoding for ML

Convert Department to numeric:

df = pd.get_dummies(df, columns=["Department"], drop_first=True)
print(df.head())

🔹 Step 9: Prepare ML Data

X = df.drop(["Name", "Salary"], axis=1)
y = df["Salary"]

print(X.head())
print(y.head())

🎯 Final Cleaned DataFrame

You now have:

  • No missing values

  • New features

  • Encoded categorical variables

  • ML-ready dataset

🚀 What You Just Practiced

✔ Data Cleaning
✔ Missing value handling
✔ Feature Engineering
✔ GroupBy
✔ Filtering
✔ Sorting
✔ Encoding
✔ ML Preparation

]


Comments

Post a Comment

Popular posts from this blog

Python Tutorial with examples

🐍 Python Syllabus (Beginner → Advanced) 1. Beginner Level Introduction What is Python, installation, IDEs Running Python scripts Basics Syntax, keywords, indentation Variables & data types Input/Output Operators (arithmetic, relational, logical) Control Flow Conditional statements ( if , elif , else ) Loops ( for , while ) Loop control ( break , continue , pass ) Collections Lists, tuples, sets, dictionaries Indexing, slicing, methods 2. Intermediate Level Functions Defining functions, arguments, return values Default & keyword arguments Lambda functions Modules & Packages Importing modules Standard libraries overview Error Handling Exceptions ( try , except , finally ) Custom exceptions File Handling Reading/writing files Working with CSV, JSON Object-Oriented Programming Classes & objects Constructors, methods Inheritance, polymorphism, encapsulation 3. Advanced Level Iterators & Generators iter() , n...
Read more

Machine learning tutorial

  📘 Definition of Machine Learning Machine learning (ML) is a subfield of artificial intelligence (AI) that focuses on building algorithms and models that can learn patterns from data and make predictions or decisions without being explicitly programmed for each task. Instead of hard-coded rules, ML systems improve their performance as they are exposed to more data. 🔑 Key Points Pattern Recognition : ML algorithms identify hidden structures in datasets and generalize them to new, unseen data. Adaptability : Models evolve dynamically as new data arrives, staying relevant in changing environments. Applications : Used in image and speech recognition, natural language processing, recommendation systems, fraud detection, autonomous vehicles, and generative AI. Types of ML : Supervised Learning : Learns from labeled data (e.g., spam detection). Unsupervised Learning : Finds patterns in unlabeled data (e.g., customer segmentation). Semi-supervised Learning : Mix of labeled a...
Read more

RAG (Retrieval-Augmented Generation) tutorial

1. What is RAG (Retrieval-Augmented Generation)? Idea: Instead of asking an LLM to answer only from its own training, you: Retrieve relevant documents from your data (PDFs, DB, website, etc.) Augment the prompt with those docs Let the Generator (LLM) answer using that context. So pipeline is: User query ➜ Retrieve top-k docs ➜ Build prompt with docs ➜ LLM answers This solves: Outdated model knowledge Hallucinations (partially) Domain customization (company docs, reports, codebase) 2. High-Level RAG Architecture Think of 2 phases: 2.1. Offline / One-Time: Build Knowledge Store Collect data PDFs, HTML pages, markdown, CSVs, Confluence, Notion, etc. Chunking Split into small text chunks (e.g., 200–500 tokens) with overlap. Why: too big ➜ irrelevant noise; too small ➜ not enough context. Embed chunks Use an embedding model (e.g., OpenAI text-embedding, or local like bge, all-MiniLM, etc.). For each chunk: embedding = embedding_model(chunk_text) Store in Vector DB Store: (id, chunk_text...
Read more