6141

Business Analytics and AI Application

Business Analytics and AI Application (Course 6141)

Course Overview

Students in this course will develop a comprehensive understanding of business analytics and artificial intelligence fundamentals, covering topics such as business requirement analysis, data collection, and machine learning. Designed specifically for undergraduate and Master of Business Administration (MBA) students with zero prior programming background, this course emphasizes intuitive, evidence-based reasoning over abstract theoretical complexity.

Through hands-on exercises, students gain operational proficiency in Python programming, progressing from core syntax to intermediate techniques essential for enterprise data projects. Students acquire data manipulation skills using foundational libraries including NumPy and Pandas for data cleaning, aggregation, and financial visualization. They also master automated web data extraction techniques using HyperText Markup Language (HTML), Cascading Style Sheets (CSS), and Beautiful Soup to collect public competitor intelligence.

With an introduction to modern machine learning tools through Scikit-Learn, participants learn to implement, interpret, and evaluate predictive models. Ethical considerations in artificial intelligence projects, algorithmic risk management, and governance frameworks are examined to ensure the responsible application of these analytical technologies.

Development Environment and AI Error Protocol

All sessions use Google Antigravity IDE with four workspace panels: [1] File Explorer, [2] Code Editor, [3] AI Agent Chat, and [4] Integrated Terminal. When an error occurs, students identify the error type in Panel [4], capture the traceback with Windows Key + Shift + S or Command + Shift + 4, paste it into Panel [3], review the explanation, apply the verified fix in Panel [2], and confirm an error-free execution with return code 0. Session 1.1 provides the complete orientation; every later lab applies the same workflow.

Module 1: Foundations of Business Analytics and Python Programming
Module 2: Web Data Extraction, Visual Dashboards, and Midterm Milestone
Module 3: Artificial Intelligence, Machine Learning Fundamentals, and Scikit-Learn
Module 4: Strategic AI Integration, Enterprise Decision-Making, and AI Governance

Module 1: Foundations of Business Analytics and Python Programming

Module 1 establishes the computational foundation for business students. Students learn to translate executive business questions into structured quantitative workflows, set up a standardized Python execution environment, master variables and arithmetic for commercial metrics, control program flow, and inspect tabular corporate filings using Pandas.

Session 1.1: Business Analytics Architecture, Google Antigravity IDE Initialization, and Core Arithmetic

Session 1.1 Summary Slide

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 1 (115/09/13 - 115/09/19) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
Session Purpose:
- Set up your Python workspace
- Replace spreadsheet cell formulas with named variables
- Calculate real business metrics in code
Expected Learning Outcomes (By the end of this session, you will be able to):
1. Run a script in Google Antigravity IDE
2. Compute operating expense and margin with variables
3. Fix a NameError or SyntaxError with AI Chat
Keywords (Core Programming Concepts):
- Python Interpreter: The computational engine that reads Python source code and executes instructions sequentially line-by-line in real time without requiring compilation.
- Variable Assignment: Storing data in computer memory using an assignment operator (=) and an intuitive descriptive label (such as net_sales = 574785.0), replacing fragile cell coordinates like C13.
- Primitive Data Types: The foundational categories of information processed by Python: whole numbers (int), decimal numbers (float), and text strings (str).
- Traceback & Exception: The automatic diagnostic navigation report generated by Python when an instruction violates a syntax or runtime rule, pinpointing the exact file and line number.
- Integrated Development Environment (IDE): A unified software workstation (Google Antigravity IDE) providing an editor, execution terminal, file explorer, and integrated AI assistant in a single interface.
Core Programming & Computational Foundations:
- Sequential Execution & Program State: Python executes code strictly line-by-line from top to bottom. Unlike spreadsheets where formulas across sheets can trigger circular reference loops, Python code maintains a clear, linear flow where each statement computes or alters memory state sequentially.
- Cell Coordinate Formulas vs. Descriptive Variables: Transitioning from Excel spatial grid thinking (such as =C13-E13) to self-documenting computational recipes (operating_expense = net_sales - operating_income). Code variables retain clear business meaning and eliminate silent formula drift.
- Interpreter Diagnostics & Traceback Mechanics: Python acts as an interactive assistant. When a syntax or naming rule is violated, Python pauses and outputs a Traceback pointing to the exact line number and cause, turning errors into immediate learning signals rather than system failures.
Hands-On Coding Micro-Examples:
- Micro-Example 1 (Variable Arithmetic): Defining commercial numbers (net_sales = 574785.0, operating_income = 36852.0) and calculating Operating Expense and Operating Margin Percentage with standard math operators (-, /, *).
- Micro-Example 2 (Dynamic Parameter Reusability): Changing a single baseline assumption (such as revising sales tax or discounting prices by 5%) and observing how Python recalculates all downstream figures instantaneously without manual cell dragging.
- Micro-Example 3 (Traceback Troubleshooting & AI Fix): Creating an intentional variable typo (netsales instead of net_sales), observing the NameError in the terminal, capturing it with Windows Key + Shift + S, and pasting into the AI chat to verify how the co-pilot identifies the typo and supplies the correction.
Lab Practice Dataset:
- File Link: data/session_01_01_dataset.csv
- Simple 4-row tabular financial dataset used to verify calculations:
  - Segment_Name: North America, International, AWS, Consolidated Total.
  - Net_Sales_USD_M: Revenue in millions of U.S. Dollars.
  - Operating_Income_USD_M: Operating income in millions of U.S. Dollars.
Official Python Documentation & References:
Integrated Development Environment Architecture (Google Antigravity IDE):
- Students use Google Antigravity IDE, a modern AI-powered development environment designed for business analytics, financial modeling, and collaborative data science.
- The 4 Workspace Panels:
  - [1] File Explorer (Left Panel): Workspace file hierarchy. Used to organize corporate datasets (data/), slide decks (slides/), Python scripts (.py), and curriculum documentation (.md).
  - [2] Code Editor & Notebook Canvas (Center Top): Primary workspace for writing Python code, defining financial variables, and executing arithmetic business formulas.
  - [3] AI Agent Chat Canvas (Right Panel): An interactive AI pair-programming assistant powered by Gemini. Students paste code snippets or error screenshots directly into this window to receive plain-English root cause explanations and verified code corrections.
  - [4] Integrated Terminal & Console (Center Bottom): Command line interface running the Python execution kernel. Displays script output results and detailed system error tracebacks.
- AI Error Diagnosis Protocol (Capture and Resolve in Seconds):
  - Zero Fear Principle: In Python, an error is not a system failure or a test penalty. It is simply a diagnostic navigation clue (Traceback) showing the exact line number and rule violation.
  - Step A (Identify): When code execution stops, look at the last line in Panel [4] (Terminal), which names the error type (e.g., NameError, SyntaxError, TypeError).
  - Step B (Capture): Take a fast screen capture of the error message using Windows Key + Shift + S (Windows) or Command + Shift + 4 (macOS).
  - Step C (Paste & Prompt): Click inside Panel [3] (AI Agent Chat), press Ctrl + V to attach the captured image, and ask: “Why did this error happen, and what is the exact corrected code?”
  - Step D (Apply): Review the AI explanation, understand the typo or missing parenthesis, and copy the verified fix back into Panel [2].
Step-by-Step Lab Protocol (Exercise 1: Foundational Practice):
- Step 1: Open Environment & Dataset: Launch Google Antigravity IDE. Place data/session_01_01_dataset.csv in your working directory and open the raw file via Panel [1] (File Explorer) to observe comma-separated tabular records.
- Step 2: Inspect Primary Sources & Financial Terminology: Confirm against Note 15 of Amazon’s Form 10-K that Consolidated Net Sales for FY2023 equaled $574,785 million and Operating Income equaled $36,852 million.
- Step 3: Define Variables and Execute First Python Calculations: In your first Python notebook cell, type the following arithmetic statements and run the cell:
```
# Amazon Consolidated Financial Metrics for FY2023 (in USD Millions)
net_sales = 574785.0
operating_income = 36852.0

# Calculate Operating Expenses
operating_expense = net_sales - operating_income

# Calculate Operating Margin Percentage
operating_margin_pct = (operating_income / net_sales) * 100.0

print("Operating Expense (USD M):", operating_expense)
print("Operating Margin (%):", round(operating_margin_pct, 2))
```
- Step 4: Bridge Excel Spreadsheet Mechanics to Python Code:
  - In Microsoft Excel, if Net_Sales is in cell C13 (574785.00) and Operating_Income is in cell E13 (36852.00), the expense formula in cell D13 is =C13-E13, and the margin formula in cell F13 is =(E13/C13)*100.
  - Notice that Python variable assignment (operating_income / net_sales) performs the identical mathematical operation as cell references, but uses memorable descriptive English words instead of abstract grid coordinates like E13.
- Step 5: Compare Segment Profitability: Execute a comparative calculation for Amazon Web Services (AWS) in FY2023:
```
aws_sales = 90757.0
aws_income = 24631.0
aws_margin_pct = (aws_income / aws_sales) * 100.0
print("AWS Operating Margin (%):", round(aws_margin_pct, 2))
```
  Confirm that the AWS operating margin (27.14%) is over four times higher than the consolidated corporate operating margin (6.41%), demonstrating how cloud infrastructure drives corporate operating cash flow.
Step-by-Step Applied Coding Practice (Exercise 2+: Hands-On Programming & Debugging):
- Coding Challenge: In Google Antigravity IDE, create a new script named session_1_1_practice.py to calculate financial metrics across multiple business units and practice error debugging.
- Task 1 (Segment Margin Calculation): Define variables for Amazon Web Services (AWS) using FY2023 figures (aws_sales = 90757.0, aws_income = 24631.0). Compute aws_margin_pct and print the formatted result with round(..., 2).
- Task 2 (Intentional Syntax Error & AI Diagnosis): Deliberately omit the closing parenthesis in your print() statement (e.g., print("AWS Margin:", round(aws_margin_pct, 2)). Run the script in Panel [4] (Terminal), observe the SyntaxError, capture the terminal message using Windows Key + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and verify the AI’s explanation and fix.
- Task 3 (Multi-Unit Comparison Output): Combine consolidated and cloud margin calculations into a single clean script that prints both metrics formatted with labels. Verify that the script executes with return code 0.

Session 1.2: Essential Python Syntax Completion: Control Flow, Collections, and Business Automation

Session 1.2 Summary Slide

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 2 (115/09/20 - 115/09/26) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
Session Purpose:
- Turn written business rules into Python logic
- Let code decide and repeat automatically
- Stop copying formulas row by row
Expected Learning Outcomes (By the end of this session, you will be able to):
1. Apply tiered rules with if / elif / else
2. Process client records with lists, dictionaries, and loops
3. Fix an IndentationError in a full script
Keywords (Core Programming Concepts):
- Boolean Expression: A logical comparison that evaluates strictly to True or False, used by computers to test commercial conditions (such as annual_spend >= 50000.0).
- Conditional Branching (else): Directing code execution down distinct logical pathways depending on operational thresholds, eliminating complex spreadsheet nested logic.
- List Collection: An ordered, mutable sequence of business records enclosed in square brackets (e.g., daily_orders = [120, 85, 94, 142]), indexed sequentially starting from zero.
- Dictionary Collection: A key-value associative data structure enclosed in curly braces (e.g., {"client_id": "CUST-101", "annual_spend_USD": 145000.0}), enabling instant data retrieval by descriptive business keys rather than numeric indices.
- Definite Iteration (for loop): A programmatic control structure that automatically iterates over every record in a collection, executing identical business rules across thousands of entries in milliseconds.
Core Programming & Computational Foundations:
- Eliminating Fragile Nested Spreadsheets: In Excel, multi-tier decision policies require nested formulas such as =IF(C2>100000, "Platinum", IF(C2>50000, "Gold", IF(C2>20000, "Silver", "Standard"))). These formulas quickly become unreadable, impossible to audit, and vulnerable to formula drift. In Python, conditional branching (if-elif-else) structures business rules into clean vertical indentation blocks that mirror managerial decision trees.
- Deterministic Repetition over Human Fatigue: When processing transactional ledgers with thousands of rows, human analysts experience fatigue, eye strain, and copy-paste errors. A Python for loop executes repetitive calculations across 10 records or 1,000,000 records with identical mathematical precision and zero cognitive degradation.
- Real-World Entity Modeling with Dictionaries: Modeling complex commercial entities as structured computational objects. A single customer account is captured as a dictionary of labeled attributes (name, industry, spend, risk score), while an enterprise client portfolio is structured as an iterable list of dictionaries.
- Indentation as Syntax Structure (The Whitespace Discipline): Unlike older programming languages that use curly brackets {} or BEGIN/END tokens, Python enforces clean code formatting through whitespace indentation (4 spaces). An IndentationError occurs when code blocks are misaligned, serving as an automated guardrail for code readability.

Hands-On Coding Micro-Examples:

Micro-Example 1 (Conditional Customer Tiering): Evaluating client annual spend to determine customer loyalty tiers and discount rates:

annual_spend = 82000.0

if annual_spend >= 100000.0:
    tier = "Platinum"
    discount_rate = 0.15
elif annual_spend >= 50000.0:
    tier = "Gold"
    discount_rate = 0.10
elif annual_spend >= 20000.0:
    tier = "Silver"
    discount_rate = 0.05
else:
    tier = "Standard"
    discount_rate = 0.0

print(f"Customer Tier: {tier} | Approved Discount: {discount_rate * 100:.0f}%")

Micro-Example 2 (Iterative Revenue Accumulation): Looping through a list of daily store sales to calculate cumulative revenue and average daily performance:

daily_sales = [4200.0, 5100.0, 3900.0, 6200.0, 7800.0, 8400.0, 4900.0]
total_revenue = 0.0

for sale in daily_sales:
    total_revenue += sale

avg_daily_sales = total_revenue / len(daily_sales)
print("Total Weekly Revenue (USD):", round(total_revenue, 2))
print("Average Daily Sales (USD):", round(avg_daily_sales, 2))

Micro-Example 3 (Dictionary Client Profile & Safe Retrieval): Storing commercial account attributes in a dictionary and computing available credit limits:

client = {
    "client_id": "CUST-104",
    "company_name": "Apex Logistics",
    "annual_spend_USD": 215000.0,
    "credit_limit_USD": 75000.0,
    "outstanding_balance_USD": 23400.0
}

available_credit = client["credit_limit_USD"] - client["outstanding_balance_USD"]
print(f"{client['company_name']} Available Credit: ${available_credit:,.2f}")

Lab Practice Dataset:
- File Link: data/session_01_02_dataset.csv
- Standard 10-row commercial client dataset used for batch automation and loop exercises:
  - Customer_ID: Unique client identification code (e.g., CUST-101).
  - Industry_Segment: Commercial vertical (Retail, Technology, Healthcare, Finance, Manufacturing).
  - Annual_Spend_USD: Total historical purchase volume in U.S. Dollars.
  - Order_Count: Total orders completed in the fiscal year.
  - Avg_Order_Value_USD: Average transaction size in U.S. Dollars.
  - Payment_Terms_Days_Count: Invoiced credit settlement period (30, 60, 90 days).
  - Risk_Score_Count: Internal risk assessment index (1 to 100).
Official Python Documentation & References:

Step-by-Step Lab Protocol (Exercise 1: Guided Practice - Client Portfolio Automation):

Step 1: Create Practice File: Launch Google Antigravity IDE and create a new script named session_1_2_automation.py in your working directory.

Step 2: Build a Client Portfolio List: Define an iterable list of customer dictionaries containing audited client records from data/session_01_02_dataset.csv:

clients = [
    {"id": "CUST-101", "segment": "Retail", "spend": 145000.0, "risk": 15},
    {"id": "CUST-102", "segment": "Technology", "spend": 82000.0, "risk": 22},
    {"id": "CUST-103", "segment": "Healthcare", "spend": 31000.0, "risk": 38},
    {"id": "CUST-104", "segment": "Finance", "spend": 215000.0, "risk": 12},
    {"id": "CUST-105", "segment": "Manufacturing", "spend": 64000.0, "risk": 45}
]

Step 3: Automate Tier Assignment and Credit Multiplier via for Loop: Write a for loop that iterates over each client, applies conditional branching based on spend, and computes an approved credit limit:

for c in clients:
    # Tiering logic based on annual purchase volume
    if c["spend"] >= 100000.0 and c["risk"] < 30:
        c["tier"] = "Platinum"
        c["credit_limit"] = c["spend"] * 0.40
    elif c["spend"] >= 50000.0 and c["risk"] < 50:
        c["tier"] = "Gold"
        c["credit_limit"] = c["spend"] * 0.25
    else:
        c["tier"] = "Standard"
        c["credit_limit"] = c["spend"] * 0.10

    print(f"Client {c['id']} ({c['segment']}): Tier={c['tier']} | Credit Limit=${c['credit_limit']:,.2f}")

Step 4: Execute and Verify Output in Terminal: Run the script in Panel [4] (Terminal) using $ python session_1_2_automation.py. Confirm that all 5 client accounts are evaluated and printed with proper credit limits.

Step-by-Step Applied Coding Practice (Exercise 2+: Hands-On Programming & AI Debugging):
- Coding Challenge: In Google Antigravity IDE, create a new script named session_1_2_practice.py to automate a multi-tier commercial commission payout pipeline.
- Task 1 (Tiered Commission Functionality): A sales team records quarterly sales volumes: sales_reps = [{"name": "Alice", "revenue": 142000.0}, {"name": "Bob", "revenue": 68000.0}, {"name": "Charlie", "revenue": 35000.0}]. Write a loop calculating commission payouts: 12% for revenue over $100,000, 8% for revenue between $50,000 and $100,000, and 5% for revenue below $50,000.
- Task 2 (Deliberate IndentationError & AI Diagnosis): Intentionally misalign one line inside the if block (e.g., adding 2 extra spaces or removing indentation). Run the script in Panel [4] (Terminal), observe the red IndentationError: unexpected indent, capture the terminal traceback with Windows Key + Shift + S, paste the image into Panel [3] (AI Agent Chat) with Ctrl + V, and review how the AI explains whitespace alignment.
- Task 3 (Dictionary Summary Output): Correct the indentation, append the calculated commission_payout back into each rep’s dictionary, and print a formatted compensation summary with zero syntax errors. Verify that the script terminates with return code 0.

Session 1.3: Tabular Data Wrangling Part 1: NumPy Arrays, Pandas DataFrames, and CSV Ingestion

Session 1.3 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 3 (115/09/27 - 115/10/03) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- Business systems (sales registers, accounting, customer databases, Excel) share tables as Comma-Separated Values (CSV) files
- Our case: a retail chain exports monthly results for its 12 stores as a CSV file
- Each row holds one store’s sales, costs, visitors, and customer rating
- Your manager asks: which stores are most profitable, and which meet the target? Today, Python answers these questions for every store at once
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “Did the store data load correctly?”: read and inspect a DataFrame (a table in Python, like one Excel sheet)
2. “What is each store’s profit margin?”: create new columns with vectorized math (one formula runs on every row at once)
3. “Which stores meet our target?”: filter rows with .loc[] (keep rows that match a condition) and fix a KeyError (the column name does not exist)

3. Business Context: Why CSV?

Where business data comes from: sales registers, the accounting system, the customer database, and Excel reports. Each system can export its tables as one shared file.

.xlsx (Excel file) vs .csv (CSV file):

	`.xlsx` Excel file	`.csv` CSV file
Format	Packed format	Plain text
What you need to open it	A spreadsheet program such as Excel	Almost anything: Notepad, Excel, Google Sheets, Python `pandas`, AI assistants
Opened in Notepad	Unreadable symbols (starts with `PK`)	Readable: `Store_ID,Region_Name,Monthly_Sales_USD`

Trade-off: CSV keeps only values, with no formulas and no formatting.

4. Data Basics: Rows and Columns:
- Row (read across): one record. Here, all the facts about one store, such as row 1 = everything about STR-1001.
- Column (read down): one type of information for every store, such as Monthly_Sales_USD. In pandas, one column is a Series.
- Cell: one value where a row and a column meet, such as 185000.0.
- Size of a table: df.shape returns (12, 6) = 12 rows, 6 columns. Row labels (the index) start at 0, not 1.
5. Lab Practice Dataset:
- File Link: data/session_01_03_dataset.csv
- Audited 12-row commercial store dataset used for vector arithmetic and filtering:
  - Store_ID: Unique retail location code (e.g., STR-1001).
  - Region_Name: Geographic division (East, West, North, South).
  - Monthly_Sales_USD: Gross store revenue in U.S. Dollars.
  - Operating_Cost_USD: Monthly operational store overhead in U.S. Dollars.
  - Customer_Footfall_Count: Total monthly foot traffic count.
  - Customer_Satisfaction_Score: Customer review rating (1.0 to 5.0 scale).
6. Keywords (Core Programming Concepts):
- pandas and NumPy:
  - What: free Python libraries for tables (pandas) and fast number arrays (NumPy)
  - Why: they sort, filter, and calculate tables of any size
- CSV file and pd.read_csv():
  - What: a plain-text table where commas separate values; pd.read_csv() loads it
  - Why: most business systems export data as CSV
- DataFrame:
  - What: a whole table in Python, with row labels and column names
  - Why: it is the main object you clean, calculate, and filter
- Series:
  - What: one column of a DataFrame, such as df["Monthly_Sales_USD"]
  - Why: most calculations work column by column
- Vectorized Computation:
  - What: one formula applied to every row at once, with no loop
  - Why: fast, and nothing to drag down row by row
- .loc[] vs .iloc[]:
  - What: .loc[] selects by label or condition; .iloc[] selects by position number
  - Why: pull out exactly the rows a question needs
  - Example: df.loc[df["Region_Name"] == "West"] keeps the West rows; df.iloc[0:2] keeps rows 0 and 1 (0:2 starts at 0 and stops before 2)

7. Core Programming & Computational Foundations:

7.1 Columns, not cells: one formula for every row

The idea: In Excel, a formula lives in one cell. To calculate the margin of 12 stores, you type the formula once and drag it down, which creates 12 separate copies, one per row. In pandas, a calculation is written once for a whole column, and pandas applies it to every row at the same time. This is called vectorized computation: the formula works on the whole column (a “vector” of values) instead of cell by cell.

See it on real data: operating margin for the first 3 stores.

	Excel	pandas
What you write	`=(C2-D2)/C2*100` in `E2`, then drag down to `E13`	`df["Operating_Margin_Pct"] = (df["Monthly_Sales_USD"] - df["Operating_Cost_USD"]) / df["Monthly_Sales_USD"] * 100`
How many formulas exist	12, one per row	1, for the whole column
`STR-1000`	`30.99`	`30.985915`
`STR-1001`	`34.05`	`34.054054`
`STR-1002`	`27.55`	`27.551020`

Common mistake (in Excel): someone types a number over E5, for example to “fix” a value for a meeting. That row’s formula is gone, the cell still looks normal, and next month’s report is wrong without any warning sign that catches the eye. In pandas there is no per-row copy to overwrite: the next run recalculates every row from the data.

More examples:

Code	Result	What it tells you
`df["Monthly_Sales_USD"] - df["Operating_Cost_USD"]`	`44000.0` for `STR-1000`	operating profit for every store
`df["Monthly_Sales_USD"] / df["Customer_Footfall_Count"]`	`11.45` for `STR-1000`	sales per visitor: what an average visitor spends
`df["Operating_Cost_USD"] / df["Monthly_Sales_USD"] * 100`	`69.01` for `STR-1000`	cost as a share of sales, in percent
`df["Monthly_Sales_USD"] * 1.05`	`149100.0` for `STR-1000`	a “what if sales grew 5%?” scenario
`df["Monthly_Sales_USD"] / 1000`	`142.0` for `STR-1000`	sales in thousands, easier to read in a report

Remember: write the rule once for the column; pandas applies it to every row.

7.2 Inside a DataFrame: index, columns, and values

The idea: A DataFrame has three parts, like a labeled spreadsheet:
- the index: one label per row, shown at the left. By default the labels are 0, 1, 2, ..., counting from 0, not 1;
- the columns: one name per column, shown at the top, such as Store_ID;
- the values: the data itself, stored as a NumPy array (a fast block of numbers that pandas uses underneath).

See it on real data: the first 3 rows as pandas prints them.

   Store_ID Region_Name  Monthly_Sales_USD  ...
STR-1000        East           142000.0  ...
STR-1001        West           185000.0  ...
STR-1002       North            98000.0  ...

The 0, 1, 2 on the left are the index, not data. STR-1001 is in the row labeled 1, which is the second row.

One column vs a smaller table: df["Store_ID"] (single brackets) returns one column, a Series. df[["Store_ID", "Region_Name"]] (double brackets: a list inside the brackets) returns a smaller DataFrame with the listed columns.
Common mistake: counting rows from 1. In Python, the first row has position 0, so “the third row” is position 2.

More examples:

Code	Result	What it tells you
`type(df["Store_ID"])`	`Series`	single brackets give one column
`df[["Store_ID", "Region_Name"]].shape`	`(12, 2)`	double brackets give a smaller table: 12 rows, 2 columns
`df.columns.tolist()`	`['Store_ID', 'Region_Name', ...]`	the exact column names, to copy instead of typing
`df.index.tolist()`	`[0, 1, 2, ..., 11]`	the row labels start at `0`
`len(df)`	`12`	the number of rows
`df.loc[0, "Store_ID"]`	`'STR-1000'`	one value: row label `0`, column `Store_ID`

Remember: index on the left, column names on top, values in between; counting starts at 0.

7.3 Data types: every column has exactly one type

The idea: pandas gives each column one data type (dtype), and every value in that column follows it. The type decides what you can do with the column: you can add numbers, but not text.

`dtype`	Meaning	Columns in our data
`float64`	decimal numbers	`Monthly_Sales_USD`, `Operating_Cost_USD`, `Customer_Satisfaction_Score`
`int64`	whole numbers	`Customer_Footfall_Count`
`str`	text (older pandas versions show `object`)	`Store_ID`, `Region_Name`

Why it matters: if a number is typed with a comma, such as "1,200", pandas cannot read it as a number, so the whole column becomes text. Adding text joins the pieces instead of summing them: a column with "1,200" and "800" “sums” to "1,200800", not 2000. In Excel the same value would sit in its cell as text and be quietly skipped by SUM.
How to catch it: run df.dtypes right after loading. A sales column showing str instead of float64 means some values were stored as text.

More examples:

Code	Result	What it tells you
`df["Monthly_Sales_USD"].sum()`	`1829000.0`	a number column adds up as numbers
`df["Region_Name"].head(3).sum()`	`'EastWestNorth'`	a text column “adds up” by joining the words
`pd.Series(["1,200", "800"]).sum()`	`'1,200800'`	numbers stored as text are joined, not summed
`pd.Series(["1,200", "800"]).str.replace(",", "").astype(float).sum()`	`2000.0`	remove the comma, convert to `float`, and the sum is right
`df["Customer_Footfall_Count"].mean()`	`13075.0`	an `int64` column works for averages too

Remember: check the types first; a number column must show float64 or int64.

7.4 Check before you calculate: four inspection tools

The idea: A professional analyst never calculates on data they have not looked at. Four commands answer the first questions about any new file:

Command	Question it answers	Result for our file
`df.shape`	How big is the table?	`(12, 6)`: 12 rows, 6 columns
`df.head(3)`	What do the first rows look like?	`STR-1000`, `STR-1001`, `STR-1002`
`df.info()`	What are the types, and are values missing?	`12 non-null` in every column: no gaps
`df.describe()`	What are the typical, smallest, and largest values?	sales: mean `152416.67`, min `89000.0`, max `225000.0`

Common mistake: df.shape has no brackets, because it is a stored value (an attribute). df.head() and df.info() do have brackets, because they are actions (methods). Writing df.shape() raises TypeError: 'tuple' object is not callable.

More examples:

Code	Result	What it tells you
`df["Region_Name"].value_counts()`	`East 3`, `West 3`, `North 3`, `South 3`	how many stores each region has
`df.isnull().sum().sum()`	`0`	the number of empty cells in the whole table
`df["Customer_Satisfaction_Score"].min()`	`3.8`	the lowest rating
`df["Customer_Satisfaction_Score"].max()`	`4.9`	the highest rating
`df["Customer_Satisfaction_Score"].mean()`	`4.35` (rounded)	the average rating

Remember: shape, head, info, describe: look first, calculate second.

7.5 Picking rows: by label with .loc[], by position with .iloc[]

The idea: There are two ways to point at a row. .loc[] uses the label in the index, or a condition. .iloc[] uses the position, counting from 0. On a freshly loaded file the labels and positions are the same, so the two look alike. They differ as soon as the rows are reordered.

See it on real data: sort the stores by sales, from lowest to highest. The row order changes, but every row keeps its label.

Request	Meaning	Result
`sorted_df.loc[1, "Store_ID"]`	the row whose label is `1`	`STR-1001`, wherever it now sits
`sorted_df.iloc[0]["Store_ID"]`	whatever row is now first	`STR-1006`, the lowest sales

Filtering with .loc[]: .loc[] also accepts a condition, such as df.loc[df["Region_Name"] == "West"], and keeps only the rows where the condition is True. This is the form you will use most.
Common mistake: misspelling a column name. df["Monthly_Sale_USD"] (missing s) raises KeyError: 'Monthly_Sale_USD': pandas looked for a column with that exact name and found none. Copy column names from df.columns instead of typing them.

More examples:

Code	Result	What it tells you
`df.loc[df["Region_Name"] == "East", "Store_ID"]`	`STR-1000`, `STR-1004`, `STR-1008`	the East stores: a condition plus one column
`df.loc[df["Customer_Satisfaction_Score"] >= 4.5, "Store_ID"]`	`STR-1001`, `STR-1004`, `STR-1005`, `STR-1008`, `STR-1011`	the 5 best-rated stores
`df.iloc[0]["Store_ID"]`	`'STR-1000'`	the first row
`df.iloc[-1]["Store_ID"]`	`'STR-1011'`	`-1` means the last row
`df.iloc[0:3]["Store_ID"]`	`STR-1000`, `STR-1001`, `STR-1002`	rows at positions 0, 1, 2 (stops before 3)
`df.loc[2, "Region_Name"]`	`'North'`	row label `2`, column by name
`df.iloc[2, 1]`	`'North'`	the same cell by position: row 2, column 1

Remember: .loc[] = label or condition; .iloc[] = position from 0.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S13_demo_1.py), and check the dataset is in data/.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors.
9. Hands-On Coding Micro-Examples:
- Every example has three parts: 1. Input (load the data), 2. Process (calculate or select), and 3. Output (print the result).
- Name each file with your own name and the session code, for example Reacher_S13_demo_1.py (pattern: YourName_S13_demo_1.py).
- Micro-Example 1 (Load a CSV File and Inspect It): YourName_S13_demo_1.py
```
# 1. Input: load the tool and the data
import pandas as pd
df = pd.read_csv("data/session_01_03_dataset.csv")

# 2. Process: measure the table and take a preview
dimensions = df.shape
column_types = df.dtypes
preview = df.head(3)

# 3. Output: show the results
print("Dataset Dimensions (Rows, Columns):", dimensions)
print("Column Data Types:\n", column_types)
print(preview)
```
  - How it works, line by line:
    - import pandas as pd loads the pandas library and gives it the short name pd, so every pandas tool can be written as pd.something.
    - pd.read_csv(...) opens the CSV file, reads the first line as column names, and stores the table in a variable named df (short for DataFrame). The path data/... starts from the project folder.
    - df.shape, df.dtypes, and df.head(3) measure the table, list each column’s type, and copy the first 3 rows. Each result is stored in its own variable, so part 3 can print it.
    - print() shows a result in the Terminal. A script that calculates without printing shows nothing at all.
  - Read the output:
    Dataset Dimensions (Rows, Columns): (12, 6) Column Data Types: Store_ID str Region_Name str Monthly_Sales_USD float64 ... Store_ID Region_Name ... Customer_Footfall_Count Customer_Satisfaction_Score 0 STR-1000 East ... 12400 4.2
    - (12, 6) means 12 rows (stores) and 6 columns, as expected for this file.
    - The money columns are float64 (numbers), so you can calculate with them.
    - The ... in the preview is not missing data: pandas hides middle columns when the table is wider than the Terminal.
- Quick Check 1 (.tail()):
  - New tool: .tail(n) shows the last n rows. Why: new records are often added at the bottom.
  - Task: in part 2. Process, change preview = df.head(3) to preview = df.____(3) so it shows the last 3 rows.
  - Check: the preview shows index 9, 10, 11 (STR-1009, STR-1010, STR-1011).
- Micro-Example 2 (Calculate New Columns Without a Loop): YourName_S13_demo_2.py
```
# 1. Input: load the store data
import pandas as pd
df = pd.read_csv("data/session_01_03_dataset.csv")

# 2. Process: profit and margin for every store at once
df["Operating_Profit_USD"] = (
    df["Monthly_Sales_USD"] - df["Operating_Cost_USD"]
)
df["Operating_Margin_Pct"] = (
    df["Operating_Profit_USD"] / df["Monthly_Sales_USD"] * 100.0
)

# 3. Output: show the first three stores
print(df[["Store_ID", "Monthly_Sales_USD",
          "Operating_Margin_Pct"]].head(3))
```
  - How it works, line by line:
    - df["Monthly_Sales_USD"] - df["Operating_Cost_USD"] subtracts two columns. pandas lines them up row by row, so each store’s cost is subtracted from that same store’s sales (section 7.1).
    - df["Operating_Profit_USD"] = ... stores the result as a new column. If the name does not exist yet, pandas adds it at the right end of the table.
    - The second calculation uses the new profit column straight away: profit / sales x 100.
    - The round brackets around each calculation only let the formula continue on the next line; they do not change the result.
    - df[["Store_ID", "Monthly_Sales_USD", "Operating_Margin_Pct"]] (double brackets) keeps just those 3 columns, and .head(3) keeps the first 3 rows.
  - Read the output:
    Store_ID Monthly_Sales_USD Operating_Margin_Pct 0 STR-1000 142000.0 30.985915 1 STR-1001 185000.0 34.054054 2 STR-1002 98000.0 27.551020
    - STR-1000: (142000 - 98000) / 142000 x 100 = 30.99%. Of every $100 of sales, about $31 is left as operating profit.
    - STR-1002 sells the least and also keeps the smallest share as profit (27.55%).
    - The long decimals are correct but hard to read; Quick Check 2 rounds them.
- Quick Check 2 (.round()):
  - New tool: .round(n) rounds every number in a column to n decimal places. Why: long decimals such as 30.985915 are hard to read in a report.
  - Task: add one line at the end of part 2. Process: df["Operating_Margin_Pct"] = df["Operating_Margin_Pct"].____(1)
  - Check: the margins print as 31.0, 34.1, and 27.6.
- Micro-Example 3 (Filter Rows and Fill Missing Values): YourName_S13_demo_3.py
```
# 1. Input: load the store data
import pandas as pd
df = pd.read_csv("data/session_01_03_dataset.csv")

# 2. Process: keep West stores with sales of $150,000+
west_high_sales = df.loc[
    (df["Region_Name"] == "West")
    & (df["Monthly_Sales_USD"] >= 150000.0)
]
#    then fill missing satisfaction scores with the median
median_score = df["Customer_Satisfaction_Score"].median()
df["Customer_Satisfaction_Score"] = (
    df["Customer_Satisfaction_Score"].fillna(median_score)
)

# 3. Output: show the filtered stores and the median
print(west_high_sales[["Store_ID", "Monthly_Sales_USD"]])
print("Median Satisfaction Score Imputed:", median_score)
```
  - How it works, line by line:
    - (df["Region_Name"] == "West") checks every row and gives True for West stores and False for the others. Note the double ==: it compares, while a single = stores a value.
    - (df["Monthly_Sales_USD"] >= 150000.0) gives True for stores selling $150,000 or more.
    - & combines the two answers row by row: a row is True only if both are True. Each condition sits in its own brackets so that Python compares first and combines second.
    - df.loc[...] keeps the True rows and stores them in west_high_sales; the original df is not changed.
    - .median() finds the middle value of the satisfaction scores: half the stores score higher, half lower. .fillna(median_score) puts that value into any empty cell (NaN, “Not a Number”).
  - Read the output:
    Store_ID Monthly_Sales_USD 1 STR-1001 185000.0 5 STR-1005 175000.0 9 STR-1009 160000.0 Median Satisfaction Score Imputed: 4.35
    - All 3 West stores sell $150,000 or more, so all 3 pass. With a higher limit, such as $170,000, only 2 would. The index labels 1, 5, 9 show where each store sat in the original table: filtering keeps the labels.
    - The median of the 12 scores is 4.35. This dataset has no empty scores, so .fillna() changes nothing today, but the same line protects the script when next month’s file has gaps.
- Quick Check 3 (len()):
  - New tool: len() counts items; for a DataFrame, it returns the number of rows. Why: it answers “how many?” questions in one step. Example: len(df) returns 12.
  - Task: add one line to part 3. Output: print("Stores found:", ___(west_high_sales))
  - Check: the Terminal shows Stores found: 3.
10. Official Python Documentation & References:
11. Exercise 1: Guided Practice - Retail Performance Wrangling:
- Step 1: Open Workspace: In Google Antigravity IDE, create a new script named YourName_S13_wrangling.py, replacing YourName with your own name (for example, Reacher_S13_wrangling.py). Verify that data/session_01_03_dataset.csv is present in Panel [1].
- Step 2: Load, Calculate, and Summarize: Type the following Python statements:
```
# 1. Input: load the store data
import pandas as pd
df = pd.read_csv("data/session_01_03_dataset.csv")

# 2. Process: add profit and margin columns, then summarize
df["Operating_Profit_USD"] = (
    df["Monthly_Sales_USD"] - df["Operating_Cost_USD"]
)
df["Operating_Margin_Pct"] = (
    df["Operating_Profit_USD"] / df["Monthly_Sales_USD"] * 100.0
)
summary = df.describe()

# 3. Output: show summary statistics for sales and margin
print("Descriptive Financial Statistics:\n",
      summary[["Monthly_Sales_USD", "Operating_Margin_Pct"]])
```
  - .describe() returns summary statistics: count, mean, standard deviation, min, max, and quartiles.
- Step 3: Compare Excel Mechanics with Pandas:
  - In Excel, you type =(C2-D2)/C2*100 in cell E2 and drag it down, so every row gets its own copy of the formula. If someone types a number over E5, that row is wrong, and the mistake is easy to miss.
  - In Pandas, one line of df["Operating_Margin_Pct"] code covers every row, so there is no copy to overwrite.
- Step 4: Execute in Terminal: Run the script via Panel [4] (Terminal) using python YourName_S13_wrangling.py. Confirm that the summary statistics are printed.
12. Exercise 2: Applied Practice - Find the Top Stores:
- Coding Challenge: In YourName_S13_practice.py, find the top stores: Operating_Margin_Pct >= 30.0 and Customer_Satisfaction_Score >= 4.5.
- Task 1 (Targeted Filtering): Use .loc[] to select the qualified stores and output only Store_ID, Region_Name, and Operating_Margin_Pct.
- Task 2 (Deliberate KeyError & AI Diagnosis): Deliberately misspell a column name (e.g., df["Monthly_Sale_USD"] instead of "Monthly_Sales_USD"). Read the KeyError (the column name does not exist), capture the traceback with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and review the AI’s explanation.
- Task 3 (Clean CSV Export): Correct the column name, then sort with .sort_values(); add ascending=False for the highest margin first. Save the table with .to_csv() to data/top_performing_stores.csv.
- Success Check: the result has 5 stores, led by STR-1011 (South, 36.89%), and the script ends with return code 0.
13. Exercise 3: Advanced Practice - Store Scorecard:
- Goal: Build a one-page scorecard that sorts all 12 stores into three tiers, so the regional manager can see at a glance which stores to reward and which to watch.
- What you need:
  - File: data/session_01_03_dataset.csv (12 rows, described in 5. Lab Practice Dataset).
  - Script name: YourName_S13_challenge.py.
  - Tools you already know: pd.read_csv(), column math, .round(), &, .loc[], .sort_values(), len(), and .to_csv().
- Requirements (follow them in order):
  1. Add Operating_Profit_USD = Monthly_Sales_USD - Operating_Cost_USD.
  2. Add Operating_Margin_Pct = Operating_Profit_USD / Monthly_Sales_USD x 100, rounded to 1 decimal place.
  3. Add Sales_per_Visitor_USD = Monthly_Sales_USD / Customer_Footfall_Count, rounded to 2 decimal places. It shows how much an average visitor spends.
  4. Add a Tier column using these rules, in this order:
    - Every store starts as "Standard".
    - "Gold": Operating_Margin_Pct >= 33.0 and Customer_Satisfaction_Score >= 4.5.
    - "Watch": Operating_Margin_Pct < 30.0.
  5. Print how many stores are in each tier, one line per tier, for example Gold stores: 4.
  6. Print all 12 stores sorted by Operating_Margin_Pct, highest first, showing only Store_ID, Region_Name, Operating_Margin_Pct, Sales_per_Visitor_USD, Customer_Satisfaction_Score, and Tier.
  7. Save the same sorted table to data/store_scorecard.csv without row labels.
  8. Answer in a comment at the end of your script: which tier holds the store with the highest Sales_per_Visitor_USD, and what does that tell the manager?
- New tool for this exercise: set a value for selected rows only:
  - df["Tier"] = "Standard" writes one value into every row of a new column.
  - df.loc[mask, "Tier"] = "Gold" changes only the rows where mask is True. mask is a condition like the one in Micro-Example 3, with each part wrapped in ( ) and joined with &.
- Hints:
  - To count a tier, filter first, then use len(): len(df.loc[df["Tier"] == "Gold"]).
  - Pass a list of column names to df[[ ... ]] to keep only the columns you need.
- Self-Check (expected results):
  - Gold stores: 4 (STR-1011, STR-1004, STR-1008, STR-1001), Watch stores: 5, Standard stores: 3.
  - The first row of the sorted table is STR-1011 (South, 36.9), and the last row is STR-1006 (North, 23.6).
  - The highest Sales_per_Visitor_USD is STR-1002 (11.95), a Watch store with a margin of 27.6. High spending per visitor does not guarantee a high profit margin.

Session 1.4: Tabular Data Wrangling Part 2: Advanced Filtering, Multi-Segment GroupBy, and Financial Auditing

Session 1.4 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 4 (115/10/04 - 115/10/10) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- Each division gets a spending budget every quarter, and finance compares it with what was actually spent
- Our case: 4 divisions x 4 quarters = 16 rows in one CSV file
- The finance director asks: who overspent, by how much, and who is responsible? Today, Python answers this for the whole year at once
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “Which quarters went over budget?”: filter rows with a Boolean mask (a True/False value for every row) and &
2. “How much did each division overspend this year?”: summarize groups with groupby() (split rows into groups, like an Excel pivot table, which is a summary table with one row per group) and .agg()
3. “Who is responsible?”: join tables with pd.merge() (join two tables on a shared column, like Excel’s VLOOKUP formula, which looks up a value in another table) and fix a KeyError

3. Business Context: Budget vs Actual:

Four terms, one example (Consumer division, Q1):

Term	Meaning	Example
Budget	planned spending	`780,000`
Actual	real spending	`810,000`
Variance	Actual - Budget	`+30,000`
Variance %	Variance / Budget x 100	`+3.85%`

Positive variance = overspent; negative variance = under budget.
Cost center: the accounting code a division’s spending is charged to, such as CC-202 for Consumer. Each cost center has a responsible VP (Vice President).
Company rule in our data: a quarter is flagged (Audit_Status = Variance_Flag) when the variance is more than 2%.
Consumer division by quarter: Q1 +3.85% (flagged), Q2 +1.92% (over, within the rule), Q3 +5.77% (flagged), Q4 -0.64% (under budget).

4. Data Basics: From Excel Tools You Know to pandas:
- Three Excel tools, in one line each:
  - Filter: hides the rows that do not match a rule, such as showing only the Consumer division.
  - Pivot Table: summarizes a large table by group. For example, put Division_Name in Rows and Actual_Expense_USD in Values to get total spending per division: 4 rows instead of 16.
  - VLOOKUP: a formula that looks up a value, such as a cost center code, in another table and returns a value from the same row, such as the VP’s name. You tell it which column to return by number (for example, the 2nd column).
- The same three jobs in pandas:
  - Filter -> Boolean mask df[mask]: the rule is written in code, so it repeats exactly next quarter.
  - Pivot Table -> df.groupby().agg(): one line; run it again when new data arrives.
  - VLOOKUP -> pd.merge(): VLOOKUP returns a column by number; pd.merge() matches rows by column name, such as Cost_Center_Code.
5. Lab Practice Dataset:
- File Link: data/session_01_04_dataset.csv
- Audited 16-row quarterly budget dataset (4 divisions x 4 quarters; each row is one division’s spending in one quarter):
  - Transaction_ID: Unique record identifier (e.g., TXN-5001).
  - Division_Name: Operating business unit (Cloud, Consumer, Enterprise, Logistics).
  - Cost_Center_Code: Accounting cost center (CC-101, CC-202, CC-303, CC-404).
  - Fiscal_Quarter: Financial reporting quarter (Q1, Q2, Q3, Q4).
  - Budget_Expense_USD: Authorized spending budget in U.S. Dollars.
  - Actual_Expense_USD: Realized expenditure in U.S. Dollars.
  - Audit_Status: Internal audit classification (Verified, or Variance_Flag when spending is more than 2% over budget).
6. Keywords (Core Programming Concepts):
- Boolean Mask:
  - What: one True or False per row
  - Why: picks exactly the rows that meet a rule
- & | ~:
  - What: “and”, “or”, “not” for masks
  - Why: Python’s and / or fail on whole columns
- Split-Apply-Combine:
  - What: split rows into groups, calculate each group, combine the results
  - Why: the idea behind every pivot table
- groupby() + .agg():
  - What: groupby() splits; .agg() sums, counts, or averages each group
  - Why: yearly totals per division in one step
- pd.merge():
  - What: joins two tables row by row
  - Why: adds facts from another table, such as the VP’s name, with no copy-paste
- Key Column:
  - What: a column both tables share, such as Cost_Center_Code
  - Why: tells pandas which rows belong together

7. Core Programming & Computational Foundations:

7.1 Combining rules: &, not and

The idea: A rule on a column, such as df["Division_Name"] == "Consumer", does not give one answer. It gives one True or False for every row: a whole column of answers, called a Boolean mask. To combine two rules, Python must compare the two columns row by row: row 1 with row 1, row 2 with row 2, and so on. pandas uses & (“and”), | (“or”), and ~ (“not”) for this row-by-row work.

See it on real data: the first 4 rows of our dataset.

Row	Division	`is_consumer`	`is_over`	Both true (`&`)	Either true (or)
`TXN-5000`	Cloud	`False`	`False`	`False`	`False`
`TXN-5001`	Consumer	`True`	`True`	`True`	`True`
`TXN-5002`	Enterprise	`False`	`False`	`False`	`False`
`TXN-5003`	Logistics	`False`	`True`	`False`	`True`

Here is_consumer = df["Division_Name"] == "Consumer" and is_over = df["Actual_Expense_USD"] > df["Budget_Expense_USD"]. With is_consumer & is_over, a row is kept only when both columns say True (TXN-5001). With is_consumer | is_over, one True is enough (TXN-5001 and TXN-5003).

Common mistake 1, using and: Python’s own and can only judge one True or False at a time. Given a whole column, it does not know whether you mean “all rows”, “any row”, or “row by row”, so it stops with ValueError: The truth value of a Series is ambiguous. “Ambiguous” means “it could mean several things”.
Common mistake 2, missing brackets: Python works out & before == and >. So in df["Division_Name"] == "Consumer" & df["Actual_Expense_USD"] > df["Budget_Expense_USD"], it first tries "Consumer" & df["Actual_Expense_USD"], which makes no sense, and raises a TypeError. Brackets force the right order: (df["Division_Name"] == "Consumer") & (df["Actual_Expense_USD"] > df["Budget_Expense_USD"]).
In these examples, is_over = df["Actual_Expense_USD"] > df["Budget_Expense_USD"].

More examples:

Code	Result	What it tells you
`(df["Division_Name"] == "Logistics") & is_over`	`Q1`, `Q3`	the Logistics quarters that overspent
`(df["Fiscal_Quarter"] == "Q4") & is_over`	`Enterprise`	only one division overspent in Q4
`(df["Audit_Status"] == "Variance_Flag") & (df["Fiscal_Quarter"] == "Q3")`	`TXN-5008`, `TXN-5009`, `TXN-5011`	the flagged quarters in Q3
`(~is_over).sum()`	`8`	`~` flips the rule: 8 quarters stayed within budget
`df["Division_Name"].isin(["Cloud", "Enterprise"]).sum()`	`8`	`.isin()` is a short way to write “Cloud or Enterprise”

Remember: one rule gives one True / False per row; combine rules with &, |, or ~, and put each rule in its own brackets.

7.2 Split-apply-combine: how groupby() builds a summary

The idea: Most management questions are about groups: “How much did each division spend?” Answering them always takes the same three steps, whether in an Excel pivot table or in pandas:
1. Split the rows into groups, one per division.
2. Apply a calculation inside each group, such as a sum.
3. Combine the answers into a new, smaller table with one row per group.

See it on real data: the Consumer division’s 4 quarterly rows go into one group, and "sum" adds them up:

Step	Consumer division
Split	Q1 `810000.0`, Q2 `795000.0`, Q3 `825000.0`, Q4 `775000.0`
Apply (`"sum"`)	`810000.0 + 795000.0 + 825000.0 + 775000.0 = 3205000.0`
Combine	one row: `Consumer` `3205000.0`

The same steps run for Cloud, Enterprise, and Logistics at the same time, so the 16 rows become 4: Cloud 1785000.0, Consumer 3205000.0, Enterprise 1280000.0, Logistics 2468000.0.

What you choose: the column to split by (groupby("Division_Name")), the column to calculate on, and the calculation: "sum" for totals, "mean" for averages, "count" for how many rows, "max" for the largest value.

More examples:

Code	Result	What it tells you
`df.groupby("Fiscal_Quarter")["Actual_Expense_USD"].sum()`	Q1 `2185000.0`, Q2 `2171000.0`, Q3 `2235000.0`, Q4 `2147000.0`	total spending per quarter: Q3 was the most expensive
`df.groupby("Division_Name")["Transaction_ID"].count()`	`4` for each division	a quick check that no quarter is missing
`df.groupby("Cost_Center_Code")["Actual_Expense_USD"].max()`	`CC-202` `825000.0` (largest)	the most expensive quarter of each cost center
`df.groupby("Audit_Status")["Transaction_ID"].count()`	`Variance_Flag` `6`, `Verified` `10`	how many quarters were flagged

Remember: groupby() does the split, .agg() does the apply, and pandas does the combine for you.

7.3 Joining tables: match by name, not by position

The idea: Information often sits in two tables: spending in one, the responsible VP in another. Joining means bringing the matching row from the second table next to each row of the first, using a column that both share: the key column, here Cost_Center_Code.
Why Excel’s VLOOKUP is fragile: VLOOKUP is told to return “the 2nd column” of the lookup table. If someone later inserts a new column, such as Region, the 2nd column is now Region, and every formula quietly returns a region instead of a VP name. No error appears, so the mistake is easy to miss.
Why pd.merge() is safer: pd.merge(spending, vp_info, on="Cost_Center_Code") matches rows by the name of the key column and brings over every other column by name. Moving or inserting columns changes nothing.
In these examples, ledger = pd.merge(df, vp_info, on="Cost_Center_Code", how="left"), as in Micro-Example 3, and vp_renamed is vp_info with its key column renamed to CostCenter.

More examples:

Code	Result	What it tells you
`ledger.shape`	`(16, 8)`	still 16 rows; one new column, `VP_Lead`, added (7 + 1)
`ledger.loc[ledger["Division_Name"] == "Logistics", "VP_Lead"]`	`David Kim` in all 4 rows	one VP row is matched to every quarter of that cost center
`pd.merge(df, vp_renamed, on="Cost_Center_Code")`	`KeyError: 'Cost_Center_Code'`	`vp_renamed` calls the key `CostCenter`: the names must match exactly

Remember: name the key column with on=; the key must have exactly the same name and the same values in both tables.

7.4 What happens to rows with no match: how="left" vs how="inner"

The idea: Sometimes a row in the main table has no partner in the other table, for example a new cost center that has not been given a VP yet. The how= setting decides what happens to that row.

See it on an example: suppose our spending table also had a new cost center, CC-505, which is not in vp_info:

`Cost_Center_Code`	`Variance_USD`	`VP_Lead` with `how="left"`	with `how="inner"`
`CC-202`	`85000.0`	Marcus Vance	Marcus Vance
`CC-404`	`28000.0`	David Kim	David Kim
`CC-505`	`5000.0`	`NaN` (empty)	row removed

NaN stands for “Not a Number”, pandas’ mark for a missing value.

Why it matters: with how="inner", the CC-505 overspend silently disappears from the report, and nobody is asked to explain it. With how="left", the row stays and the empty VP_Lead tells you that someone must be assigned.
In these examples, vp_3_rows is vp_info without the CC-404 row (as if Logistics had no VP yet), and merged is the result of the left merge.

More examples:

Code	Result	What it tells you
`pd.merge(df, vp_3_rows, on="Cost_Center_Code", how="left")`	`16` rows, `4` with `NaN` in `VP_Lead`	`vp_3_rows` lacks `CC-404`: its 4 quarters stay, with no VP
`pd.merge(df, vp_3_rows, on="Cost_Center_Code", how="inner")`	`12` rows	the 4 Logistics quarters silently disappear
`merged["VP_Lead"].isnull().sum()`	`4`	count the gaps after a left merge

Remember: use how="left" when every row of your main table must stay in the report, then look for NaN to find the gaps.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S14_demo_1.py), and check the dataset is in data/.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors.
9. Hands-On Coding Micro-Examples:
- Every example has three parts: 1. Input, 2. Process, and 3. Output. Name each file with your own name and the session code, for example Reacher_S14_demo_1.py (pattern: YourName_S14_demo_1.py).
- Micro-Example 1 (Find Overspending with a Boolean Mask): YourName_S14_demo_1.py
```
# 1. Input: load the quarterly budget data
import pandas as pd
df = pd.read_csv("data/session_01_04_dataset.csv")

# 2. Process: Consumer quarters that spent more than budget
mask = (
    (df["Division_Name"] == "Consumer")
    & (df["Actual_Expense_USD"] > df["Budget_Expense_USD"])
)
overruns = df[mask]

# 3. Output: count and list them
print("Consumer overrun quarters:", len(overruns))
print(overruns[["Fiscal_Quarter", "Budget_Expense_USD",
                "Actual_Expense_USD"]])
```
  - How it works, line by line:
    - (df["Division_Name"] == "Consumer") gives one True / False per row: True for the 4 Consumer rows.
    - (df["Actual_Expense_USD"] > df["Budget_Expense_USD"]) compares the two money columns row by row: True where a quarter spent more than planned.
    - & keeps True only where both are True (section 7.1). The result, mask, is a column of 16 True / False values. Naming it makes the rule easy to read and reuse.
    - df[mask] keeps the rows where mask is True and stores them in overruns.
    - len(overruns) counts those rows.
  - Read the output:
    Consumer overrun quarters: 3 Fiscal_Quarter Budget_Expense_USD Actual_Expense_USD 1 Q1 780000.0 810000.0 5 Q2 780000.0 795000.0 9 Q3 780000.0 825000.0
    - The Consumer division overspent in 3 of its 4 quarters; only Q4 (775000.0) stayed within the 780000.0 budget.
    - Q3 has the largest overrun: 825000 - 780000 = 45000, or 5.77% over budget.
    - The labels 1, 5, 9 are the original row positions in the 16-row file.
- Quick Check 1 (~):
  - New tool: ~ (not) turns every True into False, and every False into True. Why: find the opposite group, such as quarters that stayed within budget.
  - Task: in part 2. Process, change & (df["Actual_Expense_USD"] > df["Budget_Expense_USD"]) to & _(df["Actual_Expense_USD"] > df["Budget_Expense_USD"]).
  - Check: the script prints Consumer overrun quarters: 1, and the table shows Q4 (budget 780000.0, actual 775000.0). The label still says “overrun”, since only the rule changed.
- Micro-Example 2 (Total Each Division with groupby()): YourName_S14_demo_2.py
```
# 1. Input: load the quarterly budget data
import pandas as pd
df = pd.read_csv("data/session_01_04_dataset.csv")

# 2. Process: yearly totals for each division
div_summary = df.groupby("Division_Name").agg(
    Total_Budget_USD=("Budget_Expense_USD", "sum"),
    Total_Actual_USD=("Actual_Expense_USD", "sum"),
    Quarters=("Transaction_ID", "count"),
)
div_summary["Variance_USD"] = (
    div_summary["Total_Actual_USD"]
    - div_summary["Total_Budget_USD"]
)

# 3. Output: one row per division
print(div_summary[["Total_Actual_USD", "Variance_USD"]])
```
  - How it works, line by line:
    - df.groupby("Division_Name") splits the 16 rows into 4 groups, one per division (section 7.2).
    - Inside .agg(...), each line has the form New_Name=("Source_Column", "calculation"). For example, Total_Budget_USD=("Budget_Expense_USD", "sum") adds up the budget of each division and calls the result Total_Budget_USD.
    - Quarters=("Transaction_ID", "count") counts the rows in each group; it is a quick check that every division has 4 quarters.
    - The next line subtracts two of the new columns to get each division’s yearly variance.
  - Read the output:
    Total_Actual_USD Variance_USD Division_Name Cloud 1785000.0 -15000.0 Consumer 3205000.0 85000.0 Enterprise 1280000.0 0.0 Logistics 2468000.0 28000.0
    - The division names are now the row labels: groupby() turns the group names into the index.
    - Consumer overspent by 85000.0 for the year and Logistics by 28000.0; Enterprise spent exactly its budget; Cloud stayed 15000.0 under.
    - Consumer had one quarter under budget, yet it is still the largest yearly overspender: quarterly flags and yearly totals answer different questions.
- Quick Check 2 (.sort_values()):
  - New tool: .sort_values() reorders rows by one column; ascending=False puts the largest first. Why: the director reads the biggest overspender first.
  - Task: add one line at the end of part 2. Process: div_summary = div_summary.___________("Variance_USD", ascending=False)
  - Check: the first row is Consumer (85000.0), then Logistics (28000.0).
- Micro-Example 3 (Add VP Names with pd.merge()): YourName_S14_demo_3.py (VP names are fictional)
```
# 1. Input: budget data and a VP list (fictional names)
import pandas as pd
df = pd.read_csv("data/session_01_04_dataset.csv")
vp_info = pd.DataFrame({
    "Cost_Center_Code": ["CC-101", "CC-202", "CC-303", "CC-404"],
    "VP_Lead": ["Sarah Chen", "Marcus Vance",
                "Elena Rostova", "David Kim"],
})

# 2. Process: add VP names, then flag spending over budget
ledger = pd.merge(df, vp_info, on="Cost_Center_Code", how="left")
ledger["Over_Budget"] = (
    ledger["Actual_Expense_USD"] > ledger["Budget_Expense_USD"]
)

# 3. Output: show the first three rows
print(ledger[["Transaction_ID", "VP_Lead", "Over_Budget"]].head(3))
```
  - How it works, line by line:
    - pd.DataFrame({...}) builds a small table directly in code. Inside the curly brackets, each column name is followed by its list of values, in row order. The two lists must have the same length (here 4).
    - pd.merge(df, vp_info, on="Cost_Center_Code", how="left") takes each of the 16 rows of df, looks up its cost center in vp_info, and adds the matching VP_Lead (section 7.3). how="left" keeps every row of df, even one with no match (section 7.4).
    - ledger["Actual_Expense_USD"] > ledger["Budget_Expense_USD"] compares row by row and stores True / False in a new Over_Budget column.
  - Read the output:
    Transaction_ID VP_Lead Over_Budget 0 TXN-5000 Sarah Chen False 1 TXN-5001 Marcus Vance True 2 TXN-5002 Elena Rostova False
    - Every row now carries the name of the person responsible for that spending; no VP name was copied by hand.
    - TXN-5001 (Consumer, Q1) is over budget, and Marcus Vance is accountable for it.
    - The table still has 16 rows; .head(3) only shows the first 3.
- Quick Check 3 (.value_counts()):
  - New tool: .value_counts() counts how often each value appears in a column. Why: answers “how many quarters were over budget?” in one line.
  - Task: add one line to part 3. Output: print(ledger["Over_Budget"].____________())
  - Check: the Terminal shows False 8 and True 8: half of the 16 quarters went over budget.
10. Official Python Documentation & References:

11. Exercise 1: Guided Practice - Quarterly Division Audit:

Step 1: Open Script: In Google Antigravity IDE, create a new script named YourName_S14_audit.py, replacing YourName with your own name (for example, Reacher_S14_audit.py).

Step 2: Group by Division and Quarter: Type the following code:

# 1. Input: load the quarterly budget data
import pandas as pd
df = pd.read_csv("data/session_01_04_dataset.csv")

# 2. Process: variance per row, then group by two columns
df["Variance_USD"] = df["Actual_Expense_USD"] - df["Budget_Expense_USD"]
df["Variance_Pct"] = df["Variance_USD"] / df["Budget_Expense_USD"] * 100.0
quarterly_audit = df.groupby(["Division_Name", "Fiscal_Quarter"]).agg(
    Total_Actual=("Actual_Expense_USD", "sum"),
    Total_Variance=("Variance_USD", "sum"),
    Mean_Variance_Pct=("Variance_Pct", "mean"),
)

# 3. Output: one row per division and quarter
print("Quarterly Division Audit Summary:\n", quarterly_audit)

Grouping by a list of two columns gives one row per division and quarter: 16 rows, labeled by both values.

Step 3: Verify Output Integrity: Confirm that quarters with overruns show positive Total_Variance values. 8 of the 16 rows should be positive.
Step 4: Execute in Terminal: Run the script via Panel [4] (Terminal) using python YourName_S14_audit.py and confirm return code 0.

12. Exercise 2: Applied Practice - Annual Cost Center Overruns:
- Coding Challenge: In YourName_S14_practice.py, find the cost centers that overspent for the whole year.
- Task 1 (Annual Cost Center Aggregation): Group the data by Cost_Center_Code, sum budget and actual spending, compute the yearly variance %, and keep cost centers more than 1.0% over budget.
- Task 2 (Deliberate KeyError & AI Diagnosis): Merge the result with vp_info (from Micro-Example 3), but type on="Cost_Center" on purpose. Read the KeyError (the column name does not exist), capture the traceback with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and review the AI’s explanation of join keys.
- Task 3 (Audit Brief Output): Fix the key to on="Cost_Center_Code", then print the cost center, the VP, and the overrun in USD.
- Success Check: 2 cost centers: CC-202 (Marcus Vance, 85000.0, 2.72%) and CC-404 (David Kim, 28000.0, 1.15%); the script ends with return code 0.
13. Exercise 3: Advanced Practice - VP Exception Report:
- Goal: Build an exception report that tells the finance director which VPs must explain their spending. A VP is escalated when the cost center has repeated flagged quarters and overspent for the whole year.
- What you need:
  - File: data/session_01_04_dataset.csv (16 rows, described in 5. Lab Practice Dataset).
  - The vp_info table from Micro-Example 3 (VP names are fictional).
  - Script name: YourName_S14_challenge.py.
  - Tools you already know: Boolean masks, groupby() + .agg(), pd.merge(), .sort_values(), and .to_csv().
- Requirements (follow them in order):
  1. Add Variance_USD = Actual_Expense_USD - Budget_Expense_USD and Variance_Pct = Variance_USD / Budget_Expense_USD x 100, rounded to 2 decimal places.
  2. Add My_Flag = True when Variance_Pct > 2.0. Check your rule: count the rows where My_Flag disagrees with Audit_Status == "Variance_Flag". The count must be 0.
  3. Flagged summary: keep only the flagged rows, then group by Cost_Center_Code and compute Flagged_Quarters (count of rows), Flagged_Overrun_USD (sum of Variance_USD), and Worst_Variance_Pct (largest Variance_Pct).
  4. Yearly summary: using all 16 rows, group by Cost_Center_Code, sum budget and actual spending, and compute Year_Variance_Pct = (actual - budget) / budget x 100, rounded to 2 decimal places.
  5. Start from vp_info and merge in the yearly summary, then the flagged summary, both with on="Cost_Center_Code" and how="left". If a cost center had no flagged quarter, its empty values must become 0.
  6. Add Escalate = True when Flagged_Quarters >= 2 and Year_Variance_Pct > 1.0.
  7. Sort by Flagged_Overrun_USD, then by Worst_Variance_Pct, both largest first. Print the table and save it to data/vp_exception_report.csv without row labels.
- New tools for this exercise:
  - .reset_index(): after groupby(), the group names (here, the cost center codes) become the row labels. .reset_index() turns them back into a normal column, so you can merge on Cost_Center_Code.
  - .fillna(0): replaces empty values (NaN) with 0, for example report["Flagged_Quarters"] = report["Flagged_Quarters"].fillna(0).
  - Sorting by two columns: .sort_values(["Flagged_Overrun_USD", "Worst_Variance_Pct"], ascending=False). The second column only decides ties in the first.
  - "max" inside .agg() returns the largest value in each group, just as "sum" returns the total.
- Hints:
  - Build the two summaries as two separate tables first, print each one, and only then merge.
  - (df["My_Flag"] != (df["Audit_Status"] == "Variance_Flag")).sum() counts disagreements, because each True counts as 1.
- Self-Check (expected results):
  - Rule check: 0 disagreements; 6 of the 16 quarters are flagged.
  - Table order: CC-202 Marcus Vance (2 flagged, 75000.0, worst 5.77), CC-404 David Kim (2, 45000.0, 4.92), CC-303 Elena Rostova (1, 10000.0, 3.12), CC-101 Sarah Chen (1, 10000.0, 2.22).
  - Escalate is True only for Marcus Vance (Year_Variance_Pct 2.72) and David Kim (1.15). Elena Rostova and Sarah Chen each had one flagged quarter but finished the year at or under budget.

Module 2: Web Data Extraction, Visual Dashboards, and Midterm Milestone

Module 2 shifts focus from internal corporate records to external market intelligence and visual storytelling. Students learn to extract public competitor price data using web scrapers, construct publication-grade business charts, and present audited exploratory findings during the midterm milestone.

Session 2.1: Web Data Extraction: Scraping Financial and Competitor Intelligence with Beautiful Soup

Session 2.1 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 5 (115/10/11 - 115/10/17) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- Our pricing team watches 3 competitor websites, where prices sit on web pages, not in CSV files
- Copying hundreds of prices by hand is slow and error-prone
- The pricing manager asks: what does each competitor charge, and who discounts most? Today, Python reads a web page and turns it into a table
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “How does a web page reach my computer?”: explain an HTTP (HyperText Transfer Protocol) GET request and read HTML (HyperText Markup Language: the text with tags that describes a web page)
2. “What is BetaMart charging?”: extract prices with Beautiful Soup (a Python library that reads HTML) and CSS (Cascading Style Sheets) selectors (an address for tags, such as .price)
3. “What if a product has no stock tag?”: handle a missing tag (the search returns None, meaning nothing found) and fix an AttributeError
3. Business Context: Why Scrape?
- By hand: open each competitor page, copy each price into Excel, and do it all again tomorrow. 3 sites x 100 products = 300 copy-pastes a day, with typos.
- With a Python script: read every product on the page in seconds, get the same result every run, and run it again every morning. The output is a clean table, ready for pandas.
- Rules of the road (scrape responsibly):
  - Public pages only, never behind a login
  - Check the site’s robots.txt and terms of use. robots.txt is a short public file on a website (for example, betamart.example/robots.txt) that lists which pages automated scripts may and may not read.
  - Pause between requests, for example time.sleep(2) (wait 2 seconds), so the site is not overloaded
  - Name your script in the request. The User-Agent is a short label sent with every request that tells the site who is asking.
4. Data Basics: From Web Request to HTML:
- 1. Your browser or Python script sends an HTTP GET request: “send me this page”.
- 1. The web server answers with the page as HTML text.
- The browser draws that text as a page for people; Python reads the text itself. In the example below, data-sku stores the product’s SKU (Stock Keeping Unit), the store’s product code:
```
<div class="product" data-sku="SKU-801">
  <h3 class="name">Office Desk Chair</h3>
  <span class="price">$49.50</span>
  <span class="stock">120 units</span>
</div>
```
5. Lab Practice Dataset:
- Saved web page: data/session_02_01_catalog.html, a fictional BetaMart catalog page with 4 product cards (SKU-807 has no stock tag).
- File Link: data/session_02_01_dataset.csv
- Audited 12-row competitor pricing dataset compiled from automated extraction (competitor names are fictional):
  - Product_SKU: Stock Keeping Unit, the product code (e.g., SKU-801).
  - Competitor_Name: Competitor (AlphaRetail, BetaMart, GammaGlobal).
  - Category_Name: Merchandise category (Electronics, Office Supplies, Hardware, Appliances).
  - List_Price_USD: Extracted competitor catalog price in U.S. Dollars.
  - Promotion_Discount_Pct: Advertised promotional markdown percentage.
  - Stock_Availability_Units: Available warehouse stock units.
6. Keywords (Core Programming Concepts):
- HTTP GET:
  - What: a request, using the HyperText Transfer Protocol (HTTP), that asks a web server for a page
  - Why: every page your browser or script reads starts with one
- HTML Tag:
  - What: a label around content, such as <span class="price">
  - Why: tags tell you where each price or name sits
- DOM Tree:
  - What: Document Object Model: tags nested inside tags, like folders
  - Why: one product card holds its name, price, and stock
- Beautiful Soup (bs4):
  - What: a Python library that reads HTML into a searchable tree
  - Why: find tags by name or class in one line
- CSS Selector:
  - What: a short address for tags: .price = class “price”
  - Why: .select() / .select_one() use it to pick tags
- Ethical Scraping:
  - What: follow robots.txt, pause between requests, stay on public pages
  - Why: protects the website and your company

7. Core Programming & Computational Foundations:

7.1 A web page is text with tags, not a table

The idea: When you open a web page, your browser receives a long piece of text written in HTML and draws it as a page. The prices you see are inside that text, wrapped in tags such as <span class="price">$49.50</span>. A tag has three parts: an opening tag (<span ...>), the content ($49.50), and a closing tag (</span>). The class="price" part is a label that the website’s designers put on the tag, and it is what we use to find it.

See it on real data: one product card from our saved BetaMart page.

<div class="product" data-sku="SKU-801">
  <h3 class="name">Office Desk Chair</h3>
  <span class="price">$49.50</span>
  <span class="stock">120 units</span>
</div>

What we want	Where it is in the HTML
Product code	the `data-sku` setting of the `div` tag: `SKU-801`
Name	inside the `h3` tag with class `name`: `Office Desk Chair`
Price	inside the `span` tag with class `price`: `$49.50`
Stock	inside the `span` tag with class `stock`: `120 units`

More examples:

Code	Result	What it tells you
`<h3 class="name">Office Desk Chair</h3>`	tag `h3`, class `name`, content `Office Desk Chair`	the product name
`<span class="stock">120 units</span>`	tag `span`, class `stock`, content `120 units`	the stock level, as text
`<div class="product" data-sku="SKU-801">`	tag `div`, class `product`, setting `data-sku`	the card that holds one product; its code is a setting, not content
`<h1>BetaMart Weekly Prices</h1>`	tag `h1`, no class, content `BetaMart Weekly Prices`	the page heading

Remember: scraping means finding the right tags and taking the text inside them.

7.2 Tags inside tags: the DOM tree

The idea: Tags are nested: the div card contains the name, price, and stock tags, just as a folder contains files. This nesting is called the DOM tree (Document Object Model). Beautiful Soup reads the HTML text and builds this tree, so you can ask for “the price tag inside this card” instead of searching through raw text.
Why nesting matters: our page has 4 cards, and each one has its own price. If you search the whole page for .price, you get all 4 prices. If you first take one card and then search inside it, you get exactly that card’s price, already paired with the right name.

See it on real data:

Search	Result
`soup.select(".price")` (whole page)	4 tags: `$49.50`, `$320.00`, `$429.00`, `$195.00`
`card.select_one(".price")` (one card)	1 tag: `$49.50`, for `SKU-801` only

More examples:

Code	Result	What it tells you
`soup.select(".stock")`	3 tags: `120 units`, `38 units`, `15 units`	whole page: only 3, because one card has no stock tag
`soup.select(".product")[1].find("h3").text`	`Wireless Router`	the name inside the second card (position `1`)
`soup.select(".product")[3]["data-sku"]`	`SKU-810`	the code of the fourth card
`soup.select_one(".product .stock").text`	`120 units`	a space in a selector means “inside”: a stock tag inside a product card

Remember: loop over the cards, then search inside each card; that keeps every value with its own product.

7.3 Four ways to find tags

Method	Finds	Returns	Example
`.find("h3")`	the first tag with that name	one tag	`Office Desk Chair`
`.find_all("h3")`	every tag with that name	a list of tags	4 product names
`.select_one(".price")`	the first tag matching a CSS selector	one tag	`$49.50`
`.select(".price")`	every tag matching a CSS selector	a list of tags	4 prices

The idea: find methods search by tag name; select methods use CSS selectors (Cascading Style Sheets), the same short addresses web designers use: .price means “class price”, and div.product means “a div tag with class product”.
Getting the content: a tag is not yet text. .text takes the words inside it ("$49.50"), and tag["data-sku"] reads a setting written inside the opening tag ("SKU-801"). .strip() removes spaces and line breaks around the text.

More examples:

Code	Result	What it tells you
`soup.find("span").text`	`$49.50`	the first `span` on the page happens to be a price
`len(soup.find_all("span"))`	`7`	4 price tags + 3 stock tags
`len(soup.select("div.product"))`	`4`	the number of product cards
`soup.select_one(".discount")`	`None`	no tag on the page has the class `discount`
`soup.h1.text`	`BetaMart Weekly Prices`	a short way to get the first tag of a kind

Remember: one tag: find / select_one; all tags: find_all / select; then .text for the words.

7.4 Text is not a number yet

The idea: Everything scraped is text, even prices. "$49.50" cannot be added or averaged. Two steps turn it into a number: remove the dollar sign with .replace("$", ""), which gives "49.50", then convert it with float(), which gives 49.5.
Common mistake: forgetting the conversion. Text “added” together is joined, not summed, and float("$49.50") with the $ still there raises ValueError: could not convert string to float.

More examples:

Code	Result	What it tells you
`float("320.00")`	`320.0`	clean text converts directly
`float("$1,299.99".replace("$", "").replace(",", ""))`	`1299.99`	remove every symbol first: the `$` and the thousands comma
`int("120 units".split()[0])`	`120`	`.split()` cuts the text at spaces; `[0]` takes the first piece, `"120"`
`"49.5" + "10"`	`'49.510'`	text is joined, not added
`float("49.5") + 10`	`59.5`	after conversion, the math works

Remember: clean the text first, then float().

7.5 Select by class name, not by position

The idea: You could tell a script “take the 2nd tag in each card”. That works until the website adds something, such as a red “SALE” badge before the price; then the 2nd tag is the badge and the script quietly collects the word “SALE”. Selecting by class name (.price) follows the label, not the position, so it keeps working when the layout changes.

More examples: the same card before and after the website adds a “SALE” badge in front of the price.

Code	Before the badge	After the badge	What it tells you
`card.find_all("span")[0].text`	`$49.50`	`SALE`	“the first `span`” now points at the badge
`card.select_one(".price").text`	`$49.50`	`$44.50`	the class name still finds the price

Remember: use class names; still recheck the script whenever the site is redesigned, because class names can change too.

7.6 When a tag is missing: None

The idea: Real pages are irregular. On our page, the card for SKU-807 has no stock tag. When .select_one(".stock") finds nothing, it returns None, Python’s word for “nothing”. Asking for the text of nothing, None.text, stops the script with AttributeError: 'NoneType' object has no attribute 'text'.

The fix: store the result first, check it, then read it:

stock_tag = card.select_one(".stock")
if stock_tag is not None:
    stock = stock_tag.text.strip()
else:
    stock = "not shown"

More examples:

Code	Result	What it tells you
`soup.select(".product")[2].select_one(".stock")`	`None`	the third card (`SKU-807`) has no stock tag
`[c["data-sku"] for c in soup.select(".product") if c.select_one(".stock") is None]`	`['SKU-807']`	list every card with a missing stock tag
`soup.select(".product")[0].select_one(".stock").text`	`120 units`	the first card has one, so `.text` works

Remember: any tag might be missing; check for None before you read .text.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S21_demo_1.py), and check the files are in data/.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors. First time only: pip install beautifulsoup4.
9. Hands-On Coding Micro-Examples:
- Every example has three parts: 1. Input, 2. Process, and 3. Output. Name each file with your own name and the session code, for example Reacher_S21_demo_1.py (pattern: YourName_S21_demo_1.py).
- First time only: install Beautiful Soup in the Terminal with pip install beautifulsoup4.
- The examples read a saved copy of a fictional competitor page, data/session_02_01_catalog.html, so they work the same in class without internet access. On a live site, requests.get(url).text returns the same kind of HTML text.
- Micro-Example 1 (Read a Web Page with Beautiful Soup): YourName_S21_demo_1.py
```
# 1. Input: read the saved web page (HTML text)
from bs4 import BeautifulSoup
html_text = open("data/session_02_01_catalog.html").read()

# 2. Process: build the tree, then find the first product name
soup = BeautifulSoup(html_text, "html.parser")
first_name = soup.find("h3").text

# 3. Output: show what we found
print("Page title:", soup.title.text)
print("First product:", first_name)
```
  - How it works, line by line:
    - from bs4 import BeautifulSoup loads Beautiful Soup. The library is installed as beautifulsoup4 but imported as bs4.
    - open("data/...html").read() opens the saved page and reads the whole file as one long piece of text, stored in html_text. On a live website, requests.get(url).text would give you the same kind of text.
    - BeautifulSoup(html_text, "html.parser") reads that text and builds the DOM tree (section 7.2), stored in soup. "html.parser" is the HTML reader that comes with Python.
    - soup.find("h3") returns the first h3 tag on the page, and .text takes the words inside it.
    - soup.title.text reads the page’s title, the text shown on the browser tab.
  - Read the output:
    Page title: BetaMart Online Catalog (fictional) First product: Office Desk Chair
    - The title confirms that the script read the right page.
    - Only one name is printed, because .find() stops at the first match; Quick Check 1 counts all of them.
- Quick Check 1 (.find_all()):
  - New tool: .find_all() returns every matching tag as a list; .find() returns only the first. Why: count how many products the page shows.
  - Task: add one line to part 3. Output: print("Products on page:", len(soup.________("h3")))
  - Check: the Terminal shows Products on page: 4.
- Micro-Example 2 (Turn Product Cards into Rows): YourName_S21_demo_2.py
```
# 1. Input: read and parse the saved web page
from bs4 import BeautifulSoup
html_text = open("data/session_02_01_catalog.html").read()
soup = BeautifulSoup(html_text, "html.parser")

# 2. Process: one dictionary per product card
products = []
for card in soup.select(".product"):
    price_text = card.select_one(".price").text
    products.append({
        "sku": card["data-sku"],
        "name": card.select_one(".name").text.strip(),
        "price_usd": float(price_text.replace("$", "")),
    })

# 3. Output: one line per product
for item in products:
    print(item)
```
  - How it works, line by line:
    - products = [] starts an empty list that will collect one dictionary per product.
    - for card in soup.select(".product"): repeats the indented lines once for each product card; on each pass, card is one card (section 7.2).
    - card.select_one(".price").text finds the price tag inside this card and takes its text, such as "$49.50".
    - products.append({...}) adds one dictionary to the list. A dictionary pairs names with values: "sku" with card["data-sku"], "name" with the cleaned name, and "price_usd" with the price as a number.
    - float(price_text.replace("$", "")) removes the dollar sign and converts the text to a number (section 7.4).
    - The last loop prints each dictionary on its own line.
  - Read the output:
    {'sku': 'SKU-801', 'name': 'Office Desk Chair', 'price_usd': 49.5} {'sku': 'SKU-804', 'name': 'Wireless Router', 'price_usd': 320.0} {'sku': 'SKU-807', 'name': 'Compact Refrigerator', 'price_usd': 429.0} {'sku': 'SKU-810', 'name': 'Cordless Drill', 'price_usd': 195.0}
    - 4 cards became 4 rows, each with its code, name, and price kept together.
    - 49.5 has no quotation marks: it is now a number. pd.DataFrame(products) would turn this list straight into a table with columns sku, name, and price_usd.
- Quick Check 2 (.get_text(strip=True)):
  - New tool: .get_text(strip=True) returns the words inside a tag, with spaces and line breaks removed from both ends. Why: one call does the work of .text.strip().
  - Task: in part 2. Process, change "name": card.select_one(".name").text.strip(), to "name": card.select_one(".name").________(strip=True),
  - Check: the output does not change; the first line is still {'sku': 'SKU-801', 'name': 'Office Desk Chair', 'price_usd': 49.5}.
- Micro-Example 3 (From Scraped Prices to a Comparison Table): YourName_S21_demo_3.py
```
# 1. Input: prices already scraped from 3 competitor sites
import pandas as pd
df = pd.read_csv("data/session_02_01_dataset.csv")

# 2. Process: price after the advertised discount
df["Effective_Price_USD"] = (
    df["List_Price_USD"] * (1 - df["Promotion_Discount_Pct"] / 100)
).round(2)
electronics = df.loc[df["Category_Name"] == "Electronics"]

# 3. Output: compare competitors, then save the table
print(electronics[["Competitor_Name", "List_Price_USD",
                   "Effective_Price_USD"]])
electronics.to_csv("data/electronics_prices.csv", index=False)
```
  - How it works, line by line:
    - The CSV holds prices already scraped from 3 competitor sites, one row per product.
    - df["List_Price_USD"] * (1 - df["Promotion_Discount_Pct"] / 100) calculates the price a customer actually pays after the advertised discount, for all 12 rows at once. Example: 299.99 x (1 - 10 / 100) = 269.99.
    - .round(2) keeps 2 decimals, as for money.
    - df.loc[df["Category_Name"] == "Electronics"] keeps the 3 Electronics rows, one per competitor.
    - electronics.to_csv("data/electronics_prices.csv", index=False) saves those rows as a new CSV file; index=False leaves out the row labels, which are not data.
  - Read the output:
    Competitor_Name List_Price_USD Effective_Price_USD 0 AlphaRetail 299.99 269.99 4 BetaMart 320.00 281.60 8 GammaGlobal 310.00 263.50
    - AlphaRetail has the lowest list price, but GammaGlobal is cheapest after its 15% discount (263.50).
    - A comparison on list prices alone would pick the wrong competitor: always compare the price the customer actually pays.
- Quick Check 3 (.min()):
  - New tool: .min() returns the smallest value in a column; .max() returns the largest. Why: answers “what is the lowest competitor price?” in one step.
  - Task: add two lines to part 3. Output: lowest = electronics["Effective_Price_USD"].___() and print("Lowest price:", lowest)
  - Check: the Terminal shows Lowest price: 263.5 (GammaGlobal).
10. Official Python Documentation & References:
11. Exercise 1: Guided Practice - Competitor Price Benchmark:
- Step 1: Open Script: In Google Antigravity IDE, create a new script named YourName_S21_benchmark.py, replacing YourName with your own name (for example, Reacher_S21_benchmark.py).
- Step 2: Benchmark Competitors by Category: Type the following code:
```
# 1. Input: prices scraped from 3 competitor sites
import pandas as pd
df = pd.read_csv("data/session_02_01_dataset.csv")

# 2. Process: effective price, then average by category and competitor
df["Effective_Price_USD"] = (
    df["List_Price_USD"] * (1 - df["Promotion_Discount_Pct"] / 100)
).round(2)
benchmark = df.groupby(["Category_Name", "Competitor_Name"]).agg(
    Avg_Discount_Pct=("Promotion_Discount_Pct", "mean"),
    Avg_Effective_Price=("Effective_Price_USD", "mean"),
)

# 3. Output: one row per category and competitor
print("Competitor Category Pricing Benchmark:\n", benchmark)
```
- Step 3: Analyze Business Insights: In Electronics and Hardware, find the competitor with the deepest discount and the competitor with the lowest effective price. Are they the same?
- Step 4: Execute in Terminal: Run python YourName_S21_benchmark.py in Panel [4] (Terminal) and confirm return code 0.
- Success Check: Electronics: GammaGlobal has both the deepest discount (15%) and the lowest price (263.5). Hardware: GammaGlobal has the deepest discount (15%), but AlphaRetail has the lowest price (157.5). The deepest discount is not always the lowest price.
12. Exercise 2: Applied Practice - Scrape Stock Levels Safely:
- Coding Challenge: In YourName_S21_practice.py, extend Micro-Example 2 so each row also has the stock level.
- Task 1 (Add the Stock Field): Inside the loop, add "stock": card.select_one(".stock").text.strip() to the dictionary.
- Task 2 (Read the AttributeError & AI Diagnosis): Run the script. The SKU-807 card has no stock tag, so .select_one(".stock") returns None and Python stops with AttributeError: 'NoneType' object has no attribute 'text'. Capture the traceback with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and review the AI’s explanation.
- Task 3 (Safe Extraction and Export): Save the tag first with stock_tag = card.select_one(".stock"). If stock_tag is not None, use stock_tag.text.strip(); else use "not shown". Save the table with pd.DataFrame(products).to_csv("data/competitor_price_audit.csv", index=False).
- Success Check: the CSV file has 4 rows, SKU-807 shows not shown, and the script ends with return code 0.
13. Exercise 3: Advanced Practice - Detect a Price Change on a New Site:
- Goal: Scrape a second competitor page that uses different tags, compare it with last week’s data in the CSV, and report which product’s discount changed.
- What you need:
  - Saved web page: data/session_02_01_gamma.html, a fictional GammaGlobal page with 4 products.
  - Last week’s data: data/session_02_01_dataset.csv (use only the rows where Competitor_Name is "GammaGlobal").
  - Script name: YourName_S21_challenge.py.
  - Tools you already know: BeautifulSoup, .select(), .select_one(), None checks, float(), pd.DataFrame(), pd.merge(), and .to_csv().
- How the GammaGlobal page is built (open the file in the Code Editor to see it):
  - Each product is an li tag with class item, and its SKU is stored in data-sku.
  - The product name is in a tag with class title.
  - Every product has a list-price tag, such as $189.00.
  - A product on sale also has a sale-price tag. A product without a sale has no sale-price tag.
- Requirements (follow them in order):
  1. For each product, collect sku, name, List_Price_Web (the list price as a number), and Current_Price_Web: the sale price if the sale-price tag exists, otherwise the list price.
  2. Add Discount_Pct_Web = (List_Price_Web - Current_Price_Web) / List_Price_Web x 100, rounded to 1 decimal place.
  3. Turn the list into a DataFrame. Merge it with the GammaGlobal rows of the CSV, matching sku on the web side with Product_SKU on the CSV side.
  4. Add Discount_Changed = True when Discount_Pct_Web is not equal to Promotion_Discount_Pct.
  5. Print sku, name, Current_Price_Web, Discount_Pct_Web, Promotion_Discount_Pct, and Discount_Changed, then save the table to data/gamma_price_check.csv without row labels.
  6. Print one summary line, for example Products with a changed discount: 1.
- New tools for this exercise:
  - .select(".item") works for any class name; only the name changes from the BetaMart page (.product) to this page (.item).
  - Merging when the key columns have different names: pd.merge(web_df, gamma_df, left_on="sku", right_on="Product_SKU", how="left"). left_on names the key in the first table and right_on the key in the second.
  - != means “not equal”. On two columns, it gives one True / False per row.
- Hints:
  - Read the sale tag first with sale_tag = card.select_one(".sale-price"), then use if sale_tag is not None: to choose the current price.
  - Filter the CSV before merging: gamma_df = df.loc[df["Competitor_Name"] == "GammaGlobal"].
- Self-Check (expected results):
  - Current prices on the page: SKU-802 160.65, SKU-805 45.0 (no sale tag), SKU-808 263.5, SKU-811 391.0.
  - Only SKU-811 (Electric Oven) changed: the web shows 15.0% off, while last week’s CSV says 10.0%.
  - Summary line: Products with a changed discount: 1. Business meaning: GammaGlobal deepened its oven discount this week, so our pricing team should review our appliance prices.

Session 2.2: Business Data Visualization: Exploratory Plotting and Executive Dashboards

Session 2.2 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 6 (115/10/18 - 115/10/24) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- The marketing team keeps 12 months of results in one table: revenue, marketing spend, cost per new customer, and customer loyalty
- Executives often have only a minute or two per chart, and a table of 72 numbers hides the trend
- The CMO (Chief Marketing Officer) asks: is our marketing spend paying off, and are customers getting happier? Today, Python turns the table into charts that answer this at a glance
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “Is revenue growing month by month?”: draw a line chart with Matplotlib (Python’s main charting library) using fig, ax = plt.subplots() (fig is the whole image; ax is one chart inside it)
2. “In which months did we spend the most on marketing?”: draw a bar chart with Seaborn (a library built on Matplotlib for quick, clean statistical charts)
3. “Which numbers move together?”: read a correlation heatmap (a colored grid of correlations, each from -1 to +1), and fix an IndexError in a 2x2 dashboard
3. Business Context: A Chart Answers One Question:
- A table makes the reader hunt for the pattern; a chart shows it. In our table, revenue rises from 12.4 to 25.8 (USD millions), but the jump in November and December is easy to miss among 72 numbers.
- Five rules for charts that executives trust:
  - One message per chart, and the title states it, for example “Revenue More Than Doubled from Jan to Dec”
  - Label every axis with its unit, such as “USD M” (millions of U.S. Dollars)
  - Bar charts start at zero; a cut-off axis makes small differences look huge
  - No 3D effects or decoration; every mark on the chart should carry data
  - Use colors that still work in black-and-white print and for color-blind readers

4. Data Basics: Pick the Chart from the Question:

Business question	Chart	Python
How does a number change over time?	Line chart	`ax.plot()`
Which category is biggest?	Bar chart	`sns.barplot()` or `ax.bar()`
Do two numbers move together?	Scatter plot	`ax.scatter()`
Which of many numbers move together?	Correlation heatmap	`sns.heatmap()`

Correlation measures how closely two columns move together, on a scale from -1 to +1:
- +1: when one goes up, the other always goes up
- 0: no straight-line pattern
- -1: when one goes up, the other always goes down
Correlation is not causation. In our data, marketing spend and revenue have a correlation of 0.996, but both also simply grew through the year. A high correlation alone does not prove that the spending caused the revenue.

5. Lab Practice Dataset:
- File Link: data/session_02_02_dataset.csv
- Audited 12-row monthly performance dataset (fictional teaching data), one row per month:
  - Month_Index: Month number, 1 to 12.
  - Month_Name: Three-letter month name, Jan through Dec.
  - Organic_Revenue_USD_M: Organic revenue in millions of U.S. Dollars. “Organic” means revenue from the company’s own operations, not from buying other companies.
  - Paid_Marketing_Spend_USD_M: Paid advertising spend in millions of U.S. Dollars.
  - Customer_Acquisition_Cost_USD: CAC (Customer Acquisition Cost): the average marketing cost to win one new customer, in U.S. Dollars.
  - Net_Promoter_Score: NPS (Net Promoter Score): a customer loyalty score from -100 to +100, based on how likely customers are to recommend the company. Our values range from 58 to 74.
6. Keywords (Core Programming Concepts):
- Matplotlib and Seaborn:
  - What: Python charting libraries; Seaborn adds quick, clean statistical charts on top of Matplotlib
  - Why: they turn a DataFrame into a chart image with a few lines of code
- Figure and Axes:
  - What: the Figure is the whole image; each Axes is one chart inside it
  - Why: you draw on a specific Axes, so a dashboard can hold several charts
- Line Chart:
  - What: points joined in time order
  - Why: shows trends, such as monthly revenue
- Bar Chart:
  - What: one bar per category; the bar length is the value
  - Why: compares sizes, such as spend by month
- Scatter Plot:
  - What: one dot per row, with one column on each axis
  - Why: shows whether two numbers move together
- Correlation Heatmap:
  - What: a grid of correlations, colored from -1 to +1
  - Why: spots which of many numbers move together, in one picture

7. Core Programming & Computational Foundations:

7.1 Figure and Axes: the canvas and the charts on it

The idea: Matplotlib draws in two layers. The Figure is the whole image, like a blank sheet of paper. An Axes is one chart drawn on that sheet, with its own x-axis, y-axis, title, and data. One Figure can hold one Axes or many. fig, ax = plt.subplots() creates both at once and gives you a name for each, so every later instruction says exactly which chart it changes.

See it on real data: the revenue chart in Micro-Example 1.

Instruction	Goes to	Effect
`fig, ax = plt.subplots(figsize=(10, 5))`	creates both	a 10 x 5 inch image with one chart area
`ax.plot(df["Month_Name"], df["Organic_Revenue_USD_M"])`	the chart	draws 12 points joined by a line
`ax.set_title("Revenue More Than Doubled from Jan to Dec")`	the chart	the chart’s title
`ax.set_ylabel("Organic Revenue (USD M)")`	the chart	the y-axis label
`fig.savefig("data/revenue_trend.png")`	the whole image	writes the image to a file

Common mistake: mixing the two styles. Many online examples write plt.plot() and plt.title(), which draw on “whatever chart is active”. With several charts, that is often the wrong one. Always draw on a named ax.

More examples:

Code	Result	What it tells you
`fig, ax = plt.subplots()`	one `Figure`, one `Axes`	the most common start for one chart
`fig, (ax1, ax2) = plt.subplots(1, 2)`	2 charts side by side	1 row, 2 columns; each gets its own name
`fig, axes = plt.subplots(2, 2)`	`axes.shape` is `(2, 2)`	4 charts, stored as a 2 x 2 table
`len(axes.flatten())`	`4`	`.flatten()` lines the 4 charts up in one row, for a loop
`ax.set_xlabel("")`	no x-axis label	remove a label that adds nothing

Remember: the Figure is the page; each Axes is a chart; draw on a named ax.

7.2 Choose the chart from the question

The idea: Each chart type answers one kind of question well. Start from the question the manager asks, then pick the chart; do not start from the chart you like.

Question	Chart	Example from our data
How does a number change over time?	line (`ax.plot()`)	revenue from Jan (`12.4`) to Dec (`25.8`)
Which category is biggest?	bar (`sns.barplot()`, `ax.bar()`)	marketing spend peaks in Dec (`6.4`)
Do two numbers move together?	scatter (`ax.scatter()`)	cost per new customer vs loyalty
Which of many numbers move together?	heatmap (`sns.heatmap()`)	all 4 measures at once

Common mistake: a line chart for categories that have no order (such as regions), or a pie chart with many slices. A line suggests a trend from one point to the next; it only makes sense when the x-axis has an order, such as months.

More examples:

Question	Best chart	Why
Did revenue grow every month?	line	the eye follows the slope from month to month
Which 3 months had the most spend?	bar, sorted	lengths are easy to compare
Does a higher CAC go with a higher NPS?	scatter	one dot per month shows the pattern
Is any measure moving against the others?	heatmap	a blue cell would stand out among red ones

Remember: question first, chart second.

7.3 Correlation: how closely two columns move together

The idea: The correlation of two columns is one number from -1 to +1. Near +1, when one goes up the other goes up too; near -1, when one goes up the other goes down; near 0, there is no straight-line pattern. df["A"].corr(df["B"]) computes it for two columns, and df[cols].corr() for every pair at once.

See it on real data:

Pair of columns	Correlation	Meaning
marketing spend and revenue	`0.996`	they rose almost perfectly together
month number and revenue	`0.946`	revenue grew through the year
CAC and NPS	`0.976`	months with costlier customers also had happier customers
month number and revenue per marketing dollar	`-0.95`	each dollar of marketing brought in less as the year went on

Common mistake: reading correlation as cause. Spend and revenue both grew month by month, so they correlate strongly, but the data cannot tell whether the spend caused the revenue, or whether both simply followed the holiday season.

More examples:

Code	Result	What it tells you
`pd.Series([1, 2, 3]).corr(pd.Series([2, 4, 6]))`	`1.0`	a perfect straight-line rise
`pd.Series(range(1, 13)).corr(-pd.Series(range(1, 13)))`	`-1.0`	a perfect opposite movement
`df["Month_Index"].corr(df["Net_Promoter_Score"])`	`0.963`	loyalty also rose through the year
`df[cols].corr()`	a 4 x 4 table	every pair at once; the diagonal is always `1.0`

Remember: correlation measures moving together, not cause and effect.

7.4 Saving a chart: size and sharpness

The idea: fig.savefig(file, dpi=...) writes the chart to an image file. The final size in pixels is the figure size in inches times the dpi (dots per inch). More dots mean a sharper image and a larger file.

See it on real data: a figsize=(10, 5) chart saved three ways.

`dpi`	Image size in pixels	Good for
`100`	1000 x 500	a quick look
`150`	1500 x 750	screens and slides
`300`	3000 x 1500	printed reports

Other settings: bbox_inches="tight" trims empty white borders. The file type follows the name: .png for images, .pdf for documents that stay sharp at any zoom.
Common mistake: calling plt.savefig() after plt.show(). In many setups, showing the chart clears it, so the saved file is empty. Save first.

More examples:

Settings	Result	What it tells you
`figsize=(6, 4)`, `dpi=100`	600 x 400 pixels	a small image for a quick check
`figsize=(6, 4)`, `dpi=300`	1800 x 1200 pixels	the same chart, sharp enough for print
`figsize=(10, 5)`, `dpi=150`, `bbox_inches="tight"`	1233 x 674 pixels	smaller than 1500 x 750, because the empty border was trimmed
`fig.savefig("data/chart.pdf")`	a PDF file	stays sharp at any zoom, good for reports

Remember: inches x dpi = pixels; save before you show.

7.5 Grids of charts: rows and columns start at 0

The idea: fig, axes = plt.subplots(2, 2) makes 4 charts, and axes is a small table of them with 2 rows and 2 columns. You reach one chart with its row number and column number, both counted from 0, just like .iloc[] in Session 1.3.

Position	Code
top left	`axes[0, 0]`
top right	`axes[0, 1]`
bottom left	`axes[1, 0]`
bottom right	`axes[1, 1]`

Common mistake: axes[2] asks for a third row, which does not exist, so Python stops with IndexError: index 2 is out of bounds for axis 0 with size 2. “Axis 0” means the rows; “size 2” means there are only rows 0 and 1.
Spacing: plt.tight_layout() moves the charts apart so that titles and axis labels do not overlap.

More examples:

Code	Result	What it tells you
`axes[1, 0]`	bottom-left chart	row 1 (second row), column 0 (first column)
`axes[-1, -1]`	bottom-right chart	`-1` means “the last”, for rows and columns alike
`axes.flatten()[2]`	bottom-left chart	after flattening, the charts are numbered 0 to 3, row by row
`fig, axes = plt.subplots(3, 1)`	`axes.shape` is `(3,)`	one column of 3 charts: now `axes[2]` exists
`axes[2]` with a 2 x 2 grid	`IndexError`	there is no third row

Remember: axes[row, column], both from 0.

7.6 Honest charts: bars start at zero

The idea: A bar’s length is read as its size. If the y-axis starts above zero, the bars are cut off and small differences look huge.

See it on real data: marketing spend in Oct (4.1) and Nov (5.2).

y-axis starts at	Nov bar compared with Oct bar	What a reader believes
`0`	`1.27` times as long	Nov spend was about a quarter higher (true)
`4.0`	`12.0` times as long	Nov spend was 12 times higher (false)

Other rules: one message per chart, stated in the title; every axis labeled with its unit; no 3D effects; colors that still work in black-and-white print.

More examples:

Chart	What the reader sees	Honest?
Jan (`2.1`) vs Dec (`6.4`), axis from `0`	Dec bar `3.05` times as long	true: spend tripled
the same, axis from `2.0`	Dec bar `44.0` times as long	false: the cut axis exaggerates fifteenfold
revenue line chart, axis from `12`	the line fills the chart	acceptable: a line shows change, not size, so its axis need not start at `0`
a 3D bar chart	bars at the back look smaller	avoid 3D: depth distorts the lengths

Remember: if the axis is cut, the bar chart lies.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S22_demo_1.py), and check the dataset is in data/. Saved charts appear here as .png files; click one to open it.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors. First time only: pip install matplotlib seaborn.
9. Hands-On Coding Micro-Examples:
- Every example has three parts: 1. Input, 2. Process, and 3. Output. Name each file with your own name and the session code, for example Reacher_S22_demo_1.py (pattern: YourName_S22_demo_1.py).
- The examples save each chart as a .png file in data/. Open the file in Panel [1] to see the chart.
- Micro-Example 1 (Line Chart: Revenue Trend): YourName_S22_demo_1.py
```
# 1. Input: load the monthly results
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv("data/session_02_02_dataset.csv")

# 2. Process: one figure with one chart, revenue by month
fig, ax = plt.subplots(figsize=(10, 5))
ax.plot(df["Month_Name"], df["Organic_Revenue_USD_M"],
        color="#991B1B", marker="o")
ax.set_title("Revenue More Than Doubled from Jan to Dec")
ax.set_ylabel("Organic Revenue (USD M)")

# 3. Output: save the chart as an image file
fig.savefig("data/revenue_trend.png", dpi=150, bbox_inches="tight")
print("Saved: data/revenue_trend.png")
```
  - How it works, line by line:
    - import matplotlib.pyplot as plt loads Matplotlib’s charting tools under the short name plt.
    - plt.subplots(figsize=(10, 5)) creates the Figure (10 inches wide, 5 tall) and one Axes, named fig and ax (section 7.1).
    - ax.plot(x, y, color="#991B1B", marker="o") draws the line: month names along the x-axis, revenue up the y-axis. color is a color code (#991B1B is a dark red) and marker="o" puts a dot on each month.
    - ax.set_title(...) writes the message of the chart, not just its topic: a reader knows the conclusion before looking at the line.
    - fig.savefig(..., dpi=150, bbox_inches="tight") writes a 1500 x 750 pixel image and trims the white border (section 7.4).
  - Read the output:
    Saved: data/revenue_trend.png
    - The Terminal only confirms the file; the chart itself is in data/revenue_trend.png. Open it from Panel [1].
    - The line climbs from 12.4 in Jan to 25.8 in Dec, 2.08 times higher, with the steepest rise in Nov and Dec.
- Quick Check 1 (ax.grid()):
  - New tool: ax.grid(True, linestyle=":") adds light dotted guide lines behind the chart. Why: readers can line up a point with its value on the axis.
  - Task: add one line at the end of part 2. Process: ax.____(True, linestyle=":")
  - Check: open data/revenue_trend.png; dotted grid lines now sit behind the revenue line.
- Micro-Example 2 (Bar Chart: Marketing Spend by Month): YourName_S22_demo_2.py
```
# 1. Input: load the monthly results
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
df = pd.read_csv("data/session_02_02_dataset.csv")

# 2. Process: one bar per month
fig, ax = plt.subplots(figsize=(10, 4))
sns.barplot(data=df, x="Month_Name", y="Paid_Marketing_Spend_USD_M",
            color="#475569", ax=ax)
ax.set_title("Marketing Spend Peaks in November and December")
ax.set_xlabel("")
ax.set_ylabel("Paid Marketing Spend (USD M)")

# 3. Output: save the chart
fig.savefig("data/marketing_spend_bar.png", dpi=150, bbox_inches="tight")
print("Saved: data/marketing_spend_bar.png")
```
  - How it works, line by line:
    - import seaborn as sns loads Seaborn, which draws common statistical charts with less code than Matplotlib alone.
    - sns.barplot(data=df, x="Month_Name", y="Paid_Marketing_Spend_USD_M", ...) reads the columns by name from the DataFrame and draws one bar per month. ax=ax tells Seaborn to draw on our named chart.
    - Seaborn’s bars start at zero, so their lengths can be compared honestly (section 7.6).
    - ax.set_xlabel("") removes the automatic x-axis label (Month_Name), because the month names already explain the axis.
  - Read the output:
    Saved: data/marketing_spend_bar.png
    - In the image, the bars grow through the year, and the last two are clearly the tallest: Nov 5.2 and Dec 6.4, against 2.1 in Jan.
    - The title states this finding, so the chart can go straight into a report.
- Quick Check 2 (.nlargest()):
  - New tool: df.nlargest(3, "Column_Name") returns the 3 rows with the largest values in that column. Why: confirms the chart’s message with exact numbers.
  - Task: add one line to part 3. Output: print(df.________(3, "Paid_Marketing_Spend_USD_M")[["Month_Name", "Paid_Marketing_Spend_USD_M"]])
  - Check: the Terminal lists Dec (6.4), Nov (5.2), and Oct (4.1).
- Micro-Example 3 (Correlation Heatmap): YourName_S22_demo_3.py
```
# 1. Input: load the monthly results
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
df = pd.read_csv("data/session_02_02_dataset.csv")

# 2. Process: correlation between every pair of 4 columns
cols = ["Organic_Revenue_USD_M", "Paid_Marketing_Spend_USD_M",
        "Customer_Acquisition_Cost_USD", "Net_Promoter_Score"]
corr = df[cols].corr().round(2)
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(corr, annot=True, cmap="RdBu_r", vmin=-1, vmax=1, ax=ax)
ax.set_title("All Four Measures Rose Together in 2024")

# 3. Output: print the numbers, then save the chart
print(corr)
fig.savefig("data/kpi_correlation_heatmap.png", dpi=150,
            bbox_inches="tight")
```
  - How it works, line by line:
    - cols = [...] lists the 4 measures to compare. A list keeps the code short and makes it easy to add a column later.
    - df[cols].corr() computes the correlation of every pair, a 4 x 4 table (section 7.3). .round(2) keeps 2 decimals.
    - sns.heatmap(corr, annot=True, cmap="RdBu_r", vmin=-1, vmax=1, ax=ax) colors each cell: red for positive, blue for negative, white near zero. annot=True writes the number in each cell, and vmin / vmax fix the color scale from -1 to +1, so a pale cell really means a weak correlation.
    - print(corr) shows the same numbers in the Terminal.
  - Read the output:
    Organic_Revenue_USD_M ... Net_Promoter_Score Organic_Revenue_USD_M 1.00 ... 0.98 Paid_Marketing_Spend_USD_M 1.00 ... 0.97 Customer_Acquisition_Cost_USD 1.00 ... 0.98 Net_Promoter_Score 0.98 ... 1.00
    - The ... hides middle columns that do not fit the Terminal width; the image shows all 16 cells.
    - Every value is between 0.97 and 1.00: all four measures rose together through the year, so the heatmap is almost entirely dark red.
    - This is a reminder of section 7.3: when everything grows over time, everything correlates, and correlation alone cannot say what caused what.
- Quick Check 3 (.corr() on two columns):
  - New tool: column_a.corr(column_b) returns one correlation number for two columns. Why: answers a single question without a whole heatmap.
  - Task: add one line to part 3. Output: print(round(df["Paid_Marketing_Spend_USD_M"].____(df["Organic_Revenue_USD_M"]), 3))
  - Check: the Terminal shows 0.996.
10. Official Python Documentation & References:

11. Exercise 1: Guided Practice - Two-Panel Executive Dashboard:

Step 1: Open Script: In Google Antigravity IDE, create a new script named YourName_S22_dashboard.py, replacing YourName with your own name (for example, Reacher_S22_dashboard.py).

Step 2: Build Two Charts Side by Side: Type the following code:

# 1. Input: load the monthly results
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv("data/session_02_02_dataset.csv")

# 2. Process: one figure, two charts side by side
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))

# Left chart: revenue trend
ax1.plot(df["Month_Name"], df["Organic_Revenue_USD_M"],
         color="#991B1B", marker="s", linewidth=2)
ax1.set_title("Monthly Revenue (USD M)")
ax1.grid(True, linestyle=":")

# Right chart: cost per new customer vs customer loyalty
ax2.scatter(df["Customer_Acquisition_Cost_USD"],
            df["Net_Promoter_Score"], color="#1E293B", s=80)
ax2.set_title("Loyalty (NPS) vs Cost per New Customer (CAC)")
ax2.set_xlabel("CAC (USD)")
ax2.set_ylabel("Net Promoter Score")
ax2.grid(True, linestyle=":")

# 3. Output: save the dashboard
plt.tight_layout()
fig.savefig("data/executive_dashboard_2panel.png", dpi=200)
print("Dashboard saved: data/executive_dashboard_2panel.png")

plt.subplots(1, 2) makes 1 row with 2 charts, and (ax1, ax2) names them left and right.
ax.scatter(x, y) draws one dot per month; s=80 sets the dot size.

Step 3: Inspect Output: Open data/executive_dashboard_2panel.png in Panel [1] and check that titles and axis labels do not overlap.
Step 4: Execute in Terminal: Run python YourName_S22_dashboard.py in Panel [4] and confirm return code 0.
Success Check: the image has 2 charts. The right chart’s dots climb from bottom-left to top-right: months with a higher cost per new customer also had higher loyalty, which matches their correlation of 0.98.

12. Exercise 2: Applied Practice - Four-Panel Boardroom Dashboard:
- Coding Challenge: In YourName_S22_practice.py, build a 2x2 dashboard with 4 charts: revenue trend (line), marketing spend by month (bar), cost per new customer by month (line), and the correlation heatmap.
- Task 1 (2x2 Layout): Create the grid with fig, axes = plt.subplots(2, 2, figsize=(16, 10)). Draw each chart on its own position: axes[0, 0], axes[0, 1], axes[1, 0], and axes[1, 1]. Pass ax=axes[1, 1] to sns.heatmap().
- Task 2 (Deliberate IndexError & AI Diagnosis): Change one position to axes[2].plot(...). Run the script and read IndexError: index 2 is out of bounds for axis 0 with size 2. Capture the traceback with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and review the AI’s explanation of row and column numbers.
- Task 3 (Fix and Export): Fix the position, call plt.tight_layout(), and save the dashboard to data/boardroom_dashboard.png with dpi=300.
- Success Check: data/boardroom_dashboard.png shows 4 charts with no overlapping titles, and the script ends with return code 0.
13. Exercise 3: Advanced Practice - Is Marketing Getting Less Efficient?:
- Goal: Answer the CMO’s question with a new measure and one honest chart: does each marketing dollar still bring in as much revenue at the end of the year as at the start?
- What you need:
  - File: data/session_02_02_dataset.csv (described in 5. Lab Practice Dataset).
  - Script name: YourName_S22_challenge.py.
  - Tools you already know: column math, .round(), .mean(), .sort_values(), a for loop with if / else (Session 1.2), fig, ax = plt.subplots(), and fig.savefig().
- Requirements (follow them in order):
  1. Add Revenue_per_Marketing_USD = Organic_Revenue_USD_M / Paid_Marketing_Spend_USD_M, rounded to 2 decimal places. It shows how many dollars of revenue came with each dollar of paid marketing in that month.
  2. Compute the 12-month average of that column and round it to 2 decimal places.
  3. Build a list of 12 colors with a for loop: "#991B1B" (red) for months below the average, "#94A3B8" (grey) for all others.
  4. Draw a bar chart with ax.bar(): month names on the x-axis, Revenue_per_Marketing_USD as the bar heights, and your color list.
  5. Add a dashed horizontal line at the average, then a title that states your finding and a y-axis label with its unit (“USD of revenue per USD of marketing”).
  6. Save the chart to data/marketing_efficiency.png with dpi=150.
  7. Print the average, then the best and the worst month with their values.
- New tools for this exercise:
  - ax.bar(x_values, heights, color=color_list): Matplotlib’s own bar chart. When color is a list, bar 1 gets color 1, bar 2 gets color 2, and so on.
  - ax.axhline(y=5.08, linestyle="--", color="black"): draws a horizontal line across the chart at height y, useful for marking an average or a target.
  - Building a list in a loop: start with colors = [], then colors.append("#991B1B") adds one item to the end.
- Hints:
  - Loop over the column itself: for value in df["Revenue_per_Marketing_USD"]:.
  - For the best and worst month, sort the DataFrame by the new column and read the first and last rows, for example with .head(1) and .tail(1).
- Self-Check (expected results):
  - Average: 5.08. Best month: Jan (5.9). Worst month: Dec (4.03).
  - The 6 red bars are Jul through Dec: every month in the second half of the year is below average.
  - Business meaning: spending rose faster than revenue. Revenue doubled, but each marketing dollar brought in about 32% less revenue in December than in January, and the cost per new customer rose from 48.5 to 68.0. The CMO should ask whether the November and December campaigns were worth their extra cost.

Session 2.3: Midterm Applied Analytics Formulation Part 1: Problem Structuring and Data Audit

Session 2.3 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 7 (115/10/25 - 115/10/31) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- An IT (Information Technology) services company holds 15 enterprise client contracts worth $10.7 million a year
- The account director has a vague worry: “some clients are unhappy and may leave”, but a worry is not something data can test
- The account director asks: which contracts are at risk of leaving, and how much revenue is at stake? Today, you turn that worry into a testable question, check that the data can be trusted, and measure the starting point. This is step 1 of your midterm project
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “What exactly should we test?”: turn a vague business problem into a testable hypothesis (a clear statement that data can support or prove wrong)
2. “Can we trust this data?”: audit a dataset for missing values (.isnull()), duplicates (.duplicated()), and impossible values, such as a percentage above 100
3. “How do we stop bad data automatically?”: write assert checks (rules the data must pass) and handle an AssertionError with try / except

3. Business Context: From a Vague Worry to a Testable Question:

Managers rarely hand you a clean question. A useful analysis moves through four steps:

Step	Example
Vague worry	“Some clients are unhappy and may leave.”
Business question	“Is poor service linked to clients at risk of leaving?”
Testable hypothesis	“Accounts below 95% SLA compliance have more billing disputes and a higher retention risk than other accounts.”
Measure today (baseline)	“Revenue in High-risk accounts today: $930,000.”

SLA (Service Level Agreement): the service promise in a contract, such as “the system is available 99% of the time”. SLA compliance is the share of that promise actually delivered.
Baseline: today’s value of the measure you care about. Without it, nobody can prove later that an action improved anything.
Midterm Project, Step 1 of 2: your Question Card. Choose a business problem with a dataset you can audit (for example, any dataset from Sessions 1.3 to 2.2), and fill in five lines:
1. Vague worry: what the manager is concerned about
2. Business question: what decision the answer supports
3. Hypothesis: one sentence the data can support or prove wrong
4. Data and audit: the file you use, and the result of the three checks in section 4
5. Baseline: one number that describes today
Bring your completed Question Card and your audit script to Session 2.4, where you present your findings.

4. Data Basics: Three Checks Before Any Analysis:

Bad data produces confident but wrong answers (“garbage in, garbage out”). Run three checks before you analyze:

Check	Question	pandas
Completeness	Are any values missing?	`df.isnull().sum()`
Uniqueness	Does each ID appear only once?	`df["Account_ID"].duplicated().sum()`
Validity	Are all values possible?	`(df["SLA_Compliance_Pct"] > 100).sum()`

NaN (“Not a Number”) is how pandas shows an empty cell. .isnull() marks each NaN as True.
A rule for validity comes from business logic: a percentage must be between 0 and 100, and a contract value must be above 0.

5. Lab Practice Dataset:
- Two versions of the same 15 client contracts (company names are fictional):
  - Clean file: data/session_02_03_dataset.csv, 15 rows that pass every check. Use it for analysis.
  - Raw file: data/session_02_03_raw.csv, 16 rows as they arrived from the billing system, with a few problems for you to find. Use it for auditing.
- Columns (the same in both files):
  - Account_ID: Unique client account code (e.g., ACC-301).
  - Enterprise_Client: Client company name (Titan Tech, Beacon Health, Vanguard Retail, and others). One company can hold several accounts.
  - Contract_Value_USD: Yearly contract value in U.S. Dollars.
  - SLA_Compliance_Pct: Share of the service promise delivered, in percent.
  - Billing_Discrepancy_Count: Number of billing disputes (invoices the client challenged) during the contract.
  - Client_Retention_Risk: The account team’s rating of how likely the client is to leave: Low, Medium, or High.
6. Keywords (Core Programming Concepts):
- Hypothesis:
  - What: one clear statement that data can support or prove wrong
  - Why: turns a vague worry into a test with a yes or no answer
- Exploratory Data Analysis (EDA):
  - What: a first look at the data with counts, averages, and charts before any formal test
  - Why: shows what the data can answer, and where it looks strange
- Data Audit:
  - What: the three checks: completeness, uniqueness, and validity
  - Why: one bad row can change a total by millions
- Baseline:
  - What: today’s value of the measure you care about
  - Why: the starting point for proving any later improvement
- assert:
  - What: a line that stops the script with an AssertionError when a rule is broken, such as assert (df["Contract_Value_USD"] > 0).all()
  - Why: bad data is caught at the door, before it reaches a report
- try / except:
  - What: runs code in try; if a chosen error happens, the code in except runs instead of stopping the script
  - Why: the script can report the problem clearly and keep going

7. Core Programming & Computational Foundations:

7.1 From a worry to a hypothesis you can test

The idea: A hypothesis is one sentence that data can support or prove wrong. A useful hypothesis names what you measure, the groups you compare, and the difference you expect. “Some clients are unhappy” cannot be tested; “accounts below 95% SLA compliance have more billing disputes than the others” can, because the data has a column for each part.

See it on real data:

Part of the hypothesis	In our data
What is measured	`Billing_Discrepancy_Count`
The two groups	SLA below 95% (`4` accounts) vs 95% or above (`11` accounts)
The expected difference	the first group has more disputes
The result	`4.75` vs `0.64` disputes on average: supported

Common mistake: a hypothesis that cannot fail, such as “service quality matters”. If no result could ever prove it wrong, it does not help anyone decide.

More examples:

Vague worry	Testable hypothesis	Column needed
“Our big clients are risky.”	Accounts above $800,000 have a higher share of High risk ratings.	`Contract_Value_USD`, `Client_Retention_Risk`
“Billing is a mess.”	Accounts with 3 or more disputes are rated Medium or High risk.	`Billing_Discrepancy_Count`, `Client_Retention_Risk`
“Service is slipping.”	High-risk accounts have a lower average SLA than Low-risk accounts.	`SLA_Compliance_Pct`, `Client_Retention_Risk`

In our data, the third one is supported (High-risk accounts average 90.97% SLA, Low-risk 98.84%), while the first is not: none of the 6 accounts above $800,000 is rated High risk. A hypothesis that turns out wrong is still useful: it rules out a wrong explanation.

Remember: a good hypothesis names the measure, the groups, and the expected difference.

7.2 The three data checks

The idea: Before any analysis, check that the data is complete (no missing values), unique (no record counted twice), and valid (every value is possible). Each check is a rule on a column that gives True / False per row; .sum() counts the True values, because Python counts True as 1.

See it on real data: the raw file (16 rows).

Check	Code	Result	The problem
Completeness	`raw.isnull().sum().sum()`	`1`	`ACC-306` has no SLA value
Uniqueness	`raw["Account_ID"].duplicated().sum()`	`1`	`ACC-304` appears twice
Validity	`(raw["SLA_Compliance_Pct"] > 100).sum()`	`1`	`ACC-311` shows `105.0`%
Validity	`(raw["Contract_Value_USD"] <= 0).sum()`	`1`	`ACC-313` has a negative value

Why it matters: these 4 small problems change the totals. The raw file adds up to $11,290,000 of contracts; the corrected file to $10,710,000. The average SLA looks like 97.45% in the raw file but is really 96.48%.
Common mistake: .duplicated() marks only the second copy as True. To see both copies of ACC-304, use raw.duplicated("Account_ID", keep=False).

More examples:

Code	Result	What it tells you
`raw.isnull().sum()`	`1` in `SLA_Compliance_Pct`, `0` elsewhere	which column has the gap
`raw.loc[raw["SLA_Compliance_Pct"].isnull(), "Account_ID"]`	`ACC-306`	which account has the gap
`raw["Account_ID"].is_unique`	`False`	a one-word answer to “are the IDs unique?”
`raw.loc[raw["SLA_Compliance_Pct"] > 100, "Account_ID"]`	`ACC-311`	which account has an impossible percentage
`(~raw["SLA_Compliance_Pct"].between(0, 100)).sum()`	`2`	`.between()` treats a missing value as outside the range too

Remember: complete, unique, valid; check all three before you calculate.

7.3 Rules the data must pass: assert

The idea: assert rule, "message" checks a rule. If the rule is True, nothing happens and the script continues. If it is False, the script stops with an AssertionError that shows your message. Rules turn your data checks into a gate: bad data cannot reach the report unnoticed.

See it on real data:

Code	On the raw file	On the clean file
`assert (df["Contract_Value_USD"] > 0).all(), "Contract value must be above 0"`	stops: `AssertionError: Contract value must be above 0`	passes
`assert df["Account_ID"].is_unique, "Account IDs must be unique"`	stops	passes

.all() and .any(): a rule on a column gives many True / False values, but assert needs one answer. .all() is True only if every value is True; .any() is True if at least one is.
Common mistake: leaving out the message. assert rule alone stops with an empty AssertionError, and the reader has to guess which rule failed.

More examples:

Rule	On the raw file	On the clean file
`assert (df["SLA_Compliance_Pct"] <= 100).all()`	fails (`105.0`, and a missing value counts as `False`)	passes
`assert len(df) == 15`	fails (16 rows)	passes
`assert df["Client_Retention_Risk"].isin(["Low", "Medium", "High"]).all()`	passes	passes
`assert not df.isnull().any().any()`	fails (1 gap)	passes

Remember: one assert per rule, always with a message.

7.4 Handling a failure: try / except

The idea: Sometimes you want the script to report the problem and keep going, for example to finish a cleaning step. Put the risky lines under try:. If an AssertionError happens, Python jumps to the lines under except AssertionError as error: instead of stopping, and error holds the message.
```
try:
    assert (raw["Contract_Value_USD"] > 0).all(), "Contract value must be above 0"
except AssertionError as error:
    print("Data problem:", error)
```
On the raw file this prints Data problem: Contract value must be above 0, and the script continues.
Common mistake: catching every error with a bare except:. It also hides typos and real bugs. Name the error you expect.

More examples:

Code inside `try:`	Error raised	The matching `except` line
`int("abc")`	`ValueError: invalid literal for int() with base 10: 'abc'`	`except ValueError:`
`df["Missing"]`	`KeyError: 'Missing'`	`except KeyError:`
`1 / 0`	`ZeroDivisionError: division by zero`	`except ZeroDivisionError:`
`assert False, "Bad data"`	`AssertionError: Bad data`	`except AssertionError:`

Remember: try the check, except the expected error, and report it clearly.

7.5 The baseline: today’s number

The idea: A baseline is the value of your measure today. It is the starting point that any later improvement is compared with. Without it, nobody can prove that an action worked.

See it on real data:

Baseline	Value today
Revenue in High-risk accounts	`$930,000`
Share of all revenue at High risk	`8.7`%
Accounts below 95% SLA	`4` of `15` (`26.7`%)
Billing disputes in total	`26`

If the company improves service next quarter and the High-risk revenue falls to, say, $500,000, the baseline shows the size of the improvement.

Common mistake: measuring the baseline on raw data. With the raw file, High-risk revenue looks like $310,000 (a negative typo hides $620,000): any later “improvement” would be measured from a false start.

More examples:

Code	Result	What it tells you
`df["SLA_Compliance_Pct"].mean()`	`96.48`	average service level today
`df["Billing_Discrepancy_Count"].mean()`	`1.73`	average disputes per account
`len(df[df["Client_Retention_Risk"] == "High"])`	`3`	accounts rated High risk
share of revenue in Low-risk accounts	`72.5`%	the safe part of the portfolio

Remember: audit first, then record the baseline.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S23_demo_1.py), and check both files are in data/.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors.
9. Hands-On Coding Micro-Examples:
- Every example has three parts: 1. Input, 2. Process, and 3. Output. Name each file with your own name and the session code, for example Reacher_S23_demo_1.py (pattern: YourName_S23_demo_1.py).
- Micro-Example 1 (Audit the Raw File): YourName_S23_demo_1.py
```
# 1. Input: the raw file, as it arrived
import pandas as pd
raw = pd.read_csv("data/session_02_03_raw.csv")

# 2. Process: the three checks, as counts
missing_values = raw.isnull().sum().sum()
duplicate_ids = raw["Account_ID"].duplicated().sum()
sla_above_100 = (raw["SLA_Compliance_Pct"] > 100).sum()
value_not_positive = (raw["Contract_Value_USD"] <= 0).sum()

# 3. Output: one line per check
print("Rows:", len(raw))
print("Missing values:", missing_values)
print("Duplicate account IDs:", duplicate_ids)
print("SLA above 100%:", sla_above_100)
print("Contract value of 0 or less:", value_not_positive)
```
  - How it works, line by line:
    - The script reads the raw file, as it came from the billing system, before anything is cleaned.
    - raw.isnull() marks every empty cell True. The first .sum() counts per column, and the second adds the columns together (section 7.2).
    - raw["Account_ID"].duplicated() marks a row True when its ID already appeared above; .sum() counts them.
    - (raw["SLA_Compliance_Pct"] > 100) and (raw["Contract_Value_USD"] <= 0) mark impossible values; .sum() counts them.
    - Part 3 prints one line per check, so the report reads like a checklist.
  - Read the output:
    Rows: 16 Missing values: 1 Duplicate account IDs: 1 SLA above 100%: 1 Contract value of 0 or less: 1
    - The file has 16 rows, but only 15 real contracts: one row is a copy.
    - Each check finds exactly 1 problem, 4 in total. Section 7.2 shows which accounts they are.
    - None of these problems raises an error by itself: the file loads normally. Only the checks reveal them.
- Quick Check 1 (.drop_duplicates()):
  - New tool: raw.drop_duplicates() returns the table without rows that exactly repeat an earlier row. Why: a copied row counts the same contract twice.
  - Task: add one line to part 3. Output: print("Rows after removing duplicates:", len(raw.________________()))
  - Check: the Terminal shows Rows after removing duplicates: 15.
- Micro-Example 2 (Test the Hypothesis on the Clean File): YourName_S23_demo_2.py
```
# 1. Input: the clean file
import pandas as pd
df = pd.read_csv("data/session_02_03_dataset.csv")

# 2. Process: mark SLA breaches, then compare the two groups
df["SLA_Breach"] = df["SLA_Compliance_Pct"] < 95.0
comparison = df.groupby("SLA_Breach").agg(
    Accounts=("Account_ID", "count"),
    Avg_Disputes=("Billing_Discrepancy_Count", "mean"),
).round(2)

# 3. Output: breach (True) vs no breach (False)
print(comparison)
```
  - How it works, line by line:
    - The script reads the clean file: analysis is done on data that passed the checks.
    - df["SLA_Compliance_Pct"] < 95.0 marks each account True (breach) or False, and stores the result as a new column, SLA_Breach.
    - df.groupby("SLA_Breach") splits the accounts into the 2 groups (Session 1.4). .agg() counts the accounts and averages the disputes in each group.
    - .round(2) keeps 2 decimals.
  - Read the output:
    Accounts Avg_Disputes SLA_Breach False 11 0.64 True 4 4.75
    - The 4 accounts below the 95% promise average 4.75 billing disputes; the other 11 average 0.64, about 7 times fewer.
    - The first part of the hypothesis is supported: poor service and billing disputes go together. Quick Check 2 tests the second part, the link with retention risk.
- Quick Check 2 (pd.crosstab()):
  - New tool: pd.crosstab(column_a, column_b) counts the rows for every pair of values, such as breach yes or no against risk Low, Medium, or High. Why: tests the second half of the hypothesis in one table.
  - Task: add one line to part 3. Output: print(pd.________(df["SLA_Breach"], df["Client_Retention_Risk"]))
  - Check: in the True row, High is 3 and Medium is 1: 3 of the 4 breached accounts are rated High risk, and no non-breached account is.
- Micro-Example 3 (Measure the Baseline): YourName_S23_demo_3.py
```
# 1. Input: the clean file
import pandas as pd
df = pd.read_csv("data/session_02_03_dataset.csv")

# 2. Process: contract value by risk rating, and the High-risk share
value_by_risk = df.groupby("Client_Retention_Risk")["Contract_Value_USD"].sum()
high_risk_value = value_by_risk["High"]
high_risk_share = high_risk_value / df["Contract_Value_USD"].sum() * 100

# 3. Output: today's baseline
print(value_by_risk)
print("High-risk share of revenue (%):", round(high_risk_share, 1))
```
  - How it works, line by line:
    - df.groupby("Client_Retention_Risk")["Contract_Value_USD"].sum() adds up contract value for each risk rating. Choosing one column after groupby() gives a simple list of totals instead of a table.
    - value_by_risk["High"] reads one value from that list, by its label.
    - The share divides the High-risk total by all contract value and multiplies by 100.
  - Read the output:
    Client_Retention_Risk High 930000.0 Low 7770000.0 Medium 2010000.0 Name: Contract_Value_USD, dtype: float64 High-risk share of revenue (%): 8.7
    - $930,000 a year, 8.7% of revenue, sits in accounts rated High risk. This is the baseline (section 7.5).
    - Most revenue ($7,770,000) is in Low-risk accounts, so the business is not in immediate danger, but the 3 High-risk accounts are where action pays off first.
    - The last line of the list, Name: ... dtype: float64, is pandas describing the list: its name and its type. It is not data.
- Quick Check 3 (f-string number format):
  - New tool: f"${value:,.0f}" writes a number with thousands separators and no decimals, such as $930,000. Why: managers read money faster in this form.
  - Task: add one line to part 3. Output: print(f"Revenue at High risk: ${high_risk_value:___}")
  - Check: the Terminal shows Revenue at High risk: $930,000.
10. Official Python Documentation & References:

11. Exercise 1: Guided Practice - Hypothesis Test Script:

Step 1: Open Script: In Google Antigravity IDE, create a new script named YourName_S23_hypothesis.py, replacing YourName with your own name (for example, Reacher_S23_hypothesis.py).

Step 2: Test Two Hypotheses and Measure the Baseline: Type the following code:

# 1. Input: the clean file
import pandas as pd
df = pd.read_csv("data/session_02_03_dataset.csv")

# 2. Process: two tests and one baseline
# Hypothesis 1: higher-risk accounts have more billing disputes
disputes_by_risk = (
    df.groupby("Client_Retention_Risk")["Billing_Discrepancy_Count"]
    .mean().round(2)
)
# Hypothesis 2: lower SLA compliance goes with more disputes
sla_vs_disputes = df["SLA_Compliance_Pct"].corr(
    df["Billing_Discrepancy_Count"]
)
# Baseline: revenue in High-risk accounts today
high_risk_value = df.loc[
    df["Client_Retention_Risk"] == "High", "Contract_Value_USD"
].sum()

# 3. Output: a short report
print("Hypothesis 1: average disputes by risk rating\n", disputes_by_risk)
print(f"Hypothesis 2: correlation of SLA and disputes: {sla_vs_disputes:.3f}")
print(f"Baseline: revenue at High risk: ${high_risk_value:,.0f}")

df.loc[rows, "Column_Name"] selects the rows and one column at the same time.
{sla_vs_disputes:.3f} shows the number with 3 decimals.

Step 3: Interpret: Write two comment lines at the end of the script: does each result support its hypothesis?
Step 4: Execute in Terminal: Run python YourName_S23_hypothesis.py in Panel [4] and confirm return code 0.
Success Check: average disputes are 5.33 for High, 2.0 for Medium, and 0.25 for Low; the correlation is -0.982 (lower SLA goes with more disputes); the baseline is $930,000. Both hypotheses are supported.

12. Exercise 2: Applied Practice - Data Gatekeeper:
- Coding Challenge: In YourName_S23_practice.py, write checks that stop bad data, then clean the raw file so it passes.
- Task 1 (Write the Rules): Load data/session_02_03_raw.csv. Write three assert lines with messages:
  - assert raw["SLA_Compliance_Pct"].between(0, 100).all(), "SLA must be between 0 and 100"
  - assert (raw["Contract_Value_USD"] > 0).all(), "Contract value must be above 0"
  - assert not raw["Account_ID"].duplicated().any(), "Account IDs must be unique"
  - .between(0, 100) is True when a value is from 0 to 100; a missing value counts as False.
- Task 2 (Read the AssertionError & AI Diagnosis): Run the script. It stops at the first rule with AssertionError: SLA must be between 0 and 100. Capture the traceback with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and ask which rows break the rule.
- Task 3 (Clean, Re-check, Save): Put the three assert lines inside try: and print the message under except AssertionError as error:. Then build clean from raw: remove duplicate rows with .drop_duplicates(), keep only rows where the SLA is between 0 and 100, and keep only rows where the contract value is above 0. Run the three checks again on clean, and save it to data/contracts_clean.csv with index=False.
- Success Check: the first run prints SLA must be between 0 and 100; clean has 12 rows and passes all three checks; the script ends with return code 0.
13. Exercise 3: Advanced Practice - How Much Did Bad Data Hide?:
- Goal: Show the account director, in numbers, how the raw file’s problems distort the risk report, and list exactly which accounts the data team must fix.
- What you need:
  - Files: data/session_02_03_raw.csv (with problems) and data/session_02_03_dataset.csv (the corrected version of the same contracts).
  - Script name: YourName_S23_challenge.py.
  - Tools you already know: the three checks, groupby(), .sum(), Boolean masks with |, .loc[], pd.DataFrame(), and f-strings.
- Requirements (follow them in order):
  1. Write a function audit(table) that returns a dictionary with four counts: "missing", "duplicate_ids", "sla_out_of_range" (SLA above 100), and "value_not_positive" (contract value of 0 or less).
  2. Call audit() on both files and print both dictionaries.
  3. Make a list of every Account_ID in the raw file that breaks at least one rule. Each account should appear only once, sorted from smallest to largest.
  4. For each file, sum Contract_Value_USD by Client_Retention_Risk. Put the two results side by side in one DataFrame with the columns Raw_USD and Correct_USD, and add Difference_USD = Raw_USD - Correct_USD.
  5. Print the table and one sentence that states the biggest distortion in plain words, using an f-string with thousands separators.
- New tools for this exercise:
  - Defining a function: def audit(table): followed by indented lines, ending with return {...}. Calling audit(raw) runs those lines on raw and gives back the dictionary.
  - | means “or” for masks: (rule_1) | (rule_2) is True when at least one rule is True. .isnull() on one column marks its missing values.
  - Unique, sorted values: sorted(set(raw.loc[mask, "Account_ID"])). set() keeps each value once; sorted() puts them in order.
  - Side-by-side columns: pd.DataFrame({"Raw_USD": raw_sums, "Correct_USD": correct_sums}), where both sums come from groupby(), lines up the values by risk rating.
- Hints:
  - For requirement 3, build one mask that is True for a duplicate, a missing value, an SLA above 100, or a value of 0 or less. Use raw.duplicated("Account_ID", keep=False) so that both copies of a duplicated account are marked.
  - Print each intermediate result before you combine them.
- Self-Check (expected results):
  - Raw file: {"missing": 1, "duplicate_ids": 1, "sla_out_of_range": 1, "value_not_positive": 1}; clean file: all four counts are 0.
  - Accounts to fix: ['ACC-304', 'ACC-306', 'ACC-311', 'ACC-313'].
  - Differences: High -620000.0, Low 1200000.0, Medium 0.0.
  - Business meaning: one negative typo (ACC-313) made High-risk revenue look like $310,000 instead of $930,000, hiding two-thirds of the real risk, while one duplicated row added $1,200,000 to Low risk. The data must be fixed before anyone acts on the risk report.

Session 2.4: Midterm Analytics Pipeline Defense Part 2: Exploratory Findings and Executive Review

Session 2.4 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 8 (115/11/01 - 115/11/07) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- An online marketplace earns a fee on every sale its sellers make, in three regions: North America, Europe, and Asia Pacific
- Big sales are not the same as cash in the bank: the region that sells the most may not be the one that generates the most cash
- The CFO (Chief Financial Officer) asks: which region turns its sales into the most cash, and where should we invest next year? Today, you answer with one script that anyone can rerun, and present your midterm findings in the same way
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “Can someone else get the same numbers?”: build a reproducible pipeline (one script that goes from the raw file to the final report with no manual steps), organized into functions
2. “Which region turns sales into the most cash?”: calculate revenue and the operating cash flow margin by region and quarter
3. “How do I explain it to the CFO?”: write findings as finding, evidence, and recommendation; review a classmate’s script; and fix a TypeError (mixing text and numbers)

3. Business Context: How a Marketplace Makes Money and Cash:

Four terms, one example (North America, 2024-Q1):

Term	Meaning	Example
GMV (Gross Merchandise Value)	the total value of everything sold on the platform	`145.2` USD M
Take rate	the share of GMV the platform keeps as its fee	`12.5`%
Revenue	GMV x take rate / 100: the platform’s own income	`18.15` USD M
OCF (Operating Cash Flow)	the cash the business actually generated after paying its operating costs	`5.6` USD M

OCF margin = OCF / revenue x 100. It answers “of every dollar of revenue, how many cents became cash?” Example: 5.6 / 18.15 x 100 = 30.9%.
Midterm Project, Step 2 of 2: present your findings. In a short presentation, walk through:
1. Your Question Card from Session 2.3 (worry, question, hypothesis, data, baseline)
2. Your audit result: what you checked and what you fixed
3. Your findings, each written in three parts:
  - Finding: one sentence a manager can repeat. “North America turns the most revenue into cash.”
  - Evidence: the number behind it. “Its OCF margin is 31.0%, against 24.7% in Europe and 18.5% in Asia Pacific.”
  - Recommendation: what to do. “Study North America’s cost structure before expanding in Asia Pacific.”
4. A live run of your script, from the raw file to the final report

4. Data Basics: What Makes a Pipeline Reproducible:

A reproducible pipeline runs from top to bottom and produces the same numbers on any computer. It has four stages:

Stage	What it does	Example in today’s script
Load	Read the raw file	`pd.read_csv("data/session_02_04_dataset.csv")`
Check	Stop if the data breaks a rule	`assert (df["Gross_Merchandise_Value_USD_M"] > 0).all()`
Calculate	Add the measures	revenue and OCF margin
Report	Print and save the result	`summary.to_csv("data/region_summary.csv")`

Three habits that make a script reproducible:
- Use relative paths such as data/..., never paths that only exist on your computer, such as C:/Users/Reacher/Desktop/...
- Calculate every number from the data; never type a result into the code by hand
- Before presenting, delete your output files and run the script again from the start

5. Lab Practice Dataset:
- File Link: data/session_02_04_dataset.csv
- Audited 12-row quarterly marketplace dataset (fictional teaching data): 3 regions x 4 quarters, one row per region and quarter:
  - Quarter_Code: Reporting quarter, 2024-Q1 through 2024-Q4.
  - Market_Segment: Region: North_America, Europe, or Asia_Pacific.
  - Gross_Merchandise_Value_USD_M: GMV (Gross Merchandise Value) in millions of U.S. Dollars.
  - Platform_Take_Rate_Pct: The platform’s fee as a percentage of GMV.
  - Operating_Cash_Flow_USD_M: OCF (Operating Cash Flow) in millions of U.S. Dollars.
6. Keywords (Core Programming Concepts):
- Reproducible Pipeline:
  - What: one script that goes from the raw file to the final report, with no manual steps
  - Why: anyone can rerun it and check your numbers
- Function (def / return):
  - What: a named block of code that takes inputs and gives back a result
  - Why: each pipeline stage gets a clear name, and you can reuse it
- Relative Path:
  - What: a file path that starts from the project folder, such as data/session_02_04_dataset.csv
  - Why: works on every computer that has the project folder
- Data Type and TypeError:
  - What: text (str) and numbers (float, int) are different types; "Total: " + 204.3 raises a TypeError
  - Why: printing a result is where this mistake happens most
- Executive Finding:
  - What: a finding, its evidence, and a recommendation
  - Why: managers act on clear statements backed by numbers, not on code
- Peer Review:
  - What: a classmate checks your script against a short checklist
  - Why: a second reader catches mistakes the author no longer sees

7. Core Programming & Computational Foundations:

7.1 Functions: name a step, then reuse it

The idea: A function is a named block of code. You give it inputs (parameters), it does its work, and return hands back the result. Once a step is a function, you can run it again with one line, on any data, and a reader can understand the pipeline from the function names alone.
```
def revenue_usd_m(gmv, take_rate_pct):
    return gmv * take_rate_pct / 100
```
def starts the function, the name follows, then the parameters in brackets and a colon. The indented lines are the body.

See it on real data:

Call	Result	What happened
`revenue_usd_m(145.2, 12.5)`	`18.15`	North America, Q1: `145.2 x 12.5 / 100`
`revenue_usd_m(160.8, 13.0)`	`20.9` (rounded)	North America, Q4: same function, new inputs
`revenue_usd_m(df["Gross_Merchandise_Value_USD_M"], df["Platform_Take_Rate_Pct"])`	12 values	the same function works on whole columns

Common mistake 1, forgetting return: a function without return gives back None, Python’s “nothing”. The calculation happens, but the result is thrown away.
Common mistake 2, using an inside name outside: a variable created inside a function exists only there. After def f(x): y = x * 2, the line print(y) outside the function raises NameError: name 'y' is not defined. Use return y and store the result: y = f(3).

More examples:

Function	Call	Result
`def ocf_margin(ocf, revenue): return round(ocf / revenue * 100, 1)`	`ocf_margin(5.6, 18.15)`	`30.9`
`def load_data(path): return pd.read_csv(path)`	`load_data("data/session_02_04_dataset.csv")`	the 12-row DataFrame
`def is_weak(margin, limit=20): return margin < limit`	`is_weak(18.6)`	`True`; `limit` uses its default value, `20`
the same	`is_weak(18.6, limit=15)`	`False`; a different limit passed in

Remember: def to define, return to hand back, and store the result when you call it.

7.2 A reproducible pipeline: load, check, calculate, report

The idea: A pipeline is one script that goes from the raw file to the final report with no manual step in between. “Reproducible” means anyone can run it again, on any computer, and get exactly the same numbers. That is what lets a manager trust your midterm results.

Stage	Function in Micro-Example 2	Why it is separate
Load and check	`load_data(path)`	bad data stops here, before any number is calculated
Calculate	`add_metrics(df)`	one place for every formula
Summarize	`summarize(df)`	the table the manager reads
Report	`print(summary)` and `.to_csv()`	the result on screen and in a file

Three habits: use relative paths such as data/... (a path like C:/Users/Reacher/Desktop/... only exists on one computer); calculate every number from the data, never type a result by hand; before you present, delete the output files and run the script again from the top.
Common mistake: running cells or lines out of order while developing, so the script works on your screen but fails when run from the start. The test of a pipeline is a clean run, top to bottom.

More examples:

Path or habit	Works on	Verdict
`"data/session_02_04_dataset.csv"`	every computer with the project folder	relative path: use this
`"C:/Users/Reacher/Desktop/data.csv"`	only Reacher’s computer	absolute path: breaks for everyone else
a total typed into the code as `204.3`	only until the data changes	hard-coded result: calculate it instead
output files left from an earlier run	hides missing steps	delete outputs, then rerun from the top

Remember: one script, top to bottom, same numbers every time.

7.3 Text and numbers do not mix: TypeError

The idea: Python keeps text (str) and numbers (int, float) apart. + adds two numbers or joins two texts, but it refuses to combine a text with a number, because it cannot know whether you want to add or to join.

See it on real data:

Code	Result
`"Total: " + 204.3`	`TypeError: can only concatenate str (not "float") to str`
`"Rows: " + len(df)`	`TypeError: can only concatenate str (not "int") to str`
`"Total: " + str(204.3)`	`Total: 204.3`
`f"Total: {204.3}"`	`Total: 204.3`

“Concatenate” means “join texts together”.

Formatting numbers inside f-strings: after the value, a colon and a format code control how the number looks.

Code	Result	Format
`f"{204.3049:.2f}"`	`204.30`	2 decimals
`f"{1234567.8:,.0f}"`	`1,234,568`	thousands separators, no decimals
`f"{0.2467:.1%}"`	`24.7%`	a share shown as a percentage

More examples:

Code	Result	What it tells you
`str(12) + " stores"`	`12 stores`	convert the number to text first
`f"{12} rows"`	`12 rows`	an f-string converts for you
`"Q" + str(4)`	`Q4`	build a label from text and a number
`int("12") + 1`	`13`	convert text to a number before adding
`float("18.15") * 2`	`36.3`	decimal text becomes a `float`
`"5" * 3`	`555`	a surprise: text times a number repeats the text

Remember: put numbers into text with an f-string, and format them for the reader.

7.4 Hard-coded numbers: believable but wrong

The idea: A hard-coded number is a value typed directly into the code, such as 0.125 for the take rate. It is risky because it looks reasonable and produces believable results, even when the real values differ from row to row.

See it on real data: revenue for the year, by region, with the real take rate of each row vs a fixed 12.5%.

Region	With each row’s take rate	With a fixed `0.125`	Error
North America	`65.57`	`67.16`	`+1.59`
Europe	`68.34`	`69.28`	`+0.94`
Asia Pacific	`70.39`	`70.49`	`+0.10`

Our take rates range from 10.5% to 13.5%, so no single number is right. Nothing in the output warns you: the error is easy to miss.

More examples:

What you need	From the data	The hard-coded risk
total revenue for the year	`204.31`	`206.92` with a fixed `0.125`
number of quarters	`len(df)` reads `12`	typing `12` breaks when a 13th row arrives
list of regions	`df["Market_Segment"].unique()`	a typed list misses a new region
a business rule	`MARGIN_LIMIT = 20` at the top of the script	a named rule is visible and easy to change

Remember: read every input from the data; type a number into code only when it is a rule, and give it a name, such as MARGIN_LIMIT = 20.

7.5 Writing findings for managers

The idea: Managers act on clear statements, not on code. Write each finding in three parts: the finding (one sentence a manager can repeat), the evidence (the number behind it), and the recommendation (what to do).

See it on real data:

Part	Weak version	Strong version
Finding	“Margins vary by region.”	“North America turns the most revenue into cash.”
Evidence	“See the table.”	“Its OCF margin is 31.0%, against 24.7% in Europe and 18.5% in Asia Pacific.”
Recommendation	“Look into it.”	“Study North America’s cost structure before expanding in Asia Pacific.”

Common mistake: averaging percentages. The 4 quarterly margins of Asia Pacific average 18.1%, but its true margin for the year, total OCF / total revenue, is 18.5%. Always compute a group’s ratio from its totals.

More examples:

Finding	Evidence	Recommendation
Europe has a mid-level cash margin.	`24.7`%, between North America and Asia Pacific	Keep Europe’s investment level unchanged.
Asia Pacific is improving fast.	its margin rose from `15.0`% in Q1 to `22.0`% in Q4: `+7.0` points	Keep investing, with a margin target for next year.
Q4 was the strongest quarter.	GMV `524.4` and OCF `17.9`, both the year’s highest	Plan capacity for next year’s Q4 peak.

Remember: finding, evidence, recommendation, and every number from the data.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S24_demo_1.py), and check the dataset is in data/.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors.
9. Hands-On Coding Micro-Examples:
- Every example has three parts: 1. Input, 2. Process, and 3. Output. Name each file with your own name and the session code, for example Reacher_S24_demo_1.py (pattern: YourName_S24_demo_1.py).
- Micro-Example 1 (From GMV to Cash, Row by Row): YourName_S24_demo_1.py
```
# 1. Input: load the quarterly results
import pandas as pd
df = pd.read_csv("data/session_02_04_dataset.csv")

# 2. Process: revenue, then OCF margin, for every row
df["Revenue_USD_M"] = (
    df["Gross_Merchandise_Value_USD_M"] * df["Platform_Take_Rate_Pct"] / 100
).round(2)
df["OCF_Margin_Pct"] = (
    df["Operating_Cash_Flow_USD_M"] / df["Revenue_USD_M"] * 100
).round(1)

# 3. Output: the 3 rows with the lowest margin
lowest = df.sort_values("OCF_Margin_Pct").head(3)
print(lowest[["Quarter_Code", "Market_Segment", "OCF_Margin_Pct"]])
```
  - How it works, line by line:
    - The first calculation multiplies each row’s GMV by that row’s take rate and divides by 100: revenue in USD millions. No take rate is typed by hand (section 7.4).
    - The second divides each row’s OCF by its revenue: the OCF margin, the share of revenue that became cash. .round(1) keeps 1 decimal.
    - df.sort_values("OCF_Margin_Pct") orders the rows from the lowest margin to the highest, and .head(3) keeps the 3 lowest.
  - Read the output:
    Quarter_Code Market_Segment OCF_Margin_Pct 8 2024-Q1 Asia_Pacific 15.0 5 2024-Q2 Asia_Pacific 16.8 2 2024-Q3 Asia_Pacific 18.6
    - The 3 weakest quarters all belong to Asia Pacific: in Q1, only 15.0 cents of every revenue dollar became cash.
    - The margins rise from Q1 to Q3: Asia Pacific is weak but improving. Quick Check 1 shows Q4 too.
- Quick Check 1 (.pivot_table()):
  - New tool: df.pivot_table(index="A", columns="B", values="C") makes a grid with one row per value of A and one column per value of B, like an Excel pivot table. Why: shows all 12 margins in one readable grid.
  - Task: add one line to part 3. Output: print(df.___________(index="Market_Segment", columns="Quarter_Code", values="OCF_Margin_Pct"))
  - Check: the Asia_Pacific row reads 15.0, 16.8, 18.6, 22.0: its margin improved every quarter.
- Micro-Example 2 (The Same Work as a Pipeline of Functions): YourName_S24_demo_2.py
```
# 1. Input: a function that loads and checks the raw file
import pandas as pd

def load_data(path):
    df = pd.read_csv(path)
    assert (df["Gross_Merchandise_Value_USD_M"] > 0).all(), "GMV must be above 0"
    return df

# 2. Process: functions that add measures and summarize by region
def add_metrics(df):
    df["Revenue_USD_M"] = (
        df["Gross_Merchandise_Value_USD_M"] * df["Platform_Take_Rate_Pct"] / 100
    ).round(2)
    return df

def summarize(df):
    summary = df.groupby("Market_Segment").agg(
        Revenue_USD_M=("Revenue_USD_M", "sum"),
        OCF_USD_M=("Operating_Cash_Flow_USD_M", "sum"),
    )
    summary["OCF_Margin_Pct"] = (
        summary["OCF_USD_M"] / summary["Revenue_USD_M"] * 100
    ).round(1)
    return summary

# 3. Output: run the stages in order, then print and save
df = load_data("data/session_02_04_dataset.csv")
df = add_metrics(df)
summary = summarize(df)
print(summary)
summary.to_csv("data/region_summary.csv")
```
  - How it works, line by line:
    - load_data(path) reads the file and checks with assert that every GMV is above 0; bad data stops the pipeline here (section 7.2).
    - add_metrics(df) adds the revenue column and returns the table.
    - summarize(df) groups by region, sums revenue and OCF, and computes each region’s margin from its totals (section 7.5).
    - The last 5 lines run the stages in order. Read them alone and you know what the whole script does: load, add metrics, summarize, print, save.
    - summary.to_csv("data/region_summary.csv") saves the result so the numbers can be checked later.
  - Read the output:
    Revenue_USD_M OCF_USD_M OCF_Margin_Pct Market_Segment Asia_Pacific 70.39 13.0 18.5 Europe 68.34 16.9 24.7 North_America 65.57 20.3 31.0
    - Asia Pacific earns the most revenue (70.39) but keeps the least as cash (18.5%).
    - North America earns the least revenue (65.57) but generates the most cash (20.3), with a margin of 31.0%.
    - Bigger is not the same as better: the CFO’s question needs both columns.
- Quick Check 2 (Docstring):
  - New tool: a docstring, a text in triple quotes on the first line inside a function. help(function_name) prints it. Why: a reviewer learns what each stage does without reading its code.
  - Task: add one line directly under def add_metrics(df):, indented like the line below it: """Add revenue in USD M from GMV and the take rate.""". Then add help(add_metrics) to part 3. Output.
  - Check: the Terminal prints a help page that includes Add revenue in USD M from GMV and the take rate.
- Micro-Example 3 (Write the Findings for the CFO): YourName_S24_demo_3.py
```
# 1. Input: load the data and build the region summary
import pandas as pd
df = pd.read_csv("data/session_02_04_dataset.csv")
df["Revenue_USD_M"] = (
    df["Gross_Merchandise_Value_USD_M"] * df["Platform_Take_Rate_Pct"] / 100
).round(2)
summary = df.groupby("Market_Segment").agg(
    Revenue_USD_M=("Revenue_USD_M", "sum"),
    OCF_USD_M=("Operating_Cash_Flow_USD_M", "sum"),
)

# 2. Process: margin, then the best region by margin
summary["OCF_Margin_Pct"] = (
    summary["OCF_USD_M"] / summary["Revenue_USD_M"] * 100
).round(1)
ranked = summary.sort_values("OCF_Margin_Pct", ascending=False)
best_region = ranked.index[0]
best_margin = ranked["OCF_Margin_Pct"].iloc[0]

# 3. Output: finding, evidence, recommendation
print(f"Finding: {best_region} turns the most revenue into cash.")
print(f"Evidence: its OCF margin is {best_margin}%.")
print("Recommendation: study its cost structure before expanding elsewhere.")
```
  - How it works, line by line:
    - Part 1 rebuilds the region summary, as in Micro-Example 2.
    - summary.sort_values("OCF_Margin_Pct", ascending=False) puts the best margin first.
    - ranked.index[0] reads the first row label, here the name of the best region. After groupby(), the region names are the row labels.
    - ranked["OCF_Margin_Pct"].iloc[0] reads the first value of the margin column by position.
    - The three print() lines write the finding, the evidence, and the recommendation (section 7.5), with the numbers filled in by f-strings.
  - Read the output:
    Finding: North_America turns the most revenue into cash. Evidence: its OCF margin is 31.0%. Recommendation: study its cost structure before expanding elsewhere.
    - The region and the margin come from the data, not from typing: if next year’s file changes, the sentence changes with it.
    - The recommendation line is still written by a person. The script supplies the evidence; the judgement stays with the analyst.
- Quick Check 3 (.idxmax()):
  - New tool: column.idxmax() returns the row label of the largest value, in one step. Why: finds “which region?” without sorting.
  - Task: add one line to part 3. Output: print("Most revenue:", summary["Revenue_USD_M"].______())
  - Check: the Terminal shows Most revenue: Asia_Pacific. The region with the most revenue is not the one with the best margin.
10. Official Python Documentation & References:
11. Exercise 1: Guided Practice - Quarterly Review Pipeline:
- Step 1: Open Script: In Google Antigravity IDE, create a new script named YourName_S24_defense.py, replacing YourName with your own name (for example, Reacher_S24_defense.py).
- Step 2: Build the Quarterly Review: Type the following code:
```
# 1. Input: load the quarterly results
import pandas as pd
df = pd.read_csv("data/session_02_04_dataset.csv")

# 2. Process: revenue per row, then totals per quarter
df["Revenue_USD_M"] = (
    df["Gross_Merchandise_Value_USD_M"] * df["Platform_Take_Rate_Pct"] / 100
).round(2)
quarterly = df.groupby("Quarter_Code").agg(
    GMV_USD_M=("Gross_Merchandise_Value_USD_M", "sum"),
    Revenue_USD_M=("Revenue_USD_M", "sum"),
    OCF_USD_M=("Operating_Cash_Flow_USD_M", "sum"),
)
quarterly["OCF_Margin_Pct"] = (
    quarterly["OCF_USD_M"] / quarterly["Revenue_USD_M"] * 100
).round(1)

# 3. Output: one row per quarter
print("Quarterly review:\n", quarterly)
```
- Step 3: Write Two Findings: Under the code, write two comment lines in the three-part form (finding, evidence, recommendation): one about the quarter with the highest GMV, and one about how the total OCF margin changed during the year.
- Step 4: Execute in Terminal: Run python YourName_S24_defense.py in Panel [4] and confirm return code 0.
- Success Check: 2024-Q4 has the highest GMV (524.4) and OCF (17.9); Q2 has the lowest GMV (312.4). The total OCF margin is 23.7% in Q1, 23.7% in Q2, 24.6% in Q3, and 25.8% in Q4.
12. Exercise 2: Applied Practice - Peer Review a Classmate’s Script:
- Coding Challenge: A classmate sends you this script for their midterm. Copy it into YourName_S24_practice.py, review it, and fix it.
```
# Classmate's script: this script has mistakes on purpose
import pandas as pd
df = pd.read_csv("data/session_02_04_dataset.csv")
df["Revenue_USD_M"] = df["Gross_Merchandise_Value_USD_M"] * 0.125
total_revenue = df["Revenue_USD_M"].sum()
print("Total revenue (USD M): " + total_revenue)
```
- Task 1 (Run It & AI Diagnosis): Run the script. It stops with TypeError: can only concatenate str (not "numpy.float64") to str. Capture the traceback with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and ask why text and a number cannot be joined with +. (numpy.float64 is the number type that pandas returns; it behaves like a normal float.)
- Task 2 (Fix the TypeError): Change the last line to an f-string, rounding to 2 decimals: f"Total revenue (USD M): {total_revenue:.2f}".
- Task 3 (Apply the Peer Review Checklist): Check the script against these five questions, and fix every “no”:
  1. Does it run from top to bottom without errors?
  2. Does it use a relative path (data/...)?
  3. Is every number calculated from the data, not typed in by hand?
  4. Are outputs labeled with units?
  5. Would a manager understand the printed result?
- Success Check: the hard-coded 0.125 fails question 3, because each row has its own take rate. With df["Platform_Take_Rate_Pct"] / 100, the script prints Total revenue (USD M): 204.31. The hard-coded version prints 206.93: a believable number that is wrong.
13. Exercise 3: Advanced Practice - Where Should We Invest Next Year?:
- Goal: Build the region report the CFO needs for next year’s budget: size, cash efficiency, and trend for every region, plus a list of weak quarters.
- What you need:
  - File: data/session_02_04_dataset.csv (described in 5. Lab Practice Dataset).
  - Script name: YourName_S24_challenge.py.
  - Tools you already know: functions with def / return, groupby() + .agg(), .pivot_table(), .loc[], .sort_values(), f-strings, and .to_csv().
- Requirements (follow them in order):
  1. Write load_data(path) that reads the file and checks with assert that GMV, take rate, and OCF are all above 0.
  2. Write add_metrics(df) that adds Revenue_USD_M (2 decimals) and OCF_Margin_Pct (1 decimal) to every row.
  3. Write region_report(df) that returns one row per region with GMV_USD_M, Revenue_USD_M, OCF_USD_M (sums), OCF_Margin_Pct (from the sums, 1 decimal), and Margin_Change_Pts: the region’s Q4 margin minus its Q1 margin, in percentage points, 1 decimal.
  4. Write weak_quarters(df, limit) that returns the rows with OCF_Margin_Pct below limit, sorted from lowest to highest. Call it with limit=20.
  5. Sort the region report by OCF_Margin_Pct, highest first. Print it, print the weak quarters, and save the report to data/region_investment_report.csv.
  6. Print one recommendation line built with an f-string from the report’s numbers (not typed in by hand).
- New tools for this exercise:
  - Percentage points: the difference between two percentages. A margin moving from 15.0% to 22.0% rose by 7.0 percentage points (pts), not by 7%.
  - Reading one cell of a pivot table: margins = df.pivot_table(index="Market_Segment", columns="Quarter_Code", values="OCF_Margin_Pct"), then margins["2024-Q4"] - margins["2024-Q1"] gives the change for every region at once.
  - Adding a column that lines up by row label: report["Margin_Change_Pts"] = (margins["2024-Q4"] - margins["2024-Q1"]).round(1) works because both use the region names as row labels.
  - A function parameter with a value: weak_quarters(df, limit=20) passes 20 into limit.
- Hints:
  - Build and print each function’s result before you write the next function.
  - The margin of a region must come from its total OCF divided by its total revenue, not from averaging the 4 quarterly margins.
- Self-Check (expected results):
  - Report order: North_America (margin 31.0, change +0.2 pts), Europe (24.7, -1.1 pts), Asia_Pacific (18.5, +7.0 pts).
  - Weak quarters (margin below 20%): Asia_Pacific in 2024-Q1 (15.0), 2024-Q2 (16.8), and 2024-Q3 (18.6).
  - Business meaning: Asia Pacific is the biggest region by GMV (563.9) and revenue, but the least cash-efficient, and also the fastest-improving (+7.0 pts). North America is the cash engine. A sound recommendation: keep investing in Asia Pacific’s growth, tied to a target such as reaching North America’s cost structure, and protect North America’s margin.

Module 3: Artificial Intelligence, Machine Learning Fundamentals, and Scikit-Learn

Module 3 introduces managerial artificial intelligence and machine learning frameworks. Students explore how machine learning models generate enterprise value, prepare data through feature engineering, build predictive models using Scikit-Learn, and evaluate model trade-offs between precision, recall, and financial cost.

Session 3.1: Foundations of Artificial Intelligence: Enterprise Value Creation and Decision Automation

Session 3.1 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 9 (115/11/08 - 115/11/14) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- Five departments have proposed 10 AI projects, costing between $80,000 and $220,000 each
- The total request is $1.5 million, but this year’s AI budget is only $500,000
- The COO (Chief Operating Officer) asks: which AI projects should we fund with $500,000, and how fast will they pay back? Today, you learn what AI can and cannot do, and use Python to compare the proposals on the same financial terms
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “Is this project really AI?”: tell apart AI (Artificial Intelligence), ML (Machine Learning), and DL (Deep Learning), and decide when a simple rule is enough
2. “Which projects pay back fastest?”: calculate the payback period (months until the savings repay the cost) and the 3-year ROI (Return on Investment)
3. “Which projects fit in $500,000?”: allocate a fixed budget with a loop, and fix a ZeroDivisionError (dividing by zero)

3. Business Context: How to Judge an AI Project:

Every AI proposal is an investment: money now, savings later. Five numbers make proposals comparable (example: AI-PRJ-101, Supply Chain):

Term	Meaning	Example
Implementation cost	one-time cost to build the system	`150,000`
Running cost	yearly cost to keep it working (cloud computing, updates); here 15% of the build cost	`22,500`
Net annual benefit	yearly savings minus the running cost	`294,000 - 22,500 = 271,500`
Payback period	months until the benefit repays the build cost: cost / net benefit x 12	`6.6` months
3-year ROI	(net benefit x 3 - cost) / cost x 100	`443.0`%

Savings are forecasts, not facts. Two habits keep forecasts honest: include running costs, and compare every project with the same formula.
Labor hours saved are not the same as money saved: hours only become savings if the freed time is used for other valuable work.

4. Data Basics: AI, ML, DL, and When a Rule Is Enough:

Three nested ideas:

Term	What it is	Business example
AI (Artificial Intelligence)	any computer system that does a task we link to human thinking	a chatbot answering customers
ML (Machine Learning)	AI that learns a rule from past examples instead of being told the rule	flagging likely fraud from past fraud cases
DL (Deep Learning)	ML with many-layered neural networks, for images, sound, and text	reading scanned invoices

Rule or ML? Ask how stable and clear the decision is:
- Use a rule when the policy is fixed and written down: “sales tax is 5% of the price”. A rule is cheaper, exact, and easy to explain.
- Use ML when the pattern is complex or changes over time, and you have many past examples with known outcomes: “which invoices look like fraud?”
ML makes a prediction; a person or a business rule still decides what to do with it.

5. Lab Practice Dataset:
- File Link: data/session_03_01_dataset.csv
- Audited 10-row AI project proposal dataset (fictional teaching data), one row per proposal:
  - Project_ID: Proposal code (e.g., AI-PRJ-101).
  - Department_Name: Sponsoring department (Supply Chain, Customer Service, Marketing, Finance, Fraud Operations).
  - Implementation_Cost_USD: One-time build cost in U.S. Dollars.
  - Annual_Labor_Hours_Saved_Count: Staff hours the system is expected to save per year.
  - Projected_Cost_Savings_USD: Expected yearly savings in U.S. Dollars.
  - Payback_Period_Months_Count: The department’s own payback estimate. It ignores running costs, so it is shorter than the payback you calculate today (for example, 6.1 against 6.6 months for AI-PRJ-101).
6. Keywords (Core Programming Concepts):
- Artificial Intelligence (AI):
  - What: computer systems that do tasks we link to human thinking
  - Why: the umbrella term in every vendor pitch; ask which kind of AI is inside
- Machine Learning (ML):
  - What: a system that learns a decision rule from past examples
  - Why: handles patterns too complex to write down by hand
- Deep Learning (DL):
  - What: ML with many-layered neural networks
  - Why: needed for images, sound, and free text; costly to build and run
- Rule-Based System:
  - What: code that follows rules a person wrote, such as if amount > 250000:
  - Why: the right choice when the policy is clear and fixed
- Payback Period:
  - What: months until the net benefit repays the build cost
  - Why: shows how fast the money comes back
- ROI (Return on Investment):
  - What: total gain compared with the cost, in percent
  - Why: compares projects of different sizes on one scale

7. Core Programming & Computational Foundations:

7.1 Rule or learning: two ways to make a decision

The idea: A computer can decide in two ways. In a rule-based system, a person writes the rule, such as “flag every invoice above $250,000”. In a machine learning system, the computer finds the rule itself by trying many candidate rules on past examples with known answers and keeping the one that makes the fewest mistakes. Machine learning is useful when the right rule is hard to see by eye; a rule is better when the policy is clear and fixed.

See it on real data: the 10 past invoices in Micro-Example 2 (5 honest, 5 fraud).

Approach	The rule	Mistakes on the 10 invoices
Rule-based	a person chose “above 250,000”	`1` (the honest `260000` invoice is flagged)
Learned	the search tried every amount as a threshold and found “300,000 or more”	`0`

Common mistake: treating zero mistakes on past data as a promise. A rule fitted to old examples can miss tomorrow’s new pattern. Session 3.3 tests models on data they have not seen.

More examples:

Decision	Rule or ML?	Why
Add 5% sales tax to every order	rule	the policy is written down and never guesses
Approve expenses under $500 automatically	rule	a clear, fixed company policy
Which invoices look like fraud?	ML	the pattern is complex and changes as fraudsters adapt
Which customers will cancel next month?	ML	many signals combine in ways no one can write down
Read the amount on a scanned invoice	ML (deep learning)	images need many-layered models

Remember: clear, fixed policy: write a rule; complex pattern with many past answers: learn it.

7.2 Payback and ROI: comparing investments on the same terms

The idea: Every proposal is money now for savings later. To compare proposals fairly, apply the same formulas to all of them, including the yearly running cost (here 15% of the build cost).

Measure	Formula	`AI-PRJ-101`
Net annual benefit	savings - 0.15 x cost	`294000 - 22500 = 271500`
Payback (months)	cost / net benefit x 12	`150000 / 271500 x 12 = 6.6`
3-year ROI (%)	(net benefit x 3 - cost) / cost x 100	`443.0`

Common mistake: ignoring the running cost. The department’s own estimate for AI-PRJ-101 is 6.1 months, which is 150000 / 294000 x 12: it leaves out the $22,500 a year needed to keep the system working.

More examples:

Project	Cost	Net benefit	Payback	3-year ROI
`AI-PRJ-110` (Fraud Operations)	`190000`	`515500`	`4.4` months	`713.9`%
`AI-PRJ-102` (Customer Service)	`80000`	`143000`	`6.7` months	`436.2`%
`AI-PRJ-103` (Marketing)	`120000`	`178000`	`8.1` months	`345.0`%
`AI-PRJ-104` (Finance)	`200000`	`362000`	`6.6` months	`443.0`%

The cheapest project (AI-PRJ-102) is not the best return, and the most expensive one (AI-PRJ-110) pays back fastest: size alone says little.

Remember: same formula for every project, running costs included.

7.3 Division by zero: an error in Python, a silent inf in pandas

The idea: Dividing by zero has no answer. Python and pandas react differently:

Code	Result
`150000.0 / 0.0`	stops: `ZeroDivisionError: float division by zero`
`pd.Series([4200.0]) / pd.Series([0.0])`	`inf` (infinity), with no error

Why the pandas case is dangerous: inf counts as larger than every real number, so a project with zero cost would jump to the top of an “hours per dollar” ranking: sorting [3.0, inf, 1.0] from largest gives [inf, 3.0, 1.0].
The fix: check before dividing, for example if annual_benefit > 0:; for columns, remove or flag zero values before you divide.

More examples:

Code	Result	What it tells you
`pd.Series([10.0, 20.0]) / pd.Series([2.0, 0.0])`	`[5.0, inf]`	the second division silently becomes infinity
`pd.Series([0.0]) / pd.Series([0.0])`	`[NaN]`	zero divided by zero is “not a number”
`np.isinf(pd.Series([5.0, np.inf]))`	`[False, True]`	find the infinite values
`150000.0 / 0.0`	`ZeroDivisionError`	plain Python stops instead

Remember: a zero in the bottom of a fraction must be caught, not ignored.

7.4 Choosing under a budget: the greedy method

The idea: With more good projects than money, you must choose. The greedy method is the simplest plan: sort the projects by value per dollar, go down the list, and fund each project that still fits in the remaining budget.

See it on real data: sorting by hours saved per $1,000 and funding within $500,000.

Step	Project	Cost	Fits?	Budget used
1	`AI-PRJ-102`	`80000`	yes	`80000`
2	`AI-PRJ-107`	`90000`	yes	`170000`
3	`AI-PRJ-110`	`190000`	yes	`360000`
4	`AI-PRJ-105`	`175000`	no (`535000`)	`360000`
…	…	…	…	…
last	`AI-PRJ-103`	`120000`	yes	`480000`

Result: 4 projects, 16,100 hours a year.

Common mistake: assuming greedy is always best. It never looks back, so it can fill the last gap with a weak project. Exercise 3 checks every combination and finds 16,900 hours for $495,000.

More examples: the result depends on what you sort by.

Greedy order	Funded (`AI-PRJ-...`)	Cost and hours
hours saved per $1,000 (Exercise 2)	`102`, `107`, `110`, `103`	`$480,000`, `16,100` hours
3-year ROI	`110`, `105`, `102`	`$445,000`, `16,100` hours
most hours first	`110`, `105`, `107`	`$455,000`, `16,400` hours
cheapest first	`102`, `107`, `103`, `108`	`$420,000`, `12,300` hours

Remember: greedy is fast and simple; with few options, check the combinations too.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S31_demo_1.py), and check the dataset is in data/.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors.
9. Hands-On Coding Micro-Examples:
- Every example has three parts: 1. Input, 2. Process, and 3. Output. Name each file with your own name and the session code, for example Reacher_S31_demo_1.py (pattern: YourName_S31_demo_1.py).
- Micro-Example 1 (Payback and ROI for Every Proposal): YourName_S31_demo_1.py
```
# 1. Input: load the 10 proposals
import pandas as pd
df = pd.read_csv("data/session_03_01_dataset.csv")

# 2. Process: net benefit, payback, and 3-year ROI
df["Net_Annual_Benefit_USD"] = (
    df["Projected_Cost_Savings_USD"] - df["Implementation_Cost_USD"] * 0.15
)
df["Payback_Months"] = (
    df["Implementation_Cost_USD"] / df["Net_Annual_Benefit_USD"] * 12
).round(1)
df["ROI_3yr_Pct"] = (
    (df["Net_Annual_Benefit_USD"] * 3 - df["Implementation_Cost_USD"])
    / df["Implementation_Cost_USD"] * 100
).round(1)

# 3. Output: the 3 proposals with the highest ROI
top3 = df.sort_values("ROI_3yr_Pct", ascending=False).head(3)
print(top3[["Project_ID", "Department_Name", "Payback_Months", "ROI_3yr_Pct"]])
```
  - How it works, line by line:
    - df["Implementation_Cost_USD"] * 0.15 is the yearly running cost; subtracting it from the projected savings gives the net annual benefit, for all 10 proposals at once.
    - The payback divides the build cost by the net benefit and multiplies by 12 to turn years into months.
    - The 3-year ROI compares 3 years of net benefit with the build cost, in percent (section 7.2).
    - sort_values("ROI_3yr_Pct", ascending=False).head(3) keeps the 3 best returns.
  - Read the output:
    Project_ID Department_Name Payback_Months ROI_3yr_Pct 9 AI-PRJ-110 Fraud Operations 4.4 713.9 4 AI-PRJ-105 Fraud Operations 4.5 705.3 5 AI-PRJ-106 Supply Chain 6.6 445.6
    - Both Fraud Operations projects pay back in under 5 months and return about 7 times their cost over 3 years.
    - There is a big gap after them: the third-best return (445.6%) is far lower. Fraud Operations saves about $2.8 a year for every $1 of build cost; the other projects save between $1.6 and $2.0.
- Quick Check 1 (.rank()):
  - New tool: column.rank(ascending=False) gives each row its place, 1 for the largest. Equal values share the average place, such as 4.5. Why: a ranking column is easy to show in a decision memo.
  - Task: add one line at the end of part 2. Process: df["ROI_Rank"] = df["ROI_3yr_Pct"].____(ascending=False), then add "ROI_Rank" to the printed columns.
  - Check: AI-PRJ-110 has rank 1.0 and AI-PRJ-105 has rank 2.0.
- Micro-Example 2 (A Fixed Rule vs a Rule Learned from Examples): YourName_S31_demo_2.py
```
# 1. Input: 10 past invoices, checked by auditors (fictional)
import pandas as pd
past = pd.DataFrame({
    "Amount_USD": [120000, 180000, 210000, 240000, 260000,
                   300000, 330000, 410000, 450000, 520000],
    "Was_Fraud": [False] * 5 + [True] * 5,
})

# 2. Process: a rule a person chose vs a threshold learned from the data
rule_flags = past["Amount_USD"] > 250000
rule_mistakes = (rule_flags != past["Was_Fraud"]).sum()

best_threshold, best_mistakes = None, len(past)
for threshold in past["Amount_USD"]:
    flags = past["Amount_USD"] >= threshold
    mistakes = (flags != past["Was_Fraud"]).sum()
    if mistakes < best_mistakes:
        best_threshold, best_mistakes = threshold, mistakes

# 3. Output: compare the mistakes
print("Fixed rule (above 250,000) mistakes:", rule_mistakes)
print("Learned threshold:", best_threshold, "mistakes:", best_mistakes)
```
  - How it works, line by line:
    - pd.DataFrame({...}) builds 10 fictional past invoices with known answers. [False] * 5 + [True] * 5 makes a list of 5 False followed by 5 True.
    - past["Amount_USD"] > 250000 applies the person’s rule and gives one True / False per invoice.
    - (rule_flags != past["Was_Fraud"]).sum() counts the rows where the rule and the auditors disagree: the mistakes.
    - The for loop tries each invoice amount as a candidate threshold, counts its mistakes, and keeps the threshold with the fewest. This search is the core idea of machine learning (section 7.1).
    - best_threshold, best_mistakes = threshold, mistakes updates two variables in one line.
  - Read the output:
    Fixed rule (above 250,000) mistakes: 1 Learned threshold: 300000 mistakes: 0
    - The person’s rule flags the honest 260000 invoice: 1 mistake.
    - The search found that “300,000 or more” separates honest and fraudulent invoices perfectly in these examples.
    - With only 10 examples, this is an illustration of learning, not a rule to deploy.
- Quick Check 2 (.mean() of True/False):
  - New tool: .mean() on a True / False column gives the share of True values, because True counts as 1. Why: tells a manager what share of invoices a rule would send to review.
  - Task: add one line to part 3. Output: print("Share flagged by the fixed rule:", rule_flags.____())
  - Check: the Terminal shows Share flagged by the fixed rule: 0.6, so 6 of 10 invoices would go to review.
- Micro-Example 3 (A Shortlist That Does Not Fit the Budget): YourName_S31_demo_3.py
```
# 1. Input: load the proposals and add the payback period
import pandas as pd
df = pd.read_csv("data/session_03_01_dataset.csv")
df["Net_Annual_Benefit_USD"] = (
    df["Projected_Cost_Savings_USD"] - df["Implementation_Cost_USD"] * 0.15
)
df["Payback_Months"] = (
    df["Implementation_Cost_USD"] / df["Net_Annual_Benefit_USD"] * 12
).round(1)

# 2. Process: fast payback AND many hours saved
shortlist = df[
    (df["Payback_Months"] <= 7.0)
    & (df["Annual_Labor_Hours_Saved_Count"] >= 4000)
]
shortlist_cost = shortlist["Implementation_Cost_USD"].sum()

# 3. Output: compare the shortlist with the budget
print(shortlist[["Project_ID", "Department_Name", "Payback_Months"]])
print(f"Shortlist cost: ${shortlist_cost:,.0f} against a budget of $500,000")
```
  - How it works, line by line:
    - Part 1 recomputes the net benefit and payback, as in Micro-Example 1.
    - The two conditions keep projects that pay back within 7 months and save at least 4,000 hours a year; each condition is in its own brackets, joined with & (Session 1.4).
    - shortlist["Implementation_Cost_USD"].sum() adds up what the shortlist would cost.
    - The f-string ${shortlist_cost:,.0f} prints the total with thousands separators.
  - Read the output:
    Project_ID Department_Name Payback_Months 0 AI-PRJ-101 Supply Chain 6.6 3 AI-PRJ-104 Finance 6.6 4 AI-PRJ-105 Fraud Operations 4.5 5 AI-PRJ-106 Supply Chain 6.6 8 AI-PRJ-109 Finance 6.7 9 AI-PRJ-110 Fraud Operations 4.4 Shortlist cost: $1,095,000 against a budget of $500,000
    - 6 projects pass both rules, but together they cost more than twice the budget.
    - Filtering tells you which projects are good; it does not tell you which ones to fund. That needs a budget method (section 7.4).
- Quick Check 3 (.isin()):
  - New tool: column.isin([...]) is True when the value is in the list. Why: answers questions about a few named groups, such as “show only Finance and Fraud Operations”.
  - Task: add one line to part 3. Output: print(len(df[df["Department_Name"].____(["Finance", "Fraud Operations"])]))
  - Check: the Terminal shows 4.
10. Official Python Documentation & References:

11. Exercise 1: Guided Practice - AI Portfolio Ranking:

Step 1: Open Script: In Google Antigravity IDE, create a new script named YourName_S31_portfolio.py, replacing YourName with your own name (for example, Reacher_S31_portfolio.py).

Step 2: Rank the Portfolio and Compare Departments: Type the following code:

# 1. Input: load the 10 proposals
import pandas as pd
df = pd.read_csv("data/session_03_01_dataset.csv")

# 2. Process: net benefit and 3-year ROI, then the average by department
df["Net_Annual_Benefit_USD"] = (
    df["Projected_Cost_Savings_USD"] - df["Implementation_Cost_USD"] * 0.15
)
df["ROI_3yr_Pct"] = (
    (df["Net_Annual_Benefit_USD"] * 3 - df["Implementation_Cost_USD"])
    / df["Implementation_Cost_USD"] * 100
).round(1)
by_department = (
    df.groupby("Department_Name")["ROI_3yr_Pct"].mean().round(1)
    .sort_values(ascending=False)
)

# 3. Output: the full ranking and the department view
ranked = df.sort_values("ROI_3yr_Pct", ascending=False)
print(ranked[["Project_ID", "Department_Name", "ROI_3yr_Pct"]])
print("\nAverage 3-year ROI by department:\n", by_department)

Step 3: Interpret: In two comment lines, explain why Fraud Operations ranks first, and which numbers you would ask the Marketing team to double-check.
Step 4: Execute in Terminal: Run python YourName_S31_portfolio.py in Panel [4] and confirm return code 0.
Success Check: department averages are Fraud Operations 709.6, Supply Chain 444.3, Finance 440.2, Customer Service 429.0, and Marketing 342.3. Fraud Operations saves the most money per dollar spent (about $2.8 of yearly savings per $1 of build cost).

12. Exercise 2: Applied Practice - Fund Projects Within $500,000:
- Coding Challenge: In YourName_S31_practice.py, fund projects in order of hours saved per dollar until the $500,000 budget runs out, then check a new proposal’s payback safely.
- Task 1 (Greedy Allocation):
  - Add Hours_per_1000_USD = Annual_Labor_Hours_Saved_Count / Implementation_Cost_USD x 1000, and sort by it from highest to lowest.
  - Start with budget = 500000, spent = 0, and funded = [].
  - Loop over the sorted rows with for project, cost in zip(df["Project_ID"], df["Implementation_Cost_USD"]):. If spent + cost <= budget, add the project to funded and add cost to spent.
  - zip(a, b) walks through two columns side by side, giving one pair of values per row.
  - Print the funded projects, the amount spent, and the total hours saved by the funded projects.
- Task 2 (Read the ZeroDivisionError & AI Diagnosis): Write a function payback_months(cost, annual_benefit) that returns cost / annual_benefit * 12. Call it for a new proposal with no proven savings yet: payback_months(50000.0, 0.0). The script stops with ZeroDivisionError: float division by zero. Capture the traceback with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and ask why.
- Task 3 (Safe Division): Change the function: if annual_benefit <= 0, return None; otherwise return the payback. When the result is None, print "No payback: no proven savings".
- Success Check: the loop funds AI-PRJ-102, AI-PRJ-107, AI-PRJ-110, and AI-PRJ-103 for $480,000 and 16,100 hours a year; payback_months(50000.0, 0.0) now prints No payback: no proven savings; return code 0.
13. Exercise 3: Advanced Practice - Is the Greedy Choice the Best Choice?:
- Goal: Test the simple “most hours per dollar first” method from Exercise 2 against every possible combination of projects, and tell the COO whether the simple method left value on the table.
- What you need:
  - File: data/session_03_01_dataset.csv (described in 5. Lab Practice Dataset).
  - Script name: YourName_S31_challenge.py.
  - Your Exercise 2 result (16,100 hours for $480,000).
  - Tools you already know: lists, for loops, if, .sum(), .isin(), and f-strings.
- Requirements (follow them in order):
  1. List every possible group of projects: all groups of 1 project, of 2 projects, and so on, up to all 10.
  2. For each group, add up the build cost. Skip groups that cost more than $500,000.
  3. For the groups that fit, add up Annual_Labor_Hours_Saved_Count, and keep the group with the most hours. If two groups tie on hours, keep the cheaper one.
  4. Count how many groups you checked in total and how many fit the budget.
  5. Print the best group’s project IDs, cost, and hours, then the difference in hours compared with the greedy result.
  6. Print a two-line memo for the COO: what to fund, and why the simple method was not enough.
- New tools for this exercise:
  - from itertools import combinations loads a tool that lists groups. combinations(["A", "B", "C"], 2) gives ("A", "B"), ("A", "C"), and ("B", "C").
  - for size in range(1, 11): repeats with size = 1, 2, …, 10; inside it, for group in combinations(project_ids, size): walks through every group of that size.
  - list(group) turns a group into a list, so you can use df[df["Project_ID"].isin(list(group))] to select its rows.
- Hints:
  - Make project_ids = list(df["Project_ID"]) first.
  - Keep the best result in three variables, best_hours, best_cost, and best_group, and update them inside the loop.
  - With 10 projects there are 1,023 groups, so the loop finishes in a few seconds.
- Self-Check (expected results):
  - Groups checked: 1023; groups that fit the budget: 163.
  - Best group: AI-PRJ-101, AI-PRJ-102, AI-PRJ-105, AI-PRJ-107, costing $495,000 and saving 16,900 hours a year.
  - That is 800 more hours than the greedy result. Business meaning: “most hours per dollar first” picked the cheap Marketing project AI-PRJ-103 to fill the last gap, while a different mix used the budget better. With few options, checking every combination is cheap and finds the true best plan.

Session 3.2: Advanced Data Analysis: Feature Engineering, Scaling, and Business Data Preprocessing

Session 3.2 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 10 (115/11/15 - 115/11/21) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- A bank’s credit team wants a model that predicts which loan applicants will default (fail to repay their loan)
- Its applicant records mix text, gaps, and numbers on very different scales, such as an income of 65,000 next to a utilization of 24.5
- The credit risk manager asks: can we turn our applicant records into data a model can learn from, without cheating on the test? Today, you prepare the data step by step, so that Session 3.3 can train a model on it
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “What do we predict, and from what?”: split a table into features X (the inputs) and the target y (the answer to predict)
2. “How do text and gaps become numbers?”: fill missing values (imputation), turn text categories into 0/1 columns (one-hot encoding), and put numbers on a common scale (standardization)
3. “How do we test honestly?”: split the data into a training set and a test set before learning anything from it, so that no test information leaks into training (data leakage)
3. Business Context: Why Data Preparation Decides Model Quality:
- A model is only as good as the data it learns from (“garbage in, garbage out”). Data preparation usually takes more of a project’s time than building the model itself.
- A model reads only numbers, with no gaps. Our raw data breaks both rules:
  - Education_Tier is text (Bachelor, Master, Doctorate)
  - One income and one utilization value are missing
- A model must be tested on applicants it has never seen, just as a student is tested on new exam questions, not on the practice questions. Otherwise the bank believes the model is better than it is, and approves risky loans.
- Credit utilization: the share of an applicant’s credit limit that is in use. 48.2% means $48.20 of every $100 of available credit is borrowed.

4. Data Basics: From a Raw Row to a Model-Ready Row:

Applicant CLI-701, before and after preparation:

Column	Raw value	Model-ready value	Step
`Age_Years_Count`	`45`	`0.41`	standardized
`Annual_Income_USD`	`125000.0`	`0.44`	standardized
`Credit_Utilization_Pct`	`48.2`	`0.82`	standardized
`Education_Tier`	`Master`	`Education_Tier_Master` = `1`, `Education_Tier_Doctorate` = `0`	one-hot encoded

The standardized values use the training rows’ averages and spreads (Micro-Example 3). Micro-Example 3 also standardizes the 0/1 columns; Exercise 3 shows how to scale only the number columns.
One-hot encoding: one 0/1 column per category. With drop_first=True, the first category (Bachelor) gets no column: a row with 0 in both remaining columns is a Bachelor. Never code categories as 1, 2, 3; that would tell the model a Doctorate is “three times” a Bachelor.
Standardization: (value - average) / standard deviation. The result says how many standard deviations a value is above (+) or below (-) the average, so income and utilization become comparable. Standard deviation measures how spread out a column is.
Train/test split: the model learns from the training set; the test set is kept aside and used once, at the end, to measure the model honestly.

5. Lab Practice Dataset:
- File Link: data/session_03_02_dataset.csv
- Audited 12-row loan applicant dataset (fictional teaching data), one row per applicant, with 2 missing values on purpose:
  - Client_ID: Applicant code (e.g., CLI-701). An ID, not a feature: it says nothing about the applicant.
  - Age_Years_Count: Age in years.
  - Annual_Income_USD: Yearly income in U.S. Dollars (missing for CLI-707).
  - Credit_Utilization_Pct: Share of the credit limit in use, in percent (missing for CLI-709).
  - Education_Tier: Highest education: Bachelor, Master, or Doctorate.
  - Defaulted_Binary: The target: 1 = defaulted, 0 = repaid. 3 of the 12 applicants defaulted.
6. Keywords (Core Programming Concepts):
- Features (X) and Target (y):
  - What: X holds the input columns; y holds the outcome to predict
  - Why: every Scikit-Learn model expects this split
- Imputation:
  - What: filling missing values, for example with the column’s median (the middle value)
  - Why: models cannot learn from rows with gaps
- One-Hot Encoding:
  - What: one 0/1 column per text category
  - Why: turns text into numbers without inventing an order
- Standardization:
  - What: rescaling a column to an average of 0 and a standard deviation of 1
  - Why: stops large-unit columns, such as income, from drowning out small ones
- Train/Test Split:
  - What: keeping some rows aside to test the model at the end
  - Why: the only honest measure of how the model will do on new applicants
- Data Leakage:
  - What: information from the test set slipping into training
  - Why: makes a model look better than it really is

7. Core Programming & Computational Foundations:

7.1 Features and target: what goes in, what comes out

The idea: A prediction model learns a link between inputs and an answer. The inputs are called features and are stored in a table named X; the answer to predict is the target, stored in y. Every row is one example: one applicant’s features and that applicant’s real outcome.

See it on real data:

Column	Role	Why
`Age_Years_Count`, `Annual_Income_USD`, `Credit_Utilization_Pct`, `Education_Tier`	features (`X`)	facts known when the application arrives
`Defaulted_Binary`	target (`y`)	the outcome we want to predict
`Client_ID`	neither	a label; it says nothing about risk

Common mistake: leaving the target inside X. The model would “predict” default by looking at the answer itself, score perfectly in testing, and be useless on new applicants, who have no answer yet.

More examples:

Code	Result	What it tells you
`X.shape`	`(12, 4)`	12 applicants, 4 features
`y.value_counts()`	`0`: `9`, `1`: `3`	3 of 12 applicants defaulted
`y.mean()`	`0.25`	the default rate: the mean of a 0/1 column is the share of 1s
`X.isnull().sum()`	`1` in income, `1` in utilization	the gaps to fill before training

Remember: X = what you know in advance; y = what you want to predict.

7.2 Filling gaps: imputation

The idea: Models cannot learn from empty cells. Imputation fills each gap with a typical value of its column. The median (the middle value) is a safe default, because a few extreme values do not pull it, unlike the average (mean).

See it on real data: the income column, with CLI-707’s income missing.

Statistic of the 11 known incomes	Value
mean	`115727.27`
median	`110000.0`
smallest and largest	`58000.0` and `185000.0`

The 3 high incomes of the Doctorate applicants pull the mean up; the median stays in the middle.

Common mistake: deleting every row with a gap. With 12 rows, dropping 2 loses one-sixth of the data. Fill gaps when you can; drop rows only when too much of the row is missing.

More examples:

Code	Result	What it tells you
`df["Credit_Utilization_Pct"].median()`	`36.5`	the value that would fill `CLI-709`’s gap, using all rows
`SimpleImputer(strategy="median").fit(X_train).statistics_`	`[38.0, 96000.0, 33.15, 0.0, 0.0]`	the medians learned from the 9 training rows only
`SimpleImputer(strategy="mean")`	fills with averages	another option for columns without extreme values

The medians from all rows and from the training rows differ (36.5 vs 33.15), because they are computed from different applicants. Section 7.5 explains why the training version is the correct one.

Remember: fill gaps with the median, learned from the training rows.

7.3 Text categories become 0/1 columns: one-hot encoding

The idea: A model computes with numbers, so the text in Education_Tier must become numbers. Writing Bachelor = 1, Master = 2, Doctorate = 3 would wrongly tell the model that a Doctorate is “three times” a Bachelor. One-hot encoding instead makes one 0/1 column per category. With drop_first=True, the first category gets no column: it is the row with 0 everywhere.

See it on real data:

`Education_Tier`	`Education_Tier_Doctorate`	`Education_Tier_Master`
`Bachelor`	`0`	`0`
`Master`	`0`	`1`
`Doctorate`	`1`	`0`

Categories are put in alphabetical order, so Bachelor comes first and is the one dropped.

Common mistake: a category appearing in new data that was not in the training data, such as "Associate". pd.get_dummies() would create a column the model has never seen. Exercise 3 shows a tool that handles this safely.

More examples:

Code	Result	What it tells you
`df["Education_Tier"].value_counts()`	Master `5`, Bachelor `4`, Doctorate `3`	how many 1s each new column will have
`pd.get_dummies(X, columns=["Education_Tier"], drop_first=True).shape`	`(12, 5)`	1 text column replaced by 2 number columns
the same with `drop_first=False`	`(12, 6)`	3 columns, one per category

Remember: one 0/1 column per category, never 1-2-3 codes.

7.4 Putting numbers on one scale: standardization

The idea: Income runs from 58,000 to 185,000; utilization from 15.2 to 62.1. Some models compare rows by distance and would treat a difference of 1,000 dollars as far bigger than a difference of 10 percentage points, only because of the units. Standardization rewrites each value as “how many standard deviations above or below the average”, so all columns use the same scale. The standard deviation measures how spread out a column is.

See it on real data: age, average 42.5 years, standard deviation 10.66 years (all 12 applicants).

Applicant age	Calculation	Standardized
`28`	`(28 - 42.5) / 10.66`	`-1.36`
`45`	`(45 - 42.5) / 10.66`	`0.23`
`63`	`(63 - 42.5) / 10.66`	`1.92`

After standardization, every column has an average of 0 and a standard deviation of 1.

More examples: income, average 115727.27, standard deviation 43668.36 (the 11 known values).

Value	Calculation	Standardized
income `58000`	`(58000 - 115727.27) / 43668.36`	`-1.32`: the lowest income
income `110000`	`(110000 - 115727.27) / 43668.36`	`-0.13`: just below average
income `185000`	`(185000 - 115727.27) / 43668.36`	`1.59`: well above average
`df["Annual_Income_USD"].std()`	`45799.76`	pandas divides by n - 1 by default; the scaler uses n (`43668.36`)

Remember: standardized value = (value - average) / standard deviation.

7.5 Honest testing: split first, then learn

The idea: The test set plays the role of next month’s new applicants: the model and its preparation steps must not see it during training. So the order is fixed: split the rows first; fit the imputer and the scaler on the training rows only; then transform both sets with what was learned. Using information from the test set during training is called data leakage, and it makes a model look better than it really is.

See it on real data: train_test_split(X, y, test_size=0.25, random_state=42, stratify=y).

Setting	Effect here
`test_size=0.25`	3 of 12 rows go to the test set: `CLI-704`, `CLI-710`, `CLI-708`
`stratify=y`	2 defaults in training and 1 in test: the same 25% rate in both
`random_state=42`	the same split every time you run the script
`random_state=7` instead	a different test set: `CLI-710`, `CLI-705`, `CLI-701`

Common mistake: scaler.fit_transform(X_test). The scaler then learns the test rows’ own average, and the test applicants all look “average”, even though they are older and earn more than the training applicants. Exercise 2 shows the numbers.

More examples:

Setting	Train / test rows	Defaults in train / test
`test_size=0.25, stratify=y` (ours)	9 / 3	2 / 1
`test_size=0.25`, no `stratify`	9 / 3	2 / 1 here, but not guaranteed; the test set is `CLI-710`, `CLI-709`, `CLI-700`
`test_size=0.5, stratify=y`	6 / 6	2 / 1: 3 defaults cannot split evenly
scaler fitted on train, test averages	`0.74`, `0.73`, `0.57`, …	correct: the test applicants differ
scaler fitted on test (leakage)	all `0.0`	the difference has disappeared

Remember: split, fit on train, transform both.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S32_demo_1.py), and check the dataset is in data/.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors. First time only: pip install scikit-learn.
9. Hands-On Coding Micro-Examples:
- Every example has three parts: 1. Input, 2. Process, and 3. Output. Name each file with your own name and the session code, for example Reacher_S32_demo_1.py (pattern: YourName_S32_demo_1.py).
- Micro-Example 1 (Features, Target, and Gaps): YourName_S32_demo_1.py
```
# 1. Input: load the applicants
import pandas as pd
df = pd.read_csv("data/session_03_02_dataset.csv")

# 2. Process: inputs (X) and the answer to predict (y)
X = df.drop(columns=["Client_ID", "Defaulted_Binary"])
y = df["Defaulted_Binary"]

# 3. Output: size, outcome counts, and missing values
print("X shape (rows, columns):", X.shape)
print("Defaults (1) and repaid (0):\n", y.value_counts())
print("Missing values per column:\n", X.isnull().sum())
```
  - How it works, line by line:
    - df.drop(columns=["Client_ID", "Defaulted_Binary"]) returns a copy of the table without the ID and the answer: the features X (section 7.1). The original df is unchanged.
    - df["Defaulted_Binary"] picks the answer column: the target y.
    - y.value_counts() counts each answer; X.isnull().sum() counts the gaps per column.
  - Read the output:
    X shape (rows, columns): (12, 4) Defaults (1) and repaid (0): Defaulted_Binary 0 9 1 3 Missing values per column: Age_Years_Count 0 Annual_Income_USD 1 Credit_Utilization_Pct 1 Education_Tier 0
    - 4 features for 12 applicants.
    - 3 defaults out of 12: a 25% default rate, the first number a credit manager asks for.
    - One income and one utilization value are missing: they must be filled before training (section 7.2).
- Quick Check 1 (value_counts(normalize=True)):
  - New tool: .value_counts(normalize=True) gives each value’s share instead of its count. Why: the default rate is the first number a credit manager asks for.
  - Task: add one line to part 3. Output: print(y.value_counts(_________=True))
  - Check: 0 has 0.75 and 1 has 0.25: a 25% default rate.
- Micro-Example 2 (One-Hot Encoding): YourName_S32_demo_2.py
```
# 1. Input: load the applicants and keep the features
import pandas as pd
df = pd.read_csv("data/session_03_02_dataset.csv")
X = df.drop(columns=["Client_ID", "Defaulted_Binary"])

# 2. Process: one 0/1 column per education level, except the first
X_encoded = pd.get_dummies(X, columns=["Education_Tier"],
                           drop_first=True, dtype=float)

# 3. Output: the new column names and two rows
print(X_encoded.columns.tolist())
print(X_encoded.head(2))
```
  - How it works, line by line:
    - pd.get_dummies(X, columns=["Education_Tier"], drop_first=True, dtype=float) replaces the text column with 0/1 columns (section 7.3). dtype=float writes them as 0.0 and 1.0.
    - X_encoded.columns.tolist() lists the new column names.
    - .head(2) shows the first two applicants, a Bachelor and a Master.
  - Read the output:
    ['Age_Years_Count', 'Annual_Income_USD', 'Credit_Utilization_Pct', 'Education_Tier_Doctorate', 'Education_Tier_Master'] Age_Years_Count ... Education_Tier_Master 0 28 ... 0.0 1 45 ... 1.0
    - Education_Tier is gone; two number columns took its place. There is no Education_Tier_Bachelor: a Bachelor is the row with 0.0 in both.
    - Applicant 0 (CLI-700) is a Bachelor (0.0 in the Master column); applicant 1 (CLI-701) is a Master (1.0).
- Quick Check 2 (drop_first):
  - New tool: the drop_first setting. True leaves out the first category; False keeps a column for every category. Why: shows what “the first category is implied” means.
  - Task: in part 2. Process, change drop_first=True to drop_first=_____.
  - Check: a third column, Education_Tier_Bachelor, appears, and each row has exactly one 1.0 among the three education columns. Change it back to True afterwards.
- Micro-Example 3 (Split First, Then Fill and Scale): YourName_S32_demo_3.py
```
# 1. Input: load, encode, and split before learning anything
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
df = pd.read_csv("data/session_03_02_dataset.csv")
X = pd.get_dummies(df.drop(columns=["Client_ID", "Defaulted_Binary"]),
                   columns=["Education_Tier"], drop_first=True, dtype=float)
y = df["Defaulted_Binary"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

# 2. Process: learn medians and scales from the training rows only
imputer = SimpleImputer(strategy="median")
X_train_filled = imputer.fit_transform(X_train)
X_test_filled = imputer.transform(X_test)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train_filled)
X_test_scaled = scaler.transform(X_test_filled)

# 3. Output: sizes and the medians used to fill the gaps
print("Train rows:", len(X_train), "| Test rows:", len(X_test))
print("Medians learned from training rows:", imputer.statistics_.round(2).tolist())
```
  - How it works, line by line:
    - Part 1 encodes the text column, then splits the rows before anything is learned (section 7.5).
    - SimpleImputer(strategy="median") creates an empty gap-filler. .fit_transform(X_train) learns the training medians and fills the training gaps; .transform(X_test) fills the test gaps with those same training medians.
    - StandardScaler() works the same way: it learns each column’s training average and standard deviation, then applies them to both sets (section 7.4).
    - imputer.statistics_ holds the medians it learned; the trailing _ marks a value learned during fit.
  - Read the output:
    Train rows: 9 | Test rows: 3 Medians learned from training rows: [38.0, 96000.0, 33.15, 0.0, 0.0]
    - 9 applicants for training and 3 kept aside for the final test.
    - The medians are, in column order: age 38.0, income 96000.0, utilization 33.15, and 0.0 for both education columns. Any gap, in either set, is filled with these values.
- Quick Check 3 (scaler.mean_):
  - New tool: scaler.mean_ holds the column averages the scaler learned from the training rows. Why: shows exactly what “fit on the training set” means.
  - Task: add one line to part 3. Output: print("Averages learned by the scaler:", scaler._____.round(2).tolist())
  - Check: the first three values are 40.44 (age), 106444.44 (income), and 35.82 (utilization).
10. Official Python Documentation & References:

11. Exercise 1: Guided Practice - Credit Data Preparation Pipeline:

Step 1: Open Script: In Google Antigravity IDE, create a new script named YourName_S32_preprocessing.py, replacing YourName with your own name (for example, Reacher_S32_preprocessing.py).

Step 2: Build the Full Preparation Sequence: Type the following code:

# 1. Input: load, encode, and split
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
df = pd.read_csv("data/session_03_02_dataset.csv")
X = pd.get_dummies(df.drop(columns=["Client_ID", "Defaulted_Binary"]),
                   columns=["Education_Tier"], drop_first=True, dtype=float)
y = df["Defaulted_Binary"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

# 2. Process: fill, then scale; fit on training rows only
imputer = SimpleImputer(strategy="median")
scaler = StandardScaler()
X_train_scaled = pd.DataFrame(
    scaler.fit_transform(imputer.fit_transform(X_train)), columns=X.columns
)
X_test_scaled = pd.DataFrame(
    scaler.transform(imputer.transform(X_test)), columns=X.columns
)

# 3. Output: check the training columns
print("Training averages:\n", X_train_scaled.mean().round(2))
print("Training spread:\n", X_train_scaled.std(ddof=0).round(2))
print("Defaults in train:", y_train.sum(), "| in test:", y_test.sum())

.std(ddof=0) uses the same standard deviation formula as the scaler. pandas’ default formula gives a slightly larger value (1.06 here) for small tables.

Step 3: Validate the Scaling: Every training average should be 0.0 (a printed -0.0 is the same as 0.0) and every spread 1.0.
Step 4: Execute in Terminal: Run python YourName_S32_preprocessing.py in Panel [4] and confirm return code 0.
Success Check: 5 columns, each with average 0.0 and spread 1.0; 2 defaults in the training set and 1 in the test set, thanks to stratify=y.

12. Exercise 2: Applied Practice - Feature Engineering and a Leakage Check:
- Coding Challenge: In YourName_S32_practice.py, add one engineered feature, then see with your own eyes what data leakage hides.
- Task 1 (Engineer a Feature): Add High_Utilization_Flag = 1 when Credit_Utilization_Pct is above 50, else 0: (df["Credit_Utilization_Pct"] > 50).astype(int). .astype(int) turns True / False into 1 / 0. Print pd.crosstab(df["High_Utilization_Flag"], df["Defaulted_Binary"]).
- Task 2 (Read the NotFittedError & AI Diagnosis): Starting from Micro-Example 3, create a new StandardScaler() and call .transform(X_test_filled) on it before any .fit(). The script stops with NotFittedError: This StandardScaler instance is not fitted yet. Capture the traceback with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and ask what “fitted” means.
- Task 3 (See the Leakage): Scale the test rows two ways and print the column averages of each, rounded to 2 decimals:
  - Correct: scaler.transform(X_test_filled), with scaler fitted on the training rows
  - Leaky: StandardScaler().fit_transform(X_test_filled)
  - array.mean(axis=0) gives the average of each column of a Scikit-Learn result.
- Success Check:
  - Task 1: the flag splits the applicants perfectly: all 3 flagged applicants defaulted and all 9 others repaid. With only 12 rows, treat a perfect pattern as a warning sign, not as proof.
  - Task 3: the correct averages are 0.74, 0.73, 0.57, 0.27, 0.71, showing that the test applicants are older, earn more, and use more credit than the training applicants. The leaky averages are all 0.0: the difference has disappeared.
13. Exercise 3: Advanced Practice - A Reusable Preparation Pipeline:
- Goal: Build one preparation object that handles numbers and text in one step, can be fitted on training rows and reused on any new applicant, and does not crash on a category it has never seen.
- What you need:
  - File: data/session_03_02_dataset.csv (described in 5. Lab Practice Dataset).
  - Script name: YourName_S32_challenge.py.
  - Tools you already know: train_test_split (same settings as Micro-Example 3), SimpleImputer, StandardScaler, and .fit() / .transform().
- Requirements (follow them in order):
  1. Split the raw features (before any encoding) and the target with test_size=0.25, random_state=42, and stratify=y.
  2. Define numeric_cols = ["Age_Years_Count", "Annual_Income_USD", "Credit_Utilization_Pct"] and text_cols = ["Education_Tier"].
  3. Build a Pipeline for the numeric columns: fill with the median, then standardize.
  4. Combine it with a OneHotEncoder(handle_unknown="ignore") for the text column in one ColumnTransformer.
  5. Fit the combined object on the training rows only. Print the output column names, then the transformed test rows, rounded to 2 decimals.
  6. Transform one new applicant: age 36, income missing (None), utilization 57.0, education "Associate". Print the result, rounded to 2 decimals.
  7. In a comment, explain what happened to the missing income and to the unseen education level.
- New tools for this exercise:
  - from sklearn.pipeline import Pipeline: Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())]) runs its steps in order, and fits them in order.
  - from sklearn.compose import ColumnTransformer: sends each group of columns to its own tool and joins the results side by side, for example:
    ColumnTransformer([ ("num", numeric_pipeline, numeric_cols), ("cat", OneHotEncoder(handle_unknown="ignore"), text_cols), ])
  - from sklearn.preprocessing import OneHotEncoder: the Scikit-Learn version of one-hot encoding. handle_unknown="ignore" writes 0 in every category column for a value it never saw, instead of raising an error.
  - .get_feature_names_out() returns the names of the output columns.
  - A one-row DataFrame for the new applicant: pd.DataFrame({"Age_Years_Count": [36], "Annual_Income_USD": [None], "Credit_Utilization_Pct": [57.0], "Education_Tier": ["Associate"]}).
- Hints:
  - Wrap a result in pd.DataFrame(result, columns=prep.get_feature_names_out()) to read it with column names.
  - Without handle_unknown="ignore", the new applicant raises ValueError: Found unknown categories ['Associate']. Try it once to see why the setting matters.
- Self-Check (expected results):
  - Output columns (6): num__Age_Years_Count, num__Annual_Income_USD, num__Credit_Utilization_Pct, cat__Education_Tier_Bachelor, cat__Education_Tier_Doctorate, cat__Education_Tier_Master.
  - Transformed test rows: shape (3, 6); the second test row is the Doctorate applicant with 1.41, 1.63, 1.49.
  - New applicant: -0.4, -0.25, 1.4, then 0.0, 0.0, 0.0. The missing income was filled with the training median (96000.0) before scaling, and the unseen "Associate" level became all zeros instead of crashing the script. Business meaning: the same object can now prepare every future application in one call, exactly as it prepared the training data.

Session 3.3: Practical Machine Learning with Scikit-Learn: Model Training and Validation

Session 3.3 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 11 (115/11/22 - 115/11/28) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- A commercial property valuation firm prices office buildings by hand, which takes an expert several days per property
- It has the sale records of 14 recent deals: size, number of rooms, distance to the city center, tax rate, and price
- The valuation manager asks: can a model estimate a property’s sale price from its size and location, and how far off will it be? Today, you train your first machine learning model and measure its error honestly
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “Can a model learn prices from past deals?”: train a linear regression model with Scikit-Learn’s .fit() and read its coefficients (how much the price changes for one more unit of each feature)
2. “How far off are the predictions?”: measure the error with RMSE (Root Mean Squared Error) and MAE (Mean Absolute Error), and check stability with cross-validation (testing on several different splits)
3. “What will this new building sell for?”: predict the price of one new property, and fix the ValueError a one-row prediction often raises
3. Business Context: When a Price Model Is Worth Using:
- An estimate is useful if its typical error is small compared with the price. Our properties sell for about $513,000 on average, so a typical error of $10,000 (about 2%) is useful for a first quote, while a typical error of $100,000 would not be.
- A model learns from past deals only. It should not be trusted far outside them, for example for a 10,000 square-foot tower when the largest past deal was 3,400 square feet.
- Regression vs classification:
  - Regression predicts a number, such as a price. Today’s model, linear regression, is one example.
  - Classification predicts a category, such as “fraud” or “not fraud”. Logistic regression, used in Session 3.4, is one example.

4. Data Basics: How Linear Regression Works:

Linear regression finds the best straight-line formula: price = intercept + coefficient_1 x feature_1 + coefficient_2 x feature_2 + .... “Best” means the smallest total squared error on the training rows.
Reading a coefficient: the change in price for one more unit of that feature, while the other features stay the same. Example from Micro-Example 1: +180.1 per square foot.
Coefficients can surprise you. On its own, distance rises with price (correlation +0.98), because the big buildings in our data are far from the center. Once size is in the model, each extra mile lowers the price by about $1,900: compared at the same size, a farther building is cheaper.

Three error measures, in plain words:

Measure	Meaning	Lower or higher is better?
MAE (Mean Absolute Error)	the average size of the errors, in dollars	lower
RMSE (Root Mean Squared Error)	like MAE, but large errors count extra	lower
R-squared	the share of the price differences the model explains, from 0 to 1	higher

5. Lab Practice Dataset:
- File Link: data/session_03_03_dataset.csv
- Audited 14-row commercial property sales dataset (fictional teaching data), one row per sale:
  - Property_ID: Sale code (e.g., PROP-401).
  - Square_Feet_Units: Usable floor area in square feet.
  - Bedrooms_Count: Number of separate rooms or office suites.
  - Distance_To_CBD_Miles: Distance to the CBD (Central Business District, the city’s main business center), in miles.
  - Local_Tax_Rate_Pct: Local property tax rate, in percent.
  - Sale_Price_USD: The price the property sold for, in U.S. Dollars. This is the target.
6. Keywords (Core Programming Concepts):
- Estimator (.fit() / .predict()):
  - What: any Scikit-Learn model; .fit(X, y) learns from examples, .predict(X) makes predictions
  - Why: every model in the library uses the same two methods
- Linear Regression:
  - What: a model that predicts a number with a straight-line formula
  - Why: simple, fast, and its coefficients are easy to explain to a manager
- Logistic Regression:
  - What: a model that predicts the probability of a yes/no outcome
  - Why: the classification partner of linear regression (Session 3.4)
- RMSE and MAE:
  - What: the typical prediction error, in the target’s own unit (here, dollars)
  - Why: tells the manager how far off an estimate can be
- Cross-Validation:
  - What: splitting the data into K parts and testing K times, each time on a different part
  - Why: one lucky split can make a model look better than it is
- Overfitting:
  - What: a model that memorizes its training rows and does worse on new data
  - Why: the main risk when data is small; testing on unseen rows reveals it

7. Core Programming & Computational Foundations:

7.1 The estimator pattern: create, fit, predict

The idea: Every model in Scikit-Learn, from the simplest to the most advanced, is used in the same three steps. Learn them once and you can use hundreds of models:
1. Create an empty model: model = LinearRegression().
2. Fit it to examples with known answers: model.fit(X_train, y_train). This is where the model learns.
3. Predict answers for rows it has not seen: model.predict(X_test).

See it on real data:

Step	Code	What happens
Create	`model = LinearRegression()`	an empty model, nothing learned yet
Fit	`model.fit(X_train, y_train)`	learns the price formula from 10 sales
Predict	`model.predict(X_test)`	4 estimated prices for the 4 test sales
Score	`model.score(X_test, y_test)`	`0.996`, the share of price differences explained

Common mistake: calling .predict() before .fit(). The model has nothing learned yet and stops with NotFittedError.

More examples:

Code	Result	What it tells you
`model.intercept_`	`123568.1`	the formula’s starting value
`model.coef_`	`[180.1, 14703.6, -1915.8]`	one coefficient per feature, in column order
`model.score(X_train, y_train)`	`0.999`	on the training rows: always looks better
`model.score(X_test, y_test)`	`0.996`	on unseen rows: the honest measure

Remember: create, fit, predict; learned values end with _.

7.2 Reading the price formula

The idea: Linear regression learns a formula of the form price = intercept + coefficient x feature + … Each coefficient is the change in price for one more unit of that feature while the other features stay the same.

See it on real data: our formula is 123568.1 + 180.1 x square feet + 14703.6 x rooms - 1915.8 x miles. For a building of 1,500 square feet, 3 rooms, 5 miles from the center:

Part	Calculation	Amount
intercept		`123568.1`
size	`180.1 x 1500`	`270150.0`
rooms	`14703.6 x 3`	`44110.8`
distance	`-1915.8 x 5`	`-9579.0`
price	sum	about `428,250`

model.predict() gives 428,269: the small difference comes from rounding the coefficients to one decimal in the table.

More examples: change one feature at a time, starting from that building.

Change	Predicted price	Difference
none (1,500 sq ft, 3 rooms, 5 miles)	`428,269`
1,000 more square feet	`608,382`	`+180,113`: 1,000 x `180.1`
one more room	`442,973`	`+14,704`
5 more miles from the center	`418,690`	`-9,579`: 5 x `-1915.8`

Common mistake: reading a coefficient alone. On its own, distance rises with price (correlation +0.98), and a model with distance as its only feature even gives it a coefficient of +30,026.8 per mile, because the big buildings in our data are far from the center. Only when size is in the model does the true effect of distance at the same size appear: -1,915.8 per mile.
Remember: a coefficient is the effect of one feature with the others held fixed.

7.3 Measuring error: MAE, RMSE, and R-squared

The idea: An error is predicted price minus actual price. Three summaries turn the errors into one number each.

Measure	How it is computed	Our test set
MAE (Mean Absolute Error)	average of the error sizes	`$9,668`
RMSE (Root Mean Squared Error)	square each error, average, take the square root	`$11,539`
R-squared	share of the price differences the model explains	`0.996`

See it on real data: the 4 test errors are -4,732, +15,735, +16,064, and -2,141 dollars. MAE simply averages their sizes. RMSE squares them first, so the two large errors weigh more, and RMSE comes out higher.
A useful comparison: a “model” that always guesses the average training price would have a test RMSE of $186,691. Ours, at $11,539, is about 16 times better.
Common mistake: trusting R-squared on the training rows (0.999). It measures how well the model fits rows it has already seen.

More examples:

Errors	MAE	RMSE	What it tells you
errors `+10` and `-10`	`10`	`10`	equal errors: MAE and RMSE agree
errors `0` and `20`	`10`	`14.14`	one large error: RMSE grows, MAE does not
our 4 test errors	`9,668`	`11,539`	two larger errors raise RMSE
always guessing the average price		`186,691`	the benchmark any model must beat

Remember: MAE is the plain average error; RMSE punishes large errors; test on unseen rows.

7.4 One split is not enough: cross-validation

The idea: With only 14 sales, the result depends on which 4 land in the test set. K-fold cross-validation cuts the data into K parts (folds), trains K times, and tests each time on a different fold, so every sale is tested once.

See it on real data: cv=5 cuts our 14 sales into 5 folds, in file order.

Fold	Test sales	R-squared
1	`PROP-400`, `PROP-401`, `PROP-402`	`0.994`
2	`PROP-403`, `PROP-404`, `PROP-405`	`0.992`
3	`PROP-406`, `PROP-407`, `PROP-408`	`0.999`
4	`PROP-409`, `PROP-410`, `PROP-411`	`0.996`
5	`PROP-412`, `PROP-413`	`0.999`

More examples:

Setting	R-squared per fold	What it tells you
`cv=3`	`[0.994, 0.986, 0.998]`	`0.993`
`cv=5` (ours)	`[0.994, 0.992, 0.999, 0.996, 0.999]`	`0.996`
`cv=7`	`[0.982, 0.986, 0.996, 1.0, 0.959, 0.996, 0.999]`	`0.988`: smaller folds, more variation
`cv=KFold(5, shuffle=True, random_state=1)`	`[0.981, 0.999, 0.998, 0.983, 0.969]`	shuffled folds: different sales in each test

Remember: report the average over the folds, and look at the worst fold too.

7.5 One row is still a table: the 2D rule

The idea: .predict() always expects a table: rows and columns, called 2D (two-dimensional). A single property is a table with one row. A plain list like [1500, 3, 5.0] is 1D (one dimension) and is refused.

Input	Shape	Result
`[1500, 3, 5.0]`	1D	`ValueError: Expected 2D array, got 1D array instead`
`[[1500, 3, 5.0]]`	2D, no column names	a price, plus a warning about missing column names
`pd.DataFrame([[1500, 3, 5.0]], columns=features)`	2D, with names	`428,269`, no warning

Common mistake: predicting far outside the training data. A 10,000 square-foot building gets a “price” of about $1,959,228, but our largest sale was 3,400 square feet: the model has never seen such a building, and the number should not be trusted.

More examples:

Code	Result	What it tells you
`np.array([1500, 3, 5.0]).shape`	`(3,)`	1D: three values, no rows
`np.array([1500, 3, 5.0]).reshape(1, -1).shape`	`(1, 3)`	2D: one row, three columns
`model.predict(pd.DataFrame([[1500, 3, 5.0], [2500, 4, 10.0]], columns=features))`	`[428269, 613506]`	two buildings in one call
`model.predict(df[features].iloc[[0]])`	`[361064]`	one existing row; double brackets keep it a table (actual price: `345000`)

Remember: one row = a one-row DataFrame with the same column names.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S33_demo_1.py), and check the dataset is in data/.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors. First time only: pip install scikit-learn.
9. Hands-On Coding Micro-Examples:
- Every example has three parts: 1. Input, 2. Process, and 3. Output. Name each file with your own name and the session code, for example Reacher_S33_demo_1.py (pattern: YourName_S33_demo_1.py).
- Micro-Example 1 (Train a Price Model): YourName_S33_demo_1.py
```
# 1. Input: load the sales and split them
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
df = pd.read_csv("data/session_03_03_dataset.csv")
features = ["Square_Feet_Units", "Bedrooms_Count", "Distance_To_CBD_Miles"]
X = df[features]
y = df["Sale_Price_USD"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

# 2. Process: learn the price formula from the training rows
model = LinearRegression()
model.fit(X_train, y_train)

# 3. Output: the learned formula
print("Intercept:", round(model.intercept_, 1))
for name, coef in zip(features, model.coef_):
    print(f"{name}: {coef:,.1f}")
```
  - How it works, line by line:
    - features = [...] lists the 3 input columns; X = df[features] and y = df["Sale_Price_USD"] separate inputs and answer (Session 3.2).
    - train_test_split(..., test_size=0.25, random_state=42) keeps 4 of the 14 sales aside for testing.
    - model = LinearRegression() creates an empty model; model.fit(X_train, y_train) learns the formula from the 10 training sales (section 7.1).
    - zip(features, model.coef_) walks through the feature names and their coefficients side by side, so each line prints a name with its number.
    - {coef:,.1f} prints the number with thousands separators and 1 decimal.
  - Read the output:
    Intercept: 123568.1 Square_Feet_Units: 180.1 Bedrooms_Count: 14,703.6 Distance_To_CBD_Miles: -1,915.8
    - Each extra square foot adds about $180, each extra room about $14,700, and each extra mile from the center subtracts about $1,900, with the other features held fixed (section 7.2).
    - The intercept is the formula’s starting point. On its own it has no business meaning: no building has 0 square feet.
- Quick Check 1 (.score()):
  - New tool: model.score(X_test, y_test) returns R-squared on the test rows in one call. Why: a quick first check of how well the model explains prices it has not seen.
  - Task: add one line to part 3. Output: print("Test R-squared:", round(model._____(X_test, y_test), 3))
  - Check: the Terminal shows Test R-squared: 0.996.
- Micro-Example 2 (Predict the Test Sales and Measure the Error): YourName_S33_demo_2.py
```
# 1. Input: load, split, and train (same as Micro-Example 1)
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
df = pd.read_csv("data/session_03_03_dataset.csv")
features = ["Square_Feet_Units", "Bedrooms_Count", "Distance_To_CBD_Miles"]
X_train, X_test, y_train, y_test = train_test_split(
    df[features], df["Sale_Price_USD"], test_size=0.25, random_state=42
)
model = LinearRegression().fit(X_train, y_train)

# 2. Process: predict the 4 test sales and compute the RMSE
y_pred = model.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
comparison = pd.DataFrame({
    "Property_ID": df.loc[X_test.index, "Property_ID"],
    "Actual_USD": y_test,
    "Predicted_USD": y_pred.round(0),
})

# 3. Output: actual vs predicted, then the typical error
print(comparison)
print(f"Test RMSE: ${rmse:,.0f}")
```
  - How it works, line by line:
    - Part 1 loads, splits, and trains, as in Micro-Example 1. LinearRegression().fit(...) creates and fits in one line.
    - model.predict(X_test) returns 4 estimated prices.
    - mean_squared_error() averages the squared errors; np.sqrt() (square root, from NumPy) brings the result back to dollars: the RMSE (section 7.3).
    - pd.DataFrame({...}) builds a comparison table. df.loc[X_test.index, "Property_ID"] finds the IDs of the test rows, because the split keeps the original row labels.
  - Read the output:
    Property_ID Actual_USD Predicted_USD 9 PROP-409 465000.0 460268.0 11 PROP-411 330000.0 345735.0 0 PROP-400 345000.0 361064.0 12 PROP-412 780000.0 777859.0 Test RMSE: $11,539
    - The model is close on the large building (PROP-412: off by $2,141) and on PROP-409.
    - It overprices the two small buildings, PROP-411 and PROP-400, by about $16,000 each: small buildings deserve a manual check.
    - A typical error of about $11,500 on prices averaging $513,000 is about 2%: good enough for a first quote.
- Quick Check 2 (mean_absolute_error()):
  - New tool: mean_absolute_error(y_test, y_pred) returns the average error size in dollars. Why: the easiest error number to explain: “on average, we are off by this much”.
  - Task: change the import line to from sklearn.metrics import mean_squared_error, mean_absolute_error, then add one line to part 3. Output: print(f"Test MAE: ${___________________(y_test, y_pred):,.0f}")
  - Check: the Terminal shows Test MAE: $9,668, lower than the RMSE because RMSE weighs the two larger errors (about $16,000 each) more heavily.
- Micro-Example 3 (Five Splits Instead of One): YourName_S33_demo_3.py
```
# 1. Input: load the sales
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
df = pd.read_csv("data/session_03_03_dataset.csv")
features = ["Square_Feet_Units", "Bedrooms_Count", "Distance_To_CBD_Miles"]

# 2. Process: train and test 5 times, each time on a different part
scores = cross_val_score(LinearRegression(), df[features],
                         df["Sale_Price_USD"], cv=5, scoring="r2")

# 3. Output: the 5 scores and their average
print("R-squared per fold:", scores.round(3).tolist())
print(f"Average R-squared: {scores.mean():.3f}")
```
  - How it works, line by line:
    - cross_val_score(LinearRegression(), X, y, cv=5, scoring="r2") does all the work of section 7.4: it cuts the 14 sales into 5 folds, trains 5 fresh models, and tests each one on its own fold.
    - It returns 5 scores, one per fold. .round(3).tolist() prints them as a plain list.
    - scores.mean() averages them.
  - Read the output:
    R-squared per fold: [0.994, 0.992, 0.999, 0.996, 0.999] Average R-squared: 0.996
    - Every fold scores above 0.99: the model does well no matter which sales are held out, not just on one lucky split.
    - The average, 0.996, equals the single split’s score from Quick Check 1, which confirms that result.
- Quick Check 3 (RMSE per fold):
  - New tool: scoring="neg_root_mean_squared_error" makes cross_val_score() return the RMSE of each fold as a negative number (Scikit-Learn treats higher scores as better), so put a minus sign in front to read it. Why: shows the error in dollars for every fold.
  - Task: add two lines to part 3. Output: rmse = -cross_val_score(LinearRegression(), df[features], df["Sale_Price_USD"], cv=5, scoring="______________________________") and print(rmse.round(0).tolist())
  - Check: [7862.0, 15224.0, 4004.0, 8201.0, 3374.0]. The error ranges from about $3,400 to $15,200 depending on which properties are tested.
10. Official Python Documentation & References:

11. Exercise 1: Guided Practice - Property Valuation Engine:

Step 1: Open Script: In Google Antigravity IDE, create a new script named YourName_S33_regression.py, replacing YourName with your own name (for example, Reacher_S33_regression.py).

Step 2: Train, Predict, and Evaluate: Type the following code:

# 1. Input: load the sales and split them
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
df = pd.read_csv("data/session_03_03_dataset.csv")
features = ["Square_Feet_Units", "Bedrooms_Count", "Distance_To_CBD_Miles"]
X_train, X_test, y_train, y_test = train_test_split(
    df[features], df["Sale_Price_USD"], test_size=0.25, random_state=42
)

# 2. Process: train, then predict the test sales
reg = LinearRegression()
reg.fit(X_train, y_train)
y_pred = reg.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
r2 = r2_score(y_test, y_pred)

# 3. Output: the formula and the test error
print("Coefficients:", dict(zip(features, reg.coef_.round(1).tolist())))
print(f"Test RMSE: ${rmse:,.0f} | R-squared: {r2:.3f}")

Step 3: Interpret for the Manager: In three comment lines, explain each coefficient in plain words, using “while the other features stay the same”.
Step 4: Execute in Terminal: Run python YourName_S33_regression.py in Panel [4] and confirm return code 0.
Success Check: coefficients 180.1 (square feet), 14703.6 (rooms), and -1915.8 (distance); Test RMSE: $11,539 | R-squared: 0.996. Each extra mile from the center lowers the price when size and rooms stay the same.

12. Exercise 2: Applied Practice - Error per Property and a One-Row Prediction:
- Coding Challenge: In YourName_S33_practice.py, extend Exercise 1 to show the error of each test property in percent, then price one new building.
- Task 1 (Percentage Error per Property): Build a table with Property_ID, Actual_USD, Predicted_USD (rounded to whole dollars), and Error_Pct = absolute error / actual price x 100, rounded to 2 decimals. Use abs(y_test - y_pred) for the absolute error. Then print the average Error_Pct, called MAPE (Mean Absolute Percentage Error).
- Task 2 (Read the ValueError & AI Diagnosis): Predict a new 1,500 square-foot property with 3 rooms, 5.0 miles from the center, by writing reg.predict([1500, 3, 5.0]). The script stops with ValueError: Expected 2D array, got 1D array instead. Capture the traceback with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and ask what “2D” means.
- Task 3 (Fix the Shape): Wrap the new property in a one-row DataFrame with the same column names: new_property = pd.DataFrame([[1500, 3, 5.0]], columns=features), then predict and print the price with thousands separators.
- Success Check:
  - Error per property: PROP-409 1.02%, PROP-411 4.77%, PROP-400 4.66%, PROP-412 0.27%; MAPE 2.68%.
  - New property: about $428,269. The two smaller properties have the largest percentage errors, so the firm should check small-building quotes by hand.
13. Exercise 3: Advanced Practice - Does a Bigger Model Price Better?:
- Goal: The firm could collect more features for every quote, but collecting data costs money. Find out, with 5-fold cross-validation, whether extra features really reduce the pricing error.
- What you need:
  - File: data/session_03_03_dataset.csv (described in 5. Lab Practice Dataset).
  - Script name: YourName_S33_challenge.py.
  - Tools you already know: LinearRegression, cross_val_score() with scoring="neg_root_mean_squared_error" (Quick Check 3), dictionaries, for loops, and pd.DataFrame().
- Requirements (follow them in order):
  1. Define four candidate feature lists in a dictionary:
    - "A: size only": ["Square_Feet_Units"]
    - "B: size + distance": ["Square_Feet_Units", "Distance_To_CBD_Miles"]
    - "C: size + rooms + distance": ["Square_Feet_Units", "Bedrooms_Count", "Distance_To_CBD_Miles"]
    - "D: all four": the three above plus "Local_Tax_Rate_Pct"
  2. For each candidate, run 5-fold cross-validation with cv=5 and RMSE scoring, and record the average RMSE and the largest fold RMSE, both rounded to whole dollars, plus the number of features.
  3. Collect the results in one DataFrame with the columns Model, Features, Avg_RMSE_USD, and Worst_Fold_RMSE_USD, sorted by Avg_RMSE_USD from lowest to highest.
  4. Apply the firm’s rule: choose the model with the lowest average RMSE, unless a model with fewer features is within $500 of it; then choose the model with the fewest features among those within $500.
  5. Print the table and the chosen model with one sentence that explains the choice.
- New tools for this exercise:
  - Looping over a dictionary: for name, cols in candidates.items(): gives each key (name) and its value (cols) in turn.
  - Collecting rows: start with rows = [], append one dictionary per model, such as rows.append({"Model": name, "Features": len(cols), ...}), and finish with pd.DataFrame(rows).
  - .max() on the fold results gives the worst fold.
- Hints:
  - First print the 5 fold values of each model before you average them.
  - For the rule, first keep the rows with Avg_RMSE_USD <= best + 500, then sort them by Features.
- Self-Check (expected results):
  - Average RMSE: C $7,733, B $7,837, A $7,957, D $9,296. Worst folds: C $15,224, B $10,415, A $10,279, D $15,463.
  - Within $500 of the best ($7,733): C, B, and A. Chosen: A: size only.
  - Business meaning: in these 14 deals, adding rooms and distance cut the average error by only about $224, while adding the tax rate made it worse by about $1,563. The firm can quote from floor area alone and spend its data budget on collecting more deals instead.

Session 3.4: Core Concepts of Machine Learning: Supervised vs. Unsupervised Learning and Evaluation Metrics

Session 3.4 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 12 (115/11/29 - 115/12/05) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- A card issuer’s fraud model gives every transaction a risk score from 1 to 100; above a cut-off score (the threshold), the transaction is held and the customer gets a verification text
- The two possible mistakes have very different costs: a false alarm costs about $5 of checking, while a missed fraud costs the stolen money
- The fraud operations manager asks: at what risk score should we stop a transaction, and what does each kind of mistake cost us? Today, you judge a model by business cost, not by a single accuracy number
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “Which kind of learning fits this problem?”: tell supervised learning (learning from examples with known answers) from unsupervised learning (finding groups without answers), and group transactions with K-Means
2. “How good is the risk score, really?”: read a confusion matrix (a 2x2 count of right and wrong decisions) and calculate precision, recall, and ROC-AUC
3. “Where should we set the threshold?”: choose the threshold with the lowest business cost, and fix the ValueError raised when scores are passed in place of 0/1 decisions

3. Business Context: Not All Mistakes Cost the Same:

Every decision on a transaction lands in one of four boxes:

	Model says: fraud (hold it)	Model says: legitimate (let it pass)
Really fraud	TP (True Positive): fraud caught	FN (False Negative): fraud missed, the money is lost
Really legitimate	FP (False Positive): false alarm, about $5 to check	TN (True Negative): normal purchase, no cost

In our cost model, a missed fraud costs $1,500 and a false alarm costs $5, so one miss costs as much as 300 false alarms.
The accuracy trap: if only 1% of transactions are fraud, a model that says “never fraud” is 99% accurate and catches nothing. Accuracy hides the costly mistakes.

4. Data Basics: Reading a Classifier’s Report Card:

Four measures, each answering one question:

Measure	Formula	Question it answers
Accuracy	(TP + TN) / all	What share of all decisions was right?
Precision	TP / (TP + FP)	Of the transactions we held, what share was really fraud?
Recall	TP / (TP + FN)	Of all the fraud, what share did we catch?
ROC-AUC	area under the ROC curve, from 0.5 to 1.0	How well does the score rank fraud above legitimate transactions, at every threshold?

A higher threshold holds fewer transactions: fewer false alarms (higher precision) but more missed fraud (lower recall). The threshold moves this trade-off; the business cost decides where it should sit.
ROC-AUC (Area Under the Receiver Operating Characteristic Curve): 0.5 means the score ranks no better than a coin flip; 1.0 means every fraud scores above every legitimate transaction.
Supervised vs unsupervised:
- Supervised: past examples come with the answer (Actual_Fraud_Binary), and the model learns to predict it. Today’s risk score is a supervised model’s output.
- Unsupervised: no answer column; the method finds groups of similar rows by itself. K-Means is the most common example.

5. Lab Practice Dataset:
- File Link: data/session_03_04_dataset.csv
- Audited 16-row card transaction dataset (fictional teaching data), one row per transaction, checked afterwards by fraud investigators:
  - Transaction_ID: Transaction code (e.g., F-TXN-8001).
  - Transaction_Amount_USD: Purchase amount in U.S. Dollars.
  - Foreign_IP_Flag: 1 if the purchase came from a foreign IP (Internet Protocol) address, the network address of the device; 0 if domestic.
  - Failed_Login_Attempts_Count: Failed login attempts before the purchase.
  - Cardholder_Risk_Score: The fraud model’s risk score, from 1 (safe) to 100 (very risky).
  - Actual_Fraud_Binary: The investigators’ answer: 1 = fraud, 0 = legitimate. 7 of the 16 transactions were fraud.
6. Keywords (Core Programming Concepts):
- Supervised vs Unsupervised Learning:
  - What: learning with known answers vs finding groups without answers
  - Why: the first question to ask about any ML project
- Threshold:
  - What: the cut-off score above which the model’s answer becomes “yes”
  - Why: the business sets it, not the model
- Confusion Matrix:
  - What: the 2x2 count of TP, FP, FN, and TN
  - Why: shows each kind of mistake separately, so each can be priced
- Precision and Recall:
  - What: the share of holds that were fraud, and the share of fraud that was held
  - Why: the two sides of the trade-off a threshold controls
- ROC-AUC:
  - What: one number for how well a score ranks positives above negatives
  - Why: compares scoring models before any threshold is chosen
- K-Means Clustering:
  - What: an unsupervised method that splits rows into K groups of similar values
  - Why: finds customer or transaction segments when no labels exist

7. Core Programming & Computational Foundations:

7.1 From a score to a decision: the threshold

The idea: A fraud model does not say “fraud” or “not fraud”. It gives each transaction a score, here from 1 to 100. The business then chooses a threshold: every transaction scoring at or above it is held for checking. The model ranks; the business decides where to draw the line.

See it on real data: the 9 highest-scoring transactions, sorted by score.

Transaction	Score	Really fraud?	Held at 70?	Held at 75?
`F-TXN-8006`	`22`	no	no	no
`F-TXN-8009`	`71`	no	yes (false alarm)	no
`F-TXN-8005`	`74`	yes	yes	no (missed fraud)
`F-TXN-8015`	`76`	yes	yes	yes
`F-TXN-8001`	`78`	yes	yes	yes
`F-TXN-8011` to `F-TXN-8007`	`85` to `92`	yes	yes	yes

Moving the line from 70 to 75 removes one false alarm but lets one fraud through.

In code: (df["Cardholder_Risk_Score"] >= 70).astype(int) turns every score into 1 (hold) or 0 (pass).

More examples:

Threshold	Transactions held	Who they are
`50`	`8`	all 7 frauds + 1 false alarm
`70`	`8`	the same 8
`72`	`7`	exactly the 7 frauds
`80`	`4`	3 frauds slip through
`90`	`1`	only the riskiest one

Remember: the model gives scores; the threshold turns them into decisions.

7.2 The confusion matrix: four kinds of outcome

The idea: Every decision lands in one of four boxes. Counting them separately shows not only how often the model is right, but which mistakes it makes, and those mistakes have different costs.

	Held (`1`)	Passed (`0`)
Really fraud	TP: fraud caught	FN: fraud missed
Really legitimate	FP: false alarm	TN: normal purchase passed

See it on real data: at threshold 70, confusion_matrix(y_true, y_pred) returns [[8 1] [0 7]]. Scikit-Learn puts the “really legitimate” row first:

	Passed	Held
Really legitimate	TN `8`	FP `1`
Really fraud	FN `0`	TP `7`

Common mistake: reading the matrix in the wrong order. .ravel() flattens it as tn, fp, fn, tp, always in that order.

More examples: the four counts at other thresholds.

Threshold	TP	FP	FN	TN
`50`	`7`	`1`	`0`	`8`
`72`	`7`	`0`	`0`	`9`
`80`	`4`	`0`	`3`	`9`
`90`	`1`	`0`	`6`	`9`

Remember: TP and TN are right; FP and FN are the two different mistakes.

7.3 Accuracy, precision, and recall

The idea: Three measures answer three different questions:
- Accuracy = (TP + TN) / all: what share of all decisions was right?
- Precision = TP / (TP + FP): of the transactions we held, what share was really fraud?
- Recall = TP / (TP + FN): of all the fraud, what share did we catch?

See it on real data:

Threshold	Accuracy	Precision	Recall
`70`	`0.938`	`0.875`	`1.000`
`75`	`0.938`	`1.000`	`0.857`
`80`	`0.812`	`1.000`	`0.571`
`90`	`0.625`	`1.000`	`0.143`

Raising the threshold makes every hold more certain (precision up) but lets more fraud through (recall down).

The accuracy trap: a “model” that never holds anything is right on every legitimate transaction: 9 of 16, an accuracy of 0.562, while catching no fraud at all. In real card data, where fraud is often under 1% of transactions, such a useless model would be over 99% accurate.

More examples: the F1-score combines precision and recall into one number (their harmonic mean); it is high only when both are high.

Threshold	F1-score
`50`	`0.933`
`72`	`1.000`
`75`	`0.923`
`80`	`0.727`
`90`	`0.250`

Remember: precision asks “were the holds right?”; recall asks “did we catch the fraud?”

7.4 ROC-AUC: ranking quality at every threshold

The idea: ROC-AUC (Area Under the Receiver Operating Characteristic Curve) judges a score before any threshold is chosen. It answers: if you pick one fraud and one legitimate transaction at random, how often does the fraud get the higher score? 0.5 is a coin flip; 1.0 means every fraud scores above every legitimate transaction.

See it on real data: roc_auc_score(y_true, scores) takes the scores themselves, not 0/1 decisions.

Score used	ROC-AUC	Meaning
`Cardholder_Risk_Score`	`1.000`	every fraud (`74`-`92`) above every legitimate one (`12`-`71`)
`Transaction_Amount_USD`	`0.984`	amount alone ranks almost as well
`Failed_Login_Attempts_Count`	`0.984`	so do failed logins
`Foreign_IP_Flag`	`0.944`	a yes/no flag ranks less finely

Common mistake: passing 0/1 decisions to roc_auc_score(), or scores to confusion_matrix(). The first loses information; the second raises ValueError: Classification metrics can't handle a mix of binary and continuous targets.

More examples:

Code	Result	What it tells you
`roc_auc_score(y_true, 100 - scores)`	`0.0`	a score that is perfectly backwards
`roc_auc_score(y_true, [50] * 16)`	`0.5`	the same score for everyone: a coin flip
`roc_auc_score(y_true, scores / 100)`	`1.0`	rescaling does not change the ranking

Remember: AUC compares scores; precision and recall judge decisions.

7.5 Unsupervised learning: groups without answers

The idea: Supervised learning learns from examples with answers, such as investigated fraud cases. Unsupervised learning has no answer column: it looks for groups of similar rows by itself. K-Means, the most common method, places K group centers and assigns every row to its nearest center, moving the centers until the groups stop changing.

See it on real data: K-Means on amount and risk score only (the fraud column is not used).

K (number of groups)	Group sizes	Inertia (spread inside the groups; lower is tighter)
`2`	`8`, `8`	`3.93`
`3`	`4`, `8`, `4`	`0.70`
`4`	`4`, `8`, `2`, `2`	`0.38`

Inertia always falls as K grows, but the big drop is from 2 to 3 groups; after that, extra groups only split small groups further. This “elbow” suggests K = 3.

Common mistake: expecting K-Means to name the groups. It only numbers them 0, 1, 2; a person reads each group’s profile and gives it a business name, such as “small, safe purchases”.

More examples: kmeans.predict() places new rows (scaled the same way) into the existing groups.

Code or input	Result	What it tells you
a new purchase: `$5,000`, score `95`	group `2`	large, high-risk purchases
a new purchase: `$40`, score `10`	group `1`	small, safe purchases
a new purchase: `$1,900`, score `73`	group `0`	mid-size, higher-risk purchases
`kmeans.cluster_centers_.shape`	`(3, 2)`	3 group centers, 2 features each

Remember: supervised = learn the answer; unsupervised = find the groups, then name them.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S34_demo_1.py), and check the dataset is in data/.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors. First time only: pip install scikit-learn.
9. Hands-On Coding Micro-Examples:
- Every example has three parts: 1. Input, 2. Process, and 3. Output. Name each file with your own name and the session code, for example Reacher_S34_demo_1.py (pattern: YourName_S34_demo_1.py).
- Micro-Example 1 (Confusion Matrix at Threshold 70): YourName_S34_demo_1.py
```
# 1. Input: the investigators' answers and the model's scores
import pandas as pd
from sklearn.metrics import confusion_matrix, precision_score, recall_score
df = pd.read_csv("data/session_03_04_dataset.csv")
y_true = df["Actual_Fraud_Binary"]

# 2. Process: hold every transaction scoring 70 or more
y_pred = (df["Cardholder_Risk_Score"] >= 70).astype(int)
cm = confusion_matrix(y_true, y_pred)
precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)

# 3. Output: the 2x2 table, then the two key measures
print("Rows: really legitimate, really fraud | Columns: passed, held\n", cm)
print(f"Precision: {precision:.3f} | Recall: {recall:.3f}")
```
  - How it works, line by line:
    - y_true holds the investigators’ answers: 1 for fraud, 0 for legitimate.
    - (df["Cardholder_Risk_Score"] >= 70).astype(int) turns each score into a decision at threshold 70 (section 7.1).
    - confusion_matrix(y_true, y_pred) counts the four outcomes (section 7.2).
    - precision_score() and recall_score() compute the two key measures (section 7.3).
  - Read the output:
    Rows: really legitimate, really fraud | Columns: passed, held [[8 1] [0 7]] Precision: 0.875 | Recall: 1.000
    - Top row: 8 legitimate transactions passed, 1 held by mistake (F-TXN-8009, score 71).
    - Bottom row: 0 frauds missed, all 7 held.
    - Recall 1.000: every fraud was caught. Precision 0.875: 7 of the 8 holds were real fraud.
- Quick Check 1 (classification_report()):
  - New tool: classification_report(y_true, y_pred) prints precision, recall, and the F1-score (one number that balances precision and recall) for both classes, plus accuracy. Why: the full report card in one call.
  - Task: add classification_report to the import line, then add one line to part 3. Output: print(_____________________(y_true, y_pred))
  - Check: the accuracy line shows 0.94, and the row for class 1 shows precision 0.88 and recall 1.00.
- Micro-Example 2 (ROC-AUC: Ranking Quality at Every Threshold): YourName_S34_demo_2.py
```
# 1. Input: the answers and the scores
import pandas as pd
from sklearn.metrics import roc_auc_score
df = pd.read_csv("data/session_03_04_dataset.csv")
y_true = df["Actual_Fraud_Binary"]

# 2. Process: AUC uses the scores directly, not 0/1 decisions
auc = roc_auc_score(y_true, df["Cardholder_Risk_Score"])

# 3. Output: compare the score ranges of the two groups
print(f"ROC-AUC: {auc:.3f}")
print(df.groupby("Actual_Fraud_Binary")["Cardholder_Risk_Score"].agg(["min", "max"]))
```
  - How it works, line by line:
    - roc_auc_score(y_true, df["Cardholder_Risk_Score"]) uses the raw scores, because it checks every possible threshold at once (section 7.4).
    - df.groupby("Actual_Fraud_Binary")["Cardholder_Risk_Score"].agg(["min", "max"]) shows the lowest and highest score in each group, to see why the AUC is what it is.
  - Read the output:
    ROC-AUC: 1.000 min max Actual_Fraud_Binary 0 12 71 1 74 92
    - Legitimate transactions score from 12 to 71; fraud from 74 to 92. The two ranges do not overlap, so the ranking is perfect: AUC 1.000.
    - Any threshold from 72 to 74 would separate them exactly. With only 16 transactions, treat this as encouraging, not as proof.
- Quick Check 2 (accuracy_score()):
  - New tool: accuracy_score(y_true, y_pred) returns the share of correct decisions. Why: shows how a single accuracy number can look good while hiding the costly mistakes.
  - Task: add two lines to part 3. Output: from sklearn.metrics import accuracy_score and print(______________(y_true, (df["Cardholder_Risk_Score"] >= 90).astype(int)))
  - Check: the Terminal shows 0.625. At threshold 90, 62.5% of the decisions are right, yet 6 of the 7 frauds pass through.
- Micro-Example 3 (K-Means: Groups Without Answers): YourName_S34_demo_3.py
```
# 1. Input: two features only; the fraud column is NOT used
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
df = pd.read_csv("data/session_03_04_dataset.csv")
features = df[["Transaction_Amount_USD", "Cardholder_Risk_Score"]]

# 2. Process: scale, then form 3 groups of similar transactions
scaled = StandardScaler().fit_transform(features)
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
df["Cluster_ID"] = kmeans.fit_predict(scaled)

# 3. Output: the average profile of each group, and its fraud share
profile = df.groupby("Cluster_ID")[
    ["Transaction_Amount_USD", "Cardholder_Risk_Score", "Actual_Fraud_Binary"]
].mean().round(1)
print(profile)
```
  - How it works, line by line:
    - Only two features go in: amount and risk score. The fraud column is deliberately left out; this is unsupervised learning (section 7.5).
    - StandardScaler().fit_transform(features) puts both on one scale (Session 3.2), so the dollar amounts do not outweigh the scores.
    - KMeans(n_clusters=3, random_state=42, n_init=10) asks for 3 groups; n_init=10 tries 10 starting points and keeps the best; random_state=42 makes the result repeatable.
    - .fit_predict(scaled) finds the groups and returns each row’s group number, stored as Cluster_ID.
    - The groupby() table averages each group’s columns, including the fraud column, which K-Means never saw.
  - Read the output:
    Transaction_Amount_USD Cardholder_Risk_Score Actual_Fraud_Binary Cluster_ID 0 1795.0 74.8 0.8 1 52.2 16.1 0.0 2 3587.5 88.5 1.0
    - Group 1: small, low-risk purchases, no fraud. Group 0: mid-size, higher-risk purchases, 80% fraud. Group 2: large, high-risk purchases, all fraud.
    - The group numbers have no meaning in themselves; a person names the groups from these profiles.
    - Without seeing the answers, K-Means found groups that line up with fraud: a sign that amount and score carry the fraud pattern.
- Quick Check 3 (.size()):
  - New tool: df.groupby("Column").size() counts the rows in each group. Why: a segment’s size matters as much as its profile.
  - Task: add one line to part 3. Output: print(df.groupby("Cluster_ID").____())
  - Check: group sizes 4, 8, and 4.
10. Official Python Documentation & References:

11. Exercise 1: Guided Practice - Fraud Model Cost Report:

Step 1: Open Script: In Google Antigravity IDE, create a new script named YourName_S34_evaluation.py, replacing YourName with your own name (for example, Reacher_S34_evaluation.py).

Step 2: Count the Outcomes and Price the Mistakes: Type the following code:

# 1. Input: the answers and the scores
import pandas as pd
from sklearn.metrics import confusion_matrix, precision_score, recall_score
df = pd.read_csv("data/session_03_04_dataset.csv")
y_true = df["Actual_Fraud_Binary"]

# 2. Process: decisions at threshold 70, then the cost of the mistakes
y_pred = (df["Cardholder_Risk_Score"] >= 70).astype(int)
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
false_alarm_cost = 5.0
missed_fraud_cost = 1500.0
total_cost = fp * false_alarm_cost + fn * missed_fraud_cost

# 3. Output: counts, measures, and the cost
print(f"TP={tp}, FP={fp}, FN={fn}, TN={tn}")
print(f"Precision={precision_score(y_true, y_pred):.2f}, "
      f"Recall={recall_score(y_true, y_pred):.2f}")
print(f"Cost of mistakes: ${total_cost:,.2f}")

Two strings written next to each other inside print() are joined into one; this keeps long lines readable.

Step 3: Interpret: In two comment lines, explain why one missed fraud costs as much as 300 false alarms, and what that means for where the threshold should sit.
Step 4: Execute in Terminal: Run python YourName_S34_evaluation.py in Panel [4] and confirm return code 0.
Success Check: TP=7, FP=1, FN=0, TN=8; Precision=0.88, Recall=1.00; Cost of mistakes: $5.00.

12. Exercise 2: Applied Practice - Find the Cheapest Threshold:
- Coding Challenge: In YourName_S34_practice.py, test thresholds from 50 to 90 in steps of 5 and find the one with the lowest cost.
- Task 1 (Threshold Loop): Loop with for threshold in range(50, 95, 5):. For each threshold, compute the decisions, tn, fp, fn, tp, and the cost (false alarm $5, missed fraud $1,500). Append one dictionary per threshold to a list, and turn the list into a DataFrame with Threshold, TP, FP, FN, TN, and Cost_USD. range(50, 95, 5) gives 50, 55, …, 90; the stop value 95 itself is not included.
- Task 2 (Read the ValueError & AI Diagnosis): Inside the loop, try confusion_matrix(y_true, df["Cardholder_Risk_Score"] / 100), passing the scores instead of the decisions. The script stops with ValueError: Classification metrics can't handle a mix of binary and continuous targets. Capture the traceback with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and ask how to turn scores into decisions.
- Task 3 (Choose and Save): Fix the call with (score >= threshold).astype(int). Choose the threshold with the lowest cost; if several tie, choose the highest of them, because it sends the fewest customers to verification on future transactions. Save the table to data/fraud_threshold_costs.csv with index=False.
- Success Check: thresholds 50 to 70 cost $5 (1 false alarm); 75 costs $1,500; 80 and 85 cost $4,500; 90 costs $9,000. Chosen threshold: 70.
13. Exercise 3: Advanced Practice - Price the Misses with Real Amounts:
- Goal: The flat $1,500 per missed fraud is a guess. Replace it with each missed transaction’s real amount, test every possible threshold, and show the manager how much the guess was off.
- What you need:
  - File: data/session_03_04_dataset.csv (described in 5. Lab Practice Dataset).
  - Script name: YourName_S34_challenge.py.
  - Tools you already know: Boolean masks with & and ~ (Session 1.4), .sum(), for loops, range(), lists of dictionaries, and pd.DataFrame().
- Requirements (follow them in order):
  1. For every whole-number threshold from 1 to 100, mark the held transactions: held = df["Cardholder_Risk_Score"] >= threshold.
  2. Compute two costs for each threshold:
    - Flat_Cost_USD: false alarms x $5 + missed frauds x $1,500
    - Real_Cost_USD: false alarms x $5 + the sum of Transaction_Amount_USD of the missed frauds
  3. Put all 100 results in one DataFrame with Threshold, False_Alarms, Missed_Frauds, Flat_Cost_USD, and Real_Cost_USD.
  4. Print every threshold with the lowest Real_Cost_USD.
  5. Print the rows for thresholds 70, 75, 80, and 90, and add Gap_USD = Real_Cost_USD - Flat_Cost_USD.
  6. In two comment lines, recommend a threshold and explain why the flat guess is risky.
- New tools for this exercise:
  - A false alarm is a held legitimate transaction: held & (df["Actual_Fraud_Binary"] == 0). A missed fraud is a passed fraud: ~held & (df["Actual_Fraud_Binary"] == 1). ~ flips True and False.
  - Summing money over a mask: df.loc[missed, "Transaction_Amount_USD"].sum() adds up the amounts of the missed frauds only.
  - Selecting several rows by value: results[results["Threshold"].isin([70, 75, 80, 90])].
- Hints:
  - Inside the loop, compute the two masks first, then both costs, then append one dictionary.
  - range(1, 101) gives 1, 2, …, 100.
- Self-Check (expected results):
  - Lowest real cost: $0 at thresholds 72, 73, and 74: every fraud held and no false alarm.
  - Gaps: threshold 70: $0 ($5 both ways); 75: $600 ($1,500 flat vs $2,100 real); 80: $830 ($4,500 vs $5,330); 90: $6,480 ($9,000 vs $15,480).
  - Business meaning: the missed frauds at high thresholds are the large ones, so the flat guess understates the loss more and more as the threshold rises. On these 16 past transactions the safest choice is 72 to 74, but a perfect result on so few rows will not last; re-check the threshold every month on new investigated cases, and keep using real amounts in the cost.

Module 4: Strategic AI Integration, Enterprise Decision-Making, and AI Governance

Module 4 focuses on executive deployment, decision integration, and risk management. Students examine how machine learning predictions translate into operational policies, explore dynamic pricing and churn mitigation algorithms, and evaluate ethical considerations including algorithmic bias, data privacy, and artificial intelligence governance.

Session 4.1: Integration of Business Analytics and Enterprise AI Architecture: End-to-End Decision Systems

Session 4.1 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 13 (115/12/06 - 115/12/12) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- A model that lives in an analyst’s notebook creates no value; value comes only when its predictions trigger real actions, every day, without a person running the code
- The card issuer from Session 3.4 wants its fraud model to score each night’s transactions automatically and hold the risky ones
- The head of operations asks: how do we put the fraud model to work every night, and how will we know when it starts to fail? Today, you save a trained model, reuse it on new data, and build the checks that warn you when the world changes
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “How does a model get from training to daily use?”: map an end-to-end decision system (data in, checks, model, action), and save and reload a model with joblib
2. “Is new data still like the training data?”: detect data drift (a change in the incoming data that makes a model less reliable) by comparing averages
3. “What if something breaks at night?”: monitor latency and accuracy from a run log, and handle a missing model file (FileNotFoundError) with a safe fallback

3. Business Context: From a Model to a Decision System:

A decision system has four stages, run every night as one batch (a group of records processed together):

Stage	What happens	Example
Ingestion	collect the day’s data	120,000 card transactions
Validation	check it before use	no missing values, all columns present
Scoring	the model gives each record a probability	fraud probability from 0 to 1
Action	a business rule turns scores into actions	hold every transaction at 0.5 or above

Batch vs real-time:
- Batch scoring runs on a schedule, such as every night, for many records at once. It is cheaper, and fine when decisions can wait a few hours.
- Real-time scoring answers one request within milliseconds, for example while a customer waits at the checkout. It is faster but costs more to build and run.
Models get worse over time, because customers and markets change. Two warning signs to watch:
- Data drift: the new data no longer looks like the training data (for example, purchases are much larger in the holiday season)
- Performance decay: accuracy on recent, checked cases falls

4. Data Basics: Saving a Model and Watching It:
- Training is slow and done rarely; scoring is fast and done often. So the trained model is saved to a file once, then loaded by the nightly job:
  - joblib.dump(model, "models/fraud_model.joblib") saves it
  - joblib.load("models/fraud_model.joblib") loads it back, ready to predict, with no retraining
- A simple drift check compares each feature’s average in the new batch with its average in the training data:
  - change in percent = (new average - training average) / training average x 100
  - Our rule: flag a feature when the change is more than 20% in either direction
- A run log records every night’s stages with their record counts and latency (how long a stage took, in milliseconds; 1,000 ms = 1 second). Rising latency or falling accuracy is an early warning.
5. Lab Practice Dataset:
- Three files (all fictional teaching data):
  - Training data: data/session_03_04_dataset.csv, the 16 investigated transactions from Session 3.4. The model uses Transaction_Amount_USD, Foreign_IP_Flag, and Failed_Login_Attempts_Count to predict Actual_Fraud_Binary.
  - New batch: data/session_04_01_new_batch.csv, 8 new transactions from a holiday-season night, with the same three feature columns and no answer column.
  - Run log: data/session_04_01_dataset.csv, 20 rows: 5 nights x 4 stages.
- Run log columns:
  - Run_Date: The night of the run, 2024-12-01 to 2024-12-05.
  - Pipeline_Stage: Ingestion, Validation, Scoring, or Action.
  - Records_Count: Records processed by the stage. For Action, the number of held transactions.
  - Latency_Ms: Time the stage took, in milliseconds.
  - Model_Accuracy: Accuracy on that night’s investigated cases. Filled only for the Scoring stage; empty (NaN) for the others.
6. Keywords (Core Programming Concepts):
- End-to-End Decision System:
  - What: ingestion, validation, scoring, and action joined into one automatic run
  - Why: turns predictions into business actions without manual steps
- Batch vs Real-Time Scoring:
  - What: many records on a schedule vs one record on request
  - Why: the business deadline decides which one is worth its cost
- Model Persistence (joblib):
  - What: saving a trained model to a file and loading it later
  - Why: train once, score every night
- Data Drift:
  - What: new data that no longer looks like the training data
  - Why: a model is only reliable on data like the data it learned from
- Monitoring:
  - What: tracking counts, latency, and accuracy for every run
  - Why: catches a failing system before customers do
- Fallback:
  - What: a simple, safe rule used when the model is not available
  - Why: the business keeps running when something breaks

7. Core Programming & Computational Foundations:

7.1 Train once, use many times: saving a model

The idea: Training a model is slow and done rarely; using it is fast and done every night. So the trained model is saved to a file once, and the nightly job loads that file instead of training again. joblib.dump(model, path) saves it; joblib.load(path) loads it back, exactly as it was, ready to predict.

See it on real data:

Step	Code	Result
Save	`joblib.dump(model, "models/fraud_model.joblib")`	a small file, about 1.7 KB
Load	`loaded = joblib.load("models/fraud_model.joblib")`	the same trained model
Check	`(model.predict_proba(X) == loaded.predict_proba(X)).all()`	`True`: identical predictions

Why save the whole Pipeline: the scaler and the model are saved together, so the nightly job scales new data with the training averages (1371.72, 0.5, 1.56), exactly as during training. Saving only the model and re-fitting a scaler on tonight’s data would be data leakage (Session 3.2).
Common mistake: a wrong path. joblib.load("models/fraud_modle.joblib") raises FileNotFoundError: [Errno 2] No such file or directory. Check first with os.path.exists(path).

More examples:

Code	Result	What it tells you
`os.makedirs("models", exist_ok=True)`	creates the folder	`exist_ok=True`: no error if it is already there
`os.path.exists("models/fraud_model.joblib")`	`True`	the file is in place
`os.path.exists("models/nope.joblib")`	`False`	check before loading
`list(loaded.named_steps)`	`['scale', 'classify']`	the loaded pipeline still has both steps

Remember: save the whole pipeline once; load it every night.

7.2 Probabilities, not just yes or no

The idea: model.predict(X) returns a decision, 1 or 0, using a cut-off of 0.5. model.predict_proba(X) returns the probabilities behind it: two numbers per row, the probability of class 0 and of class 1, which always add up to 1. With probabilities, the business can choose its own cut-off and rank cases by risk.

See it on real data: the first two new transactions.

Transaction	`predict_proba`	`[:, 1]` (fraud probability)	`predict`
`N-TXN-9001`	`[0.138, 0.862]`	`0.862`	`1`
`N-TXN-9002`	`[0.930, 0.070]`	`0.070`	`0`

[:, 1] means “all rows (:), second column (1)”.

More examples:

Code (`proba = model.predict_proba(X)`)	Result	What it tells you
`proba.sum(axis=1)`	`1.0` for every row	the two probabilities always add up to 1
`(proba[:, 1] >= 0.9).sum()`	`2`	a strict cut-off holds only the 2 clearest cases
`(proba[:, 1] >= 0.5).sum()`	`4`	the default cut-off
`(proba[:, 1] >= 0.1).sum()`	`5`	a cautious cut-off also holds `N-TXN-9006` (`0.163`)
`proba[:, 1].mean()`	`0.499`	the expected share of fraud in tonight’s batch

Remember: use probabilities when the cut-off is a business decision.

7.3 Data drift: is tonight’s data like the training data?

The idea: A model learned its patterns from past data. If new data looks very different, for example much larger purchases in the holiday season, the model is being asked about situations it rarely saw, and its scores become less reliable. A simple check compares each feature’s average in the new batch with its average in the training data.

See it on real data:

Feature	Training average	Tonight’s average	Change
`Transaction_Amount_USD`	`1371.72`	`2555.00`	`+86.3`% (drift)
`Foreign_IP_Flag`	`0.50`	`0.50`	`0.0`%
`Failed_Login_Attempts_Count`	`1.56`	`1.75`	`+12.2`%

2 of tonight’s 8 purchases are larger than the largest training purchase (4200.0): the biggest is 5200.0.

Common mistake: checking only for increases. A feature that falls by 30% has drifted too; use the absolute change, .abs().

More examples:

Measure	Training	Tonight	What it tells you
median amount	`684.7`	`2150.0`	`+214.0`%: the median shows the shift even more clearly
largest amount	`4200.0`	`5200.0`	tonight goes beyond anything seen in training
purchases above the training maximum		`2` of `8`	scores for these two need extra caution

Remember: compare new averages with training averages, in both directions.

7.4 Monitoring: reading the run log

The idea: A system that runs every night must report on itself. A run log records each stage of each night: how many records, how long it took (latency), and, where checked cases exist, how accurate the model was. Trends in the log warn you before customers notice a problem.

See it on real data: the Scoring stage over 5 nights.

Night	Latency (ms)	Accuracy	Held as share of incoming
2024-12-01	`820`	`0.945`	`4.02`%
2024-12-02	`840`	`0.942`	`4.04`%
2024-12-03	`860`	`0.938`	`4.08`%
2024-12-04	`910`	`0.921`	`4.48`%
2024-12-05	`980`	`0.904`	`4.85`%

Scoring is slower, less accurate, and holds more transactions every night: three signals pointing the same way.

Common mistake: treating an empty value as a problem. Model_Accuracy is empty (NaN) for 15 of 20 rows by design, because only the Scoring stage has an accuracy.

More examples:

Code	Result	What it tells you
`log.groupby("Pipeline_Stage")["Latency_Ms"].mean()`	Scoring `882.0`, Ingestion `365.0`, Validation `193.0`, Action `126.0`	scoring is the slowest stage
`log["Model_Accuracy"].isnull().sum()`	`15`	empty by design outside Scoring
`log[log["Pipeline_Stage"] == "Scoring"]["Model_Accuracy"].min()`	`0.904`	the worst night

Remember: log every run, then watch the trends, not just single nights.

7.5 When something breaks: a safe fallback

The idea: A nightly system must keep the business safe even when a part fails. A fallback is a simple rule used when the model is not available, for example “hold foreign purchases with 2 or more failed logins”. It is weaker than the model, but much better than holding nothing or stopping everything.
```
if os.path.exists(model_path):
    use the model
else:
    use the fallback rule, and report that it was used
```
See it on real data: the fallback holds 3 of tonight’s transactions; the model holds 4. It misses N-TXN-9005, a foreign purchase of $3,900 with only 1 failed login.

More examples:

Transaction	Facts	Model	Fallback rule
`N-TXN-9001`	foreign, 3 failed logins	Hold	Hold
`N-TXN-9003`	foreign, 5 failed logins	Hold	Hold
`N-TXN-9005`	foreign, 1 failed login, $3,900	Hold	Pass
`N-TXN-9007`	foreign, 4 failed logins	Hold	Hold
the 4 domestic purchases	0 or 1 failed logins	Pass	Pass

Remember: plan for failure, and always report which method made the decisions.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S41_demo_1.py), and check the three files are in data/. Micro-Example 1 creates a new models/ folder; it appears here.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors. First time only: pip install scikit-learn joblib.
9. Hands-On Coding Micro-Examples:
- Every example has three parts: 1. Input, 2. Process, and 3. Output. Name each file with your own name and the session code, for example Reacher_S41_demo_1.py (pattern: YourName_S41_demo_1.py). Run Micro-Example 1 first: the other examples load the model it saves.
- Micro-Example 1 (Train Once, Save to a File): YourName_S41_demo_1.py
```
# 1. Input: the investigated transactions from Session 3.4
import os
import joblib
import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
train = pd.read_csv("data/session_03_04_dataset.csv")
features = ["Transaction_Amount_USD", "Foreign_IP_Flag",
            "Failed_Login_Attempts_Count"]

# 2. Process: scale + logistic regression, trained in one Pipeline
model = Pipeline([
    ("scale", StandardScaler()),
    ("classify", LogisticRegression()),
])
model.fit(train[features], train["Actual_Fraud_Binary"])

# 3. Output: save the trained model as a file
os.makedirs("models", exist_ok=True)
joblib.dump(model, "models/fraud_model.joblib")
print("Saved: models/fraud_model.joblib")
print("Training accuracy:", model.score(train[features], train["Actual_Fraud_Binary"]))
```
  - How it works, line by line:
    - The training data is the 16 investigated transactions from Session 3.4; the model uses 3 features.
    - Pipeline([("scale", StandardScaler()), ("classify", LogisticRegression())]) joins two steps into one object: first standardize, then classify. Each step has a name and a tool.
    - model.fit(...) fits both steps in order.
    - os.makedirs("models", exist_ok=True) creates the models folder if needed; joblib.dump() writes the trained pipeline into it (section 7.1).
    - model.score(...) measures accuracy on the training rows.
  - Read the output:
    Saved: models/fraud_model.joblib Training accuracy: 0.9375
    - The file now exists in the models folder, visible in Panel [1].
    - 0.9375 = 15 of 16 correct. It is measured on the rows the model learned from, so it is a ceiling, not an honest test.
- Quick Check 1 (os.path.exists()):
  - New tool: os.path.exists(path) returns True if the file exists. Why: the nightly job should confirm the model is there before it starts.
  - Task: add one line to part 3. Output: print(os.path.______("models/fraud_model.joblib"))
  - Check: the Terminal shows True.
- Micro-Example 2 (The Nightly Job: Load, Score, Decide): YourName_S41_demo_2.py
```
# 1. Input: the saved model and tonight's new transactions
import joblib
import pandas as pd
model = joblib.load("models/fraud_model.joblib")
batch = pd.read_csv("data/session_04_01_new_batch.csv")
features = ["Transaction_Amount_USD", "Foreign_IP_Flag",
            "Failed_Login_Attempts_Count"]

# 2. Process: fraud probability, then the business rule (hold at 0.5 or above)
batch["Fraud_Probability"] = model.predict_proba(batch[features])[:, 1].round(3)
batch["Decision"] = "Pass"
batch.loc[batch["Fraud_Probability"] >= 0.5, "Decision"] = "Hold"

# 3. Output: print and save tonight's decisions
print(batch[["Transaction_ID", "Fraud_Probability", "Decision"]])
batch.to_csv("data/nightly_decisions.csv", index=False)
```
  - How it works, line by line:
    - joblib.load(...) brings back the trained pipeline: no training happens here, so this script runs in a fraction of a second.
    - model.predict_proba(batch[features])[:, 1] gives each new transaction its fraud probability (section 7.2). The feature columns must have the same names as in training.
    - batch["Decision"] = "Pass" sets every row to “Pass”; batch.loc[batch["Fraud_Probability"] >= 0.5, "Decision"] = "Hold" changes only the risky rows.
    - batch.to_csv(...) saves tonight’s decisions for the operations team.
  - Read the output:
    Transaction_ID Fraud_Probability Decision 0 N-TXN-9001 0.862 Hold 1 N-TXN-9002 0.070 Pass 2 N-TXN-9003 0.984 Hold 3 N-TXN-9004 0.083 Pass 4 N-TXN-9005 0.789 Hold 5 N-TXN-9006 0.163 Pass 6 N-TXN-9007 0.965 Hold 7 N-TXN-9008 0.073 Pass
    - 4 of 8 transactions are held, all foreign purchases above $2,800.
    - The probabilities fall into two clear groups: above 0.78 or below 0.17. No case sits near the 0.5 cut-off, so the decisions are not borderline.
- Quick Check 2 (Run date stamp):
  - New tool: from datetime import date and date.today() give today’s date. Why: every decision file must show when the decisions were made, for audits.
  - Task: add from datetime import date to part 1. Input, and one line at the end of part 2. Process: batch["Run_Date"] = str(date._____())
  - Check: the saved data/nightly_decisions.csv has a Run_Date column with today’s date, written as YYYY-MM-DD.
- Micro-Example 3 (Drift Check: Does Tonight Look Like Training?): YourName_S41_demo_3.py
```
# 1. Input: training data and tonight's batch
import pandas as pd
train = pd.read_csv("data/session_03_04_dataset.csv")
batch = pd.read_csv("data/session_04_01_new_batch.csv")
features = ["Transaction_Amount_USD", "Foreign_IP_Flag",
            "Failed_Login_Attempts_Count"]

# 2. Process: compare the averages, feature by feature
drift = pd.DataFrame({
    "Train_Avg": train[features].mean().round(2),
    "Batch_Avg": batch[features].mean().round(2),
})
drift["Change_Pct"] = (
    (drift["Batch_Avg"] - drift["Train_Avg"]) / drift["Train_Avg"] * 100
).round(1)
drift["Drift_Flag"] = drift["Change_Pct"] > 20

# 3. Output: the drift report
print(drift)
```
  - How it works, line by line:
    - train[features].mean() and batch[features].mean() give each feature’s average in the two files.
    - Putting both into pd.DataFrame({...}) lines them up by feature name, one row per feature.
    - Change_Pct is the change in percent from the training average (section 7.3).
    - Drift_Flag marks features that changed by more than 20%.
  - Read the output:
    Train_Avg Batch_Avg Change_Pct Drift_Flag Transaction_Amount_USD 1371.72 2555.00 86.3 True Foreign_IP_Flag 0.50 0.50 0.0 False Failed_Login_Attempts_Count 1.56 1.75 12.2 False
    - Tonight’s purchases are on average 86% larger than the training purchases: amount has drifted.
    - The share of foreign purchases is unchanged, and failed logins rose only slightly.
    - The model still runs, but its scores for large purchases deserve a person’s review tonight.
- Quick Check 3 (.abs()):
  - New tool: column.abs() removes the minus sign. Why: a feature that falls by 30% has drifted too; > 20 alone misses it.
  - Task: in part 2. Process, change the flag line to drift["Drift_Flag"] = drift["Change_Pct"].___() > 20
  - Check: the flags do not change for tonight’s batch, but a change of -30.0 would now be flagged as well.
10. Official Python Documentation & References:
11. Exercise 1: Guided Practice - Read the Run Log:
- Step 1: Open Script: In Google Antigravity IDE, create a new script named YourName_S41_monitoring.py, replacing YourName with your own name (for example, Reacher_S41_monitoring.py).
- Step 2: Find the Nights That Need Attention: Type the following code:
```
# 1. Input: the run log, 5 nights x 4 stages
import pandas as pd
log = pd.read_csv("data/session_04_01_dataset.csv")

# 2. Process: health rules for the Scoring stage, then totals per night
scoring = log[log["Pipeline_Stage"] == "Scoring"].copy()
scoring["Alert"] = (
    (scoring["Latency_Ms"] > 900) | (scoring["Model_Accuracy"] < 0.93)
)
nightly = log.groupby("Run_Date").agg(
    Total_Latency_Ms=("Latency_Ms", "sum"),
    Transactions_In=("Records_Count", "first"),
)

# 3. Output: the Scoring stage per night, then the nightly totals
print(scoring[["Run_Date", "Latency_Ms", "Model_Accuracy", "Alert"]])
print(nightly)
```
  - .copy() makes scoring an independent table, so adding the Alert column does not touch log.
  - | means “or”: a night is flagged if either rule is broken.
  - "first" in .agg() takes the first value of each group; here, the Ingestion row’s count.
- Step 3: Interpret: In two comment lines, describe the trend in latency and accuracy across the 5 nights, and what the team should do next.
- Step 4: Execute in Terminal: Run python YourName_S41_monitoring.py in Panel [4] and confirm return code 0.
- Success Check: alerts on 2024-12-04 (latency 910, accuracy 0.921) and 2024-12-05 (980, 0.904); total latency is 1470 ms on the first two nights, then rises every night to 1730 ms, while incoming transactions grow from 120000 to 138900. Growing volume and falling accuracy together suggest drift: the model should be reviewed and possibly retrained.
12. Exercise 2: Applied Practice - Survive a Missing Model File:
- Coding Challenge: In YourName_S41_practice.py, make the nightly job keep working when the model file cannot be found.
- Task 1 (Read the FileNotFoundError & AI Diagnosis): Starting from Micro-Example 2, change the path to "models/fraud_modle.joblib" (a typo). The script stops with FileNotFoundError: [Errno 2] No such file or directory: 'models/fraud_modle.joblib'. Capture the traceback with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and ask how to check for a file before loading it.
- Task 2 (Add a Fallback Rule): Store the path in a variable model_path. If os.path.exists(model_path), load the model and decide as in Micro-Example 2, and set method = "model". Otherwise, use the fallback rule: hold when Foreign_IP_Flag == 1 and Failed_Login_Attempts_Count >= 2, and set method = "fallback rule". Print which method was used.
- Task 3 (Compare): Run the script once with the typo and once with the correct path. Print the held transaction IDs each time.
- Success Check: with the typo, the fallback rule holds 3 transactions (N-TXN-9001, N-TXN-9003, N-TXN-9007); with the correct path, the model holds 4 (it also holds N-TXN-9005, a foreign purchase of $3,900 with only 1 failed login). The script finishes with return code 0 both times: the fallback keeps the business safe, but it is weaker than the model, so the missing file must still be fixed quickly.
13. Exercise 3: Advanced Practice - One Function for the Whole Night:
- Goal: Combine everything into one function that the operations team could schedule to run every night: validate, check drift, score (or fall back), act, and report.
- What you need:
  - Files: data/session_03_04_dataset.csv (training), data/session_04_01_new_batch.csv (tonight), and models/fraud_model.joblib (from Micro-Example 1).
  - Script name: YourName_S41_challenge.py.
  - Tools you already know: functions (Session 2.4), assert (Session 2.3), os.path.exists(), joblib.load(), .predict_proba(), the drift table from Micro-Example 3, and the fallback rule from Exercise 2.
- Requirements (follow them in order):
  1. Write run_nightly(batch_path, model_path, train_path) that returns a dictionary summary.
  2. Validation: check with assert that the batch has the three feature columns and no missing values in them.
  3. Drift: compute the change in percent of each feature’s average, as in Micro-Example 3, and collect the names of the features whose absolute change is above 20.
  4. Scoring: if the model file exists, hold transactions with a fraud probability of 0.5 or above; otherwise, use the fallback rule from Exercise 2.
  5. Action: save the batch with Decision and Method columns to data/nightly_decisions.csv.
  6. Report: return {"records": ..., "held": ..., "method": ..., "drift_features": [...], "needs_review": ...}, where needs_review is True if any feature drifted or the fallback was used.
  7. Call the function twice, once with the correct model path and once with a wrong path, and print both summaries.
- New tools for this exercise:
  - Checking that columns exist: set(features).issubset(batch.columns) is True when every name in features is a column of batch.
  - A list of names from a Boolean column: list(drift.index[drift["Drift_Flag"]]) keeps the row labels (feature names) where the flag is True.
  - Any True in a list: len(drift_features) > 0.
- Hints:
  - Write and test each stage as a few lines first, then move them into the function.
  - Use int(...) around counts so the dictionary prints plain numbers.
- Self-Check (expected results):
  - Correct path: {"records": 8, "held": 4, "method": "model", "drift_features": ["Transaction_Amount_USD"], "needs_review": True}.
  - Wrong path: {"records": 8, "held": 3, "method": "fallback rule", "drift_features": ["Transaction_Amount_USD"], "needs_review": True}.
  - Business meaning: even when the model runs, tonight still needs a person’s review, because the amounts are far outside the training data. A decision system should report its own doubts, not just its decisions.

Session 4.2: AI Applications in Business Decision-Making Part 1: Predictive Customer Analytics and Churn

Session 4.2 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 14 (115/12/13 - 115/12/19) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- A subscription telecom company loses customers every month to churn (customers cancelling their contract)
- Its retention team can call customers or assign them an account manager, but every contact costs money, and calling customers who would stay anyway wastes it
- The head of customer retention asks: which of our current customers should we contact first, and will the campaign pay for itself? Today, you turn churn predictions into revenue at risk and a campaign plan with a measurable return
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “Who is likely to leave?”: train a churn model on past customers and score current customers with a churn probability
2. “How much money is at stake?”: calculate revenue at risk (churn probability x yearly customer value) and rank customers by it
3. “Will the campaign pay off?”: calculate a campaign’s expected net benefit and ROI (Return on Investment), choose who to target, and fix a ValueError caused by missing feature columns

3. Business Context: Pricing a Retention Campaign:

Four ideas, one example (current customer ACT-101):

Term	Meaning	Example
Churn probability	the model’s estimate that this customer leaves within 60 days	`0.876`
Annual value	monthly fee x 12: one year of revenue from this customer	`$2,399.88`
Revenue at risk	churn probability x annual value: the revenue we expect to lose	`$2,102.29`
Expected saved revenue	revenue at risk x success rate of the campaign	`$2,102.29 x 0.40 = $840.92`

CLV (Customer Lifetime Value) is the total value of a customer over the whole relationship, often several years. Today we use one year of value: simpler, and a cautious basis for spending decisions.
A campaign is worth it for a customer when expected saved revenue is larger than the contact cost. Contacting a customer who would stay anyway, or whose value is small, wastes budget.
Net benefit vs ROI: net benefit = saved revenue - cost, in dollars; ROI = net benefit / cost x 100, in percent. The plan with the highest ROI is not always the plan that earns the most money.

4. Data Basics: Learn from the Past, Act on the Present:

Two groups of customers, two files:

File	Customers	Has the answer (`Churned`)?	Used to
History	14 past customers	yes	train the model
Active	10 current customers	no, not yet known	score and target

A retention campaign only makes sense for customers who have not left yet. So the model learns from the history and predicts for the active customers.
What drives churn in our history: after scaling, the model’s coefficients are -0.718 for tenure (longer customers stay), +0.897 for support calls, and +0.878 for usage drop (more calls and falling usage signal leaving).

5. Lab Practice Dataset:
- Two files (fictional teaching data):
  - History: data/session_04_02_dataset.csv, 14 past customers with their outcome; 5 of them churned.
  - Active: data/session_04_02_active.csv, 10 current customers with the same columns except the outcome.
- Columns:
  - Subscriber_ID: Customer code (SUB-... in the history, ACT-... for active customers).
  - Tenure_Months_Count: Months as a customer.
  - Monthly_Subscription_Fee_USD: Monthly fee in U.S. Dollars.
  - Customer_Support_Calls_Count: Calls to customer support in the last 90 days.
  - Usage_Drop_Last_30_Days_Pct: How much the customer’s use of the service fell in the last 30 days, in percent.
  - Churned_Within_60_Days_Binary (history only): 1 = left within 60 days, 0 = stayed.
6. Keywords (Core Programming Concepts):
- Churn:
  - What: customers ending their relationship with the company
  - Why: in subscription businesses, keeping a customer is usually cheaper than winning a new one
- Churn Probability:
  - What: the model’s estimate, from 0 to 1, that a customer will leave
  - Why: ranks customers by risk instead of treating everyone the same
- Revenue at Risk:
  - What: churn probability x annual value
  - Why: turns a probability into dollars a manager can budget against
- Customer Lifetime Value (CLV):
  - What: the total value of a customer over the whole relationship
  - Why: decides how much a customer is worth saving
- Net Benefit and ROI:
  - What: saved revenue minus cost (dollars), and net benefit divided by cost (percent)
  - Why: the two numbers a finance team asks for before approving a campaign
- Targeting Threshold:
  - What: the minimum churn probability for a customer to be contacted
  - Why: sets the size and cost of the campaign

7. Core Programming & Computational Foundations:

7.1 Learn from the past, predict the present

The idea: A churn model needs examples with known outcomes, so it learns from past customers whose story has ended: they either left or stayed. It then scores today’s customers, whose outcome is still unknown. Two tables, one model: fit on the history, predict_proba on the active customers.

See it on real data: how the 14 past customers differ.

Average of past customers	Stayed (9)	Left (5)
Months as a customer	`36.9`	`6.8`
Support calls (last 90 days)	`1.1`	`4.6`
Usage drop (last 30 days)	`9.4`%	`46.0`%

Customers who left were newer, called support more, and had stopped using the service. The model turns these patterns into a probability for each current customer.

Common mistake: scoring the history instead of the active customers. The history has already happened; only current customers can still be saved.

More examples:

Code	Result	What it tells you
`history["Churned_Within_60_Days_Binary"].mean()`	`0.357`	5 of 14 past customers left
`model.score(history[features], history["Churned_Within_60_Days_Binary"])`	`0.929`	13 of 14 correct on the training rows
`active.shape`	`(10, 5)`	10 current customers, no answer column
`model.named_steps["classify"].coef_`	tenure `-0.718`, calls `0.897`, usage drop `0.878`	the signals that push toward leaving

Remember: fit on the past, score the present.

7.2 Expected value: turning a probability into dollars

The idea: A probability alone does not tell a manager how much is at stake. Multiply it by what the customer is worth, and you get the expected loss: what you would lose on average across many customers like this one. For one customer, the loss is either all or nothing; across a whole portfolio, the expected values add up reliably.

Customer	Churn probability	Annual value	Revenue at risk
`ACT-101`	`0.876`	`$2,399.88`	`$2,102.29`
`ACT-105`	`0.941`	`$1,199.88`	`$1,129.09`
`ACT-110`	`0.448`	`$2,399.88`	`$1,075.15`
`ACT-102`	`0.026`	`$1,799.88`	`$46.80`

What the table shows: ACT-105 is the most likely to leave, but ACT-101 has more money at risk because it pays twice as much. ACT-110 has less than a 50% chance of leaving, yet almost as much at risk as ACT-105.
Common mistake: ranking by probability alone. It sends the retention team to the most likely leavers, not to the biggest losses.

More examples:

Code	Result	What it tells you
`active["Revenue_At_Risk_USD"].sum()`	`$7,659.53`	total expected loss
top 3 customers’ share of it	`63.3`%	most of the risk sits with a few customers
`active["Annual_Value_USD"].sum()`	`$16,918.80`	all revenue from the 10 customers
revenue at risk / all revenue	`45.3`%	almost half of this group’s revenue is in danger
`active["Churn_Probability"].mean()`	`0.461`	the average customer here is close to a coin flip

Remember: revenue at risk = probability x value; rank by dollars.

7.3 The economics of a campaign: net benefit and ROI

The idea: Contacting a customer costs money, and it does not always work. For each customer, the expected saved revenue is churn probability x success rate x annual value. The campaign pays off for that customer only if this is larger than the contact cost.

See it on real data: a call costs $75 and keeps 40% of customers who would leave.

Customer	Expected saved revenue	Minus cost	Net
`ACT-101`	`0.876 x 0.40 x 2399.88 = 840.92`	`- 75`	`+765.92`
`ACT-110`	`430.06`	`- 75`	`+355.06`
`ACT-104`	`154.07`	`- 75`	`+79.07`

Two summary numbers: net benefit (saved revenue - cost, in dollars) and ROI (net benefit / cost x 100, in percent). A small, targeted campaign often has the highest ROI; a broader one can earn more dollars. Exercise 2 shows both on our data.

More examples:

Customer	Call net (cost $75, 40% success)	VIP net (cost $250, 60% success)	Verdict
`ACT-101`	`+765.92`	`+1011.38`	VIP pays more
`ACT-104`	`+79.07`	`-18.90`	only the call pays
`ACT-107`	`+132.34`	`+61.01`	the call pays more
`ACT-102`	`-56.28`	`-221.92`	no offer pays: contact costs more than it saves

Remember: contact a customer only when expected savings exceed the contact cost.

7.4 Matching the offer to the customer

The idea: Not every customer deserves the same offer. A cheap offer (a call) suits customers with modest value or moderate risk; an expensive offer (a personal account manager) pays only for customers who are both valuable and very likely to leave. Rules applied in order, from cheapest to most valuable, express this.

See it on real data: Micro-Example 3’s rules.

Rule (applied in this order)	Customers
everyone starts with “Newsletter”	all 10
probability `>= 0.4`: “Retention_Call”	overwrites 6
probability `>= 0.7` and value `>= 1500`: “VIP_Account_Manager”	overwrites 2: `ACT-101`, `ACT-108`

ACT-105 has the highest risk (0.941) but stays with a call: its annual value, $1,199.88, is below the VIP limit.

Common mistake: applying the rules in the wrong order. If the VIP rule ran first, the call rule would then overwrite it, and nobody would get a VIP manager.

More examples:

Rule change	Result	What it tells you
VIP limit lowered to `$1,000`	VIP: `ACT-101`, `ACT-105`, `ACT-108`	`ACT-105` now qualifies
VIP probability raised to `0.8`	VIP: `ACT-101`, `ACT-108`	no change for this group
call threshold lowered to `0.3`	`6` customers at or above it	the same 6 as at `0.4`: nobody sits between 0.3 and 0.4

Remember: set a default, then overwrite from the cheapest offer to the most valuable.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S42_demo_1.py), and check both files are in data/.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors.

9. Hands-On Coding Micro-Examples:

Every example has three parts: 1. Input, 2. Process, and 3. Output. Name each file with your own name and the session code, for example Reacher_S42_demo_1.py (pattern: YourName_S42_demo_1.py).

Micro-Example 1 (Learn from the Past, Score the Present): YourName_S42_demo_1.py

# 1. Input: past customers (with outcomes) and current customers
import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
history = pd.read_csv("data/session_04_02_dataset.csv")
active = pd.read_csv("data/session_04_02_active.csv")
features = ["Tenure_Months_Count", "Customer_Support_Calls_Count",
            "Usage_Drop_Last_30_Days_Pct"]

# 2. Process: train on history, then score the active customers
model = Pipeline([
    ("scale", StandardScaler()),
    ("classify", LogisticRegression()),
])
model.fit(history[features], history["Churned_Within_60_Days_Binary"])
active["Churn_Probability"] = model.predict_proba(active[features])[:, 1].round(3)

# 3. Output: the active customers, most at risk first
print(active.sort_values("Churn_Probability", ascending=False)[
    ["Subscriber_ID", "Tenure_Months_Count", "Churn_Probability"]
])

How it works, line by line:
- Two files are loaded: history (past customers, with the answer) and active (current customers, without it).
- The Pipeline scales the 3 features and fits a logistic regression on the history (section 7.1).
- model.predict_proba(active[features])[:, 1] scores the current customers: the second column is the probability of leaving.
- .sort_values(..., ascending=False) lists the riskiest customers first.

Read the output:

  Subscriber_ID  Tenure_Months_Count  Churn_Probability
     ACT-105                    7              0.941
     ACT-108                    3              0.900
     ACT-101                    5              0.876
     ACT-103                   10              0.675
     ACT-110                   18              0.448
...
     ACT-102                   40              0.026

The top 3 have all been customers for less than 8 months, like the past customers who left.
The longest-standing customers (ACT-102, 40 months; ACT-106, 50 months) are almost certain to stay: under 3%.

Quick Check 1 (named_steps and .coef_):
- New tool: model.named_steps["classify"].coef_ returns the learned coefficients of the logistic regression inside the pipeline. Why: shows which signals push a customer toward leaving.
- Task: add one line to part 3. Output: print(dict(zip(features, model.named_steps["classify"].____[0].round(3).tolist())))
- Check: tenure -0.718, support calls 0.897, usage drop 0.878. A negative coefficient lowers the churn probability; a positive one raises it.

Micro-Example 2 (Revenue at Risk): YourName_S42_demo_2.py

# 1. Input: train and score, as in Micro-Example 1
import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
history = pd.read_csv("data/session_04_02_dataset.csv")
active = pd.read_csv("data/session_04_02_active.csv")
features = ["Tenure_Months_Count", "Customer_Support_Calls_Count",
            "Usage_Drop_Last_30_Days_Pct"]
model = Pipeline([("scale", StandardScaler()),
                  ("classify", LogisticRegression())])
model.fit(history[features], history["Churned_Within_60_Days_Binary"])
active["Churn_Probability"] = model.predict_proba(active[features])[:, 1].round(3)

# 2. Process: one year of value, and the part of it at risk
active["Annual_Value_USD"] = (active["Monthly_Subscription_Fee_USD"] * 12).round(2)
active["Revenue_At_Risk_USD"] = (
    active["Churn_Probability"] * active["Annual_Value_USD"]
).round(2)
ranked = active.sort_values("Revenue_At_Risk_USD", ascending=False)

# 3. Output: the ranking and the total
print(ranked[["Subscriber_ID", "Churn_Probability", "Annual_Value_USD",
              "Revenue_At_Risk_USD"]])
print(f"Total revenue at risk: ${active['Revenue_At_Risk_USD'].sum():,.2f}")

How it works, line by line:
- Part 1 repeats the training and scoring of Micro-Example 1.
- Monthly_Subscription_Fee_USD * 12 is one year of revenue from each customer.
- Multiplying by the churn probability gives the revenue at risk (section 7.2).
- The table is sorted by revenue at risk, and the total is printed with thousands separators.

Read the output:

  Subscriber_ID  Churn_Probability  Annual_Value_USD  Revenue_At_Risk_USD
0       ACT-101              0.876           2399.88              2102.29
7       ACT-108              0.900           1799.88              1619.89
4       ACT-105              0.941           1199.88              1129.09
9       ACT-110              0.448           2399.88              1075.15
...
Total revenue at risk: $7,659.53

The order differs from Micro-Example 1: ACT-101 moves to the top because it pays the most, and ACT-110 jumps to fourth despite its lower probability.
Across all 10 customers, about $7,660 of yearly revenue is expected to be lost if nothing is done.

Quick Check 2 (.cumsum()):
- New tool: column.cumsum() gives a running total down the rows. Why: shows how much of the total risk the first few customers carry.
- Task: add one line at the end of part 2. Process: ranked["Running_Total_USD"] = ranked["Revenue_At_Risk_USD"].______().round(2), and add "Running_Total_USD" to the printed columns.
- Check: the third row shows 4851.27: the top 3 customers carry 63% of the total risk.

Micro-Example 3 (Assign a Campaign to Each Customer): YourName_S42_demo_3.py

# 1. Input: train and score, as in Micro-Example 2
import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
history = pd.read_csv("data/session_04_02_dataset.csv")
active = pd.read_csv("data/session_04_02_active.csv")
features = ["Tenure_Months_Count", "Customer_Support_Calls_Count",
            "Usage_Drop_Last_30_Days_Pct"]
model = Pipeline([("scale", StandardScaler()),
                  ("classify", LogisticRegression())])
model.fit(history[features], history["Churned_Within_60_Days_Binary"])
active["Churn_Probability"] = model.predict_proba(active[features])[:, 1].round(3)
active["Annual_Value_USD"] = (active["Monthly_Subscription_Fee_USD"] * 12).round(2)

# 2. Process: rules from the cheapest to the most valuable campaign
active["Campaign"] = "Newsletter"
active.loc[active["Churn_Probability"] >= 0.4, "Campaign"] = "Retention_Call"
active.loc[
    (active["Churn_Probability"] >= 0.7) & (active["Annual_Value_USD"] >= 1500),
    "Campaign",
] = "VIP_Account_Manager"

# 3. Output: how many customers get each campaign
print(active["Campaign"].value_counts())
print(active[["Subscriber_ID", "Churn_Probability", "Campaign"]])

How it works, line by line:
- Part 1 scores the customers and computes their annual value, as before.
- active["Campaign"] = "Newsletter" gives everyone the default.
- The first .loc[] line overwrites the default with “Retention_Call” for customers at 0.4 or above.
- The second .loc[] line overwrites again with “VIP_Account_Manager” for customers at 0.7 or above and worth $1,500 or more a year (section 7.4).
- value_counts() counts the customers per campaign.
Read the output:
```
Campaign
Newsletter             4
Retention_Call         4
VIP_Account_Manager    2
...
4       ACT-105              0.941       Retention_Call
...
7       ACT-108              0.900  VIP_Account_Manager
```
- 2 customers get a personal account manager, 4 get a call, and 4 only get the newsletter.
- ACT-105 gets a call, not a VIP manager, although it is the most likely to leave: it is not valuable enough for the expensive offer.

Quick Check 3 (pd.cut()):
- New tool: pd.cut(column, bins=[0, 0.4, 0.7, 1.0], labels=["Low", "Medium", "High"]) puts each value into a band. Why: managers think in risk bands, not in decimals.
- Task: add two lines to part 3. Output: active["Risk_Band"] = pd.___(active["Churn_Probability"], bins=[0, 0.4, 0.7, 1.0], labels=["Low", "Medium", "High"]) and print(active["Risk_Band"].value_counts())
- Check: Low 4, Medium 3, High 3.

10. Official Python Documentation & References:

11. Exercise 1: Guided Practice - Campaign Business Case:

Step 1: Open Script: In Google Antigravity IDE, create a new script named YourName_S42_churn.py, replacing YourName with your own name (for example, Reacher_S42_churn.py).

Step 2: Price a Campaign for Customers at 0.5 or Above: Type the following code:

# 1. Input: train on history, score the active customers
import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
history = pd.read_csv("data/session_04_02_dataset.csv")
active = pd.read_csv("data/session_04_02_active.csv")
features = ["Tenure_Months_Count", "Customer_Support_Calls_Count",
            "Usage_Drop_Last_30_Days_Pct"]
model = Pipeline([("scale", StandardScaler()),
                  ("classify", LogisticRegression())])
model.fit(history[features], history["Churned_Within_60_Days_Binary"])
active["Churn_Probability"] = model.predict_proba(active[features])[:, 1].round(3)
active["Annual_Value_USD"] = (active["Monthly_Subscription_Fee_USD"] * 12).round(2)

# 2. Process: target, then expected saved revenue, cost, net benefit, ROI
success_rate = 0.40
cost_per_contact = 75.0
targeted = active[active["Churn_Probability"] >= 0.5]
saved = (targeted["Churn_Probability"] * success_rate
         * targeted["Annual_Value_USD"]).sum()
cost = len(targeted) * cost_per_contact
net_benefit = saved - cost
roi_pct = net_benefit / cost * 100

# 3. Output: the business case
print(f"Customers targeted: {len(targeted)}")
print(f"Expected saved revenue: ${saved:,.2f}")
print(f"Campaign cost: ${cost:,.2f}")
print(f"Net benefit: ${net_benefit:,.2f} | ROI: {roi_pct:.1f}%")

Expected saved revenue multiplies three things per customer: the chance they would leave, the chance the call keeps them, and what they are worth.

Step 3: Interpret: In two comment lines, explain why we multiply by the churn probability instead of counting each targeted customer’s full value.
Step 4: Execute in Terminal: Run python YourName_S42_churn.py in Panel [4] and confirm return code 0.
Success Check: 4 customers targeted; expected saved revenue $2,199.68; cost $300.00; net benefit $1,899.68; ROI 633.2%.

12. Exercise 2: Applied Practice - Highest ROI or Most Money?:
- Coding Challenge: In YourName_S42_practice.py, test targeting thresholds from 0.3 to 0.8 and compare the best ROI with the best net benefit.
- Task 1 (Threshold Loop): For each threshold in [0.3, 0.4, 0.5, 0.6, 0.7, 0.8], repeat the business case from Exercise 1 and collect Threshold, Targeted, Saved_USD, Cost_USD, Net_Benefit_USD, and ROI_Pct in one DataFrame.
- Task 2 (Read the ValueError & AI Diagnosis): Score the active customers with only two features by mistake: model.predict_proba(active[["Tenure_Months_Count", "Customer_Support_Calls_Count"]]). The script stops with ValueError: The feature names should match those that were passed during fit. Capture the traceback with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and ask why the model needs the same columns it was trained on.
- Task 3 (Two Recommendations): Fix the scoring, then print two lines: the threshold with the highest ROI_Pct, and the threshold with the highest Net_Benefit_USD. If thresholds tie, report the higher threshold, since it contacts fewer customers.
- Success Check: highest ROI: threshold 0.8 (762.4%, 3 customers, net $1,715.51); highest net benefit: threshold 0.4 ($2,387.07, 6 customers, ROI 530.5%). The broader campaign earns $671.56 more, at a lower ROI. If the budget allows, net benefit is what adds to profit.
13. Exercise 3: Advanced Practice - The Best Offer for Each Customer:
- Goal: Stop using one threshold for everyone. For each current customer, choose the offer with the highest expected net value, or no offer at all, and show the retention head how much more this earns than the best threshold plan from Exercise 2.
- What you need:
  - Files: data/session_04_02_dataset.csv (history) and data/session_04_02_active.csv (active customers).
  - Script name: YourName_S42_challenge.py.
  - Tools you already know: the trained pipeline and churn probabilities (Micro-Example 1), annual value (Micro-Example 2), .loc[] with conditions, .value_counts(), and .sum().
- Requirements (follow them in order):
  1. Score the 10 active customers and add Annual_Value_USD, as in Micro-Example 2.
  2. Two possible offers:
    - Call: costs $75, keeps 40% of the customers who would otherwise leave
    - VIP (an account manager): costs $250, keeps 60%
  3. For each customer, compute Call_Net = churn probability x 0.40 x annual value - 75, and VIP_Net = churn probability x 0.60 x annual value - 250, both rounded to 2 decimals.
  4. Set Best_Offer:
    - "VIP" if VIP_Net is above 0 and above Call_Net
    - otherwise "Call" if Call_Net is above 0
    - otherwise "None"
  5. Add Expected_Net_USD: the net value of the chosen offer, or 0 for "None".
  6. Print each customer’s ID, churn probability, annual value, both net values, best offer, and expected net. Then print the count per offer, the total expected net, and the difference from Exercise 2’s best net benefit ($2,387.07).
- New tools for this exercise:
  - Ordered rules with .loc[]: set the default first (active["Best_Offer"] = "None"), then overwrite with "Call", then with "VIP". The conditions in step 4 must be written so that the last rule wins only when it should.
  - Copying values row by row from another column: active.loc[mask, "Expected_Net_USD"] = active.loc[mask, "Call_Net"].
- Hints:
  - Start Expected_Net_USD at 0.0 for every customer.
  - Check one customer by hand first, such as ACT-101: 0.876 x 0.60 x 2399.88 - 250 = 1011.38.
- Self-Check (expected results):
  - Offers: VIP for ACT-101, ACT-105, ACT-108, and ACT-110; Call for ACT-103, ACT-104, and ACT-107; None for ACT-102, ACT-106, and ACT-109.
  - Total expected net value: $2,951.44, which is $564.37 more than the best threshold plan.
  - Business meaning: matching the offer to the customer beats any single threshold. ACT-104 has only a 0.214 churn probability but a high annual value, so even a call is worth it, while the expensive VIP offer pays only for valuable, high-risk customers such as ACT-110.

Session 4.3: AI Applications in Business Decision-Making Part 2: Pricing Optimization and Revenue Management

Session 4.3 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 15 (115/12/20 - 115/12/26) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- An online retailer sells 12 seasonal products; the season ends in 12 days, and stock left after that must be sold off cheaply
- Prices were set by gut feeling: some may be too low (leaving profit on the table), others too high (leaving stock unsold)
- The pricing manager asks: which prices should we raise, which should we cut, and how do we clear the stock before the season ends? Today, you measure how customers react to price and simulate profit before changing a single price
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “How strongly do customers react to price?”: estimate the price elasticity of demand (the % change in units sold for a 1% change in price) from a price test
2. “Would a price change raise profit?”: simulate units, revenue, and gross profit for several price changes before making them
3. “How do we clear the stock in time?”: calculate the price cut each product needs to sell out before the season ends, and fix a SyntaxError (an unclosed bracket)

3. Business Context: Price, Volume, and Profit:

A lower price sells more units but earns less on each one; a higher price earns more per unit but sells fewer. Profit depends on both:

Term	Meaning	Example (`PRD-200`)
Unit margin	price - unit cost	`$49.99 - $22.50 = $27.49`
Gross margin %	unit margin / price x 100	`55.0`%
Daily gross profit	unit margin x daily units	`$27.49 x 32 = $879.68`
Days of stock	inventory / daily units	`450 / 32 = 14.1` days

Price elasticity of demand (PED): if elasticity is -1.5, a 1% price increase loses about 1.5% of units. When elasticity is between 0 and -1, customers barely react; below -1, they react strongly.
The goal is profit, not revenue: a cut can raise revenue and still lower profit, because each extra unit brings in less margin.
Seasonal stock: units left at the season’s end lose most of their value. Our retailer sells leftovers to a liquidator (a company that buys surplus stock) for 40% of their unit cost, a loss of 60% of the cost on every unsold unit.

4. Data Basics: Measuring Elasticity with a Price Test:

To measure how customers react, the retailer changed the price of PRD-202 once a week for 5 weeks and recorded daily units sold:

Week	Price	Daily units
1	`$59.99`	`27.9`
3	`$69.99`	`22.0`
5	`$79.99`	`18.1`

Elasticity is the slope of a straight line through log(units) against log(price). A logarithm (log) turns percentage changes into differences, so the slope reads directly as “% change in units for a 1% change in price”.
np.polyfit(x, y, 1) fits that straight line and returns its slope and intercept. Result: elasticity -1.51.
Using elasticity for a small price change: new units = old units x (1 + elasticity x price change). For +5%: 1 + (-1.51 x 0.05) = 0.9245, so about 7.6% fewer units. This straight-line rule is an approximation that works for changes of up to about 20%.

5. Lab Practice Dataset:
- Two files (fictional teaching data):
  - Catalog: data/session_04_03_dataset.csv, 12 products, one row per product.
  - Price test: data/session_04_03_price_test.csv, 5 weeks of tested prices for PRD-202.
- Catalog columns:
  - Product_ID: Product code (e.g., PRD-201).
  - Current_Price_USD: Our current price in U.S. Dollars.
  - Competitor_Price_USD: The main competitor’s price for the same product.
  - Inventory_Units_Count: Units in stock.
  - Historical_Daily_Sales_Units: Average units sold per day at the current price.
  - Unit_Cost_USD: What one unit costs us, delivered to the warehouse.
- Price test columns: Test_Week (1 to 5), Product_ID, Test_Price_USD, and Avg_Daily_Units (average units sold per day that week).
6. Keywords (Core Programming Concepts):
- Price Elasticity of Demand (PED):
  - What: the % change in units sold for a 1% change in price
  - Why: tells you whether a price rise loses a few customers or many
- Unit Margin and Gross Profit:
  - What: price minus unit cost, and unit margin times units sold
  - Why: the measure a pricing decision should maximize
- Price Simulation:
  - What: computing units and profit for several possible prices before choosing one
  - Why: test a decision on paper, not on real customers
- Days of Stock:
  - What: inventory divided by daily sales
  - Why: shows which products will not sell out in time
- Markdown:
  - What: a planned price cut to sell stock faster
  - Why: a smaller, earlier cut usually loses less than a fire sale at the end
- Salvage Value:
  - What: what leftover stock is still worth after the season
  - Why: decides how costly it is to be left with unsold units

7. Core Programming & Computational Foundations:

7.1 Price elasticity: how strongly customers react

The idea: Price elasticity of demand (PED) is the percentage change in units sold for a 1% change in price. It is usually negative: a higher price sells fewer units. With an elasticity of -1.51, a 1% price rise loses about 1.51% of units. Values between 0 and -1 mean customers barely react; values below -1 mean they react strongly.

See it on real data: the price test for PRD-202, week by week.

Weeks	Price change	Units change	Elasticity of that step
1 to 2	`+8.3`%	`-12.5`%	`-1.51`
2 to 3	`+7.7`%	`-9.8`%	`-1.28`
3 to 4	`+7.1`%	`-10.9`%	`-1.53`
4 to 5	`+6.7`%	`-7.7`%	`-1.15`

Each single step is noisy. Fitting one line through all 5 weeks gives one steadier estimate: -1.51.

Common mistake: trusting one week-to-week comparison. Sales also move for reasons other than price (weather, holidays, a competitor’s promotion); use several test weeks.

More examples:

Elasticity	Units after a 10% price rise	Revenue effect (1.10 x units factor)
`-0.5`	`-5`%	`+4.5`%: revenue rises
`-1.0`	`-10`%	about `-1`%: revenue almost flat
`-1.51` (ours)	`-15.1`%	`-6.6`%: revenue falls
`-2.0`	`-20`%	`-12`%: revenue falls sharply

Remember: elasticity = % change in units / % change in price; estimate it from several observations.

7.2 Why logs: percentages become straight lines

The idea: Elasticity is about percentage changes, but a straight line measures fixed changes. The natural logarithm (np.log) turns one into the other: a 10% rise in price always adds the same amount, about 0.095, to the log of the price, whatever the starting price. So on a log-log chart, constant elasticity becomes a straight line, and its slope is the elasticity.

See it on real data:

Price	`np.log(price)`
`59.99`	`4.094`
`69.99`	`4.248`
`79.99`	`4.382`

In code: slope, intercept = np.polyfit(np.log(prices), np.log(units), 1) fits the line; the 1 means a straight line. The slope is the elasticity.

More examples:

Code	Result	What it tells you
`np.log(1)`	`0.0`	the log of 1 is 0
`np.log(70) - np.log(60)`	`0.154`	close to the % change, `16.7`%: logs measure relative change
`np.log(1.10)`	`0.0953`	a 10% rise, whatever the starting price
`np.exp(0.0953)`	`1.1`	`np.exp()` undoes `np.log()`

Remember: log price against log units: the slope is the elasticity.

7.3 Simulating a price change

The idea: Before changing a price, compute what would happen. For a small change, new units = old units x (1 + elasticity x price change). Then profit = (new price - unit cost) x new units. Writing this once as a function, simulate(price_change), guarantees every scenario uses the same formula.

See it on real data: PRD-200 (price $49.99, cost $22.50, 32 units a day).

Price change	New price	Daily units	Daily revenue	Daily gross profit
`-10`%	`$44.99`	`36.83`	`$1,657.11`	`$828.39`
`0`%	`$49.99`	`32.00`	`$1,599.68`	`$879.68`
`+5`%	`$52.49`	`29.58`	`$1,552.85`	`$887.21`

The cut brings the most revenue but the least profit; the rise loses units but earns the most.

Common mistake: maximizing revenue. Every extra unit sold at a lower price adds less margin, so revenue can rise while profit falls.

More examples: the straight-line rule vs the exact constant-elasticity formula, (1 + change) ** elasticity.

Price change	Straight-line units factor	Exact factor
`+5`%	`0.9245`	`0.9290`
`+10`%	`0.8490`	`0.8660`
`-10`%	`1.1510`	`1.1724`
`+20`%	`0.6980`	`0.7593`

The two agree closely for small changes and drift apart for large ones: keep simulated changes within about 20%.

Remember: simulate profit, not revenue, and keep changes small.

7.4 Stock and time: days of stock and markdowns

The idea: Seasonal stock has a deadline. Days of stock = inventory / daily units shows how long the stock lasts at today’s price. If that is longer than the days left in the season, the product needs a markdown (a planned price cut) to sell faster.

See it on real data: PRD-202, 12 days left in the season.

Step	Calculation	Result
Days of stock now	`290 / 22`	`13.2` days: too slow
Units needed per day	`290 / 12`	`24.17`
Units must rise by	`24.17 / 22 - 1`	`9.9`%
Price change needed	`9.9% / -1.51`	`-6.5`%

The last step reverses the elasticity rule: if units must rise by a given percentage, the price must change by that percentage divided by the elasticity.

More examples:

Product	Days of stock	Units needed per day	Price change to sell out in 12 days
`PRD-200`	`14.1`	`37.50`	`-11.4`%
`PRD-210`	`12.5`	`4.17`	`-2.8`%: almost on time
`PRD-204`	`12.0`	`15.00`	`0`%: sells out exactly on time
`PRD-203`	`11.1`	`17.50`	`+5.2`%: sells out early, so the price could rise

Remember: compare days of stock with days left; cut just enough to sell out.

7.5 Leftover stock has a cost: salvage value

The idea: Units left after the season are sold to a liquidator for 40% of their cost, a loss of 60% of the cost on each one. So the true season profit is profit on units sold minus the loss on units left.

See it on real data: PRD-200 over the 12-day season (450 units in stock).

Price	Units sold	Units left	Season profit
`$49.99` (no change)	`384`	`66`	`$9,665.16`
`$44.99` (`-10`%)	`442`	`8`	`$9,832.00`

The cut earns less per unit but avoids most of the leftover loss: $167 more profit for the season.

More examples:

Product and price	Sold	Left over	Season profit
`PRD-202` at `$69.99`	`264`	`26`	`$9,530.16`
`PRD-202` at `$66.49` (`-5`%)	`284`	`6`	`$9,676.31`: the cut pays
`PRD-210` at `$149.99`	`48`	`2`	`$3,706.32`
`PRD-210` at `$142.49` (`-5`%)	`50`	`0`	`$3,574.50`: selling out is not worth the cut
`PRD-204` at `$94.49` (`+5`%)	`166`	`14`	`$8,566.96`: a rise leaves stock behind

Remember: count the cost of leftovers when you choose a price.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S43_demo_1.py), and check both files are in data/.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small. The editor highlights matching brackets when you place the cursor next to one.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors.
9. Hands-On Coding Micro-Examples:
- Every example has three parts: 1. Input, 2. Process, and 3. Output. Name each file with your own name and the session code, for example Reacher_S43_demo_1.py (pattern: YourName_S43_demo_1.py).
- Micro-Example 1 (Estimate Elasticity from the Price Test): YourName_S43_demo_1.py
```
# 1. Input: 5 weeks of tested prices for PRD-202
import numpy as np
import pandas as pd
test = pd.read_csv("data/session_04_03_price_test.csv")

# 2. Process: a straight line through log(units) against log(price)
slope, intercept = np.polyfit(np.log(test["Test_Price_USD"]),
                              np.log(test["Avg_Daily_Units"]), 1)
elasticity = round(slope, 2)

# 3. Output: the elasticity and what it means
print("Price elasticity of demand:", elasticity)
print(f"A 1% price rise loses about {abs(elasticity)}% of units sold.")
```
  - How it works, line by line:
    - import numpy as np loads NumPy, the number library that pandas is built on.
    - np.log(...) takes the logarithm of every price and every unit count (section 7.2).
    - np.polyfit(x, y, 1) fits the best straight line through the 5 points and returns its slope and intercept. The slope is the elasticity.
    - round(slope, 2) keeps 2 decimals; abs() drops the minus sign for the sentence.
  - Read the output:
    Price elasticity of demand: -1.51 A 1% price rise loses about 1.51% of units sold.
    - -1.51 is below -1: customers of this product react strongly to price.
    - A 10% rise would lose about 15% of units; whether that raises profit depends on the margin, which Micro-Example 2 checks.
- Quick Check 1 (.pct_change()):
  - New tool: column.pct_change() gives the change from the previous row, as a share (0.08 = 8%). Why: shows the week-to-week reactions behind the single slope.
  - Task: add one line to part 3. Output: print((test[["Test_Price_USD", "Avg_Daily_Units"]].__________() * 100).round(1))
  - Check: the second row shows 8.3 (price up 8.3%) and -12.5 (units down 12.5%). The first row is NaN, because week 1 has no previous week.
- Micro-Example 2 (Simulate Profit for Several Price Changes): YourName_S43_demo_2.py
```
# 1. Input: the catalog and the elasticity from Micro-Example 1
import pandas as pd
df = pd.read_csv("data/session_04_03_dataset.csv")
elasticity = -1.51

# 2. Process: one function, used for every price change
def simulate(price_change):
    price = df["Current_Price_USD"] * (1 + price_change)
    units = df["Historical_Daily_Sales_Units"] * (1 + elasticity * price_change)
    return round(((price - df["Unit_Cost_USD"]) * units).sum(), 2)

profit_curve = {}
for change in [-0.10, -0.05, 0.0, 0.05, 0.10]:
    profit_curve[f"{change * 100:+.0f}%"] = simulate(change)

# 3. Output: daily gross profit for the whole catalog at each change
for label, profit in profit_curve.items():
    print(f"{label}: ${profit:,.2f}")
```
  - How it works, line by line:
    - elasticity = -1.51 uses the value measured in Micro-Example 1 for every product. In a real business, each product group would get its own test.
    - simulate(price_change) computes the new price, the new units (section 7.3), and the total daily gross profit of the whole catalog.
    - The loop runs the function for 5 price changes and stores each result in a dictionary, labeled like +5%.
    - f"{change * 100:+.0f}%" writes the change with its sign: +5%, -10%.
  - Read the output:
    -10%: $6,881.85 -5%: $7,163.83 +0%: $7,342.85 +5%: $7,418.90 +10%: $7,391.98
    - Profit rises from -10% up to +5%, then falls again at +10%: there is a best point in between.
    - A 5% rise adds about $76 a day across the catalog. The margins are high (above 52%), so the profit on the remaining units outweighs the units lost.
- Quick Check 2 (max(..., key=...)):
  - New tool: max(profit_curve, key=profit_curve.get) returns the label with the highest profit. Why: finds the best scenario without reading the list by eye.
  - Task: add one line to part 3. Output: print("Best change:", ___(profit_curve, key=profit_curve.get))
  - Check: the Terminal shows Best change: +5%.
- Micro-Example 3 (Which Products Will Not Sell Out in 12 Days?): YourName_S43_demo_3.py
```
# 1. Input: the catalog
import pandas as pd
df = pd.read_csv("data/session_04_03_dataset.csv")
season_days = 12

# 2. Process: days of stock and our price gap to the competitor
df["Days_Of_Stock"] = (
    df["Inventory_Units_Count"] / df["Historical_Daily_Sales_Units"]
).round(1)
df["Price_Gap_Pct"] = (
    (df["Current_Price_USD"] - df["Competitor_Price_USD"])
    / df["Competitor_Price_USD"] * 100
).round(1)
at_risk = df[df["Days_Of_Stock"] > season_days]

# 3. Output: the products that need action
print(at_risk[["Product_ID", "Days_Of_Stock", "Price_Gap_Pct"]])
```
  - How it works, line by line:
    - Days_Of_Stock divides inventory by daily sales (section 7.4).
    - Price_Gap_Pct compares our price with the competitor’s: negative means we are cheaper.
    - df[df["Days_Of_Stock"] > season_days] keeps the products that will not sell out in 12 days.
  - Read the output:
    Product_ID Days_Of_Stock Price_Gap_Pct 0 PRD-200 14.1 -3.9 1 PRD-201 13.6 2.5 2 PRD-202 13.2 -2.8 9 PRD-209 13.0 3.7 10 PRD-210 12.5 -1.3 11 PRD-211 13.3 3.2
    - 6 of 12 products would still have stock at the end of the season.
    - PRD-201, PRD-209, and PRD-211 are also more expensive than the competitor: a price cut there has two good reasons.
    - PRD-200 is already cheaper than the competitor, yet still too slow: its stock is simply too large for the season.
- Quick Check 3 (np.where()):
  - New tool: np.where(condition, value_if_true, value_if_false) fills a whole column with one of two values. Why: a one-line alternative to setting a default and overwriting it with .loc[].
  - Task: add import numpy as np to part 1. Input, and one line at the end of part 2. Process: df["Action"] = np._____(df["Days_Of_Stock"] > season_days, "Markdown", "Keep"). Then print df["Action"].value_counts().
  - Check: Keep 6, Markdown 6.
10. Official Python Documentation & References:

11. Exercise 1: Guided Practice - Catalog Profit Curve:

Step 1: Open Script: In Google Antigravity IDE, create a new script named YourName_S43_pricing.py, replacing YourName with your own name (for example, Reacher_S43_pricing.py).

Step 2: Simulate a Wider Range of Price Changes: Type the following code:

# 1. Input: the catalog and the measured elasticity
import pandas as pd
df = pd.read_csv("data/session_04_03_dataset.csv")
elasticity = -1.51

# 2. Process: today's unit economics, then a profit curve from -15% to +15%
df["Unit_Margin_USD"] = df["Current_Price_USD"] - df["Unit_Cost_USD"]
df["Gross_Margin_Pct"] = (df["Unit_Margin_USD"] / df["Current_Price_USD"] * 100).round(1)
rows = []
for change in [-0.15, -0.10, -0.05, 0.0, 0.05, 0.10, 0.15]:
    price = df["Current_Price_USD"] * (1 + change)
    units = df["Historical_Daily_Sales_Units"] * (1 + elasticity * change)
    rows.append({
        "Price_Change_Pct": round(change * 100),
        "Daily_Units": round(units.sum(), 1),
        "Daily_Revenue_USD": round((price * units).sum(), 2),
        "Daily_Gross_Profit_USD": round(((price - df["Unit_Cost_USD"]) * units).sum(), 2),
    })
curve = pd.DataFrame(rows)

# 3. Output: margins today, then the curve
print("Gross margin % by product:", df["Gross_Margin_Pct"].tolist())
print(curve)

Step 3: Interpret: In two comment lines, explain why the price change with the highest revenue is not the one with the highest profit.
Step 4: Execute in Terminal: Run python YourName_S43_pricing.py in Panel [4] and confirm return code 0.
Success Check: gross margins range from 52.5% to 55.0%. Gross profit peaks at +5 ($7,418.90), while revenue is highest at -15 ($14,218.32): the change that brings in the most money is not the one that keeps the most. Cutting prices by 15% sells the most units but earns the least profit ($6,496.89).

12. Exercise 2: Applied Practice - Markdowns That Clear the Stock:
- Coding Challenge: In YourName_S43_practice.py, calculate the price cut each at-risk product needs to sell out in 12 days.
- Task 1 (Required Price Change): For the products with more than 12 days of stock (Micro-Example 3):
  - Required_Daily_Units = Inventory_Units_Count / 12
  - Price_Change_Pct = (Required_Daily_Units / Historical_Daily_Sales_Units - 1) / elasticity x 100, rounded to 1 decimal. It is negative: a cut.
  - This reverses the rule in section 4: if units must rise by a given share, the price must change by that share divided by the elasticity.
- Task 2 (Read the SyntaxError & AI Diagnosis): Write the new price line with a missing bracket: df["New_Price_USD"] = df["Current_Price_USD"] * (1 + df["Price_Change_Pct"] / 100. Run the script; Python stops before running anything with SyntaxError: '(' was never closed. Capture the message with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and ask where the bracket is missing.
- Task 3 (Fix and Export): Close the bracket, round New_Price_USD to 2 decimals, and save Product_ID, Current_Price_USD, Price_Change_Pct, and New_Price_USD for the at-risk products to data/markdown_plan.csv with index=False.
- Success Check: price changes: PRD-200 -11.4%, PRD-201 -8.7%, PRD-202 -6.5%, PRD-209 -5.5%, PRD-210 -2.8%, and PRD-211 -7.4%. The deepest cut goes to the product with the most stock left relative to its sales, not to the most expensive one.
13. Exercise 3: Advanced Practice - The Best Season Price for Each Product:
- Goal: Find, for each product, the price that earns the most over the 12-day season, counting the loss on any stock left over. Some prices may go up and some down. Show the manager how much the plan adds compared with keeping today’s prices.
- What you need:
  - File: data/session_04_03_dataset.csv (described in 5. Lab Practice Dataset).
  - Script name: YourName_S43_challenge.py.
  - Tools you already know: functions, for loops over rows and price changes, min(), max(), lists of dictionaries, and pd.DataFrame().
- Rules of the season:
  - Season length: 12 days; elasticity: -1.51; daily units at a new price: units x (1 + elasticity x change).
  - Units sold in the season: the smaller of (daily units x 12) and the inventory: you cannot sell stock you do not have.
  - Unsold units go to a liquidator at 40% of unit cost, a loss of 0.6 x unit cost each.
  - Season profit = (price - unit cost) x units sold - 0.6 x unit cost x unsold units.
- Requirements (follow them in order):
  1. Write season_profit(price, unit_cost, daily_units, inventory) that returns the season profit, following the rules above.
  2. For each product, try the price changes -20%, -15%, -10%, -5%, 0%, +5%, +10%, +15%, and +20%. Round each new price to 2 decimals before you use it.
  3. Keep, for each product, the change with the highest season profit, and also record the season profit at 0% (today’s price).
  4. Build one DataFrame with Product_ID, Best_Change_Pct, Best_Price_USD, Best_Season_Profit_USD, and Current_Season_Profit_USD, profits rounded to 2 decimals.
  5. Print the table, the total of both profit columns, and the difference.
  6. Count how many products get a price cut, how many a price rise, and how many keep today’s price.
- New tools for this exercise:
  - Looping over rows with for _, row in df.iterrows():. Each row behaves like a small dictionary: row["Unit_Cost_USD"]. The _ holds the row label, which we do not need.
  - min(a, b) returns the smaller of two numbers.
- Hints:
  - Test season_profit() on PRD-200 at 0% first: it should return about 9665.16, because 66 units are left over.
  - Inside the product loop, keep the best result so far in two variables and update them when a change earns more.
- Self-Check (expected results):
  - Best changes: cuts for PRD-200 (-10%), PRD-201, PRD-202, PRD-209, and PRD-211 (-5% each); rises of +5% for PRD-203, PRD-205, PRD-206, and PRD-208; no change for PRD-204, PRD-207, and PRD-210.
  - Total season profit: $85,969.93 with the plan against $83,590.57 at today’s prices: $2,379.36 more.
  - Business meaning: one rule for all products leaves money behind. Products with too much stock need a cut to avoid leftover losses; products that will sell out anyway can carry a higher price. Note that for PRD-210, a cut that clears all stock earns less than keeping the price and selling 2 leftover units to the liquidator.

Session 4.4: Ethical Considerations in Artificial Intelligence: Algorithmic Bias, Regulatory Compliance, and AI Governance

Session 4.4 Map

Lecture Presentation: Download Slides (PDF)
Reading (PDF): Download Reading (PDF): read before class
Schedule: Week 16 (115/12/27 - 116/01/02) In-person classes (1:2 Delivery Ratio: 60 Min Lecture + 120 Min Lab)
1. Session Purpose:
- A consumer lender approves loan applications automatically when its model’s approval score is 0.70 or higher
- Before the model’s annual review, the board’s risk committee must confirm that it treats applicant groups fairly and that every rejection can be explained to the applicant
- The chief risk officer asks: does our credit model treat applicant groups fairly, can we explain each rejection, and what should the board decide? Today, you audit the model for fairness, explain its decisions, and prepare the facts the board needs
2. Expected Learning Outcomes (By the end of this session, you will be able to):
1. “Are the groups treated alike?”: measure approval rates by group and the disparate impact ratio, and test it against the four-fifths rule
2. “Why was this applicant rejected?”: explain each decision with feature contributions (how much each feature pushed the score up or down)
3. “What should the board decide?”: compare one shared threshold’s fairness and risk, write a governance record, and fix a NameError caused by a variable that only exists inside a function
3. Business Context: Fairness, Explanations, and Accountability:
- A model learns patterns from past data. If the past was unfair, for example because some groups had less access to credit, the model can repeat that unfairness at scale, without anyone intending it.
- Two kinds of unfair treatment:
  - Disparate treatment: using a person’s group directly in the decision, such as a different threshold for each group. In lending, this is prohibited in many countries, including under the US Equal Credit Opportunity Act (ECOA).
  - Disparate impact: a rule that treats everyone the same on paper, but approves one group far less often. It is a warning sign that must be investigated and justified.
- The four-fifths rule: if one group’s approval rate is less than 80% (four-fifths) of the most-approved group’s rate, the gap needs review. It comes from US employment guidelines (1978) and is widely borrowed as a first screening test. Failing it is a reason to investigate, not proof of discrimination.
- Explanations are required: US lenders must give rejected applicants the main reasons (an adverse action notice), and the EU AI Act (European Union Artificial Intelligence Act) treats credit scoring of people as high-risk, with duties for documentation, human oversight, and transparency.
- Governance: the board stays accountable for automated decisions. It needs a written record of what was tested, what was found, and who decided what.
4. Data Basics: Three Fairness Numbers:
- Approval rate of a group = approved applicants in the group / all applicants in the group.
- Disparate impact ratio = the lower group’s approval rate / the higher group’s approval rate. 1.0 means equal rates; below 0.80 fails the four-fifths rule.
- Feature contribution (for a model that adds up features, like linear regression) = coefficient x (applicant’s value - average value). A negative contribution pulled the score down. For such a model, this number is also the feature’s SHAP value (SHapley Additive exPlanations, a widely used explanation method).
- Our approval scores come from a model whose formula we were not given. We fit a linear regression that copies it from the scores (a surrogate model) and explain that copy. It reproduces the scores well (R-squared 0.975), so its explanations are trustworthy for this model.
5. Lab Practice Dataset:
- File Link: data/session_04_04_dataset.csv
- Audited 16-row loan application dataset (fictional teaching data), one row per applicant, 8 in each group:
  - Applicant_ID: Applicant code (e.g., APP-6001).
  - Demographic_Group: An anonymized protected group, Group_A or Group_B. A protected group is defined by a characteristic such as race, sex, or age that the law forbids using against an applicant. The model does not use this column; auditors use it only to test fairness.
  - Credit_Score_Count: Credit score from a credit bureau (a company that records how people repay debts); higher is better.
  - Debt_To_Income_Pct: Monthly debt payments as a share of monthly income (DTI); lower is safer.
  - Algorithmic_Approval_Score: The model’s score from 0 to 1; approved at 0.70 or higher.
  - Adverse_Action_Notice_Flag: 1 if the applicant was rejected and must receive an adverse action notice, else 0.
6. Keywords (Core Programming Concepts):
- Disparate Impact:
  - What: a rule that approves one group much less often than another, even though it never uses the group
  - Why: the most common way bias enters automated decisions
- Disparate Impact Ratio and the Four-Fifths Rule:
  - What: the lower approval rate divided by the higher one, tested against 0.80
  - Why: a simple, widely used first test of fairness
- Disparate Treatment:
  - What: deciding differently because of a person’s group
  - Why: prohibited in lending in many countries, so it can never be the fix
- Explainability (feature contributions, SHAP):
  - What: showing how much each feature pushed a decision up or down
  - Why: applicants, regulators, and managers need reasons, not just scores
- Surrogate Model:
  - What: a simple model fitted to copy a model whose formula is hidden
  - Why: makes a hidden model explainable, as long as the copy is accurate
- AI Governance:
  - What: the rules, records, and people that keep AI decisions accountable
  - Why: the board, not the model, answers for the decisions

7. Core Programming & Computational Foundations:

7.1 Approval rates by group: the average of a 0/1 column

The idea: Fairness checks start with a simple question: what share of each group was approved? If approved applicants are marked 1 and rejected ones 0, the average of that column is the share of 1s, so df.groupby("Demographic_Group")["Approved"].mean() gives each group’s approval rate in one line.

See it on real data: threshold 0.70.

Group	Applicants	Approved	Approval rate
`Group_A`	`8`	`7`	`0.875`
`Group_B`	`8`	`2`	`0.250`

Common mistake: counting approvals instead of rates. If one group had 80 applicants and another 8, raw counts would hide the comparison; rates put groups of any size on the same scale.

More examples:

Code	Result	What it tells you
`df["Approved"].mean()`	`0.5625`	9 of all 16 applicants approved
`df.groupby("Demographic_Group")["Approved"].sum()`	A `7`, B `2`	approvals per group
`df.groupby("Demographic_Group")["Approved"].count()`	A `8`, B `8`	applicants per group
rate = sum / count	A `0.875`, B `0.250`	the same rates as `.mean()`

Remember: the mean of a 0/1 column is a rate.

7.2 The disparate impact ratio and the four-fifths rule

The idea: Divide the lower approval rate by the higher one. 1.0 means equal treatment in outcomes; the lower the number, the bigger the gap. Under the four-fifths rule, a ratio below 0.80 is a warning that must be investigated.
See it on real data: 0.250 / 0.875 = 0.286. To pass, Group B would need an approval rate of at least 0.80 x 0.875 = 0.70: 6 of its 8 applicants instead of 2.
Common mistake: treating the rule as a verdict. Failing it does not prove discrimination, and passing it does not prove fairness. It tells you where to look.

More examples: the ratio at other shared thresholds (Exercise 3 computes the full table).

Threshold	Group A rate	Group B rate	Ratio	Passes?
`0.45`	`1.000`	`1.000`	`1.000`	yes
`0.50`	`1.000`	`0.875`	`0.875`	yes
`0.60`	`1.000`	`0.500`	`0.500`	no
`0.70`	`0.875`	`0.250`	`0.286`	no
`0.75`	`0.875`	`0.000`	`0.000`	no

Remember: lower rate / higher rate; below 0.80, investigate.

7.3 Where a gap comes from: the features

The idea: Our model never sees the group column. But if the groups differ on the features the model uses, its scores and approvals will differ too. This is how disparate impact appears without anyone intending it.

See it on real data:

Group average	Group A	Group B
Credit score	`730.62`	`670.00`
Debt-to-income (DTI)	`26.58`%	`36.05`%
Approval score	`0.86`	`0.60`
Lowest and highest score	`0.66` to `0.94`	`0.49` to `0.72`

The real question: do these feature differences reflect real repayment risk, or past unequal access to credit? A credit score depends partly on how long someone has had access to credit. Data alone cannot settle this; people must investigate.

More examples:

Measure	Group A	Group B
lowest and highest credit score	`690` to `760`	`640` to `705`
share with DTI above 35%	`0.0`	`0.5`
correlation of score with credit score (all 16)	`0.987`
correlation of score with DTI (all 16)	`-0.982`

Remember: a model can be unfair without using the group column; look at the features.

7.4 Explaining one decision: feature contributions

The idea: For a model that adds up its features, like linear regression, each feature’s contribution is coefficient x (applicant’s value - average value). The contributions show how much each feature pushed this applicant’s score above or below the average score. They add up exactly: average score + all contributions = the applicant’s score.

See it on real data: our surrogate model has coefficients 0.00292 per credit point and -0.00734 per DTI point; the averages are a credit score of 700.31, a DTI of 31.31%, and a score of 0.7275.

Applicant	Credit effect	DTI effect	Average + effects	Decision
`APP-6007` (640, 41.2%)	`-0.176`	`-0.073`	`0.7275 - 0.176 - 0.073 = 0.479`	rejected
`APP-6012` (690, 34.0%)	`-0.030`	`-0.020`	`0.678`	rejected, just below `0.70`
`APP-6006` (760, 22.0%)	`+0.174`	`+0.068`	`0.970`	approved

For APP-6007, the low credit score is the main reason (-0.176), more than twice the effect of the high debt ratio.

Common mistake: explaining a model you have not checked. A surrogate’s explanations are only as good as its fit; ours reproduces the scores with an R-squared of 0.975.

More examples:

Applicant	Credit effect	DTI effect	Decision
`APP-6000` (720, 28.5%)	`+0.058`	`+0.021`	approved
`APP-6005` (705, 29.5%)	`+0.014`	`+0.013`	approved, close to average
`APP-6013` (700, 30.5%)	`-0.001`	`+0.006`	approved, almost exactly average
`APP-6001` (680, 34.2%)	`-0.059`	`-0.021`	rejected: credit score is the main reason

Remember: contribution = coefficient x (value - average); the largest negative one is the main reason.

7.5 Fair process: what a fix may and may not do

The idea: When a model fails a fairness test, the fix itself must be lawful and responsible.

Possible fix	Allowed?	Why
A lower threshold for one group	no	deciding by group is disparate treatment, prohibited in lending in many countries
One lower threshold for everyone	only with care	Exercise 3 shows it approves applicants above the risk limit
Investigate whether the features measure risk equally well for both groups	yes	addresses the cause
Add data such as rent and utility payment history	yes	gives applicants with thin credit records a fair hearing
Human review of rejections near the threshold, documented	yes	a person checks the borderline cases

More examples:

A governance record states	Example from this session
the fairness test and its result	ratio `0.286`, fails the four-fifths rule
the threshold in use	`0.70`, the lender’s current approval rule
the explanation method and its fit	surrogate model, R-squared `0.975`
the actions decided and their owners	feature investigation; human review near the threshold

Remember: never decide by group; fix the cause, add oversight, and document every decision.

8. Development Workflow in Google Antigravity IDE:
- [1] File Explorer: create your script with your name and the session code (for example, Reacher_S44_demo_1.py), and check the dataset is in data/.
- [2] Code Editor: write a few lines, run, repeat. Small steps keep errors small.
- [3] AI Agent Chat: on an error, capture the traceback (Python’s error report) with Win + Shift + S (macOS: Cmd + Shift + 4), paste it with Ctrl + V, and ask why it failed.
- [4] Terminal: run python followed by your script name. Return code 0 means the script finished with no errors.
9. Hands-On Coding Micro-Examples:
- Every example has three parts: 1. Input, 2. Process, and 3. Output. Name each file with your own name and the session code, for example Reacher_S44_demo_1.py (pattern: YourName_S44_demo_1.py).
- Micro-Example 1 (Approval Rates and the Four-Fifths Rule): YourName_S44_demo_1.py
```
# 1. Input: the 16 applications
import pandas as pd
df = pd.read_csv("data/session_04_04_dataset.csv")

# 2. Process: decisions at 0.70, approval rate per group, and the ratio
df["Approved"] = (df["Algorithmic_Approval_Score"] >= 0.70).astype(int)
rates = df.groupby("Demographic_Group")["Approved"].mean()
ratio = rates.min() / rates.max()

# 3. Output: rates, ratio, and the four-fifths test
print(rates)
print(f"Disparate impact ratio: {ratio:.3f}")
print("Passes the four-fifths rule:", ratio >= 0.80)
```
  - How it works, line by line:
    - (df["Algorithmic_Approval_Score"] >= 0.70).astype(int) turns each score into 1 (approved) or 0 (rejected).
    - df.groupby("Demographic_Group")["Approved"].mean() gives each group’s approval rate (section 7.1).
    - rates.min() / rates.max() divides the lower rate by the higher, whichever group it is (section 7.2).
    - ratio >= 0.80 gives True or False: the four-fifths test.
  - Read the output:
    Demographic_Group Group_A 0.875 Group_B 0.250 Name: Approved, dtype: float64 Disparate impact ratio: 0.286 Passes the four-fifths rule: False
    - 7 of 8 Group A applicants are approved, but only 2 of 8 in Group B.
    - The ratio, 0.286, is far below 0.80: the gap must be investigated before the model’s next review.
- Quick Check 1 (.map()):
  - New tool: column.map({1: "Approved", 0: "Rejected"}) replaces each value using a dictionary. Why: tables for the board should use words, not codes.
  - Task: add two lines to part 3. Output: df["Decision"] = df["Approved"].___({1: "Approved", 0: "Rejected"}) and print(pd.crosstab(df["Demographic_Group"], df["Decision"]))
  - Check: Group A: 7 approved, 1 rejected; Group B: 2 approved, 6 rejected.
- Micro-Example 2 (Where Does the Gap Come From?): YourName_S44_demo_2.py
```
# 1. Input: the 16 applications
import pandas as pd
df = pd.read_csv("data/session_04_04_dataset.csv")

# 2. Process: average features and score by group
profile = df.groupby("Demographic_Group")[
    ["Credit_Score_Count", "Debt_To_Income_Pct", "Algorithmic_Approval_Score"]
].mean().round(2)

# 3. Output: the two group profiles
print(profile)
```
  - How it works, line by line:
    - df.groupby("Demographic_Group")[[...]].mean() averages the two features and the score in each group (section 7.3).
    - The group column is used here only by the auditor, to compare groups; the model never sees it.
  - Read the output:
    Credit_Score_Count ... Algorithmic_Approval_Score Demographic_Group ... Group_A 730.62 ... 0.86 Group_B 670.00 ... 0.60
    - The ... hides the debt-to-income column: Group A 26.58%, Group B 36.05%. Quick Check 2 shows the full table.
    - Group B has lower credit scores and higher debt ratios on average, so it gets lower approval scores. The gap in approvals follows from the gap in features.
- Quick Check 2 (.T):
  - New tool: table.T flips a table so that rows become columns. Why: with many measures and few groups, the flipped table is easier to read in a report.
  - Task: in part 3. Output, change the line to print(profile._)
  - Check: the measures are now rows and Group_A and Group_B are the two columns.
- Micro-Example 3 (Explain Each Decision): YourName_S44_demo_3.py
```
# 1. Input: the 16 applications and the two features
import pandas as pd
from sklearn.linear_model import LinearRegression
df = pd.read_csv("data/session_04_04_dataset.csv")
features = ["Credit_Score_Count", "Debt_To_Income_Pct"]

# 2. Process: copy the score with a linear model, then contributions
surrogate = LinearRegression().fit(df[features], df["Algorithmic_Approval_Score"])
contributions = (df[features] - df[features].mean()) * surrogate.coef_
contributions.columns = ["Credit_Score_Effect", "Debt_Ratio_Effect"]
report = pd.concat([df[["Applicant_ID", "Algorithmic_Approval_Score"]],
                    contributions.round(3)], axis=1)

# 3. Output: how well the copy fits, then the explanations
print("Surrogate R-squared:", round(surrogate.score(df[features], df["Algorithmic_Approval_Score"]), 3))
print(report[df["Algorithmic_Approval_Score"] < 0.70])
```
  - How it works, line by line:
    - LinearRegression().fit(df[features], df["Algorithmic_Approval_Score"]) fits a simple model that copies the scores from the two features: the surrogate (section 7.4).
    - (df[features] - df[features].mean()) * surrogate.coef_ computes every applicant’s contribution for both features at once: each column’s average is subtracted, and each column is multiplied by its own coefficient.
    - contributions.columns = [...] gives the two results readable names.
    - pd.concat([...], axis=1) puts the IDs, scores, and contributions side by side.
    - report[df["Algorithmic_Approval_Score"] < 0.70] keeps the rejected applicants only.
  - Read the output:
    Surrogate R-squared: 0.975 Applicant_ID ... Debt_Ratio_Effect 1 APP-6001 ... -0.021 3 APP-6003 ... -0.053 7 APP-6007 ... -0.073 ... 15 APP-6015 ... -0.067
    - The surrogate explains 97.5% of the score differences, so its explanations can be trusted.
    - All 7 rejected applicants have negative effects. The hidden Credit_Score_Effect column is larger in every case, so the credit score is the main reason in each adverse action notice (Quick Check 3).
- Quick Check 3 (.idxmin(axis=1)):
  - New tool: contributions.idxmin(axis=1) returns, for each row, the column with the most negative value. Why: that column is the main reason to print on the adverse action notice.
  - Task: add one line at the end of part 2. Process: report["Main_Reason"] = contributions.______(axis=1)
  - Check: the rejected applicants all show Credit_Score_Effect as the main reason.
10. Official Python Documentation & References:

11. Exercise 1: Guided Practice - Fairness Audit Report:

Step 1: Open Script: In Google Antigravity IDE, create a new script named YourName_S44_governance.py, replacing YourName with your own name (for example, Reacher_S44_governance.py).

Step 2: Audit the Current Threshold: Type the following code:

# 1. Input: the 16 applications
import pandas as pd
df = pd.read_csv("data/session_04_04_dataset.csv")

# 2. Process: decisions, rates, ratio, and a check of the notice flags
df["Approved"] = (df["Algorithmic_Approval_Score"] >= 0.70).astype(int)
rates = df.groupby("Demographic_Group")["Approved"].mean()
ratio = rates.min() / rates.max()
notices_match = ((1 - df["Approved"]) == df["Adverse_Action_Notice_Flag"]).all()

# 3. Output: the audit findings
print(f"Group A approval: {rates['Group_A'] * 100:.1f}% | "
      f"Group B approval: {rates['Group_B'] * 100:.1f}%")
print(f"Disparate impact ratio: {ratio:.3f}")
if ratio < 0.80:
    print("ALERT: fails the four-fifths rule; investigate before the next review.")
else:
    print("Passes the four-fifths rule.")
print("Every rejected applicant is flagged for a notice:", notices_match)

1 - df["Approved"] turns approved (1) into 0 and rejected (0) into 1, matching the notice flag.

Step 3: Interpret: In two comment lines, explain why failing the four-fifths rule calls for an investigation, and why it does not by itself prove the model discriminates.
Step 4: Execute in Terminal: Run python YourName_S44_governance.py in Panel [4] and confirm return code 0.
Success Check: Group A approval: 87.5% | Group B approval: 25.0%; ratio 0.286; the alert line prints; Every rejected applicant is flagged for a notice: True.

12. Exercise 2: Applied Practice - A Reusable Audit Function and a Governance Record:
- Coding Challenge: In YourName_S44_practice.py, turn the audit into a function and save a governance record for the board.
- Task 1 (Audit Function): Write approval_audit(df, threshold) that computes the approval rates and ratio at any threshold, exactly as in Exercise 1.
- Task 2 (Read the NameError & AI Diagnosis): Call approval_audit(df, 0.70), then write print(ratio) on the next line, outside the function. The script stops with NameError: name 'ratio' is not defined. Capture the traceback with Win + Shift + S, paste it into Panel [3] (AI Agent Chat) with Ctrl + V, and ask why a variable made inside a function cannot be seen outside it.
- Task 3 (Return, Record, Save): Make the function end with return rates, ratio, and call it as rates, ratio = approval_audit(df, 0.70). Build a one-row governance record with pd.DataFrame([{...}]) and the keys Model, Threshold, Group_A_Rate, Group_B_Rate, Disparate_Impact_Ratio, Passes_Four_Fifths, and Status, where Model is "Credit_Approval_Model_v1" and Status is "Board review required" if the test fails. Save it to data/board_ai_governance_audit.csv with index=False, and print a 3-bullet memo to the board risk committee: finding, evidence, and requested decision.
- Success Check: the record shows Group_A_Rate 0.875, Group_B_Rate 0.25, Disparate_Impact_Ratio 0.286, Passes_Four_Fifths False, and Status Board review required; the script ends with return code 0.
13. Exercise 3: Advanced Practice - Can One Shared Threshold Fix the Gap?:
- Goal: A different threshold for each group would be disparate treatment, so it is not an option. Test whether a single threshold for everyone can pass the four-fifths rule, and what it would cost in credit risk. Prepare the trade-off for the board.
- What you need:
  - File: data/session_04_04_dataset.csv (described in 5. Lab Practice Dataset).
  - Script name: YourName_S44_challenge.py.
  - Tools you already know: your approval_audit() function from Exercise 2, for loops, lists of dictionaries, pd.DataFrame(), and Boolean masks.
- Requirements (follow them in order):
  1. For each shared threshold in [0.45, 0.50, 0.55, 0.60, 0.65, 0.70, 0.75, 0.80, 0.85], compute: the number approved, the approval rate of each group, the disparate impact ratio (rounded to 3 decimals), and Risky_Approvals: approved applicants with a DTI above 35%, the lender’s usual risk limit.
  2. Collect the results in one DataFrame with the columns Threshold, Approved, Group_A_Rate, Group_B_Rate, DI_Ratio, and Risky_Approvals.
  3. Find the highest threshold that passes the four-fifths rule, and report its Risky_Approvals.
  4. Find the highest threshold that passes the four-fifths rule and has 0 risky approvals. If there is none, print that clearly.
  5. Write three comment lines for the board: what the threshold can and cannot fix, and two other actions the board could request instead.
- New tools for this exercise:
  - Combining two masks: approved & (df["Debt_To_Income_Pct"] > 35) is True for applicants who are approved and above the risk limit; .sum() counts them.
  - Filtering a results table with two conditions: results[(results["DI_Ratio"] >= 0.80) & (results["Risky_Approvals"] == 0)]. An empty result has len() of 0.
  - Guarding a division: if a group has no approvals at all, its rate is 0, and the ratio is 0; check rates.max() > 0 before dividing.
- Hints:
  - Reuse approval_audit() inside the loop, so every threshold is tested the same way.
  - Print the full table before answering requirements 3 and 4; the pattern is easier to see.
- Self-Check (expected results):
  - Ratios: 0.45 1.000, 0.50 0.875, 0.55 0.625, 0.60 0.500, 0.65 0.250, 0.70 0.286, and 0.000 from 0.75 upward.
  - Highest passing threshold: 0.50, with 15 of 16 applicants approved and 3 risky approvals (DTI above 35%).
  - No threshold passes the rule with 0 risky approvals.
  - Business meaning: moving one threshold either fails the fairness test or approves applicants above the lender’s risk limit. The threshold cannot fix the gap, because the gap comes from the features themselves. The board should instead ask for an investigation of the features (do credit scores and DTI measure real repayment risk equally well for both groups?), consider additional data such as rent and utility payment history, and require human review of rejected applications near the threshold, with every decision documented.

6141

Business Analytics and AI Application (Course 6141)

Course Overview

Development Environment and AI Error Protocol

Table of Contents

Module 1: Foundations of Business Analytics and Python Programming

Session 1.1: Business Analytics Architecture, Google Antigravity IDE Initialization, and Core Arithmetic

Session 1.2: Essential Python Syntax Completion: Control Flow, Collections, and Business Automation

Session 1.3: Tabular Data Wrangling Part 1: NumPy Arrays, Pandas DataFrames, and CSV Ingestion

Session 1.4: Tabular Data Wrangling Part 2: Advanced Filtering, Multi-Segment GroupBy, and Financial Auditing

Module 2: Web Data Extraction, Visual Dashboards, and Midterm Milestone

Session 2.1: Web Data Extraction: Scraping Financial and Competitor Intelligence with Beautiful Soup

Session 2.2: Business Data Visualization: Exploratory Plotting and Executive Dashboards

Session 2.3: Midterm Applied Analytics Formulation Part 1: Problem Structuring and Data Audit

Session 2.4: Midterm Analytics Pipeline Defense Part 2: Exploratory Findings and Executive Review

Module 3: Artificial Intelligence, Machine Learning Fundamentals, and Scikit-Learn

Session 3.1: Foundations of Artificial Intelligence: Enterprise Value Creation and Decision Automation

Session 3.2: Advanced Data Analysis: Feature Engineering, Scaling, and Business Data Preprocessing

Session 3.3: Practical Machine Learning with Scikit-Learn: Model Training and Validation

Session 3.4: Core Concepts of Machine Learning: Supervised vs. Unsupervised Learning and Evaluation Metrics

Module 4: Strategic AI Integration, Enterprise Decision-Making, and AI Governance

Session 4.1: Integration of Business Analytics and Enterprise AI Architecture: End-to-End Decision Systems

Session 4.2: AI Applications in Business Decision-Making Part 1: Predictive Customer Analytics and Churn

Session 4.3: AI Applications in Business Decision-Making Part 2: Pricing Optimization and Revenue Management

Session 4.4: Ethical Considerations in Artificial Intelligence: Algorithmic Bias, Regulatory Compliance, and AI Governance