error_demo_messy_pipeline.py

Starting script entry point

91 steps Google Auth Google Sheets HTTP Client Can be run directly cli: __main__ guard

This automation reads lead information from a Google Sheet, uses external services to find contact details and generate summaries for each lead, and then updates a Google Sheet with this enriched data. It also sends a summary of its work to Slack and creates a local backup.

How It Works

Each block is a phase of the automation. Click a phase below for details.

Data Journey

Reads leads from Google Sheets → verifies and enriches their details using an HTTP Client → then updates Google Sheets and backs up the enriched information to a temporary file.

No logging framework

45 print() calls found. Python’s logging module provides log levels, timestamps, and configurable output.

Repetitive error handling

12 try/except blocks. Consider extracting a retry/error-handling helper function.

Step by Step

Setup & Data Gathering 6 steps

This phase focuses on establishing secure connections to necessary services and gathering initial data. It ensures the automation has permission to access Google Sheets, retrieves information from various web sources, and prepares a local backup file.

External Service Authenticates with Google Sheets 2 locations

Multi-service authentication — connects to 2 different services

External Service Fetches data from HTTP Client 2 locations

Batch data gathering — collects data from HTTP Client in 2 requests

Risk: The main risks in this phase include issues with securely connecting to Google Sheets, external web services imposing limits on data requests, or the initial company records lacking essential information like company names or domains.

Data Processing 24 steps

Data Processing Converts data format 2 locations

Data pipeline — transforms data through 2 processing steps

Data Processing Counts items 8 locations

Data pipeline — transforms data through 8 processing steps

Data Processing Processes data 8 locations

Data pipeline — transforms data through 8 processing steps

Storage & Delivery 4 steps

External Service Sends data to HTTP Client 3 locations

Batch updates — pushes data to HTTP Client in 3 operations

Error Handling 12 steps

Check Handles potential errors 12 locations

Repetitive error handling — 12 identical try/except blocks suggest extracting a shared helper

Reporting 45 steps (49%)

Log/Notify Displays message 45 locations

Excessive output — 45 print() calls; consider Python’s logging module for structured output

Step Detail

Click any step to see its details.

Services Used

Google Auth
Google Sheets
HTTP Client

Credentials Needed

Anymailfinder credentials
Google credentials
OpenAI credentials
Slack credentials

View technical flow diagram

Flow

Click any step to see details

graph TB
  n_error_demo_messy_pipeline_load_leads_from_sheet_compact[["load_leads_from_sheet() — 21 steps (11 Output, 5 Decision, 4 Transform, 1 API)"]]:::compact
  n_error_demo_messy_pipeline_enrich_single_lead_compact[["enrich_single_lead() — 12 steps (5 Output, 4 Decision, 3 API)"]]:::compact
  subgraph n_error_demo_messy_pipeline_generate_summary["generate_summary()"]
    n_error_demo_messy_pipeline_136(["Output: print()"]):::output
    n_error_demo_messy_pipeline_138{"try/except block"}:::decision
    n_error_demo_messy_pipeline_139["API: requests.post()"]:::api
    n_error_demo_messy_pipeline_152(["Output: print()"]):::output
    n_error_demo_messy_pipeline_136 --> n_error_demo_messy_pipeline_138
    n_error_demo_messy_pipeline_138 --> n_error_demo_messy_pipeline_139
    n_error_demo_messy_pipeline_139 --> n_error_demo_messy_pipeline_152
  end
  n_error_demo_messy_pipeline_save_to_sheet_compact[["save_to_sheet() — 17 steps (8 Output, 4 Decision, 4 Transform, 1 API)"]]:::compact
  subgraph n_error_demo_messy_pipeline_notify_slack["notify_slack()"]
    n_error_demo_messy_pipeline_204(["Output: print()"]):::output
    n_error_demo_messy_pipeline_206[["Transform: sum()"]]:::transform
    n_error_demo_messy_pipeline_207[["Transform: sum()"]]:::transform
    n_error_demo_messy_pipeline_209{"try/except block"}:::decision
    n_error_demo_messy_pipeline_210["API: requests.post()"]:::api
    n_error_demo_messy_pipeline_212[["Transform: len()"]]:::transform
    n_error_demo_messy_pipeline_216(["Output: print()"]):::output
    n_error_demo_messy_pipeline_218(["Output: print()"]):::output
    n_error_demo_messy_pipeline_204 --> n_error_demo_messy_pipeline_206
    n_error_demo_messy_pipeline_206 --> n_error_demo_messy_pipeline_207
    n_error_demo_messy_pipeline_207 --> n_error_demo_messy_pipeline_209
    n_error_demo_messy_pipeline_209 --> n_error_demo_messy_pipeline_210
    n_error_demo_messy_pipeline_210 --> n_error_demo_messy_pipeline_212
    n_error_demo_messy_pipeline_212 --> n_error_demo_messy_pipeline_216
    n_error_demo_messy_pipeline_216 --> n_error_demo_messy_pipeline_218
  end
  subgraph n_error_demo_messy_pipeline_save_backup["save_backup()"]
    n_error_demo_messy_pipeline_223(["Output: print()"]):::output
    n_error_demo_messy_pipeline_225{"try/except block"}:::decision
    n_error_demo_messy_pipeline_226[/"File: open('/tmp/lead_backup.json')"/]:::fileio
    n_error_demo_messy_pipeline_227[/"File: json.dump()"/]:::fileio
    n_error_demo_messy_pipeline_229(["Output: print()"]):::output
    n_error_demo_messy_pipeline_231(["Output: print()"]):::output
    n_error_demo_messy_pipeline_223 --> n_error_demo_messy_pipeline_225
    n_error_demo_messy_pipeline_225 --> n_error_demo_messy_pipeline_226
    n_error_demo_messy_pipeline_226 --> n_error_demo_messy_pipeline_227
    n_error_demo_messy_pipeline_227 --> n_error_demo_messy_pipeline_229
    n_error_demo_messy_pipeline_229 --> n_error_demo_messy_pipeline_231
  end
    n_error_demo_messy_pipeline_235(["Output: print()"]):::output
    n_error_demo_messy_pipeline_236(["Output: print()"]):::output
    n_error_demo_messy_pipeline_237(["Output: print()"]):::output
    n_error_demo_messy_pipeline_241{"if not leads"}:::decision
    n_error_demo_messy_pipeline_242(["Output: print()"]):::output
    n_error_demo_messy_pipeline_245(["Output: print()"]):::output
    n_error_demo_messy_pipeline_245[["Transform: len()"]]:::transform
    n_error_demo_messy_pipeline_248{"for lead in leads"}:::decision
    n_error_demo_messy_pipeline_251[["Transform: .append()"]]:::transform
    n_error_demo_messy_pipeline_252(["Output: print()"]):::output
    n_error_demo_messy_pipeline_252[["Transform: len()"]]:::transform
    n_error_demo_messy_pipeline_252[["Transform: len()"]]:::transform
    n_error_demo_messy_pipeline_258(["Output: print()"]):::output
    n_error_demo_messy_pipeline_259(["Output: print()"]):::output
    n_error_demo_messy_pipeline_260(["Output: print()"]):::output
    n_error_demo_messy_pipeline_260[["Transform: len()"]]:::transform
    n_error_demo_messy_pipeline_261(["Output: print()"]):::output
    n_error_demo_messy_pipeline_261[["Transform: sum()"]]:::transform
    n_error_demo_messy_pipeline_262(["Output: print()"]):::output
    n_error_demo_messy_pipeline_262[["Transform: sum()"]]:::transform
    n_error_demo_messy_pipeline_263(["Output: print()"]):::output
    n_error_demo_messy_pipeline_263[["Transform: sum()"]]:::transform
    n_error_demo_messy_pipeline_264(["Output: print()"]):::output
    n_error_demo_messy_pipeline_235 --> n_error_demo_messy_pipeline_236
    n_error_demo_messy_pipeline_236 --> n_error_demo_messy_pipeline_237
    n_error_demo_messy_pipeline_237 --> n_error_demo_messy_pipeline_241
    n_error_demo_messy_pipeline_241 --> n_error_demo_messy_pipeline_242
    n_error_demo_messy_pipeline_242 --> n_error_demo_messy_pipeline_245
    n_error_demo_messy_pipeline_245 --> n_error_demo_messy_pipeline_245
    n_error_demo_messy_pipeline_245 --> n_error_demo_messy_pipeline_248
    n_error_demo_messy_pipeline_248 --> n_error_demo_messy_pipeline_251
    n_error_demo_messy_pipeline_251 --> n_error_demo_messy_pipeline_252
    n_error_demo_messy_pipeline_252 --> n_error_demo_messy_pipeline_252
    n_error_demo_messy_pipeline_252 --> n_error_demo_messy_pipeline_252
    n_error_demo_messy_pipeline_252 --> n_error_demo_messy_pipeline_258
    n_error_demo_messy_pipeline_258 --> n_error_demo_messy_pipeline_259
    n_error_demo_messy_pipeline_259 --> n_error_demo_messy_pipeline_260
    n_error_demo_messy_pipeline_260 --> n_error_demo_messy_pipeline_260
    n_error_demo_messy_pipeline_260 --> n_error_demo_messy_pipeline_261
    n_error_demo_messy_pipeline_261 --> n_error_demo_messy_pipeline_261
    n_error_demo_messy_pipeline_261 --> n_error_demo_messy_pipeline_262
    n_error_demo_messy_pipeline_262 --> n_error_demo_messy_pipeline_262
    n_error_demo_messy_pipeline_262 --> n_error_demo_messy_pipeline_263
    n_error_demo_messy_pipeline_263 --> n_error_demo_messy_pipeline_263
    n_error_demo_messy_pipeline_263 --> n_error_demo_messy_pipeline_264
  click n_error_demo_messy_pipeline_136 call showStepDetail("error_demo_messy_pipeline.py", 136)
  click n_error_demo_messy_pipeline_138 call showStepDetail("error_demo_messy_pipeline.py", 138)
  click n_error_demo_messy_pipeline_139 call showStepDetail("error_demo_messy_pipeline.py", 139)
  click n_error_demo_messy_pipeline_152 call showStepDetail("error_demo_messy_pipeline.py", 152)
  click n_error_demo_messy_pipeline_204 call showStepDetail("error_demo_messy_pipeline.py", 204)
  click n_error_demo_messy_pipeline_206 call showStepDetail("error_demo_messy_pipeline.py", 206)
  click n_error_demo_messy_pipeline_207 call showStepDetail("error_demo_messy_pipeline.py", 207)
  click n_error_demo_messy_pipeline_209 call showStepDetail("error_demo_messy_pipeline.py", 209)
  click n_error_demo_messy_pipeline_210 call showStepDetail("error_demo_messy_pipeline.py", 210)
  click n_error_demo_messy_pipeline_212 call showStepDetail("error_demo_messy_pipeline.py", 212)
  click n_error_demo_messy_pipeline_216 call showStepDetail("error_demo_messy_pipeline.py", 216)
  click n_error_demo_messy_pipeline_218 call showStepDetail("error_demo_messy_pipeline.py", 218)
  click n_error_demo_messy_pipeline_223 call showStepDetail("error_demo_messy_pipeline.py", 223)
  click n_error_demo_messy_pipeline_225 call showStepDetail("error_demo_messy_pipeline.py", 225)
  click n_error_demo_messy_pipeline_226 call showStepDetail("error_demo_messy_pipeline.py", 226)
  click n_error_demo_messy_pipeline_227 call showStepDetail("error_demo_messy_pipeline.py", 227)
  click n_error_demo_messy_pipeline_229 call showStepDetail("error_demo_messy_pipeline.py", 229)
  click n_error_demo_messy_pipeline_231 call showStepDetail("error_demo_messy_pipeline.py", 231)
  click n_error_demo_messy_pipeline_235 call showStepDetail("error_demo_messy_pipeline.py", 235)
  click n_error_demo_messy_pipeline_236 call showStepDetail("error_demo_messy_pipeline.py", 236)
  click n_error_demo_messy_pipeline_237 call showStepDetail("error_demo_messy_pipeline.py", 237)
  click n_error_demo_messy_pipeline_241 call showStepDetail("error_demo_messy_pipeline.py", 241)
  click n_error_demo_messy_pipeline_242 call showStepDetail("error_demo_messy_pipeline.py", 242)
  click n_error_demo_messy_pipeline_245 call showStepDetail("error_demo_messy_pipeline.py", 245)
  click n_error_demo_messy_pipeline_245 call showStepDetail("error_demo_messy_pipeline.py", 245)
  click n_error_demo_messy_pipeline_248 call showStepDetail("error_demo_messy_pipeline.py", 248)
  click n_error_demo_messy_pipeline_251 call showStepDetail("error_demo_messy_pipeline.py", 251)
  click n_error_demo_messy_pipeline_252 call showStepDetail("error_demo_messy_pipeline.py", 252)
  click n_error_demo_messy_pipeline_252 call showStepDetail("error_demo_messy_pipeline.py", 252)
  click n_error_demo_messy_pipeline_252 call showStepDetail("error_demo_messy_pipeline.py", 252)
  click n_error_demo_messy_pipeline_258 call showStepDetail("error_demo_messy_pipeline.py", 258)
  click n_error_demo_messy_pipeline_259 call showStepDetail("error_demo_messy_pipeline.py", 259)
  click n_error_demo_messy_pipeline_260 call showStepDetail("error_demo_messy_pipeline.py", 260)
  click n_error_demo_messy_pipeline_260 call showStepDetail("error_demo_messy_pipeline.py", 260)
  click n_error_demo_messy_pipeline_261 call showStepDetail("error_demo_messy_pipeline.py", 261)
  click n_error_demo_messy_pipeline_261 call showStepDetail("error_demo_messy_pipeline.py", 261)
  click n_error_demo_messy_pipeline_262 call showStepDetail("error_demo_messy_pipeline.py", 262)
  click n_error_demo_messy_pipeline_262 call showStepDetail("error_demo_messy_pipeline.py", 262)
  click n_error_demo_messy_pipeline_263 call showStepDetail("error_demo_messy_pipeline.py", 263)
  click n_error_demo_messy_pipeline_263 call showStepDetail("error_demo_messy_pipeline.py", 263)
  click n_error_demo_messy_pipeline_264 call showStepDetail("error_demo_messy_pipeline.py", 264)
classDef api fill:#dbeafe,stroke:#2563eb,color:#1e3a5f
classDef fileio fill:#dcfce7,stroke:#16a34a,color:#14532d
classDef dbop fill:#f3e8ff,stroke:#9333ea,color:#3b0764
classDef decision fill:#ffedd5,stroke:#ea580c,color:#7c2d12
classDef output fill:#f3f4f6,stroke:#6b7280,color:#1f2937
classDef transform fill:#ccfbf1,stroke:#0d9488,color:#134e4a
classDef entry fill:#dcfce7,stroke:#16a34a,stroke-width:3px,color:#14532d
classDef compact fill:#f0f4ff,stroke:#6366f1,color:#312e81,stroke-width:2px

API Call File I/O Database Decision Output Transform

All Steps

Step Detail

Select a step from the diagram or list to see its details.

Services

Google Auth (google.oauth2)
Google Sheets (gspread)
HTTP Client (requests)

Secrets

ANYMAILFINDER_API_KEY
GOOGLE_TOKEN_JSON
OPENAI_API_KEY
SLACK_WEBHOOK_URL

Imports

External

google.oauth2.service_account
google.oauth2.service_account.Credentials
gspread
json
os
requests
time