🏗️ Agentic Feature Mart — System Architecture

End-to-end ML Feature Platform with LangGraph Multi-Agent Orchestration · CarTrade Tech

❄️ Snowflake 🤖 LangGraph 🐍 Python + dbt 🔁 Argo Workflows 🧠 GPT-4o 📍 Google Maps API ⚡ Kafka + Spark 🐳 Docker + K8s ☁️ AWS S3
① DATA SOURCES — Raw ingestion from 5 verticals
📣 Marketing CRM
Campaign events, email opens, WhatsApp delivery, ROAS signals
MySQL CDCREST API
🏷️ Seller Platform
Listing data, renewals, seller profile, pricing history
PostgreSQLBatch ETL
⚡ Clickstream
Page views, session depth, funnel events, dwell time per seller page
Kafka TopicsSpark Stream
📍 Google Maps API
POI data, geocoding, distance matrix, reverse geocoding
REST APIGeoPandas
🌐 Web + Census
Regional intelligence, population density, income proxy, market data
ScrapyCSV Ingest
Extract & Load
② ETL + FEATURE ENGINEERING + LLM AUGMENTATION — Compute, transform & enrich
🐍 Python ETL
Data extraction, type normalization, deduplication, Snowflake connector writes
pandassnowflake-connectorpydantic
🗄️ SQL + dbt
Feature computation in Snowflake SQL. dbt models with tests, lineage DAG & docs
dbt-snowflakedbt testSQL macros
⚡ Kafka + Spark
Clickstream streaming. 5-min micro-batch agg → Snowpipe into staging tables
KafkaPySparkSnowpipe
🗺️ Geo Computation
POI radius counts, polygon intersections, catchment areas, density grids
GeoPandasShapelyH3 Index
🧠 GPT-4o LLM Layer
Area quality scoring, POI context analysis, neighbourhood classification, 1536-dim embeddings
GPT-4o APItext-embed-3JSON mode
Orchestrated via CI/CD
③ PIPELINE ORCHESTRATION — CI/CD — Monthly automated, zero-touch
🔁 Argo Workflows
CronWorkflow triggers monthly. DAG with 6 sequential steps. 3-retry with backoff on failure. Slack alerts.
CronWorkflowDAG stepsretry:3
🐳 Docker + Kubernetes
Each pipeline step runs as isolated K8s pod. Namespace: feature-pipelines. Auto-scaled by Argo.
DockerK8s PodsHelm Charts
✅ Quality Gates
Null rate <2%, PSI drift check, row count sanity, schema validation. Pipeline halts if any gate fails.
Great ExpectationsPSI check
🏷️ Feature Stamping
Every row tagged with v{YYYY}.{MM} stamp. Enforces train-serve consistency. Registered in FEATURE_REGISTRY table.
version stampFEATURE_REGISTRY
☁️ AWS S3
Intermediate artifacts, pipeline logs, final feature matrix exports. Prefixed per stamp version.
S3 VersionedParquet
Write stamped features
④ SNOWFLAKE FEATURE MART — Single source of truth for all ML features

❄️ Snowflake · FEATURE_MART Schema

Warehouse: FEATURE_WH · Database: ANALYTICS_DB · Schema: FEATURE_MART

15,000+
Total Features
50+
Tables
99.1%
Non-null Rate
Monthly
Refresh Cycle
v{YYYY.MM}
Version Stamp
SELLER_GEO_FEATURES SELLER_ENGAGEMENT_FEATURES CITY_DEMOGRAPHICS_FEATURES LLM_LOCATION_FEATURES CLICKSTREAM_SELLER_90D SELLER_TREND_FEATURES PROPERTY_MARKET_FEATURES MARKETING_ENGAGEMENT_FEATURES FEATURE_REGISTRY + 41 more tables
Natural language query
⑤ AGENTIC LAYER — LangGraph Multi-Agent System — Conversational feature discovery & retrieval
🤖 Orchestrator Agent
Entry Point · Router
• GPT-4 function calling
• Parses NL feature requests
• Decomposes into sub-tasks
• Routes to specialist agents
• StateGraph shared memory
• Maintains conversation turns
LangGraphGPT-4StateGraph
→
🗂️ Table Recommender Agent
Semantic Search · Discovery
• Indexes FEATURE_REGISTRY
• Semantic match on 15K features
• Returns ranked table list
• Provides join keys & grain
• Relevance score per table
• Recommends filters
TF-IDF / EmbedFEATURE_REGISTRY
→
🔍 Data Fetcher Agent
SQL Gen · Snowflake Exec
• Auto-generates JOIN SQL
• Handles multi-grain alignment
• Executes on Snowflake
• Measures join success rates
• Returns quality report
• Null rate per column
SQL GenSnowflakeToolNode
→
🧪 Feature Engineer Agent
Enrichment · Output
• Validates base feature set
• Proposes enrichment groups
• Asks clarifying questions
• Fetches additional tables
• Applies version stamp
• Exports to S3 as Parquet
Human-in-loopS3 Export
💬 Flow: User types natural language feature request → Orchestrator decomposes → Table Recommender surfaces tables → Data Fetcher writes & runs SQL on Snowflake → Feature Engineer enriches + asks follow-ups → Final versioned 1,141-feature matrix exported in ~10 min
Versioned feature matrix
⑥ MODEL CONSUMERS — 5+ product lines training on Feature Mart
📈 Seller Prioritization
XGBoost ranker. Ranks sellers for sales outreach. ₹5Cr+ annual revenue attribution.
XGBoost1,141 features
💸 CRM Scoring
Identifies sellers who don't need WhatsApp outreach. ₹1Cr/yr savings. 40% cost reduction.
Propensitysklearn
🛻 Used Car Loan
Loan eligibility scoring using seller geo & engagement features as credit proxies.
LightGBMGeo features
🏦 Personal Loan
Buyer income proxies and engagement signals for personal loan pre-qualification.
LogRegDemographics
🧑‍💼 Buyer Model
Buyer intent & conversion scoring using clickstream and location context features.
EnsembleClickstream
Agentic Feature Mart · CarTrade Tech · ML Platform
15,000+ Features 50+ Snowflake Tables ₹6Cr+ Annual Value 5+ Product Lines Monthly Auto-Pipeline