AI & ML Overview

The Databricks AI Story

Energy Copilot demonstrates the full Databricks AI stack working together:

Foundation Model API (FMAPI): Claude Sonnet 4.5 routed through Databricks, keeping API calls within the network perimeter
MLflow Model Registry: ML models trained, tracked, versioned, and deployed via MLflow
Model Serving: inference endpoints for real-time model predictions
Genie AI/BI: natural language SQL over Gold Delta tables via 16 Genie spaces
Unity Catalog SQL Functions: UC functions provide the AI agent with structured access to data
Vector Search (RAG): AEMO market rules indexed for semantic retrieval by the copilot

AI/ML Capabilities

AI Market Intelligence Copilot

Claude Sonnet 4.5 with 58 domain-specific tools. Conversational interface for market analysis, trading, DNSP intelligence, and operational queries. Streams responses via SSE.

Genie AI/BI Spaces

16 Genie spaces covering NEM prices, generation, interconnectors, FCAS, gas, WEM, settlement, DNSP compliance, assets, DER, network planning, and workforce. Natural language → SQL → results.

NEM Price & Demand Forecasting

XGBoost multi-horizon forecasting. 5 regions, horizons from 1 to 48 hours. Demand, weather, generation, and constraint features. Outputs P10/P50/P90 confidence bounds and spike probability.

Asset Failure Prediction

XGBoost classifier. Scores all DNSP network assets on failure probability within 12 months. Feature importances and health index surfaced in Asset Intelligence Hub.

Vegetation Risk ML

XGBoost classifier for network span vegetation risk. BMO zone scoring and newly-flagged alert feed for AusNet bushfire mitigation program.

Workforce Demand Forecasting

Prophet time-series model. 18-month horizon. Bushfire seasonality modelled explicitly. Skills gap output for DNSP workforce planners.

STPIS Anomaly Detection

Isolation Forest. 6-DNSP peer group benchmarking for SAIDI/SAIFI/MAIFI anomalies. Surfaced in STPIS Calculator page.

AIO Draft Generator

Claude Sonnet 4.5 drafts AIO submission sections from structured DNSP performance data via FMAPI.

ML Models

Model	Algorithm	Purpose	Surfaces In
NEM Price Forecast	XGBoost	Spot price prediction 1–48 hr, P10/P50/P90 + spike probability	Copilot `query_price_forecasts`, price forecast dashboard
NEM Demand Forecast	Prophet	Demand MW with confidence bounds, 1–48 hr horizon	Copilot `query_demand_forecasts`, demand dashboard
Generation Forecast	XGBoost	Per-fuel-type generation prediction by region	`gold.generation_forecasts`, generation dashboard
Asset Failure Prediction	XGBoost	Failure probability within 12 months, health index, feature importances	DNSP Asset Intelligence Hub, Copilot `get_asset_health`
Vegetation Risk Classifier	XGBoost	Span-level bushfire/vegetation risk score + confidence	DNSP Vegetation Risk Hub
Workforce Demand Forecast	Prophet	18-month skills demand forecast with bushfire seasonality	DNSP Workforce Analytics Hub
STPIS Anomaly Detection	Isolation Forest	Abnormal STPIS performance vs 6-DNSP peer group	DNSP STPIS Calculator
Constraint Binding Prediction	XGBoost	Probability transmission constraints bind by hour/region, 4–48 hr	Copilot `predict_constraints`
Battery Dispatch Optimisation	Optimisation	Charge/discharge schedule maximising arbitrage revenue over 24 hr	Copilot `optimize_battery`
Bid Optimisation	ML (ML_OPTIMIZED)	Optimal 10-band generator bid; actual vs ML-optimal revenue comparison	Copilot `optimize_bid`, `compare_bid_vs_optimal`

MLflow Experiment Tracking

All ML models are tracked in MLflow experiments within Unity Catalog:

Experiment	Model Type	Target Variable	Regions
`price_forecast`	XGBoost	NEM spot price	All 5 NEM
`demand_forecast`	Prophet	Operational demand MW	All 5 NEM
`wind_forecast`	XGBoost	Wind generation MW	All 5 NEM
`solar_forecast`	XGBoost	Solar generation MW	All 5 NEM
`anomaly_detection`	Isolation Forest	STPIS anomaly	NEM-wide

Each experiment tracks:

Training hyperparameters
Evaluation metrics (MAE, MAPE, AUC, F1)
Feature importances
Chronological train/val/test split boundaries (stored as tags)
Model artifacts and registered model alias

AI Copilot Tools (58 total)

All tools are available to Claude Sonnet 4.5 via FMAPI function calling. The copilot selects and chains tools autonomously based on the user’s question.

Market Data

Tool	Description
`query_spot_prices`	Live NEM spot prices (RRP), demand, and available generation per region
`query_generation_mix`	Generation MW by fuel type, capacity factor, and emissions intensity
`query_interconnectors`	Interstate flows, export/import limits, and congestion status
`query_weather`	Temperature, wind speed, solar radiation, cloud cover for all NEM regions
`get_fcas_summary`	All 8 FCAS service prices, total cost, top providers, regional requirements
`get_gas_price`	STTM hub prices (Sydney/Adelaide/Brisbane) and DWGM Victorian market

Forecasting & Signals

Tool	Description
`query_price_forecasts`	ML price forecast with P10/P50/P90 and spike probability, 1–48 hr
`query_demand_forecasts`	ML demand forecast MW with confidence bounds
`get_constraint_forecast`	Constraint binding heatmap by hour-of-day and day-of-week
`predict_constraints`	Which transmission constraints bind in next 4–48 hrs and at what probability
`get_trading_signals`	Active algo signals: FORWARD_MISPRICING, SPOT_ARBITRAGE, CONSTRAINT_PLAY, WEATHER_SHIFT, DEMAND_SURGE, VOLATILITY_REGIME, RENEWABLE_SHORTFALL, PEAK_SPREAD
`scan_trading_opportunities`	Fresh cross-region market scan, ranked by expected profit with rationale

Anomaly & Intelligence

Tool	Description
`explain_anomaly`	Root cause analysis of price spikes / negative prices — generation changes, constraints, weather, congestion
`generate_market_brief`	Daily/weekly/flash NEM brief — overnight prices, key events, renewable share, weather watch

Portfolio & Trading

Tool	Description
`create_trade`	Parse natural language into a structured trade: “50MW peak swap VIC Q3 at $85”
`get_portfolio_position`	Net MW long/short by region and quarter
`get_forward_curve`	Monthly forward prices bootstrapped from ASX futures with seasonal shaping
`run_mtm_valuation`	Mark-to-market all trades against forward curves
`get_portfolio_pnl`	P&L attribution: price effect, volume effect, new trades, time decay
`explain_pnl_move`	Explain what drove P&L change on a specific date
`get_risk_metrics`	VaR (95%/99%), Greeks (delta/gamma/vega/theta), 1-day and 10-day horizons
`get_credit_exposure`	Counterparty exposure, PFE, credit utilisation, WARNING/CRITICAL alerts
`value_ppa`	Monte Carlo PPA valuation — NPV P10/P50/P90, breakeven strike, annual cashflows
`value_bundled_ppa`	PPA + LGC bundled valuation — energy NPV, LGC NPV, bundled premium

Stress Testing & Scenarios

Tool	Description
`run_stress_test`	Predefined scenarios: SA heatwave (+300%), wind drought (-80%), 2GW coal trip, QNI failure
`portfolio_what_if`	Natural language what-if: “What if SA prices spike 50%?”
`reverse_stress_test`	Find which scenarios produce a specified target loss

Generator Bidding

Tool	Description
`optimize_bid`	ML-optimised 10-band bid for a named generator
`suggest_rebid`	Rebid recommendation based on price change, constraint, outage, or wind shift
`get_bid_compliance`	Conformance rate, penalty count, compliance grade A/B/C
`compare_bid_vs_optimal`	Actual vs ML-optimal revenue uplift analysis

Battery & Storage

Tool	Description
`optimize_battery`	24 hr charge/discharge schedule for Hornsdale, Victorian Big Battery, Bomen
`get_spark_spread`	Spark spread, clean spark spread, and carbon cost by region

Settlement & Finance

Tool	Description
`get_settlement_summary`	AEMO vs internal reconciliation, variance, FCAS settlement, inter-regional residues
`get_settlement_runs`	Settlement run list with PRELIM/FINAL/R1–R3 status and amounts
`get_settlement_trueup`	Material variance analysis across run versions
`get_settlement_finance_summary`	GL journal totals, accruals, settlement statement

Environmental & Compliance

Tool	Description
`get_compliance_status`	NER obligations, conformance rate, overdue items, upcoming deadlines
`get_lgc_position`	LGC, ACCU, STC holdings, balances, and market prices
`get_carbon_exposure`	Emissions by fuel type, annual tCO2, estimated cost
`search_market_rules`	Semantic search over AEMO NER, market procedures, and regulatory documents
`generate_report`	DAILY_RISK, WEEKLY_MARKET, MONTHLY_COMPLIANCE, QUARTERLY_ENVIRONMENTAL, AD_HOC

Alerts

Tool	Description
`create_alert_rule`	Set PRICE_THRESHOLD / DEMAND_SURGE / FCAS_PRICE / FORECAST_SPIKE alert, evaluated every 5 min

Other Markets

Tool	Description
`get_wem_price`	Western Australian Electricity Market balancing prices and summary
`get_constraint_analysis`	Network constraint register, breach years, augmentation NPV

DNSP Intelligence

Tool	Description
`query_dnsp_summary`	Overall DNSP performance: AER compliance, BMP, connections, capex, rural, tariffs
`query_aer_compliance`	STPIS scores, RIN status, revenue cap utilisation, regulatory milestones
`query_bushfire_status`	AusNet BMP asset compliance, ELC inspections, BMO zone risk, capex vs allowance
`query_connections_queue`	NER queue: pending applications, timely connections rate, overdue, large customer pipeline
`get_asset_health`	Health index, risk factors, failure probability for a named asset
`get_network_loading`	MW loading, utilisation %, DER output by zone substation
`get_reliability_metrics`	SAIDI/SAIFI/CAIDI vs AER regulatory targets by region

Distribution Network Operations

Tool	Description
`get_outage_summary`	Active outages, causes, and affected customers by region
`get_hosting_capacity`	Available DER hosting capacity kW and limiting constraint by feeder
`get_curtailment_analysis`	Curtailed MWh, affected customers, reasons, and financial impact
`forecast_ev_impact`	Peak load delta, assets at risk, upgrade requirements by EV scenario and year
`get_vpp_performance`	VPP dispatch target vs actual response, accuracy, and revenue
`get_der_fleet`	Solar MW, battery MW/MWh, EV chargers by zone substation

How AI/ML Capabilities Work Together

User Question (Copilot)
        │
        ▼
Claude Sonnet 4.5 (FMAPI)
        │
        ├──→ query_price_forecasts   ──→ Model Serving (XGBoost)
        │                                     │
        │                                     ▼
        ├──→ query_generation_mix    ──→ Gold Table (Lakebase)
        │
        ├──→ query_aer_compliance    ──→ Gold Table (SQL Warehouse)
        │
        ├──→ get_asset_health        ──→ Gold Table + Model Serving
        │
        └──→ search_market_rules     ──→ Vector Search Index

The copilot’s agentic loop orchestrates calls to multiple tools in sequence or parallel, synthesising the results into a coherent, contextualised response.

Genie vs Copilot

Both the Copilot and Genie spaces provide natural language access to data, but serve different purposes:

Feature	AI Copilot	Genie AI/BI
Interface	Conversational chat	Direct question → SQL → result
Access to real-time data	Yes (via tool calls)	Yes (direct table access)
Custom tool calling	Yes (58 tools)	No
Multi-step reasoning	Yes (agentic loop)	Limited
ML model access	Yes	No
Best for	Complex analysis, trading actions, narratives	Direct data queries, ad hoc SQL
SQL visibility	No (abstracted)	Yes (show SQL)