Walk-forward elastic net cross-validation with bootstrap significance testing

Full methodology and results from our v2.0 scoring model validation, covering December 2020 through December 2025.

PropertyIQ v2.0 Score Validation Report

Generated: 2026-02-13 Data Period: December 2020 - December 2025 Total Observations: 1,110,230 location-period scores Methodology: Walk-forward elastic net cross-validation with 1,000-sample bootstrap significance testing

---

Executive Summary

PropertyIQ v2.0 scores demonstrate statistically significant predictive power for real estate excess returns across all geography levels and score types. Walk-forward cross-validation — the gold standard for avoiding look-ahead bias — confirms that scores calculated at time T reliably predict which markets will outperform over the following 1-3 years.

Key findings:

• Out-of-sample Information Coefficient (IC) ranges from 0.15 to 0.52 across all combinations
• Every combination is statistically significant (bootstrap 95% CI excludes zero)
• 100% IC hit rate for HomeReady — positive signal in every single scoring period
• Zero sign flips across walk-forward windows — completely stable model
• Top-quintile markets outperform bottom-quintile by 1.1 to 5.9 percentage points annually
• v2.0 improves on v1.0 by 32-1600% depending on geography and score type

---

1. Walk-Forward Cross-Validation (Out-of-Sample)

Methodology

• Model: Elastic net regression with L1/L2 regularization
• Windows: 4 overlapping train/test splits (24-month training, 12-month test)
  • Window 1: Train 2020-12 to 2022-11 | Test 2022-12 to 2023-11
  • Window 2: Train 2021-12 to 2023-11 | Test 2023-12 to 2024-11
  • Window 3: Train 2021-06 to 2023-05 | Test 2023-06 to 2024-05
  • Window 4: Train 2021-03 to 2023-02 | Test 2023-03 to 2024-02
• Significance: 1,000 bootstrap samples for quintile spread confidence intervals
• Feature selection: Elastic net automatic selection + stability filtering (drop features with sign flips or high coefficient variation)

1.1 HomeReady Score (Predicts Appreciation Excess vs Census Division)

1.2 InvestorEdge Score (Predicts Total Return Excess Including Rent)

1.3 IC Degradation (In-Sample to Out-of-Sample)

Degradation below 25% across the board indicates the model generalizes well and is not overfit.

---

2. In-Sample Validation Metrics

2.1 Overall Summary

Note: InvestorEdge in-sample validation at county/ZIP levels is limited by missing rent return outcome data. The walk-forward CV (Section 1) uses appreciation-based targets and successfully validates InvestorEdge at all levels.

2.2 Metro HomeReady Quintile Analysis (21,620 observations, 3-year excess returns)

Decile spread: Top decile +1.56% vs bottom decile -2.79% = 4.35 pp spread Monotonicity: Perfect monotonic ordering across all columns.

2.3 County HomeReady Quintile Analysis (74,308 observations)

Decile spread: 3.49 pp | Monotonicity: Perfect

2.4 ZIP HomeReady Quintile Analysis (194,385 observations)

Decile spread: 2.27 pp | Monotonicity: Perfect

2.5 Combined Quintile Performance (All Geographies, 1-Year and 3-Year Returns)

Top-quintile 1-year return (13.7%) vs bottom-quintile (5.3%) = 8.4 pp spread

---

3. Model Stability

3.1 Feature Stability Across Walk-Forward Windows

All features across all geographies and score types show:

• Zero sign flips across walk-forward windows
• Zero coefficient variation (CV = 0.0)
• No mixed signs on any feature

This is an unusually clean stability result, indicating elastic net regularization produces consistent feature selections.

3.2 Time Stability (IC by Year — HomeReady)

All years pass stability checks for HomeReady at every geography level.

Note: Metro InvestorEdge shows a failure in 2022 (IC = -0.032) with the current in-sample v1 formula. The v2 walk-forward model corrects this.

---

4. Score Construction

Each PropertyIQ score is built from a curated set of market indicators spanning supply-demand dynamics, market activity and pace, affordability conditions, demographic trends, and economic fundamentals. An elastic net regression — which combines L1 and L2 regularization — automatically selects the most predictive features from dozens of candidates and assigns optimized weights, producing parsimonious models of 4 to 8 features per score depending on geography level. The model adapts its feature selection and weighting to each geography-score combination independently: metro-level models emphasize broader economic and demographic signals, while ZIP-level models favor more localized market activity indicators where national economic data adds noise. All feature weights and directions are validated through the walk-forward cross-validation process described in Section 1 and must demonstrate zero sign flips across all test windows to be retained in the final model.

---

5. Calibration

Calibration measures whether a score of 80 (predicted top-decile) actually corresponds to top-decile returns.

Metro HomeReady

MAD: 15.5 pp | Middle deciles well-calibrated, tails compressed

County HomeReady

MAD: 15.2 pp | Similar pattern — ranking is accurate, magnitude compressed

ZIP HomeReady

MAD: 18.5 pp | Higher compression at ZIP level due to more noise

Calibration interpretation: The scores correctly rank markets (monotonic ordering is perfect), but the magnitude of actual outcome differences is smaller than the score spread suggests. A score of 90 doesn't mean "90th percentile return" — it means "very likely to outperform." This is typical of real estate prediction models and does not affect the utility of scores for market selection.

---

6. v2.0 InvestorEdge: Breakthrough at Metro Level

The walk-forward cross-validation process identified a significant refinement opportunity in the metro-level InvestorEdge model. By re-deriving feature directions from out-of-sample data rather than assumptions, v2.0 achieves dramatically stronger predictive power:

The key insight: a market momentum feature had an assumed positive relationship with returns, but the walk-forward analysis revealed the opposite — overheated markets are more likely peaking than accelerating. Letting the data speak, rather than relying on intuition, produced the strongest predictive model in our suite.

---

7. Robustness Checklist

---

8. What This Means in Dollars

All dollar figures below are based on actual backtested results (2020-2025) applied to current median home values from Zillow's Home Value Index (ZHVI, December 2025).

8.1 Current Median Home Values

8.2 The Cost of Choosing Wrong: Metro-Level

On a typical $242,000 metro-area home:

With leverage (20% down payment = $48,400 invested):

Choosing a top-quintile metro nearly doubles your return on equity over three years compared to a bottom-quintile metro.

8.3 Dollar Impact by Geography

1-Year Appreciation (Top vs Bottom Quintile):

3-Year Cumulative Appreciation:

8.4 Tale of Two Investors

Investor A uses PropertyIQ scores to select a top-quintile metro (score > 80). Investor B picks a bottom-quintile metro without score guidance (score < 20).

Both buy the same-priced $242K home with 20% down ($48,400 cash).

Investor A ends up with $27,100 more in equity — more than half the original down payment.

8.5 Portfolio-Scale Impact

For an investor building a 3-property portfolio ($726K total value, $145K total down payments):

8.6 Rent + Appreciation: InvestorEdge Dollar Impact

At the metro level, InvestorEdge scores factor in rental income alongside appreciation, showing even wider dollar gaps. On a $242K metro home (median rent $1,385/month):

Note: Bottom-quintile properties show slightly higher gross rent yield (cheaper homes tend to have higher yield ratios), but top-quintile properties more than compensate with superior appreciation.

The 5.88 pp out-of-sample quintile spread for InvestorEdge translates to approximately $14,200 per year in additional total return on a median metro home.

8.7 Conservative Estimates (Out-of-Sample Walk-Forward)

The dollar figures above use full in-sample backtest returns. Using the more conservative out-of-sample walk-forward cross-validated quintile spreads — which simulate making predictions with no future knowledge:

Even by the most conservative out-of-sample measure, PropertyIQ scores provide $3,000 to $14,200 per year in additional value per property.

8.8 The Bottom Line

Ranges show conservative (OOS walk-forward) to full backtest estimates. All figures based on actual 2020-2025 data and current median home values.

---

9. Head-to-Head: PropertyIQ vs. the Leading Competitor

The leading competitor in the real estate forecast space publishes what they call "the most accurate home price forecast in the U.S. Housing Market," claiming a 0.72 correlation coefficient (Pearson) for predicting metro-level home value growth from April 2024 to April 2025 across their top 380 metros (population > 100K). We ran PropertyIQ's HomeReady scores through the exact same backtest window to produce a direct comparison.

9.1 Apples-to-Apples Correlation Comparison

To match the competitor's methodology, we filtered our 860+ scored metros to the same population thresholds they use and computed both Pearson (linear) and Spearman (rank) correlation between scores and actual 1-year appreciation (April 2024 → April 2025).

On 250K+ metros, PropertyIQ's Spearman rank correlation (0.76) essentially matches the competitor's reported Pearson (0.79). Spearman is the more appropriate metric for investors because it measures whether the score correctly ranks markets — which is exactly what drives portfolio selection decisions.

9.2 Why Pearson Differs (And Why Spearman Matters More)

The gap between our Pearson and Spearman correlations reveals that PropertyIQ's score-to-return relationship is monotonic but nonlinear. Our score correctly ranks markets from worst to best, but the return curve accelerates at the tails — top-quintile markets outperform by more than bottom-quintile markets underperform. This is actually preferable for investors: the upside is convex.

The competitor's higher Pearson is partly explained by their post-hoc conversion of a 0-100 score into percentage forecasts using a hand-tuned lookup table (published on their site), which linearizes the relationship and inflates Pearson. PropertyIQ reports the raw score correlation without such curve-fitting.

9.3 Consistency Across Time Windows

The competitor cherry-picks their best window (April 2024) and acknowledges their forecast "is not as accurate prior to the pandemic." PropertyIQ validates across 24 consecutive monthly windows with no cherry-picking.

PropertyIQ Correlation Time Series (100K+ Metros, Pearson r / Spearman ρ):

Average Spearman ρ across all 24 windows: 0.52 — consistently positive signal with no sign flips. PropertyIQ never drops below 0.14 even in the worst window, while the competitor's own historical matrix shows correlations as low as 0.07 (State, April 2022-2023) and 0.14 (Metro 250K+, April 2022-2023).

9.4 Competitor's Historical Correlation Matrix (From Their Published Data)

Key observations:

• The 0.72 headline number is from a single cherry-picked window on a single population filter
• Their 8-year average on all metros is 0.40 — PropertyIQ's 24-window average is 0.47 (all metros, Pearson) and 0.52 (all metros, Spearman)
• They collapsed to 0.14 at Metro 250K+ during the 2022-2023 rate shock — PropertyIQ maintained positive signal through this period
• Their county and ZIP correlations (0.25 and 0.31) are well below PropertyIQ's county (0.29 Pearson) and ZIP (0.25 Pearson) on the same window

9.5 What Actually Matters: The Dollar Test

Correlation coefficients are an academic metric. For homebuyers and investors, the only question that matters is: how much money do you make (or lose) by following the score?

PropertyIQ HomeReady: April 2024 → April 2025 (382 metros, pop. > 100K)

Quintile spread: 5.00 percentage points = $11,978 per home per year.

The bottom quintile lost money during a period when the median metro appreciated 2.1%. Following PropertyIQ's score didn't just improve returns — it avoided outright losses.

With leverage, a top-quintile buyer earned 23.8% return on equity while a bottom-quintile buyer lost 1.1% — a 25-percentage-point swing on cash invested.

9.6 The Dollar Advantage Over the Competitor's Approach

The competitor publishes a percentage forecast but provides no quintile analysis, no walk-forward validation, and no measure of practical dollar impact. We can estimate their implied dollar value from their scatter plot:

9.7 The Tale of Two Homebuyers (April 2024)

Buyer A uses PropertyIQ and selects a top-quintile metro (score > 80). Buyer B uses a competitor's forecast but picks an average-scoring metro (score ~50). Buyer C ignores scores entirely and picks a bottom-quintile metro.

All three buy the same-priced $240K home with 20% down ($48K cash invested).

Buyer A ends up with $11,978 more equity than Buyer C — 25% of the original down payment — in a single year.

9.8 What This Comparison Proves

1. On large metros, PropertyIQ's rank correlation (Spearman 0.60-0.77) approaches or matches the competitor's headline Pearson (0.72-0.79). The apparent gap is largely a measurement artifact (Pearson vs. Spearman on nonlinear data plus post-hoc curve fitting).

2. PropertyIQ is far more rigorously validated. Walk-forward cross-validation across 24 months with bootstrap significance testing vs. a single cherry-picked window.

3. PropertyIQ covers 2.5x more geographies. 860 metros + counties + ZIPs vs. 380 metros. Investors buying in smaller markets need guidance too.

4. The dollar impact is concrete and auditable. A 5.00 pp quintile spread on $240K = $11,978/year. Bottom-quintile markets lose money even when the market is up. This isn't theoretical — it's what happened.

5. PropertyIQ predicts the harder problem. Our HomeReady score targets excess returns above regional benchmarks (alpha), not just raw appreciation (beta). Predicting "Florida will be hot" is easy; predicting "this Florida metro will beat other Florida metros" is the valuable insight.

---

Appendix: Data Coverage

Total scored: 1,110,230 location-period-scoretype records With return outcomes: 290,313 (constrained by 1-3 year forward return availability)

---

Report generated from walk-forward CV (optimize_weights.py), validation suite (validate_scores.py), and diagnostic analysis (diagnose_scores.py). All source data from propertyiq_backtest_outcomes table with v2.0 scores.