Until September 2026, CUSignals sold a score that claimed to tell vendors which credit unions were about to buy software. We called it Vendor Signal. On September 1 we tested it for the first time. It failed.
This post covers what we built, how we tested it, what we found, and what we got wrong along the way. It also covers what we changed and what would change our minds. If you sell to credit unions, the last section is the one to read.
What we built
Vendor Signal gave every credit union five scores from 0 to 100, one per software category:
- Lending: loan-origination systems.
- Collections and risk: collections and credit-risk tools.
- Digital banking: online and mobile banking.
- Treasury and liquidity: funding and cash-management tools.
- Core and regulatory: core systems and compliance tooling.
Each score came with a plain-English "why now". The idea behind all five was simple. Strain shows up on a balance sheet before a credit union goes shopping. Loans growing faster than the staff who underwrite them should come before a lending-software purchase. Delinquency rising faster than the loan book should come before a collections purchase. We made that argument at length in an earlier post, which now carries a correction.
We set every weight by hand, from that idea. Nothing had been measured. There was no outcome to measure against either: the event we claimed to predict, a credit union starting to buy software, is not in any public filing. We see no purchases of our own.
Our model card said all of this. The rest of the site did not. We described the product to search engines by its buying-intent scores, and we sold them on the pricing page as a reason to subscribe.
How we tested it
To test a prediction you need the outcome. The closest public record of a software purchase is an announcement: a vendor or a credit union saying, with a date, that the credit union adopted a product. Our daily news layer had resolved 963 of those events to specific credit unions. After removing duplicates, that left 232 adoptions by 203 credit unions, from March 2022 to August 2026.
We picked two test dates, June 2024 and June 2025. For each one, we took the scores exactly as they stood on that date. Then we counted which credit unions announced a purchase in the following year. We tested every credit union with at least $50M in assets.
We graded the broadest version of the claim: did the credit union buy anything at all? That is the easiest test for a buying score to pass.
What we found
Picture pairs of credit unions: one that went on to announce a purchase and one that did not. A useful score ranks the buyer higher most of the time. A score with no information ranks it higher about 50 times out of 100, which is a coin flip.
| Test date | Credit unions | Announced a purchase | Pairs ranked right, of 100 | Top 25 that bought |
|---|---|---|---|---|
| June 2025 | 2,421 | 75 | 51 (AUC 0.509) | 0 |
| June 2024 | 2,460 | 49 | 54 (AUC 0.543) | 0 |
Vendor Signal ranked the buyer higher 51 and 54 times out of 100. That is a coin flip. In both years, none of the 25 highest-scored credit unions announced a purchase.
The top of the list did not help either. In June 2025, the top 10% of scores held 1.33 times the buyers a random list would. In June 2024, they held 0.61 times as many, which is worse than random. A signal that points one way one year and the other way the next is not a signal.
Then we compared it with two rankings that know nothing about buying:
| Ranking | June 2025, of 100 | June 2024, of 100 |
|---|---|---|
| Vendor Signal | 51 (AUC 0.509) | 54 (AUC 0.543) |
| Total assets | 68 (AUC 0.677) | 67 (AUC 0.665) |
| Members | 68 (AUC 0.675) | 65 (AUC 0.651) |
Sorting credit unions by size beat our model in both years.
That comparison is kinder to our model than it looks. Announcements lean toward large credit unions, because large ones make news. So asset size partly measures being announced, not only buying, and that inflates the size ranking. It was a generous opponent, and our score still lost to it twice.
The worst result came where it should have been strongest. Credit unions with $1B to $10B in assets have the biggest software budgets. In that tier, with 22 announced purchases in the June 2025 test, the score ranked the buyer higher only 37 times out of 100 (AUC 0.368). It was pointing the wrong way.
What the test cannot tell us
It counts announcements, not purchases. A credit union that bought quietly counts as a miss here. So the true hit rate is at least what we measured, and it may be higher. That qualifies the result. It does not rescue it.
It says little about individual categories. Most announcements do not name a category we score, and most categories had too few events to grade:
| Category | June 2025 events | June 2024 events | What we can say |
|---|---|---|---|
| Digital banking | 43 | 29 | The two years disagree: 63 then 51 of 100 (AUC 0.626, 0.514) |
| Core and regulatory | 12 | 8 | Too few to grade |
| Collections and risk | 5 | 0 | Too few to grade |
| Lending | 1 | 3 | Too few to grade |
| Treasury and liquidity | 0 | 0 | Nothing to grade |
Digital banking is the only category with enough events to compute anything, and its two years disagree. Either figure would need roughly 60 events to be worth defending. So the claim that the score predicts purchases in general was tested, and it failed. The claims about individual categories were never tested at all.
What we got wrong along the way
We sold the score before we could test it. Our model card called it unvalidated from the start. Our home page, our search description and our pricing page called it buying intent. A reader who trusted the marketing had no way to know the difference.
We let it rule accounts out. Every credit-union profile on this site had a "Should you reach out?" verdict. On most of about 4,300 profiles, that verdict came from Vendor Signal alone, and it sometimes told readers to spend the hour on a warmer account. A score that cannot find buyers cannot rule them out either. On September 8 we rebuilt that verdict on dated events, such as a new CEO or a completed merger. It no longer rules any account out on this score's say-so.
We misread an empty count. Our first version of the test said two categories, collections and treasury, could not be observed from announcements at all. That was wrong. Our list of vendors had no supplier assigned to either category, so the count was zero, and we read that zero as a fact about the world. It was a fact about our list. We corrected it on September 2, and every category is now graded on its actual event count.
We kept selling it after the test. The test ran on September 1, and we reported the failure on the home page. But the site's description of itself, the first picture on the home page, a pricing-card bullet and the pitch on every profile page still led with buying intent. We removed all of those on September 10 and 11.
What we changed
Vendor Signal no longer ships as a ranked list. Its result stays on the home page, marked "Failed the test". We kept it there on purpose rather than quietly deleting it.
Displacement Lists replaced it. A displacement list is a record, not a prediction. It shows which credit union runs which vendor, since when, taken from dated announcements. There is no score to argue with. It covers about 200 credit unions and 42 vendors. That is under 10% of the roughly 2,400 credit unions above $50M in assets, and every list says so. Where we can cite a contract term, it shows when a contract may come up for renewal. Today that is 35 of 358 relationships. Using conventional but uncited contract terms would have produced 213 renewal dates. We decline that trade.
The scores still exist inside the app, labeled as context. Subscribers can still see them. The Vendor Signal section of the app now says the score did not rank buyers when tested, and should be read as context, not as a call list.
Two older posts carry notices. The balance-sheet argument opens with a correction. The Q1 2026 vendor counts are marked superseded.
What would change our minds
A real outcome to test against. Announcements are a thin stand-in for purchases. The right test uses actual purchase records from vendors. It also has to separate two questions: did the credit union buy in the category, and did it buy from you? A deal lost to a competitor confirms the first and fails the second. A test that mixes them measures the wrong thing.
Enough events per category. Roughly 60 purchases per category, per test year, before any category-level claim is worth making.
Until both exist, we will not describe any score of ours as predicting purchases.
What this means for you
If you sell to credit unions, start from what is observable. Use a displacement list to find who runs a competitor's system and when the contract may come up. Treat any buying-intent score, ours included, as background at most. Never use one to cross an account off your list.
If you are evaluating someone else's intent score, ask three questions. What real outcome was it tested against? How many events were in the test? Did it beat a plain ranking by asset size? Ours did not.
The full statistics are on our disclosures page. These are results on historical data, not a guarantee about any future period, and nothing here is advice about any particular credit union.