The company that calculates Idaho Reading Indicator scores for kindergarten through third-grade students produced inaccurate results, causing a months-long delay in the release date.
The error is being blamed on an algorithm problem.
Teachers typically use IRI results to guide their instructional next steps. Now, teachers are back in the classroom either unaware that spring testing results were inaccurate or questioning whether the new recalculated results are reliable.
The State Department of Education releases an IRI-based report card in July each year to show reading scores for some 90,000 Idaho students. This year’s public report has been delayed and won’t be released until October.
“Our preliminary analysis indicates that over 90% of Idaho districts and charter schools had one or more students who were affected by (the) corrections,” wrote Ryan Cantrell, chief deputy superintendent, in a letter to administrators last week.
Ensuring that all Idaho students learn to read by third grade is one the state’s top priorities and accurate scores are critical to that $72 million-a-year early literacy effort. The SDE relies on IRI results to measure reading achievement. Imagination Station, the vendor that administers Idaho’s statewide reading assessment, is beginning its sixth year providing a screener, diagnostic, monitoring progress, automatic scoring, and school, district and state reporting.
“We are assessing the impact,” said Scott Graf, the SDE’s communications director.
Despite a nearly four-month delay caused by the mathematical error, Istation won’t be penalized. The Dallas-based company is set to receive $723,570 this year, according to the SDE.
Once the problem was identified, Istation recalculated the data and provided updated results for state validation in late August. That process is currently underway and expected to be completed in the fall.
The IRI is a norm-referenced assessment that provides educators with a way to measure reading progress. Students test in the fall and again in the spring. Those results indicate how well Idaho students read in relation to students of their age or grade. They also measure growth over the school year.
“I think the more immediate concern is that if those scores aren’t right for some of those students, you’re not getting an accurate picture of where that student is,” said Greg Wilson, state superintendent Debbie Critchfield’s chief of staff.
IRI reading results are released in proficiency tiers: at grade level, near grade level or below grade level.
The algorithm problem created a situation where the original scores were lower than they actually were, Graf said. The inaccurate 2023 data was never publicly released statewide.
But before the statewide release, educators have access to the results from the Istation website. That’s when educators started asking questions.
“They said, ‘Hey, this isn’t making sense. I’m looking at Johnny’s scores. Johnny scored proficient in every subcategory. But he’s not proficient overall. Mathematically, that doesn’t add up,” Cantrell recalled. “(Istation) came back to us and said, ‘guys, we found an error in the way we re-normed the data for 2023 and it’s going to require us to recalculate all of Idaho’s 2023 scores for the spring.”
Both parents and educators are impacted by the delay
As Idaho Education News repeatedly requested the results, the release date of the statewide IRI report card has been set and changed multiple times this summer. The SDE now expects to release the full report in early October, months behind the release in past years.
For now, parents are unable to accurately evaluate their child’s reading progress last school year, nor can they gauge the teacher’s or school’s performance. The level of growth a student achieves in reading between fall and spring is one of several outcomes measured by the test results. For superintendents, there is one less tool to evaluate school and teacher performance.
The distribution model for half of the state’s early literacy money is contingent on scores. But that distribution doesn’t come until later in the school year, so the error will not have an impact, Wilson said.
“I think we’re in a good place,” Wilson said.
This is an abnormality that’s not acceptable, even for one or two kids, let alone 200 or 300, said Cantrell.
The problem started in the spring of 2023. Every four or five years, norm-referenced data has to be re-normed. During that re-norming process, an algorithm error was introduced.
“So let’s say I take a test and I get an 83. Well, an 83 might put me in the 50th percentile. But what we found is that their algorithm had an error, so my 83 really should have put me in the 52nd percentile or the 53rd percentile, because it’s norm-reference data,” Cantrell explained.
“We’ve been using Istation now for more than five years. And this is the first issue we’ve run into in five years. And they fixed it. They corrected it on their own. Everybody’s human — mistakes are going to be made. Sometimes the best we can do is fix it, be transparent about it and then move on,” he said.
