By Michael Horn, Columnist

Last week the New York Daily News came out guns blazing against New York City’s bold innovation designed to rethink how students learn math, the School of One. The Daily News’s headline declared the project “a pricey reject.” This might have been fine except that its information was based on a study that was far more nuanced in its findings and was commissioned with the purpose of helping to improve the program, not declaring with any finality the program a success or failure. Also worth noting is that the program didn’t cost taxpayer dollars, but was funded by private philanthropic dollars.

Innovation is a messy process. Although we have unearthed in our research several things that make the process of innovation far more predictable and ultimately successful, one of those fundamental findings is that using a “discovery-driven” process to identify what works—and only then scaling—is critical.

What that means is that rarely if ever does an innovation hatch out of someone’s mind as a perfectly formulated plan. The innovations that succeed are those that have the time and space to test and learn and the money left over to then iterate and tick and tack between different approaches until they nail the job of the user. Failure is actually a critical part of innovation—and its occurrence and learning from it is what allows for ultimate success. It would be awfully surprising if an innovation got everything right in its early days. The first personal computers and cell phones, to name just two, certainly didn’t.

Which brings us to New York City’s School of One. The fundamental truth that jump-started School of One’s, now Teach to One’s, approach was this: each student enters a classroom with different levels of background math knowledge and skills more generally.

Given this, teachers face an extraordinary challenge in being able to individualize for these different learning needs in the conventional classroom structure. The results are clear that today’s approach is not working for far too many students.

School of One blows apart that structure (quite literally) by administering a diagnostic assessment to each student, creating an individualized learning plan for each student, and then offering students different ways to learn based on the diagnosis of their distinct needs—be that instruction in large or small group clusters or individual online or offline work. At the end of every day, students take another diagnostic assessment to inform what they’ll do next, as School of One operates on a mastery, or competency-based, model in which students move on when they have mastered a concept.

As part of this experiment, New York City funded a series of independent studies to engage in the innovation process—that is, to test whether the innovation was really working after some early promising results that gave the city the confidence to bring the model into regular-day instruction, learn both from mistakes and the positives, and iterate appropriately.

Researchers from The Research Alliance for New York City Schools out of NYU were one group conducting a study to help School of One learn, and they recently produced a helpful and measured study, which they also released on their website. As they stated in their research, “[School of One’s] theory of action is based on the premise that students cannot learn grade-level content when they are missing precursor skills from earlier grades. Similarly, more advanced students should be able to move on to higher-level skills when they are ready. [School of One] seeks to meet each student wherever he or she is on the continuum of math knowledge and skills, while acknowledging that it may take several years to see the results of this strategy.”

At a high level, what they found was that in the first full year of operation (the 2010-11 school year), of three schools using School of One, one school did better compared with students in peer schools, one did worse, and one did the same when looking at New York State’s grade-level math test. In essence, the effect was neutral. As the Daily News reported, two of the schools dropped the program, although several new schools have adopted it. What’s interesting is that, at least preliminarily, a school that had a slight dip in its first-year results appears to have had very impressive results in its second year, in 2011-12, according to the school’s principal and this subsequent piece.

There are several missing parts of the story. First, School of One is doing exactly what we would hope an innovation process would do to get better in this respect: It put a solution out there, is studying what works and what doesn’t, and is then tweaking and, apparently, improving. The study suggests some possible questions about School of One’s approach and how we ought to improve math learning—for example, maybe students can learn easier concepts in the context of more advanced ones; maybe the School of One approach doesn’t allow students to own their learning enough; or maybe there were just operational issues that got in the way of knowing whether the premise behind School of One is sound.

Indeed, based on this state report regarding the host site that did not perform as well as peer schools and will ultimately be closed by New York City, it appears that a host of issues having nothing to do with School of One may be to blame for the first-year outcomes. In the Educational Impact Statement that the City released in advance of finalizing the closure decision, the City stated that it believed the new school that will be opened in its place should be better able to leverage School of One. In other words, School of One is far from being a reject, at least at this point.

The reporting from the Daily News misses this completely—and thereby sets back the dialogue across legislatures and boards of education, many of whose members will only ever see that first piece. It also hurts the chances that other innovative efforts like School of One will take as considered an approach to iterating around boosting student learning.

As Matthew Riggan, from the University of Pennsylvania’s Consortium for Policy Research in Education, wrote for the American Enterprise Institute’s series on the role of for-profits in education in a paper titled “Between Efficiency and Effectiveness: Evaluation in For-Profit Education Organizations,” “Although formative evaluation is critical to the development of some products or services, for some firms it can also be perilous. This type of evaluation involves identifying what is not working in order to fix it. Perhaps more important, it involves documenting those shortcomings, in the form of either quantitative outcomes or customer feedback. The intense scrutiny some private firms face—in some instances extending all the way to the subpoena of e-mails and internal documents—has made them reluctant to engage in this process for fear that politicians, policymakers, activists, and the general public will interpret and use both internal and external formative evaluation results in a summative manner, hurting both their reputations and bottom lines.”

As this sorry incident shows, non-profits will perhaps start to fear the same thing, which would not be good for innovation in education. That would not be a problem were the country’s education system succeeding for all students, but it’s certainly not. Innovation is imperative.

Another missing part of the story is one that plagues evaluation efforts of programs in all schools right now. We have a system that administers grade-level assessments and gives no credit to students who start well below grade level and make amazing progress or to those who are able to race ahead. As the study itself said, the New York State assessment used to judge School of One “focuses mostly on grade-level material. Thus, it is possible that some students made progress on lower-level math skills that were not detected by the state test.” This points to a much larger systematic problem across our education system, which is that we have to move away from a seat-time based model in which time is held as a constant and learning is highly variable and instead move to a competency-based learning model in which time becomes the variable and learning is the constant and we reward programs not only for bringing students to proficiency but also for helping students make incredible growth.

We know the system as a whole is not working, but because of this lack of nuanced data, it’s hard to figure out where and why. One hypothesis around the School of One’s apparent second-year success is that potentially the program was filling holes in the students’ learning in the first year to build a foundation for their learning and only in the second year was it able to get them up to grade level so that their gains would begin to show on the state assessments. Without more information, like NWEA’s MAP test, the sort of assessment that Teach to One will be administering going forward, it’s hard to know.

The story around the School of One points to a few other principles of innovation in education. It’s critical that we don’t dupe ourselves into thinking that there will be a panacea to solving our education woes or a silver bullet. The problems of educating students are much too complicated. Having a portfolio of approaches that are pushing different innovations and experimenting iteratively is critical—much as New York City has done by setting up its iZone initiative, of which School of One was perhaps the boldest experiment. As a result, it’s also important not to hail a new innovation as the answer until it’s actually been proven as such.

The School of One received significant hype and headlines early in its formation, and although the leadership was smart to innovate at first in areas of nonconsumption—summer school followed by after school—the attention it received in Time magazine and the steadfast championing from former New York City Schools Chancellor Joel Klein has invited a blow back. I don’t know whether School of One will ultimately be successful—although I think the premise behind it makes sense and is exciting, there are lots of operational issues to work through it seems to me—at the moment it appears to be an important research and development project, not a fully baked solution that should have been promoted as much as it was. I’d much rather see it innovate quietly and scale gradually as it proves itself (which is what New Classrooms now appears to be doing with it).

Lastly, I would like ultimately to see a system in place that rewards those programs that make the most progress with students— and therefore drives out of the market those programs that are not successful in boosting student outcomes. I’ve been consistently clear about that. A system like that would allow innovators to hone their programs against actual outcomes and gain revenue the better they became. As part of a system like this, there should be a natural mechanism that provides space for risk-taking and research that pushes the envelope in bold ways that could unearth breakthroughs in education. We should do it carefully with sound principles of innovation in place that allow us to tweak and improve the solutions; shuttle them if they show that they clearly won’t work over a reasonable amount of time; and naturally allow them to scale if they do.

Our students deserve that type of serious innovation that has created so many breakthroughs in so many other parts of society. We shouldn’t give them short shrift.

Michael Horn is Executive Director of Education at Innosight Institute, the co-author of “Disrupting Class.” This post first appeared on Forbes.com