EdCircuit Staff 
AI procurement for schools often begins with an impressive demonstration.
A platform creates a lesson in seconds, summarizes student performance, translates a family message, recommends an intervention, and predicts which students may need additional support. The presentation is polished. The promised savings—in both time and money—are difficult to ignore.
Then someone asks a simple question:
“What happens to our students’ information after they use it?”
The room changes.
The sales team may offer a general assurance that the system is secure or “compliant.” But the superintendent, technology officer, curriculum director, and legal counsel need more than reassurance. They need to know what information the platform collects, where it goes, who can access it, whether it is used to improve an AI model, and how the district can remove it.
Those questions should not be saved for the final contract review. They belong at the beginning of the conversation.
AI purchasing is not simply a technology decision. It is also an instructional, financial, privacy, security, accessibility, and governance decision. A platform may influence what teachers create, how students receive feedback, what families are told, and which students are identified for additional services or opportunities.
Before approving an AI platform, pilot, or renewal, district leaders should be able to answer ten questions.
1. What problem are we trying to solve?
A district should begin with an educational or operational need, not a product demonstration.
Is the goal to reduce the time teachers spend creating routine materials? Improve access to tutoring? Translate family communication? Help counselors organize information? Provide students with faster feedback?
“Bringing AI into the district” is not a goal.
When the problem is poorly defined, schools are more likely to purchase a broad platform and then search for ways to use it. The result can be overlapping subscriptions, unnecessary data collection, inconsistent classroom practices, and tools that create as much work as they eliminate.
Start by describing success without mentioning the product. What should improve for students, educators, or school operations? How will the district know whether that improvement occurred?
The NIST AI Risk Management Framework encourages organizations to consider an AI system’s purpose, context, benefits, and potential harms before deployment. That process may reveal that AI is the right solution. It may also reveal that a simpler option would work just as well.
2. What information does the system collect?
The answer may go far beyond names, identification numbers, and grades.
An AI platform may collect prompts, assignments, voice recordings, images, browsing activity, device information, response times, and patterns of use. It may also create new information about a student, including proficiency estimates, risk scores, behavioral classifications, or recommended interventions.
The district needs a plain-language data inventory that identifies:
- Information collected directly from students and employees
- Data imported from district systems
- Metadata created through use of the platform
- Scores, inferences, and classifications generated by the AI
- Where the information is stored
- How long each category is retained
Schools should also ask whether every data element is necessary. A writing assistant does not need a student’s disciplinary history. A scheduling tool may not need special education information. Giving a platform access to additional data may make integration easier, but convenience alone does not justify the exposure.
If a provider cannot clearly explain what its system collects, district leaders cannot meaningfully evaluate the risk.
3. Will district data be used to train an AI model?
This question requires a direct answer.
“Your data is not sold” does not necessarily mean it is excluded from model training, product development, evaluation, or other secondary uses. The platform may also rely on an outside model provider or subcontractors whose data practices differ from those of the company signing the contract.
The agreement should address student and educator prompts, uploaded documents, generated responses, feedback, and usage information. It should state whether any of that material may be used to train, fine-tune, test, or improve a model.
If some form of training is permitted, leaders need to understand what happens first. Is the information identifiable, de-identified, or aggregated? Can the district decline the use without losing essential functions? Could information submitted by a student later appear in a response provided to someone else?
NIST’s Generative AI Profile recommends examining training-data practices for privacy and other risks. For schools, that review must include every organization that may handle district information, not only the company whose name appears on the platform.
A current list of subprocessors should be part of the review, along with a requirement that the district be notified when that list changes.
4. Who controls the data, and how is it deleted?
Schools sometimes assume that because they supplied the information, they retain complete control over it. Contract language may tell a different story.
Terms that give a provider broad, permanent, or transferable rights to district content deserve close scrutiny. The agreement should preserve district control, limit use to authorized educational purposes, and prohibit unauthorized sale, disclosure, licensing, or commercial exploitation.
Under certain FERPA arrangements, third-party providers must remain under the school’s direct control regarding the use and maintenance of personally identifiable information from education records. The Department of Education’s student privacy guidance for online tools also explains that education data may not be redisclosed or used for unauthorized purposes.
The district also needs to know what happens when a student leaves, an employee account closes, or the contract ends. Can the information be exported in a usable format? When will remaining copies be deleted? Are backups included? Will the provider confirm that deletion has occurred?
The Department’s model terms-of-service guidance identifies the return and destruction of data as important practices for limiting unnecessary exposure and maintaining control over student information.
5. What evidence supports the product’s claims?
A demonstration shows what a platform can do under favorable conditions. It does not show how reliably it will perform in a real district.
If a company claims its system improves achievement, reduces teacher workload, identifies struggling students, or personalizes learning, ask to see the evidence. Who conducted the evaluation? What students and schools participated? How long was the tool used? Were the findings independently reviewed? Did the study measure educational outcomes or only user satisfaction?
Accuracy claims need the same scrutiny. A system described as “95 percent accurate” may perform differently across grade levels, subjects, languages, disabilities, and types of assignments. The remaining 5 percent also matters. A modest error rate can still produce significant harm when a tool is used across thousands of students.
Providers should be willing to discuss known limitations, common errors, and situations in which their product should not be used. That conversation is often more revealing than the demonstration itself.
6. Could the output influence a high-impact decision?
A platform may be described as providing recommendations rather than decisions. The distinction becomes meaningless if employees routinely accept its recommendations without further review.
Stronger safeguards are necessary when an AI system could influence:
- Grades, promotion, or graduation
- Academic-integrity allegations
- Discipline or behavioral consequences
- Special education identification or services
- Mental health or safety responses
- Course placement or advanced opportunities
- College and career recommendations
- Hiring, evaluation, or employment decisions
The system should never serve as the sole evidence for a consequential decision. A qualified person must review the underlying information, consider the surrounding circumstances, and have the authority to reject the recommendation.
Students, families, and employees should also have a clear way to learn that AI contributed to a decision and to question the result. A person clicking “approve” at the end of an automated process does not create meaningful oversight if that person lacks the information or time to examine what the system produced.
7. Has the system been tested for bias and accessibility?
A product can perform well overall while creating persistent problems for particular students.
Was the system tested with English learners, students with disabilities, students from different racial and cultural backgrounds, and children across the grade levels the district intends to serve? A single overall accuracy rate may conceal wide differences among groups.
Accessibility also needs to be tested in practice. Can students navigate the platform with a keyboard or screen reader? Are captions accurate? Does the interface support magnification and alternative input methods? Can students with disabilities use the central features without being pushed into a separate or inferior experience?
AI-generated translations, captions, reading-level adjustments, and suggested accommodations can be helpful, but they still require appropriate review.
The district’s responsibility continues after launch. Complaints, errors, overrides, and outcomes should be monitored for patterns. If one group of students experiences significantly more problems than another, leaders need the information and authority to investigate.
8. How is the system secured, and what happens after an incident?
“Industry-standard security” is too vague to guide a purchasing decision.
Technology leaders should examine encryption, authentication, access controls, employee permissions, vulnerability management, system logging, backups, penetration testing, and independent security assessments. They should know where information is stored and which outside companies can access it.
The contract should define a security incident and establish how quickly the district will be notified. It should also identify who investigates, who communicates with families, and what documentation the district will receive.
This becomes more complicated when several companies support one platform. A breach may occur through a hosting service, model provider, analytics company, or another subprocessor the district never evaluated directly.
The Federal Trade Commission’s updated Children’s Online Privacy Protection Rule limits indefinite retention of covered children’s information and reinforces requirements related to data minimization and protection. COPPA applies to covered online services involving children under 13. Districts must also consider FERPA, state student-privacy laws, local policies, and the specific commitments included in their contracts.
9. What will implementation actually cost?
The subscription price rarely represents the entire investment.
Districts may need to pay for integration, account management, data migration, professional development, accessibility testing, cybersecurity review, technical support, and ongoing evaluation. Staff time should be part of the calculation as well.
If teachers must constantly correct unreliable outputs, counselors must investigate excessive alerts, or technology staff must manage recurring account problems, the platform may consume the time it promised to save.
Introductory pricing and free pilots deserve a careful look. What will the system cost in its second or third year? Are privacy controls, administrative reporting, or technical support limited to premium plans? Will the district pay more as usage grows? Are there separate charges for storage, integrations, or exporting district data?
A free tool is not free if students pay for it with their data or educators pay for it with their time.
10. What happens when the product changes—or the district leaves?
AI products can change quickly. A platform approved in September may use a different model, subprocessor, or data practice by January.
The contract should require notice of material changes involving privacy, security, model providers, data use, core functionality, or known limitations. The district should have an opportunity to review significant changes and, when necessary, end the agreement without penalty.
An exit plan is equally important. Can the district retrieve its information? How long will access remain available? What happens to student accounts, prompts, outputs, analytics, backups, and information held by subprocessors?
NIST’s AI Risk Management Framework Playbook recommends continued monitoring because the performance and trustworthiness of AI systems can change after deployment. Its guidance also addresses user feedback, appeals, overrides, incident response, change management, and the need to decommission systems that exceed an organization’s risk tolerance.
A district should know how it will leave a platform before it becomes dependent on it.
Procurement Is the Beginning
Answering these questions does not guarantee that an AI system will work as promised. It gives the district a disciplined way to decide whether the likely benefit justifies the cost and risk.
Whenever possible, implementation should begin with a limited pilot built around a defined purpose. The district should identify who may use the tool, what information may be entered, what training is required, and how success will be measured. Teacher, student, and family feedback should help shape the final decision.
At the end of that pilot, leaders should be willing to walk away.
A platform may be innovative but unnecessary. It may save time while creating privacy concerns the district cannot resolve. It may perform well in a demonstration but poorly with actual students. It may address a real need but require safeguards the provider is not prepared to offer.
The U.S. Department of Education’s guidance on AI use in schools supports responsible adoption while emphasizing privacy, existing legal requirements, and engagement with affected stakeholders, especially parents.
That engagement should happen before a contract is signed—not after families begin asking what happened to their children’s information.
A polished demonstration can show what an AI platform is capable of doing. Responsible procurement determines what the district will permit it to do with students, educators, and their data.
Editor’s note: edCircuit is vendor-neutral and does not endorse products or services. Any mention of a specific solution is included for contextual and informational purposes only.