AI Assurance Framework

The proposed solution must align with UNSW’s Core Values and Strategic Plan and improve services or efficiencies. Approval for the project and related expenses, such as software development and maintenance, must be granted by the relevant budget owner or sponsor. Key review points include:

UNSW Vision: To improve lives globally through innovative research, transformative education and commitment to a just society, as outlined in the UNSW 2025 Strategy.

UNSW Values:

Demonstrates Excellence: Delivers high performance and service excellence.

Embraces Diversity: Values individual differences and promotes inclusion.

Drives Innovation: Thinks creatively and embraces change.

Displays Respect: Treats others with dignity and communicates with integrity.

Builds Collaboration: Works effectively within teams and builds relationships with stakeholders.

Improvement in services or efficiencies should be measurable, aligned with organisational objectives (2024 Strategic Priorities) and attributable to the specific project or technology. Initial and ongoing costs need to be budgeted according to the financial modelling template within the business case process.

The key issue to consider is whether the solution involves the use or creation of sensitive, commercial, or personal data.

UNSW is committed to maintaining the security, integrity, and appropriate management of its data. To ensure data is kept secure, it is classified and only used or stored in applications, systems, or platforms with the appropriate level of data protection. This is crucial when engaging with AI systems, as data must be stored in systems that match or exceed the data's classification level.

The UNSW Data Classification Standard defines data as Public, Private, Sensitive, or Highly Sensitive and should be consulted when using AI to analyse or review UNSW data. A comprehensive list of examples is available from Cyber Security.

When sharing UNSW data, even internally, a Data Sharing Agreement (DSA) is required. Sharing Personal Information (classified as Sensitive or Highly Sensitive) always requires a DSA. Additional information is available on the SharePoint page for DSAs.

Additionally, consider the Information Governance policy (still in development) when using UNSW data within AI systems. This will combine current data governance, privacy, recordkeeping, and cybersecurity policies (see Compliance).

Sensitive data considerations may apply when using or creating data about:

• Children
• Religious affiliation
• Race or ethnicity
• Political associations
• Trade union associations
• Gender and/or sexual diversity
• Criminal records
• Health or genetic information
• Other personal data

There is growing public concern about the potential for AI to cause harm. While not all uses of AI impact human rights, AI-informed decision-making can have significant consequences. Key areas of harm to consider include:

• Physical harm
• Psychological harm
• Environmental or community harm
• Unauthorised use of sensitive or personal information
• Unintended identification or misidentification
• Infringement of intellectual property rights
• Financial or commercial impact
• Incorrect advice or guidance
• Inconvenience or delays

AI-informed decision making

The term "AI-informed decision-making" refers to decisions that are significantly assisted by AI technology and have legal or significant effects on individuals. The Australian Human Rights Commission recommends that AI-informed decisions be lawful, transparent, explainable, used responsibly, and subject to human oversight, review, and intervention.

Monitoring and testing

You must closely monitor your AI system for potential harms by assessing outputs and testing results to identify unintended consequences. It is essential to quantify these effects, even during testing and pilot phases, as consequential decisions can cause real harm. Be aware that changes in the AI system's context or usage can lead to unexpected outcomes. Any planned changes should be thoroughly evaluated and monitored.

Fairness in AI systems requires safeguards to manage data bias and quality risks. For instance, can you explain why you selected specific data for your project over others? Can you justify the data's relevance and permissions? AI systems often draw from multiple datasets to find new patterns and insights, so it's crucial to decide whether you should use the data, especially if historical data was collected for different purposes.

Improving fairness in AI

• Diverse data: Ensure AI systems are trained on diverse and representative data to reduce bias. This helps the system reflect real-world diversity and consider all groups affected by its decisions, including cultural sensitivities and underrepresented populations.
• Bias detection: Use tools and techniques to detect and measure bias in AI systems. Open-source tools like Microsoft’s Fairlearn toolkit offer visualisation dashboards and algorithms to mitigate unfairness.
• Stakeholder engagement: Involve a wide range of stakeholders, especially those affected by the AI system, in the development process to identify potential unfairness and align the system with societal values.
• Ethical principles: Adhere to ethical principles such as fairness, accountability, transparency, and ethics to guide AI development.
• Governance structures: Establish governance structures to oversee AI development and deployment, ensuring fairness is prioritised throughout the system's lifecycle. The World Economic Forum discusses this in more detail.
• Continuous monitoring: Continuously monitor AI systems after deployment using performance measures and fairness targets to identify and correct biases. Determine fairness measures at the scoping stage to prioritise important decisions from the outset.

The inner workings of commercial AI systems can be inaccessible and complex, posing risks when relying on unexplainable insights or decisions. To mitigate these risks, consider the role of human judgment in assessing AI outputs.

Assessing AI-generated insights

If you can't explain how an AI system produces its insights, evaluate the potential harms, their likelihood, and the ease of reversing them. Document these considerations, especially if midrange or higher risks are identified.

Consult with the relevant user community when designing AI systems, particularly operational ones. Ensure accountability and transparency, demonstrating positive benefits and outcomes.

Transparency and accountability

Transparency requires that individuals affected by AI decisions understand the basis of these decisions and have the means to challenge them if they are unjust or unlawful. Ensure that no one loses rights, privileges, or entitlements without access to a review process.

You must have a way to explain how AI informs decisions. If the system is a “black box” or too complex to explain, human judgment should be used to intervene before acting on AI-generated insights. Document this oversight process. In low-risk environments, it may be sufficient to have mechanisms to easily reverse actions, like overriding an automated barrier.

Operator training

There is a risk of over-reliance on AI results. Ensure system operators, including those exercising judgment over insights, are trained to critically assess AI-generated outputs and understand system limitations. For operational AI systems, users should be confident they can reverse any harm or that a Responsible Officer can make informed decisions. Non-operational system users must be skilled in interpreting AI insights if they are to be relied upon.

Assigning responsibility

Identify who is responsible for:

• Using AI system insights/decisions
• Project outcomes
• Technical performance of the AI system
• Data governance

These roles should ideally be held by different individuals who are senior, skilled, and qualified.

Human oversight

For operational AI systems, ensure that humans can intervene and are accountable, which may also apply to non-operational systems. This fosters user confidence and control in your AI system.

Even small AI system projects may have privacy or security vulnerabilities. For example, an analytics project that stores commercially sensitive data in a non-secure environment unbeknown to the user. 

As with any emerging technology, AI systems can pose new cyber security risks and so it is important to be vigilant. You must comply with the mandatory requirements as identified in the Cyber Security Policy. 

Other critical policies to consider include:  

• Data Breach Policy, Policy & Procedure
• Risk Management Policy  
• Data Governance Policy  
• Privacy Policy  

There are standards which provide for better practice in all of our technology developments which remain crucial for AI system developments. 

• Cyber Security Standard, Risk Management  
• Cyber Security Standard, Data Security  
• Data Classification Standard  

A critical first step is completing a Vendor Security Risk Assessment to ensure that they meet the minimum criteria necessary to protect UNSW.   

For help on assuring your AI system remains cyber compliant please visit MyCyberHub

Just like everything else that we do, the use of AI must comply with UNSW Policies, the Ethical and Responsible Use of Artificial Intelligence at UNSW, UNSW’s guiding principles on AI, and relevant legislation. 

Critical policies to consider include:  

• Cyber Security  
• Data Breach Policy, Policy & Procedure   
• Risk Management Policy  
• Data Governance Policy  
• Privacy Policy
• IP Policy

This should include relevant standards. 

You must also make sure your data use aligns with relevant privacy, discrimination, copyright and government information and records legislation.  

In addition, any AI solution proposed to be deployed should align with UNSW Strategic Outcomes as outlined in the Business Value section below. It is also important to consider whether there is an alternative way to deliver the benefit attributed to the solution, without the use of AI. 

There are several resources available that can guide our use of AI and help us observe compliance standards. These include Australia’s AI Ethics Principles (Australia’s Artificial Intelligence Ethics Framework). 

This is supported by the Group of Eight Principles on the Use of Generative Artificial Intelligence for use by its members within higher education: 

1. Maintaining academic excellence and integrity in teaching, learning, assessment and research will continue to be a priority for Go8 universities as they adapt and lead in the responsible use of generative AI. 

2. Go8 universities will promulgate clear guidelines for the appropriate use of generative AI technologies by their academic staff, researchers and students.

3. Go8 universities will develop resources to empower students, academic and research staff to engage productively, effectively and ethically with generative AI. 

4. We will work to ensure equal access to generative AI. 

5. Go8 universities will engage in collaborative efforts to exchange and implement best practices as generative AI technology and its role in society continue to advance. 

For those participating in research and education in Europe, further resources are available in the EU Briefing on EU Legislation in Progress.  

The procurement process may be the best place to ensure the mandatory policy requirements for AI systems are considered early on and measures are taken to mitigate risk in using AI generative tools. Mechanisms for ensuring performance, ongoing monitoring and calibration as well as management of risk may be negotiated and built into contractual agreements with vendors. Please make sure you seek help from procurement experts and appropriate legal support.