In this section of the course, we are going
to discuss artificial intelligence.
The artificial intelligence section of the course
focuses on Domain 1, Governance, Risk, and Compliance,
as well as Domain 3, Security Engineering,
specifically Objectives 1.5 and 3.6.
Objective 1.5 states that you must be able
to summarize the information security challenges
associated with artificial intelligence adoption.
Objective 3.6 states that given a scenario,
you must be able to use automation to secure the enterprise.
Artificial intelligence or AI has already revolutionized
how we interact with technology.
However, with these advancements come significant challenges
such as the need for responsible use
and ethical considerations.
Additionally, as AI becomes more embedded in our lives,
it introduces new risks, including potential vulnerabilities
and the misuse of AI for malicious purposes.
Understanding and addressing these issues
and risks early is crucial to ensuring that AI
continues to contribute positively
to our day-to-day and beyond.
As we go through this section,
we will cover many topics
related to artificial intelligence,
including generative artificial intelligence,
ethical governance considerations, legal and privacy risks,
threats to the model, AI-enabled attacks,
AI manipulation attacks, AI usage risks, and AI bots.
First, we will look at
generative artificial intelligence or AI.
Generative AI refers to artificial intelligence systems
that create new content such as text, images,
code, or music.
Generative AI learns patterns from existing data
and generates outputs that resemble
or extend the original data.
Models like the generative pre-trained transformer or GPT
are trained on large data sets
to understand language, context and structure.
Based on their amount of training data,
GPTs can produce coherent and contextually-relevant content.
Baseline generative AI concepts include code assist
and documentation.
Code assist helps developers by suggesting
or completing code.
Documentation automatically generates technical documents
such as user guides, release notes,
and other technical documentation.
In practice, a developer might use generative AI
to assist in writing code by receiving realtime suggestions
or generating comprehensive documentation
based on their code base.
Next, we will explore ethical and governance considerations.
Ethical and governance considerations are used
to establish frameworks and guidelines
to ensure AI is developed and used responsibly,
and that it aligns with societal values
and organizational principles.
Ethical governance considerations
include the ethical governance of AI
and organizational policies on the use of AI.
The ethical governance of AI refers to the structures
and practices put in place
to guide the ethical deployment of AI.
Ethical governance further ensures
that this technology is used in ways that are fair,
transparent, and accountable.
Organizational policies on the use of AI
are specific rules and guidelines created by companies
to govern how AI is implemented within their organization.
Local organizational policies ensure that AI use aligns
with both legal requirements
and an organization's ethical standards.
For example, an organization might implement
an ethical governance framework that requires all AI models
to undergo bias testing, and mandates transparency
into the AI decision-making process.
This requirement can be supported by internal policies
that restrict the use of AI
in areas where it could lead to discrimination
or privacy violations.
Then we will look at legal and privacy rules.
Legal and privacy rules ensure that AI systems are designed
and deployed in compliance with laws and regulations
to protect individual rights and personal data.
Legal and privacy implications include potential misuse
and explainable versus non-explainable AI models.
The potential misuse of AI arises when AI systems are used
in ways that could lead to discrimination,
privacy violations, or other harmful outcomes.
This misuse is often exacerbated by the opacity
of the models that are used.
This infers that AI models may be explainable
or non-explainable.
Explainable models are those that provide clear,
understandable reasons for their decisions.
Non-explainable models
like many deep learning algorithms
can be difficult to interpret or understand.
Non-explainable models make it difficult to determine
why the AI made the decisions it did.
These are often referred to as black box models.
Non-explainable AI models raise challenges in legal contexts
where transparency is required.
For example, in a credit scoring application,
using a non-explainable model could result in
unfair loan denials that cannot be easily justified.
These AI decisions could lead to legal challenges
considering anti-discrimination laws.
Next, we will explore threats to the model.
Threats to the artificial intelligence or AI model
are those that can compromise the integrity,
security, or confidentiality of AI systems.
The compromise of AI models can lead to incorrect decisions
or unauthorized access to sensitive data.
These threats are potential attack paths
for malicious actors.
Threats to the model include prompt injection,
unsecured output handling, training data poisoning,
model denial of service, supply chain vulnerabilities,
model theft, and model inversion.
Prompt injection occurs when an attacker
manipulates input prompts to alter the model's behavior.
Prompt injection may then produce harmful
or unintended outputs.
Unsecured output handling can be exploited
to leak sensitive information or execute malicious commands.
Next, training data poisoning
is the intentional manipulation of training data
to skew the model's predictions or performance.
Training data poisoning can lead to bias
or incorrect decision making and operational failures,
security violations or ethical issues
that undermine the trust and reliability of the AI model.
Model denial of service attacks
aim to overwhelm the AI system,
preventing legitimate users from using it.
Next, supply chain vulnerabilities emerge
when compromised components or third party libraries
used in model development
are exploited to insert malicious code, backdoors,
or alter the functionality of the AI system.
AI supply chain attacks can lead to unauthorized access,
data breaches, or the deployment of flawed models
that behave unpredictably or maliciously in production.
Model theft is the unauthorized extraction
of a model's intellectual property.
Model theft could allow attackers to replicate
or misuse the model.
Finally, model inversion allows attackers
to reconstruct sensitive data used during AI training.
By analyzing the model's outputs,
an attacker could use model inversion to extract
and expose sensitive data, such as personal information
or proprietary details
that were used during training the AI model.
This could lead to privacy breaches and data theft.
In application, an attacker might use prompt injection
to cause a chat bot to generate harmful advice,
exploit unsecured output handling
to extract confidential information,
and engage in training data poisoning
to bias an AI model used for content moderation.
Each of these potential vulnerabilities underscore the need
for managing threats to the AI model.
Following that, we will look at AI enabled attacks.
AI enabled attacks are attacks
that leverage artificial intelligence
to enhance the sophistication, scale,
and effectiveness of malicious activities,
making them more difficult to detect and defend against.
AI enabled attacks include unsecure plugin design,
AI pipeline injectors, and automated exploit generation.
An AI plugin is an add-on or extension
that integrates artificial intelligence capabilities
into existing software or platforms.
AI plugins enable enhanced functionality such as automation,
data analysis, or personalized user experiences.
Unsecured plugin design can leave AI systems
vulnerable to exploitation, allowing attackers
to gain unauthorized access
through poorly secured extensions or integrations.
AI pipeline injectors place malicious data or code
in the AI training or deployment pipeline.
AI pipeline injection can alter the model's behavior
in harmful ways, such as introducing bias,
generating incorrect or misleading outputs,
creating backdoors for unauthorized access,
or compromising the model's integrity.
Automated exploit generation leverages AI
to quickly discover
and create exploits for unpatched vulnerabilities.
Automated exploit generation significantly increases
the speed and precision of attacks.
For example, an attacker could exploit
an unsecured AI plugin in a web application,
inject malicious code into the AI model
during its training phase,
and then use AI to automatically generate exploits
targeting the compromised system.
Then we will explore AI manipulation attacks.
AI manipulation attacks involve the intentional alteration
or deception of AI systems to produce harmful
or misleading outcomes.
AI enabled manipulation attacks
include social engineering and deepfakes
through digital media and interactivity.
Social engineering refers to the techniques
that exploit human trust
to deceive individuals into interacting
with manipulated AI outputs.
Interactivity may occur through interactive chatbots
or social media platforms.
Manipulating AI outputs could include deep fake videos
or audio that appear genuine but are entirely fabricated.
Deepfake technology uses AI to create realistic
but fake digital media.
Deep fakes can be distributed through social media,
video conferencing, and streaming services
to convincingly mimic real people, leading to misinformation
or reputational damage.
For example, an attacker might use deepfake technology
to create a video of a company executive
making false statements,
then employ social engineering tactics
to distribute the video widely,
causing significant harm to the company's reputation
and misleading the public.
Next, we will look at AI usage risks.
AI usage risks are the potential dangers
and unintended consequences
that arise from the implementation
and reliance on artificial intelligence systems.
AI usage risks may be realized when AI systems
are not properly managed or understood.
Risks of AI usage include overreliance,
sensitive information disclosure to the model,
sensitive information disclosure from the model,
and excessive agency of the AI.
Over-reliance on AI can lead to situations
where human judgment is undervalued,
potentially resulting in decisions
that overlook important contextual factors
that AI cannot calculate for.
Sensitive information disclosure to the model
occurs when private or confidential data
is input into AI systems without adequate safeguards.
Sensitive information disclosure to the model
raises the risk that disclosure data could be mishandled
or exploited by the AI model.
Sensitive information disclosure from the model
involves the unintentional release of private data
that the AI has learned during its training.
Sensitive information disclosure from the model
can lead to privacy breaches.
Finally, excessive agency of the AI refers to situations
where AI systems are given too much control or autonomy.
AIs with excessive agency can make decisions
or take actions that are harmful
or not aligned with societal values.
In poor practice and demonstrating overreliance,
a healthcare provider might rely on an AI system
for diagnosing patients,
inputting sensitive patient data
without sufficient privacy protections.
And then the risk the AI inadvertently reveals
private information during its operations,
while also allowing the AI too much autonomy
in making critical health decisions.
Finally, we will explore AI bots.
AI bots are automated software programs
that interact with users or systems
to perform specific tasks.
AI bots often simulate human behavior
in areas like customer service or data management.
AI bots may also serve as assistance and digital workers.
AI bot concepts include access and permissions, guardrails,
data loss prevention, and the disclosure of AI usage.
Access and permissions refer to the controls put in place
to ensure that AI bots only interact with
data and systems they are explicitly authorized to access.
By doing this, access and permissions
prevent unauthorized data manipulation
and security breaches.
Guardrails are predefined boundaries or rules
that restrict the actions of AI bots.
Guardrails ensure AI bots
operate with safe and intended parameters.
Data loss prevention strategies are used with AI bots
to prevent sensitive information from being accessed
or leaked inadvertently.
Finally, disclosure of AI usage involves informing users
that they are interacting with an AI bot
rather than a human.
Disclosure of AI usage is important for transparency
and maintaining trust.
In application, a customer service bot
with proper access controls and guardrails
might be used to assist users with account inquiries.
Data loss prevention measures would ensure the bot
cannot access or leak sensitive financial information,
and clear disclosure would let customers know
they're interacting with an AI system and not a human.
To finish things off, we'll take a short quiz
to see what you learned during this section of the course,
and we will review each of those quiz questions fully
to ensure you can explain why the right answers were right
and the wrong answers were wrong.
So let's get ready to dive into artificial intelligence
in this section of the course.