In this lesson,
we will learn about Data Management.
Data management involves organizing, protecting,
and analyzing data
to ensure its integrity, confidentiality,
and availability within specialized systems.
Data management concepts include aggregation
and data analytics.
Aggregation refers to the process
of collecting and combining data
from various sources to create
a single comprehensive data set
that can be analyzed for patterns, trends, or anomalies.
Data analytics involves examining aggregated data
to identify insights,
detect potential security threats,
and inform the decision-making process.
Let's learn more about aggregation and data analytics.
First, we have aggregation.
Aggregation is the process of collecting and combining data
from multiple sources into one comprehensive data set.
The purpose of this is to have a single, unified data set
that can be analyzed to find patterns,
trends, or anomalies.
Without aggregation, we would have fragmented bits
of data spread all across different systems,
making it difficult to get a full picture
of what's going on within the enterprise network.
To understand the benefits of aggregation,
let's think about an e-commerce business
that wants to better understand its customers' behavior.
The company collects data from various network sources,
customer purchases from its website,
user activity from its mobile app,
marketing data from social media campaigns,
and feedback from customer-service interactions.
Each of these data sources contains useful information,
but if kept separate,
they only provide isolated and partial snapshots
of the customer and their interactions.
But through data aggregation, all these data points,
purchase history, app usage,
social media engagement, and feedback
are pulled together into a single data set.
This aggregated data set allows businesses
to see a bigger picture.
Now, they can analyze patterns in the unified data set
to find out that this customer and others,
who engage more with their app,
also tend to purchase higher-value products
after seeing specific types of social media ads.
This can drive the marketing plan and increase sales.
From a technical standpoint,
aggregation also requires a system that can
collect data in real-time
or at regular intervals, normalize the formats,
because data from each source might be
structured a little bit differently,
and then combine it all into a central database
or repository for analysis and data analytics.
Second, we have data analytics.
Now that data aggregation has occurred,
we can move on to data analytics.
Data analytics is the process
of analyzing and processing aggregated data
to detect security incidents and potential threats.
This involves using specialized tools that sift
through large amounts of data
to identify patterns or anomalies.
The large amounts of data come
from various network sources,
like endpoint protection software,
identity and access management tools,
network traffic, threat feeds, cloud platforms,
and other applications used throughout an organization.
The primary goal of data analytics
is to identify security issues
and monitor for any suspicious activities.
A common system used for this purpose
is a Security Information and Event Management,
or SIEM, platform.
A SIEM takes aggregated data, normalizes it,
and prepares it for further analysis.
Normalization is required because data from different
systems may be in various formats,
and while data aggregation may have
initiated the process of standardization,
a SIEM ensures the data is fully normalized.
After normalization, the data is in a consistent
format that can be easily searched,
indexed, and analyzed in a centralized repository.
Once the data is collected,
it moves through what is known as a processing pipeline.
This pipeline ensures the data flows correctly
and is processed before reaching the central repository.
In the past, batch processing pipelines were common,
where data was collected over a fixed period,
such as 8 hours, and then processed in bulk.
While effective for certain tasks like
handling logs and reports,
this method introduced delays,
which were problematic for security analysts
that required real-time information.
So, to address this issue,
modern systems now use stream processing pipelines.
With stream processing,
data is collected and analyzed in near real-time,
significantly reducing the delay between
collection and analysis.
This allows security teams to detect and
respond to threats more quickly.
Once the data is in the central repository,
it must be indexed.
Indexing is important because it speeds up searches,
making the data more accessible and ensuring analysts can
find the information they need quickly and efficiently.
Another important step is log curation.
Log curation helps prioritize and filter the
most relevant data for collection.
When gathering data across a network,
it's important not to collect everything indiscriminately.
While it might seem beneficial to
collect all available data,
doing so can overwhelm the system and slow down analysis.
Collecting too much data wastes resources
in processing and indexing,
making it harder to focus on the important information.
So, by curating the log data,
we ensure that only the most important data
is collected and stored.
After the data is in storage, it's essential to protect it.
This is where database activity monitoring tools come in.
Database activity monitoring tools monitor
and secure databases
by detecting unauthorized access and suspicious activities.
They also assist in compliance audits by providing
continuous real-time monitoring of the database.
Database activity monitoring tools gather information
from sources such as database logs,
sensors attached to the database, or transaction logs,
offering visibility into database activity and
ensuring data integrity and security.
There are three main methods used
for database activity monitoring.
The first is the interception-based model,
which monitors communication between the
client and the server.
The second is the memory-based model,
where a sensor attached to the database captures
SQL statements as they are executed.
The third is the log-based model,
which monitors transactions based on database server logs.
Each method has its strengths and weaknesses.
For instance, interception and log-based methods
may miss some data,
while memory-based models capture more data
but can slow down the system.
Database activity monitoring tools can also
help with data classification,
data loss prevention, and ensure data integrity.
By using database activity monitoring tools,
organizations can protect their databases
from unauthorized access or malicious behavior.
Database activity monitoring tools combine
methods like network sniffing,
memory scraping, reading system tables,
and analyzing database logs to get a full
picture of who is accessing the system.
This makes database activity monitoring a key
part of an organization's security strategy,
ensuring that potential threats are
detected and addressed quickly.
So, remember, data management is all about organizing,
protecting, and analyzing data to maintain
its integrity and security.
One key process within data management is data aggregation,
where data from different sources is combined into
a single data set for analysis.
After aggregation, data analytics come into play,
allowing organizations to analyze
the collected data for patterns,
trends, or potential security threats.
Tools like Security Information and Event Management,
or SIEM platforms,
help normalize and process this data
in real-time or near real-time
to enable faster detection and response to issues.
Finally, database activity monitoring tools protect
and monitor the data in storage,
ensuring it remains secure and compliant.