Implementing GenAI for Effective Log Analysis and Query Automation in Amazon Security Lake (Amazon Bedrock, RAG)
Date of creation : 2024-06-14
The Only MSP to Obtain Amazon Security Lake Service Partner Qualification!
Have you seen the content related to Amazon Security Lake posted on the MegaZone Cloud homepage tech blog?
The Amazon Security Lake service is AWS's Native SIEM (Security Information and Event Management) Service that allows for the simple collection of various AWS logs and conversion into a uniform format (OCSF). In October 2023, MegaZone Cloud became the first in Korea to obtain the Security Lake Service Partner qualification, introducing it to various clients based on our technical expertise.
Amazon Security Lake is a service well-suited for collecting AWS's Native Logs, but the challenge that actual clients face is how to analyze these logs. We aim to share the process of implementing and testing an architecture that can assist in analyzing these logs using the recently trending GenAI services, which will be demonstrated at the MegaZone Cloud booth at AWS Summit Seoul 2024.
AWS Services and Technologies for High-Performance GenAI Configuration
Earlier this year, I noticed a post on the AWS Tech Blog about utilizing Amazon SageMaker and Amazon Bedrock to leverage data from Amazon Security Lake. The AWS Tech Blog illustrates how to analyze OCSF formatted logs collected through Security Lake using LLM models, enabling more efficient threat detection and quicker response tracking. It was intriguing, and I thought that if the desired performance could be achieved through actual implementation, it could be a useful tool within companies.
Therefore, our MegaZone Cloud CTC (Cloud Technology Center) referred to this blog to implement directly, conduct performance testing, and enhance the setup. Before looking into the demo configuration, let's explore the related services and technologies.
Amazon Bedrock and Amazon SageMaker
Both Amazon Bedrock and Amazon SageMaker are platforms for machine learning (ML) and artificial intelligence services provided by AWS, but they differ somewhat in purpose and functionality.
Amazon Bedrock is a fully managed service that integrates high-performance foundation models (Foundation Models, FM) from leading AI companies. Users can easily test and evaluate various large-scale AI models with Amazon Bedrock, select models according to their needs, and personalize them using techniques like Fine Tuning and RAG (Retrieval Augmented Generation). It particularly offers a variety of LLM models and emphasizes the secure management and protection of user data.
Amazon SageMaker supports the entire ML workflow and is designed to help users develop ML models more quickly and easily. In other words, it provides an integrated platform that allows users to visually manage all ML workloads from data preparation to model training, tuning, and deployment.
Both services are core to AWS's ML and AI service ecosystem, supporting users in effectively implementing AI and ML functionalities without worrying about complex infrastructure management. However, Amazon Bedrock primarily focuses on the application of LLM-based large generative AI models, while Amazon SageMaker supports a broader range of ML workflows, enabling developers to control and optimize machine learning projects of various workloads more finely. In this demo, the combination of Amazon Bedrock and Amazon SageMaker enables in-depth analysis of security data and rapid response times.
Foundation Models (Anthropic Claude – Haiku, Sonnet, Opus)
For the demo, we tested three models of Antropic Claude 3.0: Haiku, Sonnet, and Opus. We selected these three models that support Korean from the various LLM models supported by Amazon Bedrock.
The characteristics of each model are as follows:
Haiku is the smallest among the Claude models. It is optimized for efficiency and fast response times. It is suitable for basic language understanding, simple content generation, and quick response tasks and can be used with relatively limited resources. It is cost-competitive compared to other Claude models. With low computational demands and fast processing speeds, it is designed for applications or businesses with budget constraints. Small businesses, startups, or individual developers can utilize it for customer support chatbots, simple content generation, and light language processing tasks.
Sonnet is a medium-sized model that provides a balance between performance and resource consumption. It is trained on more extensive data than Haiku, offering enhanced language understanding and generation capabilities. It can handle more complex content generation, better context understanding, and more detailed language processing. Although it is priced higher than Haiku, it reflects the enhanced capabilities and is optimized for cost-performance. It is suitable for businesses that need a more advanced language model but cannot invest heavily. Therefore, it is appropriate for SMEs, tech companies, and services requiring more sophisticated language processing, such as advanced customer support, content generation, marketing automation, and interactive AI applications. It is versatile, offers good cost-performance ratios, and is suitable for various applications.
Opus is the largest and most powerful model in the Claude series, providing state-of-the-art language understanding and generation capabilities using comprehensive and diverse data. It is particularly used for advanced content generation, deep context understanding, handling complex and detailed tasks, supporting large-scale applications, detailed content generation, extensive marketing and sales automation, and research and development. However, due to extensive training and advanced features, it requires the highest cost among the Claude models. Therefore, it is suitable for large enterprises and organizations needing cutting-edge AI solutions.
In summary, Haiku is ideal for businesses needing a low-cost/high-efficiency AI solution for basic tasks. Sonnet balances performance and cost, making it suitable for businesses that require above-basic functionality within a reasonable budget. Opus targets organizations ready to invest in premium performance for high-level AI capabilities and advanced applications. Each model in the Claude series is designed to be selected based on specific requirements and budget constraints, catering to various business needs.
Text2SQL
Text2SQL (Text-To-SQL) is a technology that automatically converts questions entered in natural language into SQL (Structured Query Language) queries. This technology combines with SQL, which is widely used to extract information from databases, allowing users to ask questions in everyday language, and the system converts this into appropriate SQL queries to retrieve the necessary data.
It is particularly useful for users with complex database management or lacking technical knowledge, as it allows them to access required information by asking questions in their usual language without needing to write complex query syntax. This approach can also be beneficial in the security field, enabling non-technical security professionals to easily analyze security log data and extract desired information. This lowers the entry barrier for database usage and supports faster and more efficient data-driven decision-making, making it applicable in various fields such as business analysis, customer service management, and security monitoring.
Architecture
Next, let’s look at the actual configuration and testing conducted by MegaZone Cloud using these AWS services and technologies.
Amazon Security Lake with GenAI Architecture
During the demo, we configured only the necessary services based on the architecture outlined in the AWS blog. First, we separated the account with Amazon Security Lake activated from the account configured for GenAI, activating Amazon SageMaker and Amazon Bedrock.
Specifically, in the AWS GenAI for Security Lake Account, we configured GenAI models using Amazon SageMaker and Amazon Bedrock. In this account, we set up a SageMaker VPC and a SageMaker domain, and configured an Amazon SageMaker Instance. We also enabled data analysis through Amazon Athena, and finally configured the analysis results to be stored in Amazon S3.
Unfortunately, since Amazon Bedrock does not support the Seoul region, we built the entire services in Northern Virginia. Additionally, because Amazon Security Lake collects a large number of logs from various sources such as cloud, servers, and databases, testing all logs would require substantial resources and constraints, so we configured it to test only some cloud login VPC Flow Logs and CloudTrail Logs to conduct testing in a realistic environment.
Amazon SageMaker Architecture
SageMaker Configuration Flow
After setting up the AWS environment, we configured a Notebook Instance in Amazon SageMaker, and the basic logic of the Amazon SageMaker structure was organized in the following order: creating a template based on the table and data information to be used, calling the LLM based on the created template and outputting results, validating the output query, and explaining the analysis results.
The main modules of SageMaker are as follows, and we will take a closer look at them.
※ The code below reflects the final configuration, and there may be some differences from the code during the testing process.
● table_info_call_athena: Extracting information about the used tables and data
● create_query_prompt: Defining the basic template configuration
● task_define_template: Detailed task definition template (combined with the basic configuration of create_query_template)
● query_validation_with_record: Extracted
.## 1 Create a Template Based on the Table and Data Information
First, it starts by receiving the user's natural language input. Then, it extracts the table and data information to be used based on the user's natural language input and creates a prompt template. The prompt template contains not only the table and data information but also the business logic used in practice, which can enhance the accuracy of the Amazon Bedrock model.
# Extract table and data information and declare the Prompt template
table_name = 'cloudtrail_table'
table_info = slm.table_info_call_athena(table_name=table_name, info_type='structure')
prompt = slm.create_query_prompt(table_info=table_info, query_k=query_k, user_input=user_input, retrieve_few_shots=global)# Extract DB table information
def table_info_call_athena(info_type=None):
table_name = 'cloudtrail_table'
if info_type == 'structure':
return spt.cloudtrail_table_info# Generate query and define prompt template based on table information and user's query
def create_query_prompt(table_info=None, sample_log=None, query_k=None, user_input=None):
template = spt.task_define_template(table_info=table_info, sample_log=sample_log, query_k=query_k)
prompt = f"""
System: {template}
Human: {user_input}
Assistant:
"""
return prompt
...
def task_define_template(table_info=None, sample_log=None, query_k=None):
task_define_template = f"""
You are the chief security officer in charge of in-house security control.
You are in charge of analyzing AWS SecurityLake data.
Your role is to convert natural language requests into valid SQL queries.
<table_info>
{table_info}
</table_info>
<table_info></table_info> contains database schema and 3 sample rows from the table.
Generate a query using only the columns that exist in the database schema provided for the table information.
The DB table name is “{table_name}”.
Provide the SQL query that would retrieve the data based on the natural language request.
Always limit a query to a maximum of {query_k} results.
If you think the question is not related to the database, Answer "The requested task cannot be performed".
"""2. Call LLM Based on the Created Template and Output Results
Invoke the Claude model based on the prompt template and output the results. You can choose the Anthropic Claude model according to user preferences or business environment, which reflects the openness of Amazon Bedrock.
# Call Claude model based on the prompt template and output results
return_text, boto3_bedrock = slm.query_invoke_model(model_name=model_name, prompt=st.session_state.prompt_session, max_token=None)
# Extract only the query from the output results
model_query = slm.query_extract(return_text=return_text)3. Validate the Output Query
The generated query is executed against the actual database. If the query execution is successful, the results are validated; if it fails, the query is regenerated. The ‘query_validation_with_record’ function utilizes the execution and validation loop of the query to enhance its accuracy. While Amazon Bedrock supports a high-quality LLM model, it may occasionally struggle to provide perfect answers to various demands. Therefore, validation logic has been employed to improve the overall system's completeness.
# Query validation
query_result, st.session_state.prompt_session, valid_log, query_id = slm.query_validation_with_record(
valid_roop=valid_roop, model_query=model_query, retrieve_few_shots=globals()['retrieve_few_shots'],
table_info=table_info, query_k=query_k, session_log=st.session_state.prompt_session, model_name=model_name,
user_input=user_input, table_name=table_name, boto3_bedrock=boto3_bedrock, max_tokens=max_tokens, query_mode=query_mode)def query_validation_with_record(valid_roop=None, model_query=None, db_conn=None, model_name=None,
table_info=None, sample_log=None, query_k=None, session_log=None,
user_input=None, table_name=None, boto3_bedrock=None, max_tokens=None, query_mode=None,
retrieve_few_shots=None):
query_result = None
valid_model_query = ''
valid_log = []
for roop in range(valid_roop):
if query_result is None:
try:
query_result = query_execute_with_athena(query=model_query)
if query_result.shape[0] < 1:
print('NO DATA')
raise Exception('No data')
else:
print('EXECUTE SUCCESS')
break
except Exception as e:
query_result = None
valid_model_query += f'
Assistant : {model_query}'
valid_model_query += '
Human: An error occurs in the query you created. Please re-create the query with appropriate syntax.'
prompt = query_valid_prompt(table_info=table_info, sample_log=sample_log, query_k=query_k, user_input=user_input,
valid_model_query=valid_model_query, table_name=table_name, query_mode=query_mode)
return_text = query_invoke_model(prompt=prompt, boto3_bedrock=boto3_bedrock, max_tokens=max_tokens)
model_query = query_extract(return_text=return_text)
print(return_text)
else:
query_result = None
return query_result4. Output Analysis Results
To understand the logical basis on which the LLM generated the query, the analysis results are outputted, which is structured to enhance user satisfaction and query comprehension.
analysis_prompt = st.session_state.prompt_session
PaS_analysis = spt.result_analysis_template()
analysis_prompt += f'
Human: {PaS_analysis}
Assistant: '
message, msg = analysis_model(prompt=analysis_prompt, boto3_bedrock=boto3_bedrock, max_tokens=4096)def result_analysis_template(table_info=None, query=None, result=None):
result_analysis_template = """
Based on the last generated query and the results of the last generated query, please explain the logical basis for generating the last query.
Let's plan and solve the problem step by step.
"""
return result_analysis_template# Call result analysis model
def analysis_model(prompt=None, boto3_bedrock=None, max_tokens=None, log_f=None):
prompt_config = {
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": max_tokens,
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": prompt},
],
}
],
}
body = json.dumps(prompt_config)
modelId = "anthropic.claude-3-sonnet-20240229-v1:0"
accept = "application/json"
contentType = "application/json"
response = boto3_bedrock.invoke_model_with_response_stream(body=body, modelId=modelId, accept=accept, contentType=contentType)
msg = response.get("body")
return msg5. Return Results
If the query is validated and executed successfully, the results are returned to the user.
Answer Return Based on User Question
After completing the configuration of Amazon SageMaker, tests were conducted on the accuracy of actual questions and answers. During the testing, not only general questions (e.g., "Please tell me the Source IP with the most API Calls in the last week." or "Provide the log history using the root account.") were asked, but also questions related to actual Port Scan and Credential theft scenarios were generated to assess the capability of detecting security threats. Various queries were made regarding security logs, prioritizing Latency and query accuracy during the testing process.
Test Results
Initially, it was configured using the zero-shot learning technique in Amazon SageMaker. Queries related to general queries and threat detection scenarios concerning Port Scan and Credential theft were conducted, resulting in the following values.
User Question (related to VPC Flow Logs) resulted in a Failed Result
User Question (related to VPC Flow Logs) resulted in a Failed Result
As a result of the test, it was found that the LLM model did not generate accurate Text2SQL when conducted in a zero-shot manner as shown above.
This result suggests that Amazon Security Lake utilizes OCSF (Open Cybersecurity Schema Framework) logs containing JSON data as its dataset, and when the dataset that serves as the basis for query execution is in JSON format, the complexity of the generated query itself is high, leading to limitations in query generation. Additionally, it was determined that the lack of information regarding the OCSF schema contributed to the inaccuracies in query generation.
Phase 1 Enhancement – Few-Shot Learning
To address these issues, the JSON portion from the logs containing JSON data was separated to create a separate dataset, and tests were conducted in a new environment based on this. A Few-Shot Learning technique was applied to provide highly accurate responses based on internal data sources, and a Query Assistant was created to assist SQL queries along with the queries, allowing for retesting.
User Question (related to VPC Flow Logs) resulted in a Succeed Result
User Question (related to CloudTrail Logs) resulted in a Succeed Result
As shown in the images above, it was confirmed that appropriate answer results were produced after the improvements. To verify more accurate results from the improved configuration, tests were conducted with various query lists along with multiple queries against the base models (Haiku, Sonnet, Opus). The results showed that among the base models, Sonnet provided the most accurate responses, yielding approximately 40% accuracy. This represents a significant improvement compared to the initial configuration, but there are still areas that require enhancement.
Phase 2 Enhancement – RAG (Retrieval Augmented Generation)
After conducting tests and the first phase of enhancements as described above, the second phase of enhancement was carried out. During the second enhancement phase, it was confirmed that the test results improved significantly, and the accuracy increased greatly. Through these improvements, it was decided to demonstrate a demo at the Megazone Cloud booth during AWS Summit Seoul 2024.
The second phase of enhancement consisted of two main components: adding Few-Shot Learning data and configuring RAG (Retrieval Augmented Generation), along with modifying some prompts. First, data was added for Few-Shot Learning, and during testing, it was found that the S3 bucket name of the data used for Few-Shot Learning was utilized in the queries, leading to actual hallucinations occurring. (Hallucination refers to the phenomenon where the LLM provides answers that do not exist or are logically inconsistent, indicating low performance of the AI model.) To resolve this issue and generate more accurate and reliable responses, the implementation was carried out using RAG (Retrieval Augmented Generation) for enhancement. Let’s take a closer look at the details of the RAG implementation and the demo conducted at AWS Summit Seoul 2024.
It is a technology that combines the processes of generating and retrieving information in the field of artificial intelligence (AI). This method allows the language model to generate more accurate and useful responses by retrieving relevant information from external data sources and then using that information to generate text. The operation of RAG consists mainly of processing the user's question → information retrieval → answer generation. To explain in detail, when a user inputs a question or request, this input is converted into an embedding vector to search for similar documents. The LLM then generates the final response based on the content of the retrieved documents. In this stage, the retrieved information is combined with the knowledge of the language model (training data) to generate more accurate responses.
RAG is a technology that combines the strengths of information retrieval and generation to produce more accurate and useful text. This can significantly enhance the performance of GenAI models and can be utilized in various applications. RAG primarily consists of two components: indexing (embedding) and retrieval and generation. Indexing is a pipeline for collecting and indexing data from sources, while retrieval and generation involve searching for relevant data from the index based on user queries and then passing this data to the model to generate answers.
The RAG indexing process is illustrated in the following figure. The advantage of RAG is that it significantly enhances the response accuracy of the language model by utilizing external data. Additionally, it can reflect the latest information in real-time responses, thereby providing richer information by leveraging external knowledge beyond the limitations of the model itself.
[RAG indexing process]
After loading various data, the process of Data Chunking, which involves splitting large texts into smaller segments, is carried out. Following that, the chunked data undergoes an encoding process (Embedding) and is stored in a Vector Database to enable the segmented data to be searched.
The embedding of the user's input question is used to search for similar data in the Vector Database and is provided to the LLM. This allows the LLM to provide more efficient answers. You can understand this better by looking at the RAG diagram below.
RAG diagram
During this demo, we pre-constructed frequently used questions and their corresponding answer queries from the logs stored in Amazon Security Lake to create a Vector Database. We implemented RAG as shown below to reference the semantically most similar question queries to the user's input question and output the results accordingly.
#RAG 구현
chunking_dataset = pickle.load(open(os_path+'/pickle_dir/chunking_dataset.pkl', 'rb'))
corpus_embeddings = pickle.load(open(os_path+'/pickle_dir/corpus_embeddings.pkl', 'rb'))
query_list = pickle.load(open(os_path+'/pickle_dir/query_list.pkl', 'rb'))
# HuggingFace 한국어 임베딩 모델 로드
embeddings = SentenceTransformer("jhgan/ko-sroberta-multitask")
# FAISS 엔진을 활용한 Similarity Search(IndexFlatIP : Cosine Similarity)
index = faiss.IndexIDMap(faiss.IndexFlatIP(768))
# question embedding과 인덱스를 연결
index.add_with_ids(corpus_embeddings, np.array(range(0, corpus_embeddings.shape[0])))
# 한국어 임베딩 모델을 활용해 user_input에 대한 encoding 작업
query_vector = embeddings.encode([user_input])
# 유저의 질문과 유사한 쿼리 인덱스 리스트 출력
top_k = index.search(query_vector, few_shot_k)
docs = [chunking_dataset[_id] for _id in top_k[1].tolist()[0]]
# 유사도 검색 결과 가장 유사한 질문에 대한 정답쿼리를 prompt template에 구성
for doc, idx in zip(docs, top_k[1][0].tolist()):
temp_str = f"Human: {doc}\nAssistant: {query_list[idx]}\n\n"
print(chunking_dataset) # 문장 단위로 분할
['MFA를 비활성화한 관리 콘설 로그인이 있었나요?',
"'root' 계정 사용한 로그가 있나요?",
'어제 몇번의 로그인 시도가 있었습니까?',
'최근 일주일간 가장 많이 로그인한 IAM User를 알려주세요',
'AWS 콘솔에 로그인한 IP 중 54.216.116.106 이 아닌 IP를 알려주세요.',
'최근 일주일 동안 가장 많이 발생된 API Call이 무엇입니까?',
'최근 한달동안 error가 가장 많이 발생된 행위(API Call)가 무엇입니까?',
'최근 1일간 AWS KMS 이벤트 기록을 알려주세요',
'이틀 전 API 호출이 제일 많았던 IP를 알려주세요',
'어제 API 호출이 가장 많이 이루어진 시간대는 언제인가요?',
'KMS 등으로 가장 많이 복호화 요청이 있었던 서비스/사용자는?',
'EC2 인스턴스, RDS 인스턴스 의 기동, 중지, 삭제 등 확인이 가능한가요?',
'특정 보안 그룹의 규칙이 추가되거나 제거된 이벤트 확인이 가능한가요??',
'AWS 리소스에 대한 태그 변경 분도 확인이 가능한가요?',
'최근 일주일 동안 api call 을 발생시킨 횟수가 많은 순으로 IP를 리스팅 해주고 리스트 상위 5개 IP에서 발생한 API call 들을 IP별로 보여줘',
'Security Group 변경된 로그가 있나요?',
'AWS Management Console 인증 실패에 대한 로그가 있나요?',
"arn:aws:iam::551508107696:user/DemoUser' 가 로그인할 때 MFA를 사용했습니까?",
'Amazon RDS 인스턴스에 대한 스냅샷 복원 이벤트를 추적할 수 있나요?',
'지난 24시간 동안 API 호출이 실패한 경우를 리스팅 해주고 errorCode와 errorMessage 항목들을 보여줘',
'CMK가 키로테이션 된 일자를 알고 싶습니다.',
'새벽에 접속한 사람 있어??']
print(corpus_embeddings) # 임베딩
tensor([[-5.3411e-03, -7.4023e-02, 3.7926e-01, ..., -7.2996e-04,
-1.6781e-01, -1.9504e-01],
[ 3.5060e-02, -2.2566e-01, 4.5334e-01, ..., -2.9150e-01,
-1.2969e-01, 1.0309e-01],
[-4.2866e-01, -6.3441e-01, 5.5720e-01, ..., -9.3364e-02,
6.0054e-02, -6.7089e-01],
...,
[-9.8832e-01, 6.1827e-01, 3.7764e-02, ..., -2.4416e-01,
5.9593e-02, 1.0958e-01],
[-2.9902e-01, -1.5139e-01, 1.4176e-01, ..., 1.6358e-02,
8.6800e-01, -7.1446e-01],
[-2.9451e-01, -4.9310e-01, 9.2341e-01, ..., 6.7432e-02,
-3.1949e-01, -1.7224e-01]])
print(q_list)
['SELECT * \nFROM \n ""amazon_security_lake_glue_db_ap_northeast_2"".""security_lake_genai""\nWHERE\n eventday >=
'SELECT\n *\nFROM \n ""amazon_security_lake_glue_db_ap_northeast_2"".""security_lake_genai""\nWHERE\n eventday
'SELECT COUNT(activity_name) activity_name_count\nFROM ""amazon_security_lake_glue_db_ap_northeast_2"".""security_lake_
'SELECT user_uuid, COUNT(actor_user_uuid) user_uuid_count, COUNT(activity_name) activity_name_count\nFROM ""amazon_secu
'SELECT src_endpoint_ip, COUNT(src_endpoint_ip) src_endpoint_ip_count\nFROM ""amazon_security_lake_glue_db_ap_northeast
'SELECT api_operation, api_service_name, eventday, COUNT(api_operation) as api_count\nFROM ""amazon_security_lake_glue_
'SELECT api_operation, api_service_name, eventday, COUNT(api_operation) as api_count\nFROM ""amazon_security_lake_glue_
'SELECT api_operation, COUNT(activity_name) activity_name_count\nFROM ""amazon_security_lake_glue_db_ap_northeast_2""."
'SELECT src_endpoint_ip, COUNT(src_endpoint_ip) src_endpoint_ip_count\nFROM ""amazon_security_lake_glue_db_ap_northeast
'SELECT \n CAST(CONCAT(CAST(DATE(""time"") AS VARCHAR), \' \', CAST(HOUR(""time"") AS VARCHAR), \':00\') AS TIMESTAM
'WITH \n role_tbl AS(\n -- assumerole_table\n SELECT actor_invoked_by as actor, COUNT(actor_invoked_by
'SELECT\n api_operation\n , count(api_operation) AS operation_count\nFROM \n ""amazon_security_lake_glue_db_ap
'SELECT\n api_operation\n , count(api_operation) AS operation_count\nFROM \n ""amazon_security_lake_glue_db_ap
'SELECT\n api_operation\n , count(api_operation) AS operation_count\nFROM \n ""amazon_security_lake_glue_db_ap
'WITH ip_list_table AS (\n SELECT\n src_endpoint_ip\n , COUNT(src_endpoint_ip) AS ip_count\n FROM \
'SELECT\n *\nFROM \n ""amazon_security_lake_glue_db_ap_northeast_2"".""security_lake_genai""\nWHERE \n eventda
'SELECT\n *\nFROM \n ""amazon_security_lake_glue_db_ap_northeast_2"".""security_lake_genai""\nWHERE\n eventday
"SELECT * \nFROM security_lake_genai\nWHERE actor_user_uuid ='arn:aws:iam::551508107696:user/DemoUser' AND mfa= true\nO
'SELECT\n *\nFROM \n ""amazon_security_lake_glue_db_ap_northeast_2"".""security_lake_genai""\nWHERE \n eventda
'SELECT\n api_operation\n , api_response_error\n , api_response_message\n , api_service_name\nFROM \n ""
'SELECT\n api_operation\n , eventday\n , resources_uid\nFROM \n ""amazon_security_lake_glue_db_ap_northeast
]
By combining Security Lake and GenAI in this way, we explored the entire process of analyzing log data and generating SQL queries, ultimately adding a RAG model to enhance the stability and accuracy of the LLM.
Final Results
We demonstrated this configured demo environment on the web at the AWS Summit Seoul 2024 Megazone Cloud booth. The demonstration details can be found below.
Additional Suggestions for Customer Environment and Requirements
So far, we have looked at how the logs collected by Amazon Security Lake are configured to make access and analysis of security logs easier through Amazon Bedrock and Amazon Sagemaker. We confirmed that security log analysis has become much smoother through the actual demo environment setup and testing; however, this demo test was conducted in a limited environment, which had some disappointing aspects. Therefore, additional enhancement work will be necessary in the future. Below are the considerations that should be taken into account for future enhancements based on the demo test conducted.
First, this test was performed on a single table within the database, executing simple queries and aggregation query generation. However, in a real operational environment, it is often necessary to derive results using multiple tables. In particular, since various security logs are integrated and stored in Security Lake, it is essential to define rule sets for analyzing the relationships between each log and to conduct additional learning and tuning. In this process, the complexity of the queries increases, and performance validation for multiple join queries is required, so it is necessary to consider query configurations using various tables and the corresponding performance optimization measures.
Additionally, when generating queries based on natural language, it is important to create specific guidelines for natural language when constructing prompt templates. For example, if a query related to 'traffic' is made, the actual user's intent may relate to 'the number of network logs,' but the LLM might encounter issues such as looking for data related to 'log size.' To prevent such problems, clear guidelines for words that have the same meaning but different expressions should be established to promote consistent query generation.
Finally, when outputting results through the LLM, there may be cases where explanations are included or not, leading to inconsistencies in the output format. To resolve this, multiple tests should be conducted to ensure consistency in the output results and to enhance the reliability of the analysis results.
These considerations are essential elements that security professionals and AI/ML experts must review when applying this demo to real operational environments. By thoroughly reviewing and reflecting on these elements, a more stable and efficient security data analysis environment can be established.
Amazon Security Lake with GenAI in collaboration with Megazone Cloud
By integrating and storing data in a common format through OCSF, Amazon Security Lake and GenAI allow for security log searches and relationship analysis of each log to be conducted from a single prompt. This will provide security personnel and data scientists with opportunities to save considerable effort and resources.
I hope this blog provides useful information to engineers and data scientists interested in security data analysis, helping them extract valuable insights even from complex security datasets. Megazone Cloud will continue to research and innovate to enable users to build safe and efficient cloud environments.
Megazone Cloud plans to continuously research and develop the demo environment to enhance the ability to effectively detect and respond to security threats. We will also strive to provide detailed customized solutions tailored to the specific environments and requirements of each enterprise. As an Amazon Security Lake Service Partner, we are committed to providing the highest level of service and technical expertise, and customers interested in Amazon Security Lake are welcome to contact us at any time.
📧Contact: megazonecloud-asl@mz.co.kr
[Reference Materials]
[https://aws.amazon.com/ko/blogs/security/generate-ai-powered-insights-for-amazon-security-lake-using-amazon-sagemaker-studio-and-amazon-bedrock/ ] AWS Blog
[https://www.megazone.com/amazon_security_lake_231012/ ] AWS Service Partner Acquisition
[https://www.megazone.com/techblog_awssecuritylake_ocsfv1-1_240514/ ] Introduction to OCSF 1.1 and 1.2
[https://python.langchain.com/v0.2/docs/introduction/ ] LangChain
Written by Megazone Cloud Cloud Technology Center