Real-time Threat Detection Using Machine Learning and Apache Kafka

Streaming Audio: Apache Kafka® & Real-Time Data

Content provided by Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ro.player.fm/legal.

2y ago 29:18

MP3•Pagina episodului

Can we use machine learning to detect security threats in real-time? As organizations increasingly rely on distributed systems, it is becoming more important to analyze the traffic that passes through those systems quickly. Confluent Hackathon ’22 finalist, Géraud Dugé de Bernonville (Data Consultant, Zenika Bordeaux), shares how his team used TensorFlow (machine learning) and Neo4j (graph database) to analyze and detect network traffic data in real-time. What started as a research and development exercise turned into ZIEM, a full-blown internal project using ksqlDB to manipulate, export, and visualize data from Apache Kafka®.
Géraud and his team noticed that large amounts of data passed through their network, and they were curious to see if they could detect threats as they happened. As a hackathon project, they built ZIEM, a network mapping and intrusion detection platform that quickly generates network diagrams. Using Kafka, the system captures network packets, processes the data in ksqlDB, and uses a Neo4j Sink Connector to send it to a Neo4j instance. Using the Neo4j browser, users can see instant network diagrams showing who's on the network, allowing them to detect anomalies quickly in real time.
The Ziem project was initially conceived as an experiment to explore the potential of using Kafka for data processing and manipulation. However, it soon became apparent that there was great potential for broader applications (banking, security, etc.). As a result, the focus shifted to developing a tool for exporting data from Kafka, which is helpful in transforming data for deeper analysis, moving it from one database to another, or creating powerful visualizations.
Géraud goes on to talk about how the success of this project has helped them better understand the potential of using Kafka for data processing. Zenika plans to continue working to build a pipeline that can handle more robust visualizations, expose more learning opportunities, and detect patterns.
EPISODE LINKS

Capitole

1. Intro (00:00:00)

2. What is the Ziem Project? (00:05:03)

3. How do you use ksqlDB? (00:12:15)

4. Creating network visualizations with Neo4j and Neovis.js (00:14:00)

5. Machine learning plans with Ziem (00:17:50)

6. Supervised vs. non-supervised machine learning (00:20:07)

7. Future use cases for Ziem (00:22:30)

8. How to get started with TensorFlow (00:24:29)

9. It's a wrap! (00:27:33)

265 episoade