Artwork

Content provided by Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ro.player.fm/legal.
Player FM - Aplicație Podcast
Treceți offline cu aplicația Player FM !

What can Apache Kafka Developers learn from Online Gaming?

55:32
 
Distribuie
 

Manage episode 424666711 series 2510642
Content provided by Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ro.player.fm/legal.

What can online gaming teach us about making large-scale event management more collaborative in real-time? Ben Gamble (Developer Relations Manager, Aiven) has come to the world of real-time event streaming from an usual source: the video games industry. And if you stop to think about it, modern online games are complex, distributed real-time data systems with decades of innovative techniques to teach us.
In this episode, Ben talks with Kris about integrating gaming concepts with Apache Kafka®. Using Kafka’s state management stream processing, Ben has built systems that can handle real-time event processing at a massive scale, including interesting approaches to conflict resolution and collaboration.
Building latency into a system is one way to mask data processing time. Ben says that you can efficiently hide latency issues and prioritize performance improvements by setting an initial target and then optimizing from there. If you measure before optimizing, you can add an extra layer to manage user expectations better. Tricks like adding a visual progress bar give the appearance of progress but actually hide latency and improve the overall user experience.
To effectively handle challenging activities, like resolving conflicts and atomic edits, Ben suggests “slicing” (or nano batching) to break down tasks into small, related chunks. Slicing allows each task to be evaluated separately, thus producing timely outcomes that resolve potential background conflicts without the user knowing.
Ben also explains how he uses pooling to make collaboration seamless. Pooling is a process that links open requests with potential matches. Similar to booking seats on an airplane, seats are assigned when requests are made. As these types of connections are handled through a Kafka event stream, the initial open requests are eventually fulfilled when seats become available.
According to Ben, real-world tools that facilitate collaboration (such as Google Docs and Slack) work similarly. Just like multi-player gaming systems, multiple users can comment or chat in real-time and users perceive instant responses because of the techniques ported over from the gaming world.
As Ben sees it, the proliferation of these types of concepts across disciplines will also benefit a more significant number of collaborative systems. Despite being long established for gamers, these patterns can be implemented in more business applications to improve the user experience significantly.
EPISODE LINKS

  continue reading

Capitole

1. Intro (00:00:00)

2. Building real-time inspection machines (00:05:29)

3. Going Multiplayer with Kafka (00:07:14)

4. What does a real-time experience look like in gaming? (00:14:35)

5. Broken message offsets vs. dead letter queues (00:17:05)

6. The distributed transaction problem (00:19:15)

7. Solving for conflicting events with nano-batching (00:25:02)

8. Pooling in real-time systems (00:31:56)

9. Optimizing for latency (00:38:29)

10. Using column modeling to improve performance (00:44:08)

11. Principles for stream-processing (00:50:49)

12. It's a wrap! (00:52:46)

265 episoade

Artwork
iconDistribuie
 
Manage episode 424666711 series 2510642
Content provided by Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ro.player.fm/legal.

What can online gaming teach us about making large-scale event management more collaborative in real-time? Ben Gamble (Developer Relations Manager, Aiven) has come to the world of real-time event streaming from an usual source: the video games industry. And if you stop to think about it, modern online games are complex, distributed real-time data systems with decades of innovative techniques to teach us.
In this episode, Ben talks with Kris about integrating gaming concepts with Apache Kafka®. Using Kafka’s state management stream processing, Ben has built systems that can handle real-time event processing at a massive scale, including interesting approaches to conflict resolution and collaboration.
Building latency into a system is one way to mask data processing time. Ben says that you can efficiently hide latency issues and prioritize performance improvements by setting an initial target and then optimizing from there. If you measure before optimizing, you can add an extra layer to manage user expectations better. Tricks like adding a visual progress bar give the appearance of progress but actually hide latency and improve the overall user experience.
To effectively handle challenging activities, like resolving conflicts and atomic edits, Ben suggests “slicing” (or nano batching) to break down tasks into small, related chunks. Slicing allows each task to be evaluated separately, thus producing timely outcomes that resolve potential background conflicts without the user knowing.
Ben also explains how he uses pooling to make collaboration seamless. Pooling is a process that links open requests with potential matches. Similar to booking seats on an airplane, seats are assigned when requests are made. As these types of connections are handled through a Kafka event stream, the initial open requests are eventually fulfilled when seats become available.
According to Ben, real-world tools that facilitate collaboration (such as Google Docs and Slack) work similarly. Just like multi-player gaming systems, multiple users can comment or chat in real-time and users perceive instant responses because of the techniques ported over from the gaming world.
As Ben sees it, the proliferation of these types of concepts across disciplines will also benefit a more significant number of collaborative systems. Despite being long established for gamers, these patterns can be implemented in more business applications to improve the user experience significantly.
EPISODE LINKS

  continue reading

Capitole

1. Intro (00:00:00)

2. Building real-time inspection machines (00:05:29)

3. Going Multiplayer with Kafka (00:07:14)

4. What does a real-time experience look like in gaming? (00:14:35)

5. Broken message offsets vs. dead letter queues (00:17:05)

6. The distributed transaction problem (00:19:15)

7. Solving for conflicting events with nano-batching (00:25:02)

8. Pooling in real-time systems (00:31:56)

9. Optimizing for latency (00:38:29)

10. Using column modeling to improve performance (00:44:08)

11. Principles for stream-processing (00:50:49)

12. It's a wrap! (00:52:46)

265 episoade

Tất cả các tập

×
 
Loading …

Bun venit la Player FM!

Player FM scanează web-ul pentru podcast-uri de înaltă calitate pentru a vă putea bucura acum. Este cea mai bună aplicație pentru podcast și funcționează pe Android, iPhone și pe web. Înscrieți-vă pentru a sincroniza abonamentele pe toate dispozitivele.

 

Ghid rapid de referință