Please log in to watch this conference skillscast.
This is the story of how we built a highly available data pipeline that processes terabytes of network data every day, making it available to security researchers for security assessment and threat hunting.
Building this kind of stuff in the cloud is not that complicated, but if you have to make it near real-time, fault tolerant and 24/7 available, well... that's another story. In this talk, Luciano and Domagoj will tell you how they achieved this ambitious goal and how they missed a few good nights of sleep while trying to do that!
Spoiler alert: contains AWS, lambda, elastic search, monitoring, alerting & more!
YOU MAY ALSO LIKE:
- Cracking JWT Tokens: A Tale of Magic, Node.JS And Parallel Computing (SkillsCast recorded in July 2018)
- Leonardo De Marchi's Deep Learning Fundamentals (in London on 22nd - 23rd October 2019)
- Fast Track to RESTful Microservices (in London on 11th - 13th November 2019)
- Scala eXchange London 2019 (in London on 12th - 13th December 2019)
- Practical ML 2020 (in London on 2nd - 3rd July 2020)
- Keynote by Brian Ketelsen on Going Multicloud with Serverless (in London on 17th October 2019)
- Reinforcement Learning Journal Club (in London on 17th October 2019)
- Porcupine: Flows Your Rows with Arrows (SkillsCast recorded in October 2019)
- Automating Elaborate-Transform-Load for Busy Data Scientists (SkillsCast recorded in October 2019)
Processing TeraBytes of Data Every Day and Sleeping at Night
Domagoj Katavic (Principal Software engineer at Vectra) is a technology enthusiast coming from Split, Croatia. From his early days, Dom has enjoyed playing with technology, starting with programming robots in PBASIC, to gesture recognition and tinkering with FPGA. Dom has always liked programming so his first job was building a cloud editor Codeanywhere and teaching programming at a local university. Now he is a Software Engineer at Vectra AI where he is making the hunt for cyber attacks as easy and automated as possible.
Luciano is a software engineer born in 1987, the same year that “Super Mario Bros” was released in Europe, which, by chance is his favourite game! Luciano started coding early at the age of 12, hacking away with his father's old i386 armed only with MS-DOS and the QBasic interpreter and he has been professionally a software developer for more than 10 years. Luciano is currently a Solution Architect at Vectra AI in Dublin where he's working on automating the hunt for cyberattackers and speeding-up incident response.