Ian Barber is a London based developer advocate for Google+, a regular conference speaker, and occasional blogger. Ian has been working with PHP since 2002, maintains the PHP Support Vector Machine extension, and is involved with several open source projects, particularly the messaging library ZeroMQ. His site at http://phpir.com covers machine learning, search, natural language processing and other interesting subjects.
Building A Firehose
More and more companies are realising they have a wealth of data available to them, or they have the opportunity to aggregate information from multiple sources in near real time. While big data and batch processing systems like Hadoop are part of the tool kit of many teams, more and more are realising that a continually updating system of pipes and filters – a fire hose – of their data is a powerful and flexible tool. In this talk we will look at the challenges of building this type of system, the patterns and techniques that can help simplify it, and the difficulties of managing such a system when it’s running.