This month's topic: Lean Mobile Data and Open Source: Storage, Messaging and Analysis

This talk will provide a overview of Urban Airship's core data warehouse architecture - a system designed to handle capture, intake and analysis of data for 100s of millions of mobile devices with near real time precision. The talk will touch on Urban Airship's use of HBase, Hadoop Core, ZooKeeper, Kafka as well as home-grown services. Time permitting, the talk will also cover how Urban Airship takes a lean approach to working with volumes of data including the use of ad-hoc tools such as Pig and Cascading as well as how the company leverages the data architecture for fast customer discovery and innovations.

Speaker: Erik Onnen

Erik Onnen is the Hadoop and Analytics Lead at Urban Airship, the Portland-based leader in mobile application engagement services. He has over 10 years in distributed systems experience including the design and implementation of multiple "big data" systems. Erik joined Urban Airship in October of 2010, prior to that he was a Principal Engineer at Jive Software.

PJUG meetings start with some time to eat and socialize (pizza and beverages are provided), followed by the featured speaker, then Q&A, discussion, sometimes a drawing to give away swag. :)

Though we like knowing how many people to expect, you don't have to RSVP, on Upcoming or otherwise. Go ahead and just show up!

Many people also go for a drink and further discussion following the meeting, at a location determined ad hoc (lately, Trees restaurant in the same building).

http://twitter.com/pjug http://pjug.org/ (join our mailing list, linked from the website!)

This will be the first meet up and open to general conversation about group focus/topics/etc. If you're interested, please jump in and help get the ball rolling.

If you're unable to join but are interested, please post your suggestions for discussion at the meeting on the Google Group page.

Last minute change of plans due to presenter being under the weather.

Tonight's meeting will be a casual hack & share meeting.

Here's the simple agenda: 1) Start off with group/topic related announcements (from anyone) and then let people suggest topics or things they'd like to hack on or discuss with other people in smaller groups. 2) Break up for mingling based on what you're interested in. 3) Repeat #2 above until Eric kicks us out.

Good times. See you all tonight!

Presentation this month by Dave Revell from Urban Airship.

OVERVIEW "The main topic will be how we use Hadoop and HBase at Urban Airship for collecting and analyzing data from mobile apps. Mobile app developers integrate our client library into their apps, which send analytic events to our backend. We'll talk about the problems involved in collecting and analyzing data at scale. We use HBase for storing raw events, for feeding the analysis processes, and serving the generated reports. We use Hadoop Mapreduce for ad-hoc analysis and moving large amounts of data; we'll talk about why it's unsuitable for the kind of near-real-time incremental analysis that we do.

For the data scientists, we'll talk about our data set and types of analyses we'd like to do. Our current reports are fairly basic, but we'd like to know what factors affect user engagement with mobile push notifications and whether we can make predictions about their effectiveness."

Questions and suggestions are definitely welcome. Post on Google Groups Page: https://groups.google.com/d/forum/pdx-h-d-s

1 Hour Meeting

No presenters tonight. Instead we'll be having open conversation as a group about Big Data topics. Also, we'll discuss the future of this Hadoop / Data Science group.

Agenda: 1) Open discussion about Big Data topics, news, events, etc. Bring your own list to add to the conversation. 2) Discussion about the future of the group. This is the third meeting and we've tried a variety of formats across interests. Where do we go from here? Or do we go anywhere from here?

Questions and suggestions are definitely welcome. Post on Google Groups Page: https://groups.google.com/d/forum/pdx-h-d-s

This month's topic: Apache Drill

Apache Drill is a new Apache incubator project. Its goal is to provide a distributed system for interactive analysis of large-scale datasets. Inspired by Google's Dremel technology, it aims to process trillions of records in seconds. We will cover the goals of Apache Drill, its use cases and how it relates to Hadoop, MongoDB and other large-scale distributed systems. We'll also talk about details of the architecture, points of extensibility, data flow and our first query languages (DrQL and SQL).

Speaker: Gera Shegalov Gera Shegalov owns Hadoop MapReduce and Hadoop Core components in MapR's Hadoop Distribution. Prior to MapR, he worked at Oracle in Oracle Database High Availability on (Active) Data Guard, and in Oracle Java Platform Group on JMS backend communication and storage. Gera received Masters and PhD in Computer Science from Saarland University in Saarbruecken, Germany. His research focussed on workflow management, temporal databases, as well as application & database recovery.

Hadoop is an important batch data processing framework in use by companies of all sizes. It has a very approachable architecture and can be applied to a large group of modern computing problems. In addition, the framework supports an implementation of mapreduce which allows users to run jobs on any size cluster to fit their data size. Come learn about the architecture of this framework, management of the cluster, and how to develop mapreduce jobs.

About the speaker:

Dan Colish is a Core Data Engineer at Urban Airship. He is also a maintainer and active open source developer for Xapian and other smaller projects. He resides in Portland with his family and enjoys snowshoeing and hiking around Mt. Hood.

ApacheCon NA 2013 Portland, Oregon February 26th – 28th, 2013

First held in 1999 for developers and users of the Apache Server to meet face-to-face, ApacheCon is the official conference, trainings, and expo series of The Apache Software Foundation (ASF), and is the public showcase for Apache innovations.

Apache products power over half the Internet, petabytes of data, teraflops of operations, billions of objects, and enhance the lives of countless users and developers. ApacheCon brings developers and users together to explore key issues in building Open Source solutions "The Apache Way". With hundreds of thousands of applications deploying ASF products and code contributions by more than 3,500 Committers from around the world, the Apache community is recognized as among the most robust, successful, and respected in Open Source.

"No talks. You may opt to take up to 60 seconds to complain about Big Data. One paper per month, no obligation to read it."

We'll start by letting anyone who wants to take up to a minute to tell us what they've been doing with data lately.

This month's paper is A time-efficient, linear-space local similarity algorithm by Huang and Miller. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

Cascading is an open source workflow abstraction atop Hadoop and other Big Data frameworks, with a 5+ year history of large-scale Enterprise deployments. For example, half of Twitter's total compute uses this API, along with other large use cases at eBay, Etsy, Airbnb, LinkedIn, Apple, Climate, Nokia, Factual, Telefonica, etc. Cascading leverages some aspects of functional programming so that developers can create large-scale data pipelines which are robust and easier to operationalize. There are popular DSLs in Scala (Scalding) and Clojure (Cascalog), plus Jython, JRuby, etc. Recent support also implements DSLs for ANSI SQL (Lingual) and PMML (Pattern).

This talk will describe the technology and some of the large use cases for Cascading, plus show sample apps in Scalding, Cascalog, and go into examples of ANSI SQL with Lingual and PMML with Pattern. We'll also cover material about Mesos, a cluster scheduler akin to Google's "Borg" which is used at scale by Twitter, Airbnb, Box, etc.

Doors are open at 6:15, Talk starts at 7pm. Light snacks, beverages and brews will be available and will have a social hour following the talk.

Please RSVP via Eventbrite if you plan to attend: http://pdx-hadoop.eventbrite.com/

Talk will be led by Paco Nathan.

Paco Nathan is Chief Scientist at Mesosphere, a committer on Cascading, and an O'Reilly author for "Enterprise Data Workflows with Cascading". He is a recognized expert in Hadoop, R, cloud computing, distributed systems, machine learning, and predictive analytics, with 25+ years experience in the tech industry ranging from Bell Labs to early-stage start-ups. For the past 10+ years, Paco has led innovative Data teams building large-scale apps.

http://liber118.com/pxn/ @pacoid

Organized/Sponsored By: Aaron Betik, NIKE, Inc Global Technology Director, Consumer and Digital Analytics & BI

"No talks. You may opt to take up to 60 seconds to complain about Big Data. One paper per month, no obligation to read it."

We'll start by letting anyone who wants to take up to a minute to tell us what they've been doing with data lately.

This month's paper is Disk Aware Discord Discovery: Finding Unusual Time Series in Terabyte Sized Datasets by Yankov, Keogh and Rebbapragada. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

"No talks. You may opt to take up to 60 seconds to complain about Big Data. One paper per month, no obligation to read it."

We'll start by letting anyone who wants to take up to a minute to tell us what they've been doing with data lately.

This month's paper is Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent by Gemulla, Haas, Nijkamp and Sismanis. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

"No talks. You may opt to take up to 60 seconds to complain about Big Data. One paper per month, no obligation to read it."

We'll start by letting anyone who wants to take up to a minute to tell us what they've been doing with data lately.

This month's paper is Computational Methods for Dynamic Graphs by Cortes, Pregibon, and Volinsky. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

"No talks. You may opt to take up to 60 seconds to complain about Big Data. One paper per month, no obligation to read it."

We'll start by letting anyone who wants to take up to a minute to tell us what they've been doing with data lately.

This month's paper is Ad Click Prediction: a View from the Trenches by McMahan et al. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

"No talks. One paper per month, no obligation to read it."

This month's paper is Austerity in MCMC Land: Cutting the Metropolis-Hastings Budget by Korattikara, Chen, and Welling. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

CANCELED! Forecast is calling for 3-7 inches of snow tonight, the sinus plague is going around, and it's just damn cold. See you in March!

"No talks. One paper per month, no obligation to read it."

This month's paper is Image Mining of Historical Manuscripts to Establish Provenance by Hu et al. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

"No talks. One paper per month, no obligation to read it."

This month's paper is Image Mining of Historical Manuscripts to Establish Provenance by Hu et al. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

"No talks. One paper per month, no obligation to read it."

This month's paper is Palette Power: Enabling Visual Search through Colors by Bhardwaj et al. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

"No talks. One paper per month, no obligation to read it."

This month's paper is Profiler: Integrated Statistical Analysis and Visualization for Data Quality Assessment by Kandel et al. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

"No talks. One paper per month, no obligation to read it."

This month's paper is High-Dimensional Visual Analytics: Interactive Exploration Guided by Pairwise Views of Point Distributions by Wilkinson et al. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

"No talks. One paper per month, no obligation to read it."

This month's paper is Mondrian Forests: Efficient Online Random Forests by Lakshminarayanan et al. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

"No talks. You may opt to take up to 60 seconds to complain about Big Data. One paper per month, no obligation to read it."

We'll start by letting anyone who wants to take up to a minute to tell us what they've been doing with data lately.

This month's paper is Dynamo: Amazon’s Highly Available Key-value Store by DeCandia etal. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

"No talks. You may opt to take up to 60 seconds to complain about Big Data. One paper per month, no obligation to read it."

We'll start by letting anyone who wants to take up to a minute to tell us what they've been doing with data lately.

This month's paper is A Crowd of Your Own: Crowdsourcing for On-Demand Personalization by Organisciak etal. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

"No talks. You may opt to take up to 60 seconds to complain about Big Data. One paper per month, no obligation to read it."

We'll start by letting anyone who wants to take up to a minute to tell us what they've been doing with data lately.

This month's paper is Beyond Clicks: Dwell Time for Personalization by Yi etal. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

There will be pizza.

"No talks. You may opt to take up to 60 seconds to complain about Big Data. One paper per month, no obligation to read it."

We'll start by letting anyone who wants to take up to a minute to tell us what they've been doing with data lately.

This month's paper is Materialization Strategies in a Column-Oriented DBMS by Abadi etal. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

There will be pizza.

"No talks. You may opt to take up to 60 seconds to complain about Big Data. One paper per month, no obligation to read it."

We'll start by letting anyone who wants to take up to a minute to tell us what they've been doing with data lately.

This month's paper is Large-Scale High-Precision Topic Modeling on Twitter by Yang etal. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

There will be pizza.

"No talks. You may opt to take up to 60 seconds to complain about Big Data. One paper per month, no obligation to read it."

We'll start by letting anyone who wants to take up to a minute to tell us what they've been doing with data lately.

This month's paper is Visual Analysis of Large Heterogeneous Social Networks by Semantic and Structural Abstraction by Shen etal. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

There will be pizza.

"No talks. You may opt to take up to 60 seconds to complain about Big Data. One paper per month, no obligation to read it."

We'll start by letting anyone who wants to take up to a minute to tell us what they've been doing with data lately.

This month's paper is Geotagging One Hundred Million Twitter Accounts with Total Variation Minimization by Compton etal. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

There will be food.

"No talks. You may opt to take up to 60 seconds to complain about Big Data. One paper per month, no obligation to read it."

We'll start by letting anyone who wants to take up to a minute to tell us what they've been doing with data lately.

This month's paper is Feature Selection For High-Dimensional Clustering by Wasserman, Azizyan, and Singh. Read it or don't - the goal is just to have something to start conversations. "Did you read the paper?" will do nicely.

Mention @PDXBigData on Twitter with the link to the full paper to suggest papers for future sessions.

There will be food.