Agenda

As DoorDash business grows, the online ML prediction volume grows exponentially to support the various Machine Learning use cases, such as the ETA predictions, the Dasher assignments, the personalized restaurants and menu items recommendations, and the ranking of the large volume of search queries. In this session, we will share our journey of building and scaling our Machine Learning platform to meet the DoorDash business growth.  In addition, we will share  a few of lessons learned while optimizing the prediction service and how we measure success.

Given ML is here to stay, and growing in complexity by the day, it has become now critical that we think deeply about how to properly align our models with human interests. Doing this correctly requires a cultural shift in the way most organizations approach the model development process. Traditional metrics like F1, accuracy, or ROC on held-out test sets are woefully inadequate indicators of alignment when considered on their own. However, with the right tooling and a clear alignment framework, if we apply the principles of TDD (test-driven development) in a collaborative and transparent manner to ML development, we can not only build more performant models with greater velocity, but also instill far greater confidence in the models we are integrating into human lives.

While there is a lot of talk about the need to train AI models that are safe, robust, unbiased, and equitable - few tools have been available to data scientists to meet these goals. This session describes new open-source libraries & tools that address three aspects of Responsible AI. The first is automatically measuring a model's bias towards a specific gender, age group, or ethnicity. The second is measuring for labeling errors - i.e. mistakes, noise, or intentional errors in training data. The third is measuring how fragile a model is to minor changes in the data or questions fed to it. Best practices and tools for automatically correcting some of these issues will be presented as well, along with real-world examples of projects that have put these tools for use, focused on the medical domain where the human cost of unsafe models can be unacceptably high.

Product recommendation is at the heart of Personalization group’s efforts to help Albertsons customers. Deep Learning has become the go-to approach for recommendation. Therefore, the group has begun to put efforts into applying Deep Learning to enhance new product recommendations. First, leveraging transaction data and product catalog, we built Customer DNA and Product DNA models. Customer DNA model captures customer characteristics such as purchase behavioral pattern, dietary preference, e-com affinity, customer location, etc. and embeds into a list (vector) of numbers. Similarly, Product DNA model captures product characteristics (e.g., is product organic and/or sugar-free?) and product-product associations—e.g., bread and peanut butter are usually purchased together. Second, we leverage these models to build a next generation recommendation system—inspired by the Wide and Deep recommendation model architecture. Our experiments building the framework have generated favorable results and we will share our journey from model conception to putting it in in production to better serve our customers.

This talk is a technical overview of DeepMind and Google's recent work on reinforcement learning for controlling commercial cooling systems. Building on expertise that began with cooling Google's data centers more efficiently, we recently conducted live experiments on two real-world facilities in partnership with a building management system provider. These live experiments had a variety of challenges in areas such as evaluation, learning from offline data, and constraint satisfaction. Our paper describes these challenges in the hope that awareness of them will benefit future applied RL work. We also describe the way we adapted our RL system to deal with these challenges, resulting in energy savings of approximately 9% and 13% respectively at the two live experiment sites.

At Northwestern we have developed a system that is built to consume and capitalize on IoT infrastructure by ingesting device data and employing modern machine learning approaches to infer the status of various components of the IoT system. It is built on several open source components with state-of-the-art artificial intelligence. We will discuss its distinguishing features of being Kubernetes native and by employing our work that enables features to be specified through a flexible logic which is propagated throughout the architectural components. We will discuss select use cases implemented in the platform and the underlying benefits. The audience will learn how to build or use a streaming solution based on Kubernetes and open source components.

In fintech, it's important for companies manage their credit risk because if customers don't repay their credit, the lender loses money. In this talk, we'll explore how to prevent credit risk using a neural network model. We'll discuss not only what worked, but also what didn't work, and how the building blocks of the full machine learning system. We start with one model and discuss how to layer on more of an ensemble approach to predict risky customers and take action in a way that doesn't cause them to become even more risky business. We'll discuss features, a neural network model, evaluation, serving, monitoring, and ideas to improve.

Enjoy some machine learning laughs with Evan Wimpey, a predictive analytics comedian (and we're not just talking about his coding skills). No data topic is off-limits, so come enjoy some of the funniest jokes ever told at a machine learning conference.*

* Note the baseline.

At the RealReal we take in between 10-20k items daily. For each item 3-6 images are taken for the item. Historically those images would be "retouched" by hand which include tasks such as aligning the mannequin, cropping at one of three fixed distances based on the length of the item, digitally removing the base of the mannequin and its shoulder seams all while providing a "luxury" feel. We've learned lots along the way to automating 87% of our supply saving 10's of millions each year.

In this talk, we will talk through the journey of building an end-to-end forecasting platform with a focus on feature engineering, consolidation of features in a single place (feature store), and leveraging DNN techniques to solve different forecasting problems like Demand forecasting, Replenishment, Inventory optimization, etc. powered through the same platform

As part of the session, we will review the journey with the following stages :

• Feature Engineering: Experimentation to add critical features like Product Hierarchies, Price Rank Ratios, historical aggregated features, etc. along with their impact on the model accuracy
• MLOps : Use open-source Kubeflow to orchestrate and automate the data load, feature engineering, and Model training along with Model Deployment
• Decomposition of Forecast: Decompose forecast into Base & Promo forecast for What-if Simulation to identify the impact of Promos on Forecasts. Also, expanding the approach to inventory optimization.

Large language models are taking the machine learning industry by storm. Amid an outpouring of new research and exciting updates is the business question of how we can actually take advantage of this cutting-edge technology in the business world. Glean, an enterprise search engine that helps people find what they need and unearth things they should know across all their apps, is one of the only companies in the enterprise world which leverages LLMs jointly with enterprise data. We'll talk about lessons learnt from doing so, and how can enterprise companies build AI teams from scratch.

The largest issue ML teams face is due to AI models that silently fail. Silent failure occurs when model performances gradually degrade over time without showing any apparent signs of failure. These signs are therefore difficult to catch in time, usually leading to sudden or abrupt drops in performance after the gradual decline. This leads to a heavy impact on not just ML or business teams but also on the customer who faces the repercussions of incorrect predictions.

As more products incorporate ML, engineering leaders face a unique challenge. How do I incorporate ML experts into my organization? How can I support ML expert growth and development and drive the largest impact with my ML team?

At Paychex, we used big language models like SBERT to match client inputted job titles to a taxonomy of 800+ job categories provided by the US Bureau of Labor Statistics. Our multi-level architecture combines SBERT with neural network classifiers to achieve high matching accuracy. The SBERT model was fine-tuned in a Siamese network approach to improve its performance. The product will be available to our clients to recommend the best matching BLS codes for their inputted job titles. Normalizing job titles will provide Paychex clients with advanced wage or retention analytics.