One of the most important advantages of AI for marketing is hyper-personalization, which is the ability to understand the exact preferences of each and every individual customer or prospect. In this new world in which brand loyalty has been eroded, and new regulations about consumer privacy are rolling out, marketing teams need the ability to personalize matters more than ever. Getting granular about understanding your prospect and customer segments allows for a lot more responsive reaction mechanism and, as a result, a better connection with your audience.

What Are the Tools That Can Support Marketers to Be More AI-driven?

Becoming an AI-driven Marketing Department requires more than an intelligent marketing point solution or traditional marketing automation software. It requires you to be able to take all of your data: customers, marketing, sales, and operations, and transform it into something that will give you a fast and intelligent leading indicator of where, who, and how to engage with your customers and prospects.

Let me share with you a real-world example. Beacon Street Services is the services arm (and one of a handful of affiliates) of Stansberry Holdings, and they produce financial publications that are exclusively available through purchased subscriptions. For its marketing and sales teams, there was an opportunity to improve on previous tactics and processes of selling subscriptions, with a clearer feedback loop and signal for marketers to optimize their campaigns.

They realized there was value to applying a data science approach to this, especially given the wealth of valuable data they already had. They hoped to identify better buying criteria with a modelling approach to help their marketing team run more targeted and effective campaigns.

Their hope was true, so when they decided to use an AI and ML tool, it led to improvement in accuracy and significant time saving for the team. Doing this process only in one week — a process that used to take them more than six months. And the business impact has been significant: Beacon Street Services is on track to realizing $15 million in additional annual sales, directly attributable to augmenting their marketing with AI.

Ethical AI in Marketing

As marketing teams adopt AI, there is also a responsibility to ensure that it’s truly representative of your customer and prospects, and that bias doesn’t creep in. What do bias and fairness mean in the context of AI-driven marketing? It can mean ensuring your marketing campaigns aren’t skewing to a certain profile that isn’t truly representative of your target market or inadvertently targeting promotional offers to one group and not another.

To understand how ethics factor into AI use in marketing, I invited Sarah Ladipo from the DataRobot’s Trusted AI group to share some of her experiences. “Marketing data and use cases often use sensitive features (features that are legal to use but which would cause reputation damage if they were used like gender, and race), which brings a variety of associated ethical risks. When working with this type of data, marketers need to understand how to use this data in a way that decisions that are made are not biased and that those decisions are having a good impact on society.”

“When marketers learn how to use data that is not biased, AI systems will choose unbiased algorithms if they follow the ideals of transparency and trustworthiness,” Sarah says. “Creating explainable AI that allows users to understand how it arrived at a certain outcome allows AI to be more trustworthy. The truth is that AI is only as smart as the information it has been given so far. It is entirely up to business executives to make the most of this technology’s enormous marketing possibilities.”

Sarah continues: “A part of marketing is evaluating your product and identifying which demographic out of a large population will be interested to target. This can have a number of drawbacks, including ethical implications to consider. A line can be crossed where marketing can be exploitative. In some cases sensitive information will not be used in marketing efforts but the way data can be used can be problematic.”

I’ve Been Hearing All About Becoming AI-Driven, but How Does One Actually Start on That Journey?

First, pick a use case and ask yourself — why is this important for you? Second, define the KPIs — what is success for you? For your business? I recommend choosing something you can quickly test if you’re new to the AI journey. Third, make sure you have enough of the right data. And lastly, use an AI platform to achieve your goal.

Becoming AI-driven in marketing doesn’t require a 12-month process to get started. It should be fast, quick, and like a flywheel, where you can quickly take a few use cases and get onto new ones.

We have a whole library of AI solution accelerators that can help you understand where and how you can get started.

The post How to Be More Intelligent in Marketing appeared first on DataRobot AI Platform.

DataRobot conducted a study on AI Bias in the fall of 2021 and the results surrounding third-party ethics consulting showed conclusively that companies are relying increasingly on third-party consultants to help them make ethical decisions around AI. 47% of companies use third-party AI bias experts or consultants to define if bias exists in their algorithms or datasets, 38% of companies use third-party AI consultants to help prevent AI Bias and 80% of companies use an external advisory firm to conduct modeling audits.

The motivations for why a business might want third party help when it comes to AI ethics consulting comes down to the fact the ethics consult has come to reflect both the increasing complexity of AI’s ethical dilemmas, and our discomfort with the prospect of answering them alone. Ethics consulting, which developed in other industries, like medicine, in the 80s, is now commonplace. Medical ethics relies on standards and credentialing, and yet how medical organizations evaluate the outcomes continues to be controversial.

Sharing the tough decisions

At its core, consulting satisfies a desire to share the responsibility for harms that result from unexamined technological development.

This is where third party AI ethics has yet to hit its stride. No one has satisfactorily addressed the concerns of authority and responsibility, and creating ethics committees or internal ethics compliance programs as a proxy doesn’t really answer what an ethics committee would actually do on a day-to-day basis, besides create a divergence of governance which in turn could result in a bottleneck of decision-making at a time when the technology and its measures and metrics are in a state of rapid evolution.

There are those who eschew shared responsibility of decision-making in place of tools, platforms, or frameworks aiming to ensure explainability, fairness, and accountability in AI systems. Forrester calls this Responsible AI (RAI). In the fall of 2020 they published an overview of the ethical AI consulting landscape titled, “New Tech: Responsible AI Solutions,” that claimed the “Responsible AI Market is Showing Promise” while showing that third party risk evaluation was still low across all functionality segments. Each vendor’s technical solution had varying capabilities, from offers of “explainability engines” to platforms that purport to work on top of any existing model development functionality.

Asking what we want from an ethicist

What are we really seeking when we engage an ethics expert? Can they be replaced by a technical solution to an ethics problem that perhaps should not be automated? Tools themselves cannot solve the problem of context. That is the perspective of an ethics consultancy like ORCAA, an algorithm auditing company started by Kathy O’Neil, who has been an author and independent and outspoken data science consultant since 2012. For her, the aspects of ethics consulting are pretty straightforward: Ask yourself for whom does the system fail? Is your data even collected toward that end?

Ted Kwartler, Field CTO at DataRobot sees a middle ground. “It’s a long standing business question regarding building capabilities internally, or using (usually) expensive consultants to accomplish a task much faster. The truth of the matter is that ML/AI is becoming business critical, making speed to production more important as well.” It would be understandable to believe one straightforward solution would be to buy consultants in order to realize ethical AI value faster with their existing frameworks and technology, and as a result avoid disruption by competitors.

Kwartler cautions this is actually short-sighted.

The true issue lies with selecting both the software and people. Pure management consultants may not have any particular AI technology and instead be working in Julia, R, Python in Jupyter Notebooks and seek to deploy these models in a number of methods depending on the consultant that wrote the code. Technology variance can increase systemic risk much in the same way that airlines, like Southwest, often have a fleet of aircraft from a single manufacturer, like Boeing. It makes maintenance and monitoring standardized and well understood by personnel.

His advice? When choosing a consulting partner, make sure they know their technology.

If using consultants, why inherit their diffuse technology headaches? Large enterprises can’t have multiple tech stacks or they invite confusion among quality and audit responsibilities, resulting in slowed implementation. Likewise, if you’re confident about the tech, then the question becomes, who actually becomes responsible for scoping, building, and deploying your models?

Ethics is not a generic catch-all

Selecting a technology partner means vetting their actual expertise. If you truly believe you can’t afford the uphill slog that is developing the right resources in-house in response to specific mission directives and against the organization’s ethical values, then ask a few hard hitting questions like,

• How many models has your chosen vendor put into production?
• What’s the simplest model they ever deployed? (since simpler is often better)
• How did they recover from a project that had unexpected data problems?
• Where is their specialty within the business landscape?

Probing questions that go beyond technical solutionism will identify the notebook-only prototyping data scientist from the people that can share the horror stories and use that experience to overcome the inevitable hiccups that derail production deployments.  If organizations don’t own their AI process, which is foundational to a business operation, they don’t actually own the operation itself.

This relates to another type of ethics partner that doesn’t want to endlessly consult, but instead wants to teach, and then who is still around to provide decision-making tools and advice as a backstop when needed. This “ethics as a service” aspect provides access to a network of interdisciplinary ethics professionals who specialize in your specific situation, say in law and policy, or data use and privacy. Will Griffin, Chief Ethics Officer of Hypergiant defines EaaS as an aid to the critical thinking and risk mitigation steps necessary to blend “…ethics from the hearts and minds of designers and developers into work flows, and ultimately into the AI products released into society.”

There are other flavors of academically-oriented AI ethics consulting delivered as opinion pieces or critical reviews of scientific literature, that focus on a singular ethical question, or even a single tech sector challenge, like bias in large language models. This kind of analysis lends itself best to change management initiatives or investment decisions that have an ethical impact and become critical when adopting a new function, service, or product shift.

Ultimately, every organization understands their own needs and budgets best, and should be congratulated for elevating ethics into the mix rather than scrambling for legal advice after their model has been released into the wild. Looking to technology partners that unify their technology needs while also providing practical expertise with a proven track record of deployment will move them toward a solution that works best for both the business and the communities they serve.

The post AI Ethics Consulting; Asking for More Than Advice appeared first on DataRobot AI Platform.

But, frankly, the failure rate of enterprise investments in AI are staggering: analysis by VentureBeat suggests that only 21% of companies have AI “deployed across the business.” That sounds bad, and it is. But, on the flip side, the 13% that do succeed are seeing a big impact.

Clearly, then, increasing the number of businesses using AI/ML is a problem worth focusing on. And that’s where ML operations management, or MLOps, platforms come in.

Bogged Down in ML Operations Management

It’s interesting how many businesses get bogged down in ML Operations management. The idea of an MLOps platform is to manage the lifecycle of machine learning models – specifically  deploying, maintaining, and updating – in much the same way as application lifecycle management methods and systems are used in software development. The goal is to provide the guardrails and processes necessary to be able to deploy, manage, operate, and secure ML workloads at scale.

A big part of the reason behind enterprise AI’s relatively anemic success rate is that too many businesses still spend too much time and energy building – or attempting to build – their own ML operations management platforms.

ML operations management platforms are essential to getting models into production and keeping them there. A model or pipeline that is not in production is one that cannot provide any value (or limited value) to the business. But while they have very high operational benefits, they are not value-add from a business point of view. Technologists love building. I am one of those. But usually our mission is to use technology to move the needle for the business.

Data scientists have the most impact exploring strategies, building models, feature engineering, and working with hyperparameters. Machine learning engineers add value by making sure the deployment and scale-out of that work is as quick and reliable as possible.

Is having this talent work on building a platform for any of these steps the value-add? Is having a proprietary MLOps platform a direct competitive advantage? For most businesses, the answer is emphatically “no.”

The Machine Learning Model Lifecycle is Complicated

There’s no other way to say it: building an ML platform is a very heavy lift. Even the biggest and best-resourced teams would hesitate to take it on. It’s also an iterative process that requires version after version of improvements.

Building effective models is a challenge in itself, but to get any business value at all from them, they must be run with a production application. Making this happen requires supporting and tracking several requirements and processes at the deployment, operations, and governance and security stages of the ML model lifecycle.

At the deployment level, you must consider how to manage model versioning to track changes and enable roll-backs if needed. You also need to be able to publish models to a model catalog so that other people can find and reuse them, which is especially valuable when models have already been trained. Models don’t live in vacuums – sometimes they consist of ensembles of multiple classifiers, in other cases they are pipelines with pre- and post-processing functions – so a pipelining function is also needed to track dependencies and control development and deployment. There’s more: the source code needs managing, APIs need to be provisioned to enhance adoption and scalability, and data source connections need to be established and maintained.

Now imagine doing that for tons of frameworks and libraries, languages, processing dependencies, and data connections. Your buildout matrix has just become quite big. And that’s only the first layer of MLOps.

Next comes the operations side. Here we need to think about integrating with the CI/CD pipeline to accelerate production and leverage automated testing, deployment, and updates. Application integration is also an important consideration, whether that’s for visualizations or simply other infrastructure applications. Performance monitoring and alerting is required for every model. And where are you running this model – in the cloud, on-prem, multi-cloud… all of the above? Are you going to be able to build something that’s consistently transferrable to each environment?

Then you get down to the real nitty-gritty of production: governance and security. Can you guarantee the data security, network security, and model security? Are you correctly applying the InfoSec policy from your organization to your ML workloads? Are you doing all the right things from a permissions perspective? If you work in a regulated industry, can you package up all of the audit trails for compliance? Can you at any point determine the 4 W’s of Who called, What, When, and Why?

This is what you need to think about for a true production system. All of this is a lot of work – and that’s just to get the first version up and running. Then – wait for it – what happens when you inevitably need to upgrade something?

The right question is “build and maintain or buy?”

Hopefully you have a sense now that there is a lot more to building an MLOps platform than building.

I’ve met multiple companies that have successfully built out stunningly complex internal ML platforms. But the one constant we see is that the investment gets bigger and bigger every year. And that’s fine, as long as you know from the outset that building a solution such as this is a journey. It’s not a decision that you make at one point in time.

Many organizations start with a recipe or blueprint approach for productionizing models. That tends to break down when you have hundreds of models to control, observe, and track. It leads to a lot of “let’s go build X,” which is a distraction from value-add work and, as an operating model, is at serious risk of being blown up by changes in resourcing priorities.

Furthermore, large organizations will have dozens of teams each using dozens of tools. By definition, an MLOps platform that can run all these workloads in a consistent way has to be expensive – it has to be continuously evolving.

At risk of someone muttering “Yeah, you would say that,” 9.9 times out of 10 you will get the best value from investing in a platform that allows you to accelerate and standardize production. Not only will that platform perform better, sooner, than an internally built solution, but also it will free up your very expensive data science and ML engineering teams to build new and added-value use cases instead.

Time to Start Treating ML Development Like Software Development

One of the stock objections to buying an ML operations management platform is the perceived lack of flexibility.

In some areas of the business this kind of argument stacks up but not when it comes to ML lifecycle management. Take a look at the world of software development and look at your current stack there. For the vast majority of organizations, buying solutions and components to manage software production has not left them gripped by inflexibility – if anything there has been great success in loosely coupled but tightly integrated bought components.

It’s time to think about the ML development lifecycle like the software development lifecycle: adopt best-in-class components for every single piece of the value chain. As I noted at the start, the success rate of enterprise ML is too low for such a high investment and high potential impact technology – this is an urgent problem.

Ultimately, ML is even more iterative than software development. We need to accelerate production to get models out there fast, so the bad can die and the great can prosper. The credibility of the discipline depends on it.

The post When it Comes to MLOps, Asking “Build vs Buy” Is the Wrong Question appeared first on DataRobot AI Platform.

This data allows machines to draw insights from the patterns of life—people’s mobility habits and behaviors and how they interact with their environment. From insurance companies customizing premiums based on real-world driving to refining conservation policies based on urban sprawl, businesses and governments are beginning to understand the benefits of bringing geospatial data into their machine learning applications.

Certainly, the opportunities are huge. And we’ve only really started to scratch the surface of its potential.

What’s Driving Increased Interest in Geospatial Data?

The increased supply of analysis-ready geospatial images that covers wide areas is largely driving the increased interest in geospatial data. . Thanks to their own use of machine learning algorithms, so-called satellite-as-a-service providers have driven down the costs of this data by using algorithms to extract and classify features from satellite imagery. These algorithms have also made it much easier to regularly update these often large datasets to keep them relevant. Growing awareness of this kind of geospatial data has driven interest in other types of datasets like map data.

This wealth of clean, verified, and ready-to-go geospatial data is a game-changer. By applying these datasets to existing problems, we can reveal richer insights.

Human Geography is the study of people and their interaction with their environment. So, although the term geospatial data sounds a little cold and clinical, it is intimately tied into the human experience. There is enough of a geospatial element in every facet of our daily lives to interest almost all businesses and governments who can reveal and respond to unseen factors and influences by better understanding what’s happening in time and space. That might mean making census-style population surveys annually based on analysis of satellite imagery versus once-a-decade direct outreach campaigns. This would enable much more responsive public services, for example. Or it could simply mean using foot traffic data to identify the optimum time to put out fresh groceries.

Either way, geospatial data provides insight into the spaces around us and how they can be optimized to create better experiences or more sustainable and efficient operations. And organizations don’t need to wait for a new problem to solve in order to start making use of geospatial data.

How Can Businesses Start Working with Geospatial Data?

In every industry and in every business function, there is value to be had in adding geospatial layers to your analysis. The best way for businesses to prepare to add a geospatial element to their machine learning applications is to revisit their current problem statements. Just as with most other areas of data science and applied machine learning, it’s a process of retesting your hypotheses with geospatial factors, discovering what works and building iteratively off that. Each time you revisit your problem statement, you’re enriching your data with geospatial elements to create a more accurate and detailed picture of your problems and predictions and of your business and its customers.

Organizations may, in fact, already have spatial components to their own data, in-house but unused. Anything that identifies a space or a place, such as a zip code, can become a geospatial component. So, by classifying customers into zip code boundaries you unlock the ability to think about them spatially and for a machine learning system to apply spatial understanding. For example, tagging sensors with a location can add geospatial components in your data.

Geospatial Data in Action: What Is the Predicted Sale Price for a Home in Utah?

A more detailed example would be leveraging geospatial elements to better predict or assess house values. “What is the predicted sale price for a home in Utah?” might seem like a pretty straightforward question but accurately determining real estate values is a struggle for many firms. How do you know which variables deserve more weight than others? What factors are buyers looking for? How will the market change? Location AI can give you the answers by providing the tools to combine location variables with numerical, categorical, date, image, and text data to unlock the full potential of your geospatial data.

In the case of real estate in Utah, that means enriching typical listing information with geospatial data to see what really influences house prices. The former includes numeric (price, bedroom counts, bathroom counts, acres, etc.), categorical (garage, exterior, and roof types, etc.) and location geometry (i.e., longitude and latitude) features. Depending on your hypothesis, the latter might include select demographic variables from the U.S. Census Bureau, walkability scores, highway distance, school district scores, and distance to recreation.

As this use case illustrates, adding a spatial component to a problem is a way to better contextualize the human experience within machine learning algorithms. And as our example list of data sources above shows, there’s tons of this information out there. That’s the fun part of geospatial data; the main challenge is finding the best fit for your models.

Hopes for the Future of Geospatial Data in Machine Learning

At a very high level, machine learning seeks to make large amounts of data consumable and understandable. It aims to reveal patterns that would otherwise slip past unnoticed. And in the context of geospatial data, I have several hopes for new developments that increase its potential to significantly improve our understanding and enjoyment of the world around us.

Among these, leveraging technology to shorten the loop between extracting features from satellite imagery and on-the-ground verification would provide faster turnarounds on verified imagery data sets. (Today, this often involves a human visiting a location to physically check the accuracy of what an algorithm says is in an image.) Current barriers to entry for consumer-grade enrichment data – primarily costs – need to come down, too, to move us towards a more open geospatial data environment. OpenStreetMap is a great repository for data, but it is something of an outlier right now, and we have yet to see the wide availability of free and open data that we see in other areas.

And, finally, I hope we can use geospatial data to drive machine learning applications that are predictive rather than reactionary. In a previous role, I was a human geographer, studying the impact of events such as rising fuel prices on societal instability. If we can train machines to use geospatial data to automatically predict instability before we have to send out food or medical aid, that would have a real and immediate impact on people’s lives.

Ultimately, a future with more “live” and open geospatial data will enable greater responsiveness and accelerated time to value – and, more importantly, better human experiences.

The post Geospatial Data: Where Machine Learning Meets Life appeared first on DataRobot AI Platform.

The potential is huge – we are likely witnessing the most revolutionary technology shift we’ll see in our lifetimes. Which begs the question: why are we not seeing it in production? Why do so many machine learning (ML) projects fail?

According to Gartner, 51% of enterprises have started their AI journey, but just 10% of ML solutions get deployed. This is because ML and ML in production are two different beasts, and many people don’t fully understand the barriers they face, questions they should ask, and objectives to strive for.

In the field, we often see people focus on data collection, data cleansing, building out models, accuracy rates, and explainability, assuming those are the keys to success. The reality is many ML problems are not technology problems at all. Instead success stems from digging into infrastructure, orchestration, integration, deployment, and core business objectives.

ML isn’t the problem: it’s the people around it, managing it, and how you go about those things. So let’s break this all down into five major problem areas, the key question to ask in each case, and how to solve it.

1. Lack of process

Getting funding for a project or experiment is the easy part. Not thinking through how to get it into production is a major problem. Too many people think about achieving results, but aren’t intentional about a clear pathway to actualizing them.

Teams need to consider how a project will affect the bottom line, how it will be funded, who’ll need to be involved, and whether everyone fully understands these things. The number of projects where instigators say something will go live and “we’ll figure the other stuff out later“ is too high – which is why 90% fail.

Ask: How do we get from proof of concept to production?

The solution: Start from how you’ll deliver value to your organization and work back from there. Plan and fund the deployment up front – who’ll take it over and be responsible for it, and how it will be implemented. Set clear deployment criteria around ownership and updates.
Bring in IT/DevOps stakeholders early, because data science and ML teams on their own won’t cut through a large organization’s red tape. And build for repeatability, because the only truth in this space is that tomorrow you’ll have more models in production than you do today.

2. The wrong incentives

When ML efforts are part of innovation mandates, they’re designed to be ‘out there’, but this is optimization technology. ML is supposed to optimize business processes, increase revenue and reduce costs. It must align with the organization and enable it to meaningfully do things it couldn’t do before. Innovation alone won’t get results to the business.

Ask: What is a justifiable improvement

The solution: ML needs to be effective, integratable and usable – not ‘demoware.’ You must be able to answer how it will affect the business and whether you’ll save money on getting a model into production today. Should you find money would be lost for each day without the model, that’s incentive to fund it.
It’s useful to start an ML problem by thinking about how to improve a business objective and what your constraints are. Are you focusing on the right things? If you’ll save a specific amount, does that justify the resource? What is the process you’re optimizing and the tech stack you’ll use? Who’ll own it, what are the key drivers for performance, and do you have the right data and modeling tools?

Companies that approach this from the wrong direction fail. You must first consider changing how value will be added to the business and only then build out the stack.

3. The wrong teams

ML projects are more likely to fail when an organization doesn’t apply its skillsets in the proper place or at the right time. This might be asking the wrong people to build things – for example, having data scientists with a lack of engineering experience build infrastructure. Or a company bridging a communication gap between DevOps and data scientist teams too late.

Culture is an issue too. Bringing someone into a large enterprise and telling them to deliver stuff can be daunting if they lack experience in process within large organizations, and are used to rolling a model on a laptop and deploying it when they like.

Ask: Do I have the right people to make a solution deployable in my org?

The solution: Don’t expect people to do tasks they are not suited for, and don’t rely on finding ‘unicorns’ who know ML, production, DevOps and engineering. They exist but are hard to find, so stop chasing them. Instead, create hybrid high-performing teams that combine DevOps engineers, data scientists and software engineers.
Beyond that, ensure you use software and platforms that enhance your data science and ML teams. And look for tooling to help you with data prep, model training, scaling and ops, to figure out where to hire people or buy a solution that will fill a gap.

4. The wrong technology

All sorts of technology problems can trip up getting ML into production: lack of defined stacks/best practices; not building for repeatability, measurability and auditability; not thinking about access to data.

On the last of those, you’ll hear people argue: “If only I had access to this data, I could build a better model.” Well, if you don’t, you don’t. That’s reality and the difference between production and dev!

Ask: What’s the best ML architecture for my organization?

The solution: Design to execute at scale and for repeatability and efficiency. Tightly integrate components you’ll use in the ML stack through APIs and programmability, but ensure you can swap out, replace and upgrade them when tech, data sources and needs evolve.
Above all, be agile. What you’re using today is not what you’ll use six months from now. But also be open to integration with in-house technologies: the less you need to replace internally in your organization, the less friction you’ll have in getting ML out there. Be tactical about what you need to add.

5. A lack of champions

This final point isn’t special to ML: with deployment of new technology and potentially expensive science experiments, the lack of a champion can be a death knell. ML projects without exec sponsorship rarely see the light of day, regardless of other considerations.

Ask: How do I get buy-in from stakeholders?

The solution: You need someone who’s forward thinking and who can understand the business case for what you’re trying to do. Also, it doesn’t hurt if their own personal standing could be improved by backing your project – I’ve genuinely in the past asked stakeholders: “How do I ensure your bonus this year?”
Ultimately, you need to align ML with the organization itself, because without champions, getting ML into development is hard. Everything looks like cost, so figure out how to get buy-in, involve stakeholders up and down the command chain early on, align values and interests, and collaborate to achieve your goals.

Some prior points I’ve explored can further help you with this solution. Take into account all five and you’ll be on the path to success – one of the 10%, rather than the 90% that never go anywhere.

The post From R&D to ROI: Five Reasons ML Doesn’t Go Into Production – and How to Solve Them appeared first on DataRobot AI Platform.

I had started school to learn about hardware and CPU design. But instead, I had become infatuated with developing AI and machine learning algorithms. As my personal experiments with algorithm development escalated, it led to introductions to professors at Cornell University – where I was studying – who were researching AI and evolutionary algorithms.

At the time, nearly all AI research was focused on generating the most accurate predictions– especially around images and text. But I started to wonder if AI could help the scientific process itself instead. Could we devise algorithms that bootstrap discoveries? Could they discover answers that were not just accurate but concise and elegant? What would they find if we unleash them on new experimental data? These questions became my obsession in graduate school, and they ultimately led me to working on a new algorithm and application called Eureqa to answer them.

I knew that developing AIs to think like scientists would be a challenging problem (the clue is in the name). But I hadn’t expected to learn so much about how we – people – formulate and communicate our expectations. And why we so often get the unexpected back in return.

Eureqa and Genetic Algorithms

Genetic or evolutionary algorithms mimic natural selection, by eliminating weaker solutions to a given problem and allowing the stronger ones to be developed into future generations of possible solutions.

Eureqa uses this approach to mimic the scientific process. Fundamentally, it proposes new features and designs the right experiments to test for mathematical relationships in a given dataset. Its objective is to find the most explainable model for those relationships with the least assumptions; that’s a totally open-ended problem. (“Explainable” matters. Newton’s law is beautiful – and useful – because it’s elegant.)

This enables Eureqa and other genetic algorithms to outperform many machine learning techniques in highly complex, but real-world situations. For example, many time series forecasting problems suffer from unpredictable or spurious events in the data, and many machine learning models build extremely complex ways attempting to fit them. That extra complexity can come with a large cost, offsetting the true signals and leading to rapid model drift when used in production. Some other algorithms, like reinforcement learning, are interesting because they can continue to adapt in production. However they too suffer from this same risk of generating highly complex explanations for simple behavior, while also being risky to control in production.

That usually is not a problem for Eureqa, which instead searches for the simplest possible relationships, and uses the data that is available. It can also be given more prior knowledge than other learning algorithms, guiding it towards a particular structure or asking it only to improve specific aspects of the model.

Upping the Stakes

During the course of my research leading to Eureqa, I joined the Creative Machines AI Lab at Cornell University where I eventually went to grad school.

In that setting, I had the opportunity to add some degree of intelligence to robots, which are not really very smart, by modeling how their physical parts interact and how they themselves interact with their environment.

But during this time I was becoming more and more fascinated with Eureqa’s potential for automating and accelerating scientific discovery in physics. Nearly every law in physics is based on some sort of symmetry or conserved quantity (i.e. conservation laws). And identifying these laws are the basis for most of physics’ biggest discoveries. I was convinced Eureqa could help scientists to find them more quickly and more often.

The difficulty with algorithmic discovery of conservation laws, however, is that you’re liable to get a lot of trivial conservations. f(t)=0 is conserved over time, for example, is a super boring conservation that explains almost nothing. So, the real breakthrough is developing an algorithm that can search for and find meaningful physical relations, while avoiding trivial solutions.

And the success in optimizing Eureqa to detect these laws led to the publication in 2009 of a research paper in Science, the prestigious – and awesome – academic journal.

Navigating the Dark Arts of Feature Engineering

Part of the reason Eureqa excels in the search for conservation laws and other mathematical relationships, is that in its efforts to discover explainable models, it also automatically figures out the optimal feature transformations from scratch and without human intervention.

On its journey to provide the highest accuracy with the least complexity, it can filter through the almost limitless potential transformations you might make on data. It tests ever simpler ways to achieve similar or improved accuracy, billions of times. In one sense it’s a flashlight for the dark arts of feature engineering, the attribute that differentiates the most skilled data scientists.

Eureqa’s automatic feature transformations have turned out to be particularly powerful in working with time series datasets to forecast out the future. Time series often requires heavy feature engineering to figure out the right lags and interactions–which is where Eureqa thrives.

Through the Wormhole and into the Public Consciousness

I didn’t appreciate at the time the significance of the Science publication and how pivotal it would turn out to be.

But as a result of that paper, the work was covered in the science section of The New York Times, and I was invited onto NPR’s Radiolab and the Science Channel’s Through the Wormhole with Morgan Freeman. On those shows, we put Eureqa through its paces, challenging it to offer an underlying law of physics for how different physical systems behave (e.g. double pendulums, moving cars, etc).

The experience was a sharp lesson in how scientific work can catch the public imagination. The media latched on to the original publication with the idea that AI can help automate scientific discoveries – exactly what had captured my imagination years before — even dubbing it the a “robot scientist”.

The resulting interest prompted me to develop Eureqa as a software that people could use. I got out of school, moved in with my girlfriend in Detroit, and coded for about 12 months straight.

Incredibly, in that first year, we had 30,000 users, which allowed me to bootstrap a company around Eureqa that was eventually acquired by DataRobot six years later.

As Relevant as Ever

In the years since the Science publication, Eureqa has been mentioned or cited in almost 3,000 academic studies.

And recently, there’s been a surge in Eureqa-inspired academic research into evolutionary algorithms that compete with Eureqa (which, I have to add, still performs extremely well).

The reasons behind this interest might be society’s need to continue scaling our ability to explain complex systems. World events have emerged to create a perfect inflection point for genetic algorithms and global-scale problem solving. Certainly, they perform well solving hard optimization problems with limited info or in rapidly changing environments – like, say, a global pandemic.

COVID and the sudden changes in behavior it prompted has broken many models that were trained pre-pandemic. Retraining models to mitigate this is difficult because even after almost two years, there’s not a lot of training data available. And so, compared to machine learning and deep learning models that need months of training, Eureqa’s principle of finding the simplest model to explain a phenomena makes it more robust to things like overfitting and noise. It’s also able to use smaller datasets more effectively, which is great for when you have less history or scenarios in which things are changing quickly.

What Algorithm Development Can Teach Us About Ourselves

My experience taking Eureqa from my dorm room to the lab, into corporate America and beyond, revealed a lot to me about what you might call the “soft” skills needed to build a useful algorithm. Curiosity and the ability to get excited about its potential impact are essential complements to sound development fundamentals and a good math foundation. Coming up with ideas, testing them systematically and brainstorming new approaches is extremely creative work.

At a deeper level, I’ve worked on Eureqa for so long and become so familiar with it that there are analogies with close personal relationships and all the frustration they can sometimes throw up. You can spend a lot of time building and nurturing an algorithm. But the crazy thing about developing new genetic algorithms is that you give them an objective function (e.g., the highest accuracy with the lowest complexity), and they’ll find the most creative ways of doing that. You let them loose and they achieve their goal, but in exactly the opposite way that you hoped they would. A lot of experimentation goes into defining the right rewards and operations within these algorithms to encode what it is that you really want, like now encoded in Eureqa.

It’s a constant fight to adequately communicate your goals to the algorithm, which I think forces us to also confront how the way we articulate our requirements – to both machines and people – is often loaded with assumptions about context.

For me, that was an illuminating experience, and it’s one that’s inherent to most machine learning endeavors. You’re going to learn a lot about what your objectives really are when developing and refining algorithms. I know I did.

The post Eureqa! How a Bored Undergrad’s Algorithm Achieved 3,000 Academic Citations appeared first on DataRobot AI Platform.

• ”Hey Arjun – I feel like seeing lots of trees and birds. What hike should I go to?”
• Or “I have family visiting this weekend. What trail should I take them to? I recently did this hike (shows image of a hike on their phone..)”

I quickly realized that this would require a major undertaking to build out an AI application – going from gathering data from actual hikes, tagging them with reviews, building complex Natural Language Processing (NLP) and computer vision models, operationalizing the model, and finally making recommendations accessible through an end user interface – but I really wanted to try it out.

The How – Data, Modeling, And Predictions

Here’s how I went from reviewing some sample images on my phone to building and testing various cutting edge machine learning techniques to developing an AI app in less than a week to recommend hiking trails!

Data

I had about 200 plus scenic images across a lot of my favorite hikes and trails in the last four years of being in the Bay Area! Some of my favorites being Gray Whale Cove Trail in Montara, Golden Gate Park in San Francisco, or just going for a walk by the Embarcadero. Additionally, I tagged the trips with reviews: “I liked this trail a lot, lots of trees and birds to see.” Each review added a piece of information that would give users the ability to make a wise choice (without my help in the future).

Modeling

As a data scientist, while I had extensive experience fitting and training the more common classification and regression algorithms on tabular data, I had never developed a reliable, accurate, and integrated computer vision and a natural language processing model – simply because of the time required to iterate on such approaches, the amount of data required to get started, and the necessary skills required to accomplish such a task! So how did I test and iterate on hundreds of the most advanced techniques to develop a reliable model in a few hours? DataRobot AutoML!

Quick note if you are new to ML:In this case, machine learning is helping us learn patterns from historical hiking trips to accurately and reliably recommend which hike would be a good fit!

Deployment

Deployment can often be the hardest stage in a project. As a data scientist, you simply expect that once you build some reliable experiments and share insights your work is done – now my IT or MLOps friends can deploy the code (or operationalize this model). However, since this was a weekend project, I had to be my own MLOps team and get my model live and ready to serve real-time predictions for my users (friends).

The What — Making Recommendations and Building Apps

Why does the end-to-end process involve so many stages? As a data scientist, you already have to prepare the data, build and validate models, (educate different audiences on what you are trying to solve), and now I am supposed to be building a front end AI app? However, with some open source streamlit app examples and snippets, I could easily integrate the above API code and test my idea and solution vision!

For fellow coders – feel free to check out the app and modeling code here: Github Repository

Feel free to try out the app this weekend in the Bay Area (or now if you can’t wait like me!): Bay Area Hiking Trail Prediction App.

What’s Next

For now, I’ve built my model for nine hiking trails. But the project is really motivating me to revisit old hikes to take more pictures (and generate more data). I recently got my hands on the book, Best Hikes in the Bay Area, and there’s a lot more hiking trails that I can’t wait to explore and take pictures of! And the best part of this project is the fact the model can be retrained, updated, and made fully functional within a couple of hours on a weekend. I’ll be continuing to share my hiking passion with my friends and family through my stories and AI solutions!

If you have feedback or questions about the data, process, or my favorite hiking trails – feel free to reach out! Always looking to make incremental improvements! Comment if you would like to see a system setup for adding data or contribute to the app!

Thankful for the support from my fellow hiking friend and colleague, Austin Chou, on helping with data collection and modeling activities!

The post Can Machine Learning Recommend the Perfect Hike? appeared first on DataRobot AI Platform.

As companies continue to fill more than a million surplus jobs, one of the top roles they’re hiring for is artificial intelligence specialist. It’s great to see a strong focus on AI and machine learning as they become more prevalent across industries. However, these companies may not be searching as far and wide as they could be.

Per a World Economic Forum-LinkedIn study, women make up only 22% and 12% of global AI and machine learning professionals, respectively. What’s creating this gap? The answer, of course, is complicated. Is it an unwillingness to interview women for these types of roles? Is it a lack of opportunity for women to move up within the organization? Maybe job postings aren’t being written in appealing ways?

All those factors (and more) are at play. But one thing remains true: this gap limits the effectiveness of companies. Instead of having diverse and varied hires, there’s a danger of too much similar thinking within the organization. This can put a limit on what it can accomplish.

Dreaming Big to Bring More Women into AI

One of the coolest programs at DataRobot is Dream Big. Dream Big guides employees to determine and define their long-term career and personal goals, then puts employees on the right track to accomplish them.

A part of the program that I really enjoyed was this thought exercise; we were asked to develop our Dream Big Legacy Goal — something we wanted to accomplish several years in the future. Then, we worked backward to figure out how we could reach that goal, moving in smaller increments.

This exercise was a challenge I hadn’t fully considered before. It had to define my steps, ask for help and feedback and focus on what I wanted to achieve. In this case, I wanted to bring to life the Women in AI Camp (WaiCAMP), an introductory virtual course on artificial intelligence and applied data science — entirely in Spanish. The AI gender gap is especially large in Latin America, and we see this camp as a way to contribute to the inclusion of women in AI roles, projects and initiatives in a positive way.

Considering the thought exercise, some of my motivating factors are to contribute to the economic development of Latin America by accelerating the adoption of AI and to reduce the gender gap in AI. Those motivations require a lofty goal and breaking it down has been helpful in maintaining a solid path.

My Dream Big Legacy

My Dream Big Legacy goal is that in five years, I will have mentored and trained 1,000 women in AI in Latin America through the WaiCAMP by DataRobot University; my one-year goal was to launch the camp; my six-month goal was to reach out to women and secure enough interest to have 60 applicants.

By breaking the overarching goal down into smaller achievements, I’m more likely to accomplish each of them. However, I still get to celebrate impactful wins along the way, and in this case, I’m doing so in the presence of some incredible women!

Through these smaller achievements, I also learn what works and what doesn’t and can course correct to help reach future goals. And I’m not alone either. By having the fundamental ability to understand how to apply AI and machine learning in certain areas, other women can launch impressive projects to build on that foundation.

For example, neuroscientist and technologist Poppy Crum coined the term “hearables” and works with Dolby to develop neuroscience algorithms. Her work will continually and silently assess and anticipate people’s needs and state of mind — it’s like a USB outlet for the ear.

Or how about Dr. Radhika Dirks of XLabs? She’s developing AI-enabled strategies to improve upon social distancing and creating projects focused on accelerating vaccine and antiviral discovery for COVID-19. We may never return to the “before times” of pre-COVID life, so this work is beneficial for our well-being.

Additionally, this foundational knowledge plays an important role in reskilling and upskilling as new technologies emerge. Twenty years ago, many people didn’t fully understand AI or machine learning, and these won’t be the last technological advancements we see.

How Can We Start Closing This Gap?

The first step toward closing this gap is acknowledging it exists. Company and industry leaders must recognize this is an issue, but they can’t stop there. They must take action, too.

Here are some places to begin:

• Continue (or start) making company culture gender-inclusive. Culture builds the foundation for employee productivity and other company initiatives. If women feel like they aren’t being heard or recognized for their efforts, you may see their productivity start to slip. A gender-inclusive culture extends to hiring efforts as well. Cast a wide net during new hire searches. Don’t immediately reject someone with a “unique” or less common background and consider AI tools to ensure your job postings aren’t using discouraging language.
• Dedicate company resources to training and educating. Whether you’re bringing in outside speakers or teachers to present information or allowing women to pursue additional degrees or certifications, an environment of continuing education can lead to unparalleled success.
• Introduce opportunities to contribute to AI projects within the company. Far too often, there’s a small circle of trust when it comes to AI, which often leads to unconscious biases, opening up potential pitfalls. Allow women of all levels to participate to bring in diverse perspectives.
• Highlight women trailblazers in the AI space. It’s hard to know about some of the career paths or opportunities available if you’ve never seen anyone else accomplish them. Initiatives like Women’s History Month shine a light on women doing wonderful things, but we can continue those celebrations throughout the entire year. Bring in outside speakers, encourage the reading of women authors within AI, and look for other ways to amplify the good work of women inside and outside your organization.

Leaders must also realize the benefits of having women in AI and leadership positions. Companies with at least one diverse hire — with a focus on women — see a 44% increase in their average share price within a year of going public, compared to 13% that don’t have a diverse hire.

Additionally, having more diverse voices in AI can introduce new ideas, career paths, products, and services. This space is still so new that we’re able to shape it in exciting ways, though it won’t happen if we don’t create an inclusive environment.

As long as AI and machine learning lack diverse perspectives, they will produce biased results. It may not even be obvious, but too much of the same thinking will miss the same blind spots, limiting the effectiveness of AI models.

From an educational perspective, teachers, mentors, and even journalists can highlight how regularly we use AI, data science, and STEM topics and present them in ways that are approachable. The following organizations are doing their part in that regard:

• Women in AI combines education with research and networking and highlights why having a support system is so critical. I worked with Women in AI – Mexico in building and distributing the WaiCAMP by DataRobot University program. The Women in AI network of more than 5,000 members has strong relationships with tech companies like AWS, Microsoft, and IBM and their experience with community, events, education, and logisitics make them a perfect partner!
• GeekPack teaches the importance of gaining skills while building a community through coding. No one ever needs to go it alone on their AI, STEM, or data science journey.
• HipHopEd introduces STEM subjects through a hip-hop lens to build upon students’ existing interests. For example, it showcases the need for engineering skills to enter into a field like designing shoes. Most students would never put those two disciplines together!
• QA’s Teach the Nation to Code delivers talent and training services to help individuals and companies thrive in the digital revolution.

To be most effective, this education needs to start early—and it must be consistent.

Understanding AI in Daily Life

Before any of those advancements occur, though, we must take a step back and realize AI is penetrating our lives, sometimes in ways we don’t even recognize. From a citizen data scientist perspective, the more you understand how AI, machine learning, and algorithms work, the better it will help in your daily lives.

Chatbots such as Capital One’s Eno automate basic queries while also providing personalized recommendations and goals. These chatbots help reduce wait times (and frustration levels) of customers looking for financial information.

AI-powered mortgage advisors use machine learning to determine whether a customer will qualify for a mortgage, suggest proper tools and products, and give tips for boosting credit scores. Buying a home is a big commitment and takes a lot of research, so these tools help expedite that process.

Even the DMV — one of the most notoriously slow services we deal with — has used AI-backed tools like edge-based customer forms and interactive voice response to avoid those long lines and make our lives easier.

The above examples are just a few of the ways AI is becoming more ingrained into our lives, and we’re still only scratching the surface of what’s possible. Yet those possibilities will be limited unless we get more diversity into AI. The all-encompassing AI landscape makes it critical to have more women in the field, avoiding unconscious bias and creating more well-rounded and comprehensive AI models.

When everyone is able to meaningfully contribute to our industry, success is far more likely. I encourage you to be curious about AI — you just might be surprised where it can take you.

The post Addressing the Gender Gap in AI appeared first on DataRobot AI Platform.

Editor’s Note – This post was updated March 22nd to check in on the results of our model. 

It’s that time of year again. Since I was a kid, I’ve always been swept away by March Mania. I’m not much of a college basketball fan, but I remember printing out brackets with my dad and brothers, filling them in with our best picks and watching the games together with my family.

It was a great way to bond, and it’s part of what makes March Mania so special. You might not know your Tigers from your Wildcats or your Racers from your Volunteers, but there’s a special camaraderie that comes from completing brackets with family, friends, and colleagues.

There’s only one problem: knowing who to pick. Very few of us watch many (if any) games during the season, and there are so many aspects of gameplay to consider. It can be tough to lose to someone who picked their winners purely based on team mascots or their favorite vacation destinations. And while it’s difficult to win against people who follow every bounce of the ball, AI and machine learning can help you make picks as informed as a casual basketball fan.*

That just may be enough of an advantage to win your bracket pool. To be clear, this isn’t a strategy to make you rich by betting all your money—let Mattress Mack serve as a cautionary tale for that—but it’s a fun way to be competitive with your friends and family who spend the entire season watching college basketball and know every team.

For the past decade, I’ve used the data of past tournaments to build a machine learning model for completing March Mania brackets. These models seem to be about as well-informed as a casual basketball game, and it’s helped me compete more closely with my family members who pay a lot more attention to basketball than I do.

Here’s what’s most helpful to look at.

Building the March Mania Model

Using a logistic regression model, I’ve developed a way to predict games throughout the tournament. Of course, it’s still not possible to be correct with 100% of your picks, but this can help provide some clarity if you’re stuck on who to select when making your picks.

With 64 teams in the tournament—for our purposes, we’re discounting the First Four games on Tuesday and Wednesday—there are 4,096 potential matchups. That’s a lot of variance, so it’s best to keep it simple.

The model runs a Monte Carlo simulation of 10,000 games and consists of two main elements:

• Power ratings: These ratings are something I’ve worked to hone over the years for both the men’s and women’s tournaments. They look at data from the regular season and are based on Ken Pomeroy’s methodology.
• Point spread: For the men’s Tournament, the Las Vegas odds for the initial games are a great source of data for the first round of the bracket. If a team is favored to win by 25 points, they’re very likely—though not certain—to win. If a team is only favored to win by one or two points, the game will likely be close.

Running this model, here’s a look at my predicted bracket for the Men’s Tournament:

Here are my top 10 favorites to win this year’s Men’s Tournament, and their probability of winning.

1. Gozanga – 26%
2. Arizona – 10%
3. Kansas – 7%
4. Baylor – 7%
5. Houston – 7%
6. Kentucky – 6%
7. Tennessee – 5%
8. Auburn – 4%
9. Texas Tech – 4%
10. Villanova – 4%

If you’re looking for some potential upsets in round one, here are the most likely surprises for the men’s bracket:

• Michigan (11) over Colorado State (6) – 59% chance of upset
• Memphis (9) over Boise State (8) – 59% chance of upset

Here is my predicted bracket for the Women’s Tournament:

Here are my top 10 favorites to win this year’s Women’s Tournament, and their probability of winning:

1. South Carolina – 36%
2. NC State – 19%
3. Stanford – 16%
4. Connecticut – 7%
5. Louisville – 5%
6. Texas – 3%
7. Baylor – 3%
8. North Carolina – 2%
9. Iowa State – 1 %
10. BYU – 1%

If you’re looking for some potential upsets in round one, here are the most likely surprises for the women’s bracket:

• Princeton (11) over Kentucky (6) – 51% chance of upset
• South Florida (9) over Miami (8) – 53% chance of upset
• Georgia Tech (9) over Kansas (8) – 59% chance of upset
• Kansas State (9) over Washington State (8) – 66% chance of upset

Other models look at some of the more traditional basketball metrics. While they may provide a bit of value, they also offer some challenges:

• If you go by seeding alone, you won’t pick any upsets correctly. For example, 37.5% of 11 seed teams beat 6 seeds since 1985. Sometimes, the matchup is such that the worst seed—which would be viewed as inferior by a seeding-centric model—is actually the favorite to win the game.
• You could also look at the win-loss record of each participant. A team like, say, Gonzaga or Auburn looks really good, sporting a high winning percentage. However, both of those teams lost in the final two weeks of the regular season, so you could argue they’re not playing at their best. Similarly, other teams with shaky records may suddenly be playing very well, but it wouldn’t look that way based on record alone.
• Wins and losses also don’t account for margin of victory—a one-point victory is typically a coin flip. A team that won a lot of games by a single basket may not be nearly as strong as their record suggests.
• Other elements like the final AP or Coaches Poll rankings provide a nice overview of where teams currently are in the standings, but that’s based on their recent play, which is primarily against teams in their own conferences. Those intra-conference matchups rarely happen until later rounds, and they never occur in the first round, so poll rankings don’t provide a full picture.

Putting the Data to Work

The current format of the Tournament, with a 64-team bracket, began in 1985. Current box score data goes back to the mid-80s, and point spread data reaches back to 2003. As we play more tournaments, we’ll continue to get more data.

However, simple models seriously outperform complex models. More stats usually don’t lead to better results—in fact, it’s quite often the opposite. I’ve tried hundreds of different, crazy stats over the years, and they almost always impede the model. Selecting one or two highly informative variables is consistently a better move than trying to load up models with too much data. This is a really, really hard dataset to model—there’s a LOT of noise and very little signal. Keeping your model simple helps cut through the noise, but you will never be able to predict basketball game outcomes with a high degree of certainty.

For me, I’ve found betting spreads and power ratings provide the best results. If you’ve developed a model, you might look at the distance each team must travel to play their tournament games, or the number of senior guards a team has on the roster. But keep it simple—limit yourself to one or two highly informative variables.

This is also a good reminder that in statistics, “very rare” is not the same thing as “impossible.” For years, bracket experts have touted picking a number 1 seed to win its first-round game. Since the tournament field expanded to 64 teams in 1985, every No. 1 seed has beaten every No. 16 seed for 33 straight years.

When No. 16 seed UMBC knocked off No. 1 Virginia—a team that had only lost two other games all year—in 2018, it shocked a lot of college basketball fans. They thought such an upset couldn’t happen. But, looking at the model, Virginia had a 95% chance to win that game. In other words, UMBC had a 5% chance, or would record one win in every 20 games against Virginia. Their victory was certainly a long shot, but it wasn’t impossible. In fact, you should expect a 1 vs 16 upset to occur roughly every 5-10 years in the tournament, although it’s impossible to predict which year exactly will feature such a huge upset.

One other wrinkle from incorporating too much data—every time you add a dataset, you have to account for how all that data is entered and labeled. Let’s take the Saint Mary’s Gaels, for example. One dataset lists them as Saint Mary’s. Another may write St. Mary’s. A third may do SMC for Saint Mary’s College, while yet another does St Marys.

On top of that, you have to make sure you’re actually getting data from the St. Mary’s College of California in Moraga, not one of the other 14 St. Mary’s colleges and universities across North America.

So, again, I suggest you keep any machine learning model simple. Basketball is a noisy game, and simpler models deal well with noise. My dad likes to say, “they’re just kids!” You were likely more unpredictable when you were 18 to 21 years old, and March Mania is similarly full of crazy, wild moments. Yet, that’s part of what makes the games so fun to watch.

If you develop your own model to try and solve the Mania, don’t overcomplicate things. Pick one or two datasets to work with and see how that goes. You can always iterate down the road.

Best of luck as you fill out this year’s bracket. Now, let the games begin.

March 22nd Update:

We’re back and checking in on the performance of our model throughout the tournament. Let’s dive into the men’s tournament:

Overall Model Performance:

Loading the actual results of rounds 1 and 2, we can see how the predictions vs. actuals performs for both positive and negative class.

1. The predictions vs. actuals performed well for both positive and negative class

Class 0:

Class 1:

Actual Round Results vs Predictions:

I loaded the pairs for each round using a batch prediction job. 

My tournament data is stored in snowflake so in this case, I used a prediction job to write the results to Snowflake so I can have all the data in one place. 

This is how we can get the predictions for every round and compare them later with the actuals.

The beauty of march mania is the big surprises, everyone can win when you play one game.

Below are the results of the model’s predictions vs the actuals:

Round 1:

Successfully predicted 25 games over 32 total

The Surprises

St Peter’s beats Kentucky

No. 15 seed beat No. 2 seed. Not many predicted it. Over 12% of brackets had Kentucky playing in the national championship game, and over 6% had them winning it all.

Richmond beats Iowa

Another big surprise here, No. 12 seed beat a No. 5 seed.

Notre Dame beats Alabama

Notre Dame keeps their momentum.

New Mexico State beats Connecticut

No. 12 seed beat a No. 5 seed. The first 12-over-5 upset in the modern era actually came in the first year of the 64-team tournament, in 1985.

Close Predictions

Miami FL beats USC

USC – Miami –  very close game! These two teams are very similar. Miami won only by 2 points.

TCU beats Seton Hall

Again we see a very close prediction (No. 8 seed vs. 9 seed) TCU was underrated as number 9 and still won.

Creighton beats San Diego State

Final score 72-69; only 3 points difference. Creighton had an amazing comeback to win this game.

Iowa State beats LSU

LSU’s coach was fired one week prior to the tournament, which may have contributed to this loss.

Round 2:

Successfully predicted 11 games over 16 total

The Surprises

St Peter’s beats Murray State

St Peter’s continues to surprise!

North Carolina beats Baylor

A surprise, as Baylor is the defending national champion.

Miami FL beats Auburn

Another surprise from Miami, No. 10 seed beating a No. 2 seed.

Michigan beats Tennessee

A No. 11 seed beat a No. 3 seed.  Michigan played well on defense.

Round 3 Predictions

Here’s our predictions for the next round:

Let’s see how well we do for this round!

March 29th Update:

It’s me again! And I’m about to give you an update on the coming final four and the overall winner.

But before that let’s take a look at the predictions and actuals in rounds 3 and 4, how we did and what we can learn from it (men’s tournament):

Round 3:

This round was one of the most surprising rounds in the history of the tournament

This is the mania of march – unlikely events can happen!

With all the surprises above, there are some changes we needed to make to the point spreads and the simulation.

Round 4:

Things are back on track! Successfully predicted 3 of 4 games.

Final Four

So what are we expecting in the final four?

Mens Tournament:

Kansas has a more challenging game than Duke as it’s a seed 1 vs 2, compared to 2 vs 8.

Women’s Tournament:

With MLOps, we are still controlling our model in production, and our graphs are keeping updating for accuracy – logloss is getting better.

So far so good! Overall in 108 games, we predicted 70% of the games correctly, which is better than random1.

After we updated our simulation here are the updated winners:

Men:
Kansas – 42%
Duke – 36%
Villanova – 18%
North Carolina – 3%

Women: (based on Nate Silver’s simulation)
South Carolina – 54%
Stanford – 21%
UConn – 15%
Louisville – 10%

*DataRobot gives no warranty as to the accuracy, correctness, or completeness in live operation of any Model used by the Solution or predictions made by the Solution. The accuracy of the Models and any generated outcomes created by the Solution is dependent on the data used.

*The National Council on Problem Gambling operates the National Problem Gambling Helpline Network (1-800-522-4700). The network is a single national access point to local resources for those seeking help for a gambling problem. The network consists of 28 call centers which provide resources and referrals for all 50 states, Canada and the US Virgin Islands. Help is available 24/7 and is 100% confidential.

The post Building a March Mania Bracket Using Machine Learning appeared first on DataRobot AI Platform.

When listening to data scientists discuss the inherent flaws in existing AI models, you’ll often hear the phrase, “All models are wrong, but some are useful.” Reality may be complicated, and humans complex, but in the end it’s the illuminating results that are necessary for progress.

It is accepted wisdom that—despite being crafted with as much skill and artisanship as a ceramics master—some models, when faced with the complexities of reality, break. Likewise, kintsugi is considered a contemplative approach to dealing with the repair of something useful, in a way that highlights the cracks and is transparent to everyone.

Perhaps there are other lessons here that we can take from art and literature. Is there a way to embrace the spirit of this methodology in our system’s design so they reflect this repair? In kintsugi, the gold lines create a completely new look for the bowl. It isn’t about tweaking a model in production as much as it is about identifying what breaks when we don’t expect it to and identifying what we should do about it. In machine learning, the “art of repair” needs to address aspects of transparency in order to create trust. The gold lacquer equals the ethical bonds that make our products stronger by highlighting the efforts of the past.

Pattern Tracking is the Artistry

Both recognizing patterns and historical record-keeping have been critical to the evolution of civilizations. History is fundamentally about the passage of time. In kintsugi, the history of an object doesn’t change its fundamental shape, nor does the repair. Historically, repeating patterns deepens the pattern’s importance as historical records keep civilizations accountable for those patterns and the changes they bring about. This awareness becomes an evolutionary advantage by creating landmarks for our memories—those who forget the past are doomed to repeat it, as the aphorism goes. Kintugi incorporates its past by wrapping itself around its history. Can AI annotate itself to provide us that awareness?

To do so, we must review the errors made in our algorithms over time within the context of repairing a broken vessel. If we simply discard or hide the lessons being learned now by our algorithms, society will not benefit and learn from those mistakes in the aggregate.

Our instinctive, cultural bias against defects challenges our ability to err, adjust, and improve. When things break, we throw them away. Many of the products we develop are designed to be disposable when the repair costs as much as the original device. Our technology becomes wasteful when it could be repurposed, but this doesn’t happen.

What impact could this awareness have on the tech industry’s approach to AI? Rather than sensationalize or shame waste through “reputation management,” there should be methods to capture—and even highlight—those cracks in the model. If auditors are interested, let them easily see the repairs. Presenting them with an established willingness to “emphasize the broken pieces” opens the door to constructive, supportive feedback.

How can we design our machine learning systems to do this? By creating a virtuous loop of feedback and improvement that, in turn, can validate a trusted model.

Some possibilities include:

• When building AI, we could shut down a project and tag a model as inadequate by agreeing not to move it to production. When an organization champions this kind of decision without negative consequence, it allows for greater experimentation and more possibilities. Rather than seeking perfectionism and failing, acknowledging the lack of benefits opens the door to other AI/ML projects that could be more impactful.
• We could consider the net risks of the system instead of a single risk. What is the “cost” (financially, reputationally) of repairing all the tiny broken pieces in comparison to the benefits the system is providing to the organization and society. In engineering terms, this is similar to the concept of “technical debt” which, over time, can become too difficult to maintain but incurs hidden costs in terms of reliability and usability. Knowing at what point it is appropriate to throw in the towel (or having the bravery to admit it is time), is key to understanding the weight of one snowflake error against a world of possible good.
• We could evaluate whether a particular AI model is worth salvaging. Is this particular broken vessel absolutely necessary, or is there perhaps a better material or vessel that could entirely replace the one under repair? This lends the focus to the right solution for the problem, rather than whether or not it will work in the first place.
• Rather than focusing on the technical aspects (such as the craftsmanship of the repair), we could set the use case as the target for continuous improvement. As an anthropologist might trace the lineage of a broken pottery shard by understanding the entire environment—the application, the contents, and the history, each use case improvement adds to the overall understanding of the alterations to our systems over time.
• Maybe there is a way we could create general awareness of where the system broke down in the past. By doing so, we provide assurance that our skills and attention have created a pot that will not break again in the same place (although it may still break in a different spot or in a different way). Leveraging existing risk analysis and mitigation tasks that already provide this type of insight could be adapted to include a historical record, and strengthen trust in the system.
• Do we believe that we could allow larger fault lines to become the most visible? While breakage over time is an inevitable risk, this evaluation is less about whether the pot is broken than it is about how we communicate that repair. What is or should be most visible to the community? How can we acknowledge that repairs may not result in something usable or even recyclable without creating more discord? The largest fault lines are what create the “character” of the system. What humans track must be conscious and intentional.

Strength Begets Trust

Everyone agrees that in order to adapt AI to your needs, you have to be able to trust in its outcomes.

If repairing bias in our models makes our systems stronger, then trust is what evolves from that strength. Likewise, what we construct evolves with us.

An essential criteria of that trust is the ability to communicate meaning. Humans cultivate trust through signals and by communicating clearly: by speaking plainly and avoiding buzzwords, by anticipating issues, sharing or acknowledging difficulties, and expressing the truth, even when it’s easier not to. A trusted advisor who shares outcomes with honesty, humility, and a dash of optimism is far more compelling than one who obfuscates conclusions with bravado or hides the facts.

Do we really trust that mistakes won’t happen with our models despite our tacit awareness that changes will occur regardless? Of course not. You don’t need to have been the one to have made a mistake for it to impact you. Both the mistake’s author and the downstream casualties must all view those golden lines reflecting the errors as paths for learning together.

Imagine if trust lived inside a database—a gallery of broken lines humanity can look at, dissect, and learn from. If we can see and embrace the bias in the overall design, then we must find a way to acknowledge and codify our learning through communication.

For example, the last decade has seen a rise in the use of virtual assistant technology. Amazon, Google, and Microsoft all sell these devices and all are trained on similar natural language processing that uses sophisticated algorithms to learn from data input to become better at predicting future needs. Recently in the news, Amazon’s Alexa assistant directed a ten-year-old to do a potentially lethal challenge. Amazon specifically responded, “Customer trust is at the center of everything we do and Alexa is designed to provide accurate, relevant, and helpful information to customers,” a company spokesperson told Insider. “As soon as we became aware of this error, we quickly fixed it, and are taking steps to help prevent something similar from happening again.”

This is a reactionary response to a public demand. Could the concept of kintsugi highlight this error so that the fix is transparent to all future developers and recorded for posterity outside of the sensationalized and reputation-damaging influence of a media expose?

Last July the AI Incident Database released the First Taxonomy of AI Incidents as a way to bring transparency to the forefront. It is a brilliant starting point—allowing users worldwide to acknowledge the breaks in their vessels. Many incidents are submitted anonymously and, according to the site, more than half pertain to “systems developed by much smaller organizations.” There are no fixes included as part of this database, only a record of incidents and that damage was done. What if the next step was to propose, debate, or simply include solutions? This could be a beginning to the “art” of repair.

It’s human nature to be resistant to sharing mistakes. Countless models never make it into production because the risk of failure is so high as to be intolerable. A choice between admitting or burying failure. No organization willingly airs their blunders by offering them up for external, out-of-context headlines. The Incident Database approach could be one method of historic log keeping reflective of kintsugi. It would allow future developers to see and value experiments over time (measured in decades not moments). Technical kintsugi could provide trusted explanations and repairs for those examining AI applications in the future.

The Future of Repair

The premise of kintsugi has lasted centuries, not just because of its technical execution, but because it says something about humans as creators. For AI, the art of repair can become the fundamental principle that ultimately influences people’s trust in machine intelligence.

Data science can collectively change the AI narrative from fear to acceptance. Let’s illustrate how making better choices, and improving ourselves in the process, is possible if we can communicate trust artfully and for the long haul. Motivation should be based on concern for not serving the greater good, rather than the fear of making mistakes. We need a dialog that emphasizes continual improvement by acknowledging mistakes made in the past:

Everyone craves a more efficient transaction. Improvisation is dead and curiosity is mortally wounded. If it’s awkward then it’s bad, most people think, when in fact the opposite is true: Awkwardness signals value, a chance to discover something unusual, something unnerving, in the silence between…the cringe.” – Heather Havrilesky

When it comes to work, or intelligence, or physical appearance, or machine learning, our culture sets the standards for perfection. People are quick to judge others for missing the mark, for displaying their awkwardness. Perhaps in the future, in the spirit of kintsugi, we can create repair systems that embrace mistakes by documenting the countless examples of how things can go wrong and how, by communicating honestly and clearly, how they were repaired.

The post Kintsugi — the Art of Repair appeared first on DataRobot AI Platform.