Xavier Conort on maximising your machine learning success with innovative feature engineering

Xavier Conort is an esteemed actuary, Kaggle Grandmaster, DataRobot Chief Scientist (2013-2020), and co-founder and Chief Product Officer of FeatureByte. On his recent visit to Sydney, Xavier gave an inspiring yet practical talk on feature engineering for machine learning.

What are Data Science Sydney Meet-Ups?

Data Science Sydney Meet-Ups are not just monthly gatherings for the data science community but is also an opportunity where actuarial and non-actuarial data scientists come together to explore the potential of data through advanced statistical, computational, and analytical tools.

Participants engage in networking, knowledge sharing, and while enjoying pizza! These sessions are usually held at the Actuaries Institute, but this month due to an increase of attendees, we met at the University of Sydney Business School.

Xavier began the session by introduced himself and outlined that the topics he would cover including the creative process of feature engineering, the importance of organising ideas, and introducing FeatureByte – an open-source Python library created by Xavier to assist with feature engineering.

What is feature engineering? It’s the process of selecting, manipulating, and transforming raw data into features that be used in machine learning models.

Feature engineering: A creative process

Xavier emphasised that feature engineering is a highly creative process and clarified that creativity does not solely rely on generating original ideas.

Quoting Steve Jobs, who said, “Creativity is just connecting things,” Xavier suggested that originality isn’t the only aspect of creativity, but rather, it involves combining separate ideas.

Reflecting on his success as a Kaggle Grandmaster, Xavier shared techniques that he employed that were inspired from a range of sources; from past and present Kaggle Champions to decades old academic papers.

Xavier mentioned that while he thoroughly enjoys the creative process of feature engineering, the audience should not overlook knowledge shared by peers and established techniques. As a learnable example, Xavier shared learnings from his record on the “Give Me Some Credit” Kaggle challenge, where he built a credit scoring model that has remained at the top of the competition leader board for over 10 years. He attributed the inspiration of techniques used and his success in this challenge to his peer and previous Kaggle Grandmaster, Owen Zhang, as well as the use of stacking, which he discovered in a research paper dating back over 20 years.

Name and organise your ideas

While Xavier takes a creative approach to generating feature ideas, he enhances this process through a systematic categorisation of his ideas. Xavier found it useful to classify his thoughts and feature ideas into distinct signal types. Each signal type is tabulated below with a brief description and example.

FeatureByte

FeatureByte is a free and open-source Python package designed to empower data enthusiasts who love creating innovative features and improving model accuracy through data. It enables users to quickly create and share features, experiment seamlessly with features, and effortlessly deploy features in production without the challenges of managing data pipelines.

From a technical standpoint, the package allows users to easily create features for each of the signal types discussed earlier. Defining features is swift as no data needs to be held in memory. The SQL code required to obtain a feature is executed only when you “save” it, which is much faster than the corresponding pandas manipulations. Once users grasp the package’s syntax, it becomes a fast and easy way to develop features for machine learning models. Additionally, there is an enterprise solution built on top of the open-source package, offering a comprehensive framework for managing and organising AI operations at scale, including governance workflows and a user interface for the feature catalogue. If you’re interested, you can check it out here.

Final words from Eugene

Eugene Dubossarksy, the founder of Data Science Sydney, concluded the session by recognising Xavier as an exceptional figure in the field of data science, epitomising an extraordinary balance of humility and expertise. Eugene also reminded the audience that AI is not limited to just Large Language Models (such as GTP-4). Many organisations still have a significant need for machine learning models and will benefit from platforms like FeatureByte.

CPD: Actuaries Institute Members can claim two CPD points for every hour of reading articles on Actuaries Digital.