Dr. Niclas Kannengiesser studied industrial engineering and computer science at the University of Kassel and completed his Ph.D. in information systems at KIT in 2024. With a technical background in blockchain and decentralized applications, his research focuses on meaningful decentralization of information systems, aiming to prevent monopolies and data hoarding. He has a particular interest in collaborative distributed machine learning, which allows training parties to retain data sovereignty while still sharing data collaboratively.

The Concept of Heuristics in Hyperparameter Optimization

In his talk, Dr. Kannengiesser drew on the ideas of Gerd Gigerenzer, a prominent psychologist who has studied the role of heuristics—simple decision-making rules—in human decision processes. Gigerenzer showed that heuristics, despite their simplicity, often yield good results. Dr. Kannengiesser applies this concept to machine learning.

Hyperparameter optimization (HPO) is a critical but complex step in ML model development. Hyperparameters are settings defined before training and are key determinants of model performance. For example, the "learning rate" specifies how quickly a model learns from data, impacting the robustness of the knowledge it acquires. Choosing the right hyperparameters can be time-consuming and resource-intensive. By using heuristics, developers can streamline this process by relying on proven strategies that yield good results in specific contexts.

Methods of Hyperparameter Optimization and Practitioner Motives

Various methods for hyperparameter optimization differ in complexity and approach:

1. Manual Optimization: Developers adjust hyperparameters based on experience and intuition.
2. Grid Search: A systematic search through a predefined grid of hyperparameter combinations.
3. Random Search: Random selection of hyperparameter combinations within a defined search space.
4. Bayesian Optimization: Probabilistic modeling to predict the most promising hyperparameters.
5. Evolutionary Algorithms: Application of evolutionary principles to iteratively improve hyperparameters.

Through interviews and surveys, Dr. Kannengiesser explored the motivations behind practitioners' choices of HPO methods. He identified several key goals:

• Performance Improvement: Maximizing the accuracy and efficiency of the model.
• Understanding: Gaining in-depth insight into the effects of individual hyperparameters.
• Low Computational Cost: Minimizing required computing resources.
• Reduced Effort: Minimizing time and labor.
• Compliance: Adhering to standards and regulatory requirements.
• Target Audience Requirements: Meeting the specific expectations of clients or colleagues.

The Model: Matching Goals to Methods Based on Context

A central part of Dr. Kannengiesser’s talk was a model that shows how practitioners consider their goals and context to select the appropriate HPO method. This model is based on three main factors:

1. Knowledge: The developer's expertise in ML and HPO.
2. Social Environment: Influences like peer acceptance, competition for resources, or established practices within the community.
3. Technical Resources: Availability of computational power, time, and suitable tools.

Using these factors, developers can choose the HPO method that best suits their goals. For example:

• A developer aiming to minimize computational costs with limited resources might choose Random Search, as it requires fewer resources than more complex algorithms.
• A developer pursuing maximum performance with ample resources may opt for Bayesian optimization to efficiently find the best hyperparameters.
• If model understanding is prioritized, manual optimization may be chosen, as it allows a deeper exploration of parameters.

Dr. Kannengiesser observed that the method chosen in practice often differs from the one theoretically most suitable. This can be due to factors like lack of knowledge about alternative methods, established habits, or social environment influences. Some developers continue to use manual optimization despite the efficiency of automated methods because they feel more comfortable with it or because it is customary in their environment.

Challenges in Collaborative Machine Learning

In collaborative distributed machine learning, where multiple participants train models together without sharing raw data, hyperparameter optimization becomes even more complex. Methods like federated learning or collaborative distributed machine learning enable collaboration while preserving data sovereignty but also introduce new challenges:

• Diverse Goals and Contexts: Each participant may have different objectives, resources, and technical requirements.
• Limited Information Exchange: Since data and sometimes even model architectures are not fully shared, coordinating HPO becomes challenging.
• Trust Issues: Selecting reliable collaborators and ensuring fair contributions become essential.
• Technical Complexity: HPO must be designed to be effective under these conditions.

Dr. Kannengiesser emphasized that new approaches and methods for hyperparameter optimization are needed in such scenarios to meet the diverse requirements. His current research focuses on developing solutions that consider both the technical and social aspects.

Conclusion

Dr. Niclas Kannengiesser's research talk offered an in-depth look into the complex world of hyperparameter optimization in machine learning. He highlighted the importance of making a conscious choice of optimization method, considering one's objectives, knowledge, social environment, and technical resources. By applying heuristics and understanding the various influencing factors, developers can create more efficient and effective ML models.

Especially in collaborative settings, hyperparameter optimization presents a significant challenge requiring innovative approaches. Dr. Kannengiesser’s work contributes to understanding these challenges and developing solutions that address both the technical and social dimensions of machine learning.