CWE-1434

Insecure Setting of Generative AI/ML Model Inference Parameters
AI Translation Available

The product has a component that relies on a
generative AI/ML model configured with inference parameters that
produce an unacceptably high rate of erroneous or unexpected
outputs.

Status
draft
Abstraction
base
AI/ML Not Technology-Specific

Generative AI/ML models, such as those used for text generation, image synthesis, and other creative tasks, rely on inference parameters that control model behavior, such as temperature, Top P, and Top K. These parameters affect the model's internal decision-making processes, learning rate, and probability distributions. Incorrect settings can lead to unusual behavior such as text 'hallucinations,' unrealistic images, or failure to converge during training. The impact of such misconfigurations can compromise the integrity of the application. If the results are used in security-critical operations or decisions, then this could violate the intended security policy, i.e., introduce a vulnerability.

Common Consequences

integrity other
Impacts
varies by context unexpected state alter execution logic

Detection Methods

automated dynamic analysis manual dynamic analysis

Potential Mitigations

Phases:
implementation system configuration operation documentation
Descriptions:
• Develop and adhere to robust parameter tuning processes that include extensive testing and validation.
• Provide comprehensive documentation and guidelines for parameter settings to ensure consistent and accurate model behavior.
• Implement feedback mechanisms to continuously assess and adjust model performance.