CWE-1426
Improper Validation of Generative AI Output
AI Translation Available
The product invokes a generative AI/ML
component whose behaviors and outputs cannot be directly
controlled, but the product does not validate or
insufficiently validates the outputs to ensure that they
align with the intended security, content, or privacy
policy.
Status
incomplete
Abstraction
base
Affected Platforms
AI/ML
Not Technology-Specific
Technical Details
AI Translation
Common Consequences
integrity
Impacts
execute unauthorized code or commands
varies by context
Detection Methods
dynamic analysis with manual results interpretation
dynamic analysis with automated results interpretation
architecture or design review
Potential Mitigations
Phases:
architecture and design
operation
build and compilation
Descriptions:
•
Use "semantic comparators," which are mechanisms that
provide semantic comparison to identify objects that might appear
different but are semantically similar.
•
During model training, use an appropriate variety of good and bad examples to guide preferred outputs.
•
Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space.
•
Use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.