Skip to main content
Scorecard uses a hierarchical structure to organize your evaluation work: Organizations contain multiple Projects, and each Project contains your testsets, metrics, runs, and results.

​
Organizations

Organizations are the top-level entity in Scorecard that enables team collaboration and workspace isolation. All members of an organization can access all projects within that organization. API keys are scoped to a specific organization, not user or project.

Organization member management dialog with a few users.

​
Projects

Each project serves as a container for:
  • Testsets: Collections of Testcases for evaluation
  • Metrics: Custom evaluation criteria and scoring methods
  • Runs: Evaluation executions and their results
  • Prompts and Systems: The AI system versions being evaluated
  • Traces: Live traces and logs of AI system calls

Projects overview page for an organization.

​
Creating a New Project

To create a project, click “New Project” in the top right of the Projects page and fill in the project details.
Choose a descriptive project name for the system or use case you’re evaluating, such as “Customer Support Chatbot” or “Product Recommendation Engine.”

Creating a new project.