AI Analytics
Monitor AI generation usage, quality, users, exports, and rate limits.
Use AI Analytics to review how AI generation is being used across your organization. The page is designed for admins who need to monitor generation volume, quality, token usage, active users, and rate-limit risk.
Before you begin
You need:
- Access to
/admin-dashboard/ai-analytics. - One of these permissions:
ai:view_analytics,ai:*, oradmin:*. - The AI Analytics feature entitlement. The page is available on Growth, Professional, and Enterprise plans.
- AI generation activity in the selected time range.
If you do not have permission, the page shows You don't have permission to view AI analytics. Admin access required. If your plan does not include the feature, the page shows an upgrade prompt.
Read the dashboard
What the page tracks
AI Analytics focuses on organization-wide generation activity.
It shows:
- Total generations.
- Successful generations.
- Failed generations.
- Average quality score.
- Generation types.
- Active users.
- Top content generators.
- Token usage over time.
- Organization usage against configured limits.
- Exportable summary data.
The dashboard refreshes automatically every minute while the page is visible.
Choose a time range
Use the range selector in the AI Generation Analytics header.
Options:
| Range | Data window |
|---|---|
| Daily | Last 1 day. |
| Weekly | Last 7 days. |
| Monthly | Last 30 days. |
After changing the range, the charts and cards reload for that window. Use Refresh when you need to pull the latest data immediately.
Read the key metrics
The top metric cards summarize current generation health.
| Metric | What it means |
|---|---|
| Generations | Total AI generation requests in the selected range, with successful count shown below. |
| Quality Score | Average quality score for generated output. |
| Active Users | Number of users who generated content in the selected range. |
The Generations progress bar shows successful generations as a share of total generations. If total generations increase but successful generations do not, investigate failures before increasing usage.
Use Usage Trends
Open the Usage Trends tab to review volume patterns.
This tab includes:
- Token Usage Over Time: token consumption and generation count over the selected range.
- Generation Types: breakdown of generation activity by type.
- Hourly Distribution: distribution by hour.
Use this tab to answer questions like:
- Did AI generation spike after a launch or bulk operation?
- Which generation type is most common?
- Are admins generating content during expected working hours?
- Does usage match the operational period you intended to review?
If token usage is high, compare the spike against AI Operations to see whether a bulk operation or repeated generation run caused it.
Use Quality Metrics
Open Quality Metrics to review whether generation output is meeting the expected quality bar.
The tab shows:
- Average Quality.
- Success Rate.
- Estimation Accuracy.
- Overestimations.
- Underestimations.
The page also shows an alert when estimation accuracy is low enough to warrant adjusting the buffer percentage.
Use this tab when:
- Generated content requires too much manual cleanup.
- Users report inconsistent generated output.
- Failures are increasing.
- Cost estimates do not match observed usage.
The page guidance treats 70% as the minimum quality score. Review failed generations and source materials when quality drops below your expected bar.
Use User Activity
Open User Activity to see Top Content Generators.
For each user, the list shows:
- Rank.
- Email.
- Number of generations.
- Tokens used.
Use this tab to find:
- Heavy users who may need guidance.
- Unexpected generation activity.
- Teams creating unusually large volumes of content.
- Candidates for quota or process review.
This list is not a quality judgment by itself. Pair it with Quality Metrics and AI Operations before changing access or usage policy.
Use Rate Limits
Open Rate Limits to compare current usage with limits.
The tab shows:
- Organization Limit: organization token usage against the configured organization limit.
- Global Platform Limit: platform usage against the global limit.
When organization usage passes 80% of the limit, the page shows a warning: Organization approaching rate limit (80% used). Consider increasing limits or optimizing usage.
At or near the 80% threshold:
- Check whether a bulk generation operation is running.
- Pause non-urgent generation work.
- Review the top generators.
- Use caching and batching where possible.
- Request a limit increase before peak usage periods.
Act on analytics
Export AI analytics
- Open AI Analytics.
- Choose Daily, Weekly, or Monthly.
- Select Export.
The page downloads a CSV named like:
ai-analytics-<range>-<timestamp>.csvThe export includes summary rows for:
- Total Generations.
- Successful Generations.
- Average Quality.
- Active Users.
- Organization Usage.
- Estimation Accuracy.
Use this export for leadership summaries, usage audits, or month-end review. It is a summary export, not a row-level generation log.
Cost optimization checks
The page includes operational reminders for reducing AI generation costs:
- Use caching for similar content requests.
- Batch generation requests when possible.
- Use the least complex model that still meets the quality target.
- Use progressive generation.
- Set quality thresholds before launch so reviewers know when to pause or rework a generation workflow.
Apply these checks when token usage grows faster than content output or when a few users account for most generation activity.
Quality review checks
Use these checks when quality score or success rate drops:
- Review failed generations.
- Confirm source materials cover the topic, audience, and policy version used by the generation request.
- Use example courses for consistency.
- Validate generated learning objectives.
- Monitor user feedback before scaling the same prompt pattern.
Quality problems are often easier to fix in the source material, prompt, or generation workflow than by repeatedly regenerating the same request.
Rate-limit management checks
Use these checks before heavy generation periods:
- Monitor usage at the 80% threshold.
- Set up usage or limit alerts when your organization has alerting enabled.
- Plan for peak usage periods.
- Use user quotas when one team or program could consume the monthly limit before other planned generation work is finished.
- Request limit increases proactively.
Do not wait until generation is blocked to review limits. Rate-limit issues are most disruptive during launches, migrations, and bulk content creation.
When to escalate usage concerns
Escalate to the AI operations owner, customer success, or support when:
- Usage is near the organization limit and a launch or migration is scheduled.
- Quality score stays low after source material and prompt cleanup.
- A small number of users account for most generation activity.
- Exported summaries do not match the operational period under review.
- The team needs a plan or limit change before continuing bulk generation.
Verify and support AI analytics
Verify success
AI Analytics is working when:
- The page loads without a permission or upgrade prompt.
- Time range changes update the charts and counts.
- Refresh completes without error.
- Usage, quality, user activity, and rate-limit tabs show data for the selected range.
- Export downloads a CSV.
Troubleshooting
Check both permission and plan access. You need an AI analytics permission and the AI Analytics entitlement.
Change the time range. If Daily is empty, try Weekly or Monthly. If all ranges are empty, the organization may not have AI generation activity yet.
Check User Activity for top generators, then check AI Operations for bulk operations, progress tracking, and recent AI service exports.
Review failed generations, source materials, and generation prompts. If the same workflow keeps producing low scores, fix the input pattern before continuing to generate more content.
Pause non-urgent generation work, check whether bulk operations are running, and request a limit increase before the next peak period.