Screenshots, UI and Charts
By the Syncfusion DevTutorials team · Last updated
Turn error screenshots into complete bug reports and read charts honestly. Small error codes need OCR, summaries need a schema, and values read from bar heights must be flagged as estimates.
What Screenshot and Chart Reading Is
Screenshot and chart reading = turning a picture of a screen or a chart into facts you can act on: a bug report, or the values behind the bars.
In this lesson, you will turn error screenshots into complete bug reports, and read charts without pretending to be more exact than you are.
A customer sends a screenshot of an error on the order-tracking screen. The model writes a clear summary: "The courier's system did not answer." But it reads the error code as E-553O: the zero became the letter O.
The support team searches the logs for E-553O and finds nothing.
Press Play, or use Next to go one step at a time.
- Frame 1: the screenshot, with a small error code.
- Frame 2: the summary is right; the code is wrong.
- Frame 3: the fix: OCR text sent with the image.
Simple lesson: a model can understand a screen and still misread the one exact value on it.
A Real-World Example
Think about reporting a fault with your car.
- You tell the mechanic what happened: "it shook, then a light came on." That is the meaning.
- The mechanic plugs in a reader and gets the fault code: P0301. That is the exact value.
- If you guessed the code from memory, they would fix the wrong part.
A bug report needs both:
- The meaning = what went wrong, in plain words. The multimodal LLM is good at this.
- The exact code = the string a developer searches for. OCR is good at this.
- The form = every field filled in. You get this by asking for a schema.
Simple lesson: use the generalist for meaning and the specialist for exact text, and ask for every field by name.
What You Will Learn
By the end of this lesson, you will be able to:
- explain why multimodal models misread small exact text in screenshots
- combine OCR text with the image for exact values
- ask for a bug-report schema instead of a free-text summary
- flag chart values as estimate or exact
- pull an error code out of OCR text with a regular expression
This lesson works on the Perceive stage: what you send to the model, and what you ask it for.
Screenshots: Meaning and Exact Text
A screenshot mixes two kinds of information:
| Kind | Example | Best reader |
|---|---|---|
| Meaning | "The bank declined the card" | Multimodal LLM |
| Exact text | E-4012, CARRIER_TIMEOUT | OCR |
| Layout | Which screen, which button | Multimodal LLM |
So send both: the image, and the OCR text of the image. The LLM uses the image for meaning and layout, and copies exact strings from the OCR text.
Ask for Every Field
A free-text summary contains what the model thinks is important. That is not always what your team needs.
Step through both frames.
- Frame 1: a fluent summary with no screen name.
- Frame 2: a bug-report schema, one field at a time: nothing missing.
Easy way to remember: if you need a field, ask for it by name. A summary is not a form.
Simple lesson: a schema turns a nice paragraph into a report you can file.
Charts: Estimate or Exact
A chart is a picture of numbers. Sometimes the numbers are printed on it. Often they are not, and the model can only estimate them from the bar heights.
Step through all three frames.
- Frame 1: a chart with no numbers on the bars, read as "exact" values.
- Frame 2: the same values, flagged as estimates.
- Frame 3: a chart with printed numbers: copied and marked exact.
An estimate is fine, as long as it says so. A value stated as exact when it was guessed from a bar's height is a silent error waiting to happen: someone will copy it into a report.
Charts also invite invented numbers. If the chart has no axis values at all, the model may still give you precise numbers. Ask it to say "unknown" when there is nothing to read from.
Simple lesson: say how sure a value is, based on where it came from.
Three Fixes, One at a Time
Step through all four frames.
- The row of fixes: OCR text, a schema, estimate flags.
- The score: bug-report fields right, and chart values flagged right.
Click the button and look at the Pipeline tab.
- The list: six screenshots (🖥) and four charts (📊).
- The table on the right: what the model read against the real values.
Lab Exercise: Bug Reports and Honest Chart Values
Your task: change the Perceive settings until the screenshots and charts pass. Then press Check.
Click the button and open the Pipeline tab.
- Perceive: the bug-report schema, OCR text, and estimate flags.
- To pass: at least 90% of bug-report fields right, and every chart value flagged correctly as estimate or exact.
Build It in TypeScript
When OCR text comes with the screenshot, you do not even need a model for the error code. A regular expression finds it exactly.
export function errorCode(ocrText: string): string | null {
// Codes look like E-4012: "E-" and four digits.
const match = ocrText.match(/\bE-\d{4}\b/);
return match ? match[0] : null;
}
// Use the exact code from OCR, and the meaning from the LLM.
const report = {
code: errorCode(ocrText) ?? llmReport.code,
screen: llmReport.screen,
meaning: llmReport.meaning,
};Screenshots can contain secrets. Error screens often show emails, order numbers or tokens. Lesson 12 covers removing personal data before an image leaves the device.
This is a pattern you will use again and again: the specialist (or plain code) for exact values, the generalist for meaning, and your code to join them.
Simple lesson: when plain code can read a value exactly, do not ask a model to guess it.
TL;DR
- Screenshots mix meaning (LLM) and exact text (OCR). Send both.
- Small exact text (codes, IDs) is where the LLM slips: 0 becomes O.
- A schema asks for every field by name. A summary leaves things out.
- Charts: printed numbers are exact; values read from bar heights are estimates.
- Flag estimates, and say "unknown" when there is nothing to read.
- Plain code (a regular expression) beats a model for exact patterns.
Common mistakes
| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Trusting the code in a summary | "It read the whole screen." | Take exact strings from OCR. | The team searches for a code that does not exist. |
| Asking for a summary | "It will mention what matters." | Ask for a schema with every field. | Reports without the screen or the steps. |
| Stating chart values as exact | "The model read the chart." | Flag values read from bar heights. | Estimates end up in reports as facts. |
| Using a model for a pattern | "The model can find the code." | Use a regular expression on OCR text. | Slower, dearer and less exact. |
| Sending only the image | "The model can see the text." | Send the image and its OCR text. | Small text is misread. |
Check Your Understanding
Exercises
What to Remember
- A screenshot has meaning and exact text. Use the right reader for each.
- Send OCR text with the image for exact values.
- Ask for a schema, not a summary.
- Values read from bar heights are estimates. Flag them.
- When plain code can read a value exactly, use plain code.
Further Reading
- Regular expressions, on MDN: find exact patterns in OCR text.
- Transformers.js documentation, by Hugging Face: OCR and vision models in the browser.
Next: Verification and Grounding, where you catch the wrong values that the model was sure about, before they reach a customer.