Screenshots are frequently shared on social media, via personal communications, and in academic papers. Unfortunately, existing screenshot tools strip away semantics useful for making the content accessible, leaving only pixels. For example, a screenshot of a table removes the structural information useful for conveying it.

Demo

  1. Download the APK
  2. Install. You will need to enable external apps and accept warnings.
  3. The app should ask for storage permissions. Click ok.
  4. There should be a list of demo images in the home screen.
  5. If talkback is enabled you should be able to experience the interface on top of the screenshot.

For a full demo that includes screenshot capture, build from master branch from git.

Implementation

We aim to provide a seamless experience identical to existing screenshot tools. To allow semantic information to be automatically propagated along with pixels, we choose to embed it inside the image itself in standard metadata fields (Exif). If an editing tool strips away this information, then it can also be transmitted externally as a separate file. Although our proofof-concept implementation runs on Android, other platforms such as Windows and MacOS have equivalent APIs and can support similar implementations. For more details refer to the paper.

[Code] [Paper] [APK download]