From Samples to Scenes: Processing Images, Audio and Video

A slide looks sharp in the original, but its small characters blur in the stream. The video plays smoothly, yet the subtitles lag behind. These errors arise at different points in media processing. They still affect the same situation: can someone understand the explanation?

Computer graphics and audiovisual data processing connects the computation of visual content with the processing of existing signals. For our example media library, creating suitable playback variants, preparing audio and aligning derived information in time are particularly relevant.

Digitization is a limited description

Sampling captures a signal at discrete intervals. Quantization describes its values with limited precision. The sampling theorem states conditions for reconstructing band-limited signals. Without suitable prefiltering, frequency components above half the sampling rate can appear as false lower frequencies: aliasing. A subsequent reduction in sampling rate therefore also requires appropriate filtering. Smith: The Scientist and Engineer’s Guide to Digital Signal Processing, “The Sampling Theorem”

For our library, the audio track would therefore need deliberate preparation. A smaller file can simplify processing, but not every reduction preserves all the information needed. Speech transcription can have different requirements from archiving a music lecture.

One dedicated test case would be a recording with technical terms, background noise and brief speaker changes. Processing would be checked both technically and substantively: was the file created correctly? Does speech remain intelligible? Does the resulting transcript deteriorate?

Computer graphics creates images from a scene

A virtual scene describes geometry, materials, lighting and a camera. Ray tracing investigates rays and their interactions with the scene. Path tracing approximates light transport through stochastically selected paths. A limited number of samples can produce visible noise. The textbook Physically Based Rendering connects these foundations to their implementation. Pharr, Jakob and Humphreys, 2023: Physically Based Rendering, 4th edition

For an interactive learning application, this could produce a rotatable three-dimensional representation, for example. A time budget then matters too: targeting 60 frames per second leaves approximately 16.7 milliseconds per frame. Rendering, interaction and other work must together fit within the available budget.

In our library, such a feature would be a separate extension. Its learning purpose would have to be established first. An elaborate representation is worthwhile only if its additional interaction is needed for that task. The UX perspective in Part 3 provides the evaluation framework.

Compression changes the available information

Lossless compression allows exact recovery of the original data. Lossy methods reconstruct an approximation and can achieve greater reductions. Which deviations are acceptable depends on the purpose. Smith: The Scientist and Engineer’s Guide to Digital Signal Processing, Chapter 27, “Data Compression”

For our educational videos, checking moving camera footage alone would be insufficient. Small text, fine diagram lines and displayed source code must also remain legible. A variant that looks good for a conversation may be unsuitable for a screen recording.

The example design therefore retains an original and generates derived variants from it. We distinguish the container, which organizes tracks and metadata, from the methods used to encode the image and audio signals it contains. A file extension alone would provide too little information for selecting a processing pipeline.

Several playback variants could later be delivered adaptively. The web platform in Part 2 would then handle the appropriate delivery. Production and delivery remain distinct responsibilities.

Timestamps need a shared basis

Our search result should jump to the correct statement. Video playback and the transcript must therefore use the same time base. Removing an introduction changes every subsequent position. A reference to the video title alone does not preserve this relationship.

In the design, each content edit receives its own media version. The transcript references exactly that version. Playback variants must preserve its timing or have an explicitly described mapping. This rule is anchored in the data model in Part 4.

For acceptance, we would check several marked passages at the beginning, middle and end. A constant offset and a deviation that grows over time are different failure patterns. Checking only the beginning of a video could miss the latter.

Media becomes evidence for search

Further processing steps could detect scene changes, select preview images and recognize text in individual frames. In our example, these results would be treated as clues. A camera cut need not indicate a topic change; a slide heading does not necessarily describe the whole explanation.

One possible pipeline is: validate the original, generate playback variants, prepare audio, create a transcript, check timestamps and derive search segments. Each step records its input version and method. An incorrect search result can then be traced back to the point where information was lost or changed.

Text search in Part 5 and the learned methods in Part 6 can only work with information this pipeline actually preserves.