DOI : 10.17577/IJERTCONV14IS110034- Open Access

- Authors : Thimmaiah K K S, Jyothiba Salunkhe S
- Paper ID : IJERTCONV14IS110034
- Volume & Issue : Volume 14, Issue 11, CONFLUENCE – 2026
- Published (First Online) : 20-11-2026
- ISSN (Online) : 2278-0181
- Publisher Name : IJERT
- License:
This work is licensed under a Creative Commons Attribution 4.0 International License
An Intelligent Multimodal System for Continuous Lecture Note Generation Using Voice and Visual Content
Thimmaiah K K S1, Student, Bachelor of Computer Applications (BCA)
St. Francis Degree College, Koramangala, Bengaluru Affiliated to Bengaluru City University, Karnataka, India
Jyothiba Salunkhe S2 , Assistant Professor, Bachelor of Computer Applications (BCA) St. Francis Degree College, Koramangala, Bengaluru Affiliated to Bengaluru City University,
Karnataka, India, salunkhejyothiba@gmail.com
Abstract – In this paper, I describe VoxBox, an Android application intended to capture both sources of lecture content and produce structured study notes while retaining the original evidence for later review. A student follows spoken explanations and material on the board or screen, including slides, equations and diagrams, at the same time. Voice recorders and transcription tools preserve speech without the board; photographs preserve the board without the explanation that gives it meaning. Voice mode serves discussion-led sessions, whereas Live Board mode adds periodic CameraX frames to continuous audio capture. Recorded in bounded 20-second chunks, audio remains on the device until transcription succeeds. Before any frame leaves the phone, a local visual gate and on-device optical character recognition classify it as a duplicate, a continuation or a new scene. The remote extraction service receives only finalised board segments. Transcripts and board images are stored separately from the AI-generated notes and are never rewritten. For note generation, the system uses revision-guarded Markdown, bounded syllabus context and an authenticated backend proxy, with provenance linking each statement to its source. A survey of sixteen recent studies (20222025) highlights the gaps targeted by the design. In a two-minute device session, 18 frames were captured, of which 15 were rejected as duplicates and 2 merged as continuations. This left one representative image and no board extraction request. All 193 automated tests pass in the current build. Classroom-scale evaluation of accuracy and endurance remains future work.
Keywords: Multimodal lecture notes; automatic speech recognition; optical character recognition; evidence preservation; responsible educational technology
-
INTRODUCTION
In a lecture, reasoning and worked explanations come through speech, alongside definitions, equations and diagrams on the board or slide. The student follows both at once, so taking handwritten notes on the two sources divides attention. A recent meta-analysis suggests that handwritten notes are linked to higher achievement, although typing captures more content (Flanigan et al., 2024). Existing digital aids appear to cover only one source: voice recorders and transcription services omit the board, while photographs preserve it without the explanation that gives it meaning.
Evidence preservation is the defining principle of VoxBox, the Android application I present here. Its raw transcript and board images are stored before any AI interpretation; those records are never rewritten and remain available whenever a generated statement is in doubt. The application generates structured study notes from spoken explanations and board or screen content during a lecture. The paper makes three contributions:
-
A structured survey of sixteen recent studies (20222025) on lecture analysis, speech recognition, optical character recognition (OCR) and grounded text generation (Section 2);
-
An architecture in which evidence is preserved and an on-device board-filtering pipeline classifies every frame before any remote request is made (Section 4); and
-
Device-level results reported with explicit claim boundaries, together with an evaluation protocol for the measurements still outstanding (Sections 57).
-
-
LITERATURE SURVEY
A multimodal lecture-note system draws on four areas of research: (A) Note-taking and lecture understanding, (B) Speech recognition, (C) Visual text and mathematics recognition, and (D) Grounded and verifiable text generation. This survey covers all four areas through sixteen studies published between 2022 and 2025. Each study was selected for its relevance to one stage of the VoxBox pipeline. Table 1 presents their methods and the gaps most relevant to continuous lecture note generation.
Table 1. Literature survey of related work
Sl.
No.
Paper Name
Authors
Method Used
Key Gaps & Limitations
A. Note-taking and lecture understanding
1
Typed Versus Handwritten Lecture Notes and College Student
Flanigan, A. E., Wheeler, J., Colliot, T., et al. (2024)
A meta-analysis of 24 studies from 21 articles compares typed and
Handwritten notes were linked to higher achievement (g = 0.248), although typing produced
Sl.
No.
Paper Name
Authors
Method Used
Key Gaps & Limitations
Achievement: A Meta- Analysis
Educational Psychology Review
handwritten lecture notes among college students.
more notes (g = 0.919). The analysis covers manual note-taking only and does not study AI- generated notes.
2
Lecture Presentations Multimodal Dataset: Towards Understanding Multimodality in Educational Videos IEEE/CVF ICCV
Lee, D. W.,
Ahuja, C.,
Liang, P. P., et
al. (2023)
The dataset contains 180+ hours of lecture video and 9,000+ slides aligned with speech. The work benchmarks cross-modal retrieval and proposes the PolyViLT transformer.
The authors report weak slidespeech alignment, novel visual media, technical language and long sequences as open challenges. The recordings show slides rather than live boards.
3
M3AV: A Multimodal, Multigenre, and Multipurpose Audio- Visual Academic Lecture Dataset
ACL
Chen, Z., Liu,
H., Yu, W., et al. (2024)
About 367 hours of academic lecture video are accompanied by human- annotated slide text and speech. Benchmarks cover contextual ASR, speech synthesis, and slide and script generation.
The work provides a dataset and benchmark for recorded presentations with projected slides rather than handwritten classroom boards. It is not an end-to-end note system.
4
NoTeeline: Supporting Real-Time, Personalized Notetaking with LLM- Enhanced Micronotes ACM IUI
Huq, F., Samee, A., Lin, D. C.-
E., et al. (2025)
As a video plays, an LLM expands short user-written micronotes into full notes in the user's style. The system was evaluated in a within-subjects study (n = 12).
Factual correctness was 93.2%, so errors still need to be identified. The sample was small and the videos pre-recorded.
Students must continue writing micronotes during the lecture.
B. Speech recognition
5
Robust Speech Recognition via Large- Scale Weak Spervision ICML
Radford, A., Kim, J. W., Xu,
T., et al. (2023)
Whisper is an encoder decoder Transformer trained on 680,000 hours of weakly supervised audio. It decodes long recordings sequentially in 30 s windows.
For long-form audio, the authors report repeat loops, missed words at segment edges and hallucinated text. The system processes audio only and provides no speaker attribution.
6
WhisperX: Time-Accurate Speech Transcription of Long-Form Audio Interspeech
Bain, M., Huh,
J., Han, T., &
Zisserman, A. (2023)
WhisperX combines voice-activity cut-and- merge into ~30 s chunks,
batched Whisper inference and forced phoneme alignment for word-level timestamps.
WhisperX requires additional VAD and language-specific alignment models. It uses batch GPU processing of complete recordings rather than live mobile capture.
7
Turning Whisper into Real-Time Transcription System
IJCNLP-AACL System
Demonstrations
Macháek, D., Dabre, R., & Bojar, O. (2023)
A local-agreement policy with self-adaptive latency adapts Whisper for streaming transcription. Latency is about 3.3 s on long-form speech. The transcriber was demonstrated at a live conference.
The transcriber optimises the latency of a rolling transcript without adding board content, note structure or speaker identity.
Sl.
No.
Paper Name
Authors
Method Used
Key Gaps & Limitations
8
Careless Whisper: Speech- to-Text Hallucination Harms
ACM FAccT
Koenecke, A., Choi, A. S. G.,
Mei, K. X., et
al. (2024)
An audit of Whisper output on 13,140 AphasiaBank segments identifies hallucinations and categorises them by type of harm.
1.4% of transcriptions contained hallucinations, 38% of them harmful, more often with long pauses. The work diagnoses the problem without a mitigation pipeline.
9
SlideSpeech: A Large Scale Slide-Enriched Audio-Visual Corpus IEEE ICASSP
Wang, H., Yu, F., Shi, X., et al. (2024)
SlideSpeech contains 1,705 conference videos
(1,000+ hours, 473 transcribed) with synchronised slides. Slide keywords provide context for speech recognition.
Projected slides from recorded talks supply text only as recognition context. Handwritten boards and note generation are not included.
C. Visual text and mathematics recognition
10
TrOCR: Transformer- Based Optical Character Recognition with Pre- trained Models
AAAI
Li, M., Lv, T.,
Chen, J., et al. (2023)
TrOCR's pre-trained image-Transformer encoder and text- Transformer decoder are fine-tuned for printed, handwritten and scene text.
A separate detector is needed because recognition operates on cropped text lines.
Continuous use on a phone is costly, and there is no layout or equation structure.
11
CoMER: Modeling Coverage for Transformer-Based
Handwritten Mathematical Expression Recognition ECCV
Zhao, W., &
Gao, L. (2022)
A Transformer decoder recognises handwritten mathematical expressions using coverage attention (attention refinement, self- and cross-coverage).
Expression recognition rate is about 5963% on CROHME benchmarks. The method recognises isolated expressions rather than full camera-captured boards.
12
Nougat: Neural Optical Understanding for Academic Documents ICLR
Blecher, L., Cucurull Preixens, G., Scialom, T., & Stojnic, R. (2024)
A Visual Transformer converts images of scientific document pages into markup, including mathematical expressions.
The model is designed for digital academic documents and does not address handwriting, camera perspective or glare on classroom boards.
13
OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models
Science China Information Sciences
Liu, Y., Li, Z.,
Huang, M., et al. (2024)
A benchmark of 29 datasets evaluates large multimodal models such as GPT-4V and Gemini. It covers text recognition, VQA, information extraction and handwritten mathematics.
Models show clear weaknesses with handwritten text, non- semantic text and mathematical expressions, all typical board content.
D. Grounded and verifiable text generation
14
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation EMNLP
Min, S.,
Krishna, K., Lyu, X., et al. (2023)
FActScore splits generated text into atomic facts and measures the share supported by a knowledge source through human and automated evaluation.
On biographies, ChatGPT reached only 58%. Facts are checked against an external source rather than the input used to generate a note.
Sl.
No.
Paper Name
Authors
Method Used
Key Gaps & Limitations
15
Enabling Large Language Models to Generate Text with Citations
EMNLP
Gao, T., Yen,
H., Yu, J., &
Chen, D. (2023)
In ALCE, LLM systems retrieve evidence and answer with citations. The benchmark scores fluency, correctness and citation quality.
On ELI5, even the best models lacked complete citation support about half the time. The task is question answering rather than lecture notes.
16
Retrieval-Augmented Generation for Large Language Models: A Survey
arXiv preprint
Gao, Y., Xiong,
Y., Gao, X., et al. (2024)
The survey covers naive, advanced and modular RAG, including retrieval, generation, augmentation and evaluation methods.
Retrieval mitigates hallucination but does not eliminate it. Retrieved text is reference knowledge, not evidence of what a lecturer said.
-
Synthesis of the Survey
In Table 1, I identify three patterns. Recent lecture datasets and note-taking tools (Chen et al., 2024; Huq et al., 2025; Lee et al., 2023) work with recorded presentations and projected slides. None of them appears to capture a live classroom from a student's phone, where speech and a handwritten board change continuously. The components are strong but fallible: recognisers hallucinate during pauses and miss words at segment edges (Koenecke et al., 2024; Radford et al., 2023). Streaming calls for careful boundary handling (Macháek et al., 2023); slide text can aid recognition (Wang et al., 2024). Both specialised recognisers and multimodal models remain weak on handwriting and mathematics (Liu et al., 2024; Zhao & Gao, 2022). LLM output contains unsupported statements even when it cites sources (T. Gao et al., 2023; Min et al., 2023). Retrieved context improves factuality without establishing what was actually said (Y. Gao et al., 2024). I take these findings to mean that no single stage should be trusted on its own. Evidence should therefore be retained between stages, deterministic work run locally and remote output kept traceable to its evidence.
-
-
RESEARCH GAP AND OBJECTIVES
Transcript-centred tools miss board equations and diagrams. Camera-centred tools miss the reasoning spoken while the board changes. Summarisation-centred tools obscure whether a statement came from speech, the board or the model. None of the surveyed systems appears to combine live, two-channel capture on a mobile device with an auditable separation between evidence and generated notes. These gaps lead me to four questions:
RQ1 How can a phone capture speech and board evidence together without losing the final partial speech segment or sending redundant frames?
RQ2 How can notes be restructured freely while the underlying evidence remains intact and traceable?
RQ3 Which privacy, provenance and recovery controls must be in place before AI-generated notes can be trusted?
RQ4 What must be measured before claims can be made about accuracy, reliability or cost?
The objective is an Android lecture companion whose output is portable Markdown notes backed by auditable evidence. The architecture answers RQ1RQ3; an evaluation protocol answers RQ4.
-
METHODOLOGY AND SYSTEM ARCHITECTURE
The work follows a design-science method: problem identification, technology review, artefact construction, device verification and an evaluation protocol for claims not yet measured. I present the architecture in Figure 1. A session starts when the user selects a subject, a mode (Voice or Live Board) and a note policy, and capture runs only while the application is in the foreground. Independent workers handle audio, local frames and remote extraction. Camera load therefore cannot delay ordered speech. Durable state is stored on the phone, and a hosted proxy mediates every model call. The provider credentials are held only by the proxy.
Figure 1. System architecture of VoxBox: independent capture workers, a local evidence store, a revision-guarded note layer and an authenticated proxy boundary
-
Audio Evidence Pipeline
Android's AudioRecord API records audio as 16 kHz mono PCM in 20-second chunks. The platform SpeechRecognizer was rejected on the basis of Android's own documentation, which advises against using it for continuous recognition on battery and bandwidth grounds. Before upload, each chunk is saved privately as a WAV file. Consecutive chunks overlap slightly so that a word falling on a boundary is less likely to be lost (Macháek et al., 2023; Radford et al., 2023). The final partial chunk is drained on stop rather than discarded. The on-device Room database stores transcripts as immutable, timestamped segments. Only after its transcript has been persisted is a WAV file deleted. A transcription failure leaves the file available for recovery. Assigned per chunk, speaker labels serve only as a local hint about the dominant speaker, never as an identity.
-
Live Board Filtering Pipeline
Live Board mode adds periodic frame capture through the CameraX library. Boards rarely change all at once; most updates add a single word or equation line. Sending every frame to a vision model would therefore be wasteful. Below, I describe the three on-device stages each frame passes through first:
-
Visual gate. The gate compares a 64 × 64 luminance sample with the baseline using exposure-compensated novelty, changed-pixel fraction, edge statistics and a 64-bit difference hash. It rejects flicker, sensor noise and pointer movement without OCR.
-
On-device OCR. The bundled ML Kit text recogniser reads the surviving frames, which are then compared with the open segment by token, edit-distance and line-containment similarity.
-
Segment state machine. Each frame is classified as shown in Table 2. Only a finalised segment, represented by its best frame, is sent for remote extraction of text, equations and diagram regions.
Table 2. Frame classification and local action
Classification
Typical board event
Local action
Duplicate
Unchanged board, exposure flicker or a small pointer movement
Discard the redundant image after persistence and keep the clearer representative; no remote request.
Continuation
A word, bullet or equation line is added
Merge the new text into the open segment; defer the remote request.
New scene
Slide change or erased board
Finalise the open segment for extraction and start a new one.
Classification
Typical board event
Local action
Non-text change
A diagram grows without new text
Keep the frame in the open segment as a diagram candidate.
Uncertain
Large visual change with little readable text
Hold one further frame; unresolved frames are retained for review.
The heuristic score used to select the representative frame considers sharpness, exposure, obstruction, completeness, OCR length and diagram visibility. A segment is flushed on a new scene, on stop or manual capture, or when it reaches its maximum age (180 s by default) or capacity. Recovery after process death also triggers a segment flush. Checkpointing allows an interrupted session to resume without replaying frames.
-
-
Note Generation and Provenance
Users can select one of three note policies. Verbatim formats timestamped speech and board text without AI rewriting. Evidence strict rewrites conservatively, adds no claims from the syllabus or general knowledge and flags uncertain gaps. Study enhanced, the default policy, is instructed to merge repeated explanations, reconcile speech with board text, repair fragments and remove unrelated conversation. Every change is recorded as provenance.
While capture is in progress, live sections are composed locally and marked as unrefined. After stop-and-drain, a single finalisation request sends the complete evidence graph (transcripts, board segments, ranked syllabus excerpts and the existing note) to the proxy. The proxy returns a rewritten note with each section labelled as captured, normalised, corrected or AI- supplemented. Correction decisions give transcriptboard agreement priority over repeated lecture context and the syllabus. Each correction records the original and proposed wording, a reason, a confidence value and its evidence. Contradictions are flagged for review. Syllabus text is never presented as something the lecturer said. Every write, whether from the model or the user, commits only if its base revision and hash match the current note. Stale responses are rejected.
-
-
IMPLEMENTATION AND RESULTS
The re-architected build was verified on a Redmi 2411DRN47I running Android 16 (Figure 2), using a loopback mock proxy with the provider transport disabled. No billable request was made. These runs therefore do not establish provider accuracy or note quality. Table 3 lists each result and the claims it does not establish.
Table 3. Implementation results and claim boundaries
Area
Observed result
Not established
Automated tests
115/115 Android unit, 66/66 backend and 12/12 on- device instrumentation tests pass (193/193).
Classroom accuracy.
Audio capure
20 s chunks transcribed in order; final partial chunk drained at stop; recovery files cleared after success.
Word error rate (transcription used the mock).
Board filtering (device)
Two-minute session: 18 frames captured, 15 duplicates skipped, 2 continuations merged, 0 extraction requests, 1 representative image kept.
Performance on changing boards; the scene was largely static.
Board filtering (replay)
30 synthetic captures over 3 topics: 9 local OCR passes, 3 segments; 6 of 9 extraction requests avoided
(66.67%).
Monetary cost or real error rates.
Camera zoom
3.53× held across 18 captures and 75 s; an automated test held 2.0× for 60 s and across activity recreation.
Behaviour on other phone models.
Export
ZIP of UTF-8 Markdown with YAML front matter, relative asset paths and separate evidence sections.
Import into an external vault (not re-tested).
Repetition removal, gap filling and correction remain unverified because the mock returns one fixed sample per audio chunk. In field trials, an earlier prototype connected to a live provider transcribed real speech with timestamps and speaker labels. This suggests that the approach is feasible, but its accuracy has yet to be established.
Figure 2. VoxBox on a physical device (Android 16): the library, a Live Board session with its diagnostics panel, and the reviewed note
-
PRIVACY, SECURITY AND RESPONSIBLE USE
The user starts and stops each session. Capture runs only in the foreground, and the application requests only the permissions needed for the selected mode when capture begins. Users are responsible for obtaining consent before recording a class. Provider credentials never enter the
application. The HTTPS proxy requires a bearer token, enforces size, rate and budget limits, rejects requests that do not conform to a strict schema, and does not write submitted media, syllabi or notes to disk. The client token can be extracted from the APK, so its risk is bounded by those limits rather than by secrecy. Speaker labels are never used to identify people, as identification would require consent-aware enrolment. Generated notes are presented as revision aids, with their source evidence available for inspection.
-
EVALUATION PLAN
Accuracy claims require a labelled corpus fixed before any tuning (Table 4). Each run will record the device, build, model configuration and network condition. Results will include sample sizes and uncertainty. The evaluation will compare voice-only and multimodal capture to measure concept coverage. Uploading every changed frame will be compared with local filtering to measure requests avoided versus changes missed. Direct summarisation will be compared with evidence-linked generation for unsupported statements and review effort.
Table 4. Planned evaluation dimensions and metrics
Dimension
Test material
Metrics
Speech
Quiet and noisy rooms, competing speakers, long silences, short final chunks
Word error rate, timestamp error, diarisation error rate, final-chunk preservation
Board
Static frames, progressive reveals, equations, diagrams, exposure shifts, erasures, slide changes
Duplicate-rejection precision, new-scene recall, OCR and equation accuracy, diagram-region IoU, requests avoided
Note quality
Reference concept lists linked to evidence identifiers
Concept coverage, duplicate-concept rate, unsupported-statement rate, correction precision
Resources
30- to 60-minute sessions on stated hardware
Latency, network requests, storage growth, battery and thermal behaviour
Usability
Task-based study with students
Task completion, correction time, accessibility (large text, TalkBack), export validity
-
DISCUSSION AND LIMITATIONS
VoxBox's main strength is that it retains the evidence alongside the generated notes. A statement in doubt can be traced to its transcript segment or board frame, and media from a failed request remain available for reprocessing. Local filtering should make session cost depend on the number of distinct board states rather than lecture length, although this has yet to be measured. The main limitation is that accuracy remains unmeasured. Each component has documented failure modes (Table 1), and verification has so far used a mock provider on a single phone model. The design also has limits: OCR covers Latin script only, syllabus import
accepts only UTF-8 text and Markdown, diagrams are cropped without interpretation, and very long lectures are rejected rather than consolidated. No 30-minute endurance, battery, thermal or process-death test has yet been run. Results from one classroom, subject or device may not generalise.
-
CONCLUSION AND FUTURE WORK
VoxBox generates continuous lecture notes from voice and visual content and keeps the evidence those notes derive from. The survey suggests that the required components exist but are individually fallible, and that recent lecture systems work with recorded presentations rather than live classrooms. VoxBox responds with visible capture, recoverable audio chunks, on- device board filtering, immutable evidence, revision-guarded generation and a credential- isolating proxy. The build passes all 193 automated tests. In one device session, redundant board frames were eliminated before any remote request was made.
Future work will proceed in order of evidential value: a live-provider session with speech and a changing board comes first, then a labelled corpus and scorer for Table 4, then endurance and recovery testing. Handwriting and equation recognition, long-lecture consolidation and consent-aware speaker identification come last. The system should claim only what its evidence supports.
REFERENCES
-
Bain, M., Huh, J., Han, T., & Zisserman, A. (2023). WhisperX: Time-accurate speech transcription of long-form audio. In Proceedings of Interspeech 2023 (pp. 44894493). International Speech Communication Association. https://doi.org/10.21437/Interspeech.2023-78
-
Blecher, L., Cucurull Preixens, G., Scialom, T., & Stojnic, R. (2024). Nougat: Neural optical understanding for academic documents. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024). https://proceedings.iclr.cc/paper_files/paper/2024/hash/a39a9aceda771cded859ae7560530e09-Abstract-
Conference.html
-
Chen, Z., Liu, H., Yu, W., Sun, G., Liu, H., Wu, J., Zhang, C., Wang, Y., & Wang, Y. (2024). M3AV: A
multimodal, multigenre, and multipurpose audio-visual academic lecture dataset. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 90419060).
Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.acl-long.489
-
Flanigan, A. E., Wheeler, J., Colliot, T., Lu, J., & Kiewra, K. A. (2024). Typed versus handwritten lecture notes and college student achievement: A meta-analysis. Educational Psychology Review, 36(3), Article 78. https://doi.org/10.1007/s10648-024-09914-w
-
Gao, T., Yen, H., Yu, J., & Chen, D. (2023). Enablinglarge language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 6465 6488). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.398
-
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2024).
Retrieval-augmented generation for large language models: A survey (arXiv:2312.10997). arXiv. https://doi.org/10.48550/arXiv.2312.10997
-
Huq, F., Samee, A., Lin, D. C.-E., Tang, A. X., & Bigham, J. P. (2025). NoTeeline: Supporting real-time, personalized notetaking with LLM-enhanced micronotes. In Proceedings of the 30th International Conference on Intelligent User Interfaces (pp. 10641081). Association for Computing Machinery. https://doi.org/10.1145/3708359.3712086
-
Koenecke, A., Choi, A. S. G., Mei, K. X., Schellmann, H., & Sloane, M. (2024). Careless Whisper: Speech- to-text hallucination harms. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (pp. 16721681). Association for Computing Machinery. https://doi.org/10.1145/3630106.3658996
-
Lee, D. W., Ahuja, C., Liang, P. P., Natu, S., & Morency, L.-P. (2023). Lecture presentations multimodal dataset: Towards understanding multimodality in educational videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 2003020041). IEEE. https://doi.org/10.1109/ICCV51070.2023.01838
-
Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z., & Wei, F. (2023). TrOCR:
Transformer-based optical character recognition with pre-trained models. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 1309413102. https://doi.org/10.1609/aaai.v37i11.26538
-
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., & Bai, X. (2024).
OCRBench: On the hidden mystery of OCR in large multimodal models. Science China Information Sciences, 67(12), Article 220102. https://doi.org/10.1007/s11432-024-4235-6
-
Macháek, D., Dabre, R., & Bojar, O. (2023). Turning Whisper into real-time transcription system. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations (pp. 1724). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.ijcnlp-demo.3
-
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023). FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 1207612100). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp- main.741
-
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning (Vol. 202, pp. 2849228518). PMLR. https://proceedings.mlr.press/v202/radford23a.html
-
Wang, H., Yu, F., Shi, X., Wang, Y., Zhang, S., & Li, M. (2024). SlideSpeech: A large scale slide-enriched audio-visual corpus. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 1107611080). IEEE. https://doi.org/10.1109/ICASSP48485.2024.10448079
-
Zhao, W., & Gao, L. (2022). CoMER: Modeling coverage for transformer-based handwritten mathematical expression recognition. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Computer vision ECCV 2022 (Lecture Notes in Computer Science, Vol. 13688, pp. 392408). Springer. https://doi.org/10.1007/978-3-031-19815-1_23
