StreamSQL-Repair-Bench: Does Execution Feedback Let Agents Fix Event-Time Streaming SQL?
DOI:
https://doi.org/10.66238/fsrma97Keywords:
large language models, agents, execution feedback, self-repair, streaming SQL, event time, benchmarkAbstract
Writing a streaming query that runs is easy; writing one that is correct under event-time semantics is not. Watermarks and late-data policies are subtle, and the kind of temporal reasoning they demand is something large language models (LLMs) handle poorly. The dominant fix in agentic coding is execution feedback: let the model run its code, see what comes out, and revise. We ask whether that loop actually rescues streaming-SQL errors. StreamSQL-Repair-Bench hands a model a streaming-SQL query with a single planted bug, an intent description, and a few input/output tests, and asks it to repair the query; ground truth comes from a small reference engine implementing Dataflow-model event-time semantics, so we grade by execution accuracy without running a cluster. We separate bugs into shallow (readable straight off the I/O) and deep (require simulating event time), and compare one-shot repair, blind retrying, and repair with execution feedback. Across five LLMs, execution feedback barely moves shallow bugs (approx.0.9 either way) but roughly doubles deep-bug accuracy (e.g. Claude Haiku 0.14->0.53, Gemini Flash 0.28->0.56; aggregate 0.24->0.49). A blind-retry control stays at or below one-shot, so the gain comes from the executed result, not from extra attempts. Yet feedback is not a cure: it repairs structural mistakes (wrong window type or size) but leaves watermark and late-data bugs unfixed, the same semantics that resist one-shot reasoning. The benchmark, reference engine, and harness need no streaming infrastructure.
References
[1] M. Pourreza and D. Rafiei, "DIN-SQL: Decomposed in-context learning of text-to-sql with self-correction," in Advances in Neural Information Processing Systems (NeurIPS), 2023.
[2] D. Gao, H. Wang, Y. Li, X. Sun, Y. Qian, B. Ding, and J. Zhou, "Text-to-sql empowered by large language models: A benchmark evaluation," Proc. VLDB Endowment, vol. 17, no. 5, pp. 1132-1145, 2024.
[3] T. Akidau, R. Bradshaw, C. Chambers, S. Chernyak, R. J. Fernandez-Moctezuma, R. Lax, S. McVeety, D. Mills, F. Perry, E. Schmidt, and S. Whittle, "The dataflow model: A practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing," Proc. VLDB Endowment, vol. 8, no. 12, pp. 1792-1803, 2015.
[4] A. Arasu, S. Babu, and J. Widom, "The CQL continuous query language: Semantic foundations and query execution," The VLDB Journal, vol. 15, no. 2, pp. 121-142, 2006.
[5] Q. Tan, H. T. Ng, and L. Bing, "Towards benchmarking and improving the temporal reasoning capability of large language models," in Proc. 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023, pp. 14,820-14,835.
[6] X. Chen, M. Lin, N. Scharli, and D. Zhou, "Teaching large language models to self-debug," in Int. Conf. on Learning Representations (ICLR), 2024.
[7] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, "Reflexion: Language agents with verbal reinforcement learning," in Advances in Neural Information Processing Systems (NeurIPS), 2023.
[8] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, "ReAct: Synergizing reasoning and acting in language models," in Int. Conf. on Learning Representations (ICLR), 2023.
[9] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark, "Self-refine: Iterative refinement with self-feedback," in Advances in Neural Information Processing Systems (NeurIPS), 2023.
[10] Z. Gou, Z. Shao, Y. Gong, Y. Shen, Y. Yang, N. Duan, and W. Chen, "CRITIC: Large language models can self-correct with tool-interactive critiquing," in Int. Conf. on Learning Representations (ICLR), 2024.
[11] T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev, "Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task," in Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), 2018, pp. 3911-3921.
[12] J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo, X. Zhou, C. Ma, G. Li, K. C.-C. Chang, F. Huang, R. Cheng, and Y. Li, "Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls," in Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023.
[13] P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas, "Apache flink: Stream and batch processing in a single engine," Bull. IEEE Computer Society Tech. Committee on Data Engineering, vol. 38, no. 4, pp. 28-38, 2015.
[14] F. Lei, J. Chen, Y. Ye, R. Cao, D. Shin, H. Su, Z. Suo, H. Gao, W. Hu, P. Yin, V. Zhong, C. Xiong, R. Sun, Q. Liu, S. Wang, and T. Yu, "Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows," in Int. Conf. on Learning Representations (ICLR), 2025.
[15] H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi, "CodeRL: Mastering code generation through pretrained models and deep reinforcement learning," in Advances in Neural Information Processing Systems (NeurIPS), 2022.
[16] T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, "Toolformer: Language models can teach themselves to use tools," in Advances in Neural Information Processing Systems (NeurIPS), 2023.
[17] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, "SWE-bench: Can language models resolve real-world github issues?" in Int. Conf. on Learning Representations (ICLR), 2024.
[18] T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama, "Is self-repair a silver bullet for code generation?" in Int. Conf. on Learning Representations (ICLR), 2024.
[19] A. Younesi, Z. Najafabadi Samani, and T. Fahringer, "AutoStreamPipe: LLM-assisted automatic generation of data stream processing pipelines," arXiv preprint arXiv:2510.23408, 2025.
[20] M. Armbrust, T. Das, J. Torres, B. Yavuz, S. Zhu, R. Xin, A. Ghodsi, I. Stoica, and M. Zaharia, "Structured streaming: A declarative API for real-time applications in apache spark," in Proc. ACM SIGMOD Int. Conf. on Management of Data, 2018, pp. 601-613.
Toor, M. J., Shabbir, S., Rauf, A., Nisar, A., Safdar, S., & Zaib-Un-Nisa. (n.d.). The role of structured-communication-technique in developing pragmatic language capability among ASD-children. https://doi.org/10.63878/cjssr.v3i2.785
Toor, M. J., Mehmood, H., Yasmeen, A., Honey, Q., Waseem, M., & Wasif, M. (n.d.). Exploring the effect of verbal encouraging phrases (VEP) on Chinese language learners’ academic achievement at university level.
Toor, M. J., Arafat, M., Ayoub, M., Ali, M., & Alessi Longa, F. E. (n.d.). From coherent self to fragmented persona: Digital identity fragmentation and academic self-concept among Generation Z, with comparative insights from Generation Alpha. https://doi.org/10.63878/cjssr.v4i2.2318
Katsitadze, N., Kharadze, N., & Pirtskhalaishvili, D. (2025). The impact of social media on students’ effective time management. Baltic Journal of Economic Studies, 11(1), 200–210. https://doi.org/10.30525/2256-0742/2025-11-1-200-210
Toor, M. J., Zafar, S., Omer, F., Sarwar, M., & Hussain, A. K. (n.d.). Teachers’ perspectives on effectiveness of translanguaging in enhancing English language learning at secondary school level.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Zhuoxi Wang, Jingwen Hou
This work is licensed under a Creative Commons Attribution 4.0 International License.
License Terms (Recommended – CC BY 4.0)
All articles published in Fundamental Scientific Reports in Multidisciplinary Areas are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
This license permits unrestricted use, distribution, and reproduction in any medium, provided the original author(s) and source are properly cited. Users are allowed to copy, redistribute, remix, transform, and build upon the material for any purpose, including commercial use, as long as appropriate credit is given.
Authors retain copyright of their work while granting the journal the right of first publication. The journal provides immediate open access to its content on the principle that making research freely available supports a greater global exchange of knowledge.
For more information about this license, visit:
https://creativecommons.org/licenses/by/4.0/
