Management Science and Information Technology

Management Science and Information Technology

SynapseDoc: Automated Documentation with a Multi-LLM-as-Judges Evaluation Framework for Critical Systems

Document Type : SI; OMCI 2025

Authors
Engineering School of Statistics and Information Analysis, University of Carthage, Tunis, Tunisia
10.22034/ISS.2026.10440.1071
Abstract
Objective. This study introduces SynapseDoc, an automated documentation generation system for critical software systems, coupled with a novel multi-LLM-as-Judges evaluation framework. The research aims to automate the conversion of source code into pedagogically structured tutorials and establish a scalable, unbiased method for assessing the quality of AI-generated documentation.
Methods. The system is built on a modular pipeline architecture using the PocketFlow orchestration framework. It processes code through sequential nodes for repository fetching, abstraction identification, relationship analysis, chapter ordering, content generation, and final assembly. The core innovation is the evaluation framework, which employs four different Large Language Models as independent judges. They assess generated documentation across nine weighted metrics (including Accuracy, Faithfulness, Completeness, Clarity, and Hallucination) using a rigorous scoring methodology that includes normalization, weighted aggregation, and content density analysis.
Results. The framework was tested on PyMedPhys, a real-world open-source medical physics library. Quantitative evaluation revealed a clear performance hierarchy among eight state-of-the-art LLMs. The Gemini family outperformed others significantly. These results demonstrate the critical impact of model selection on documentation quality, particularly for healthcare applications where factual reliability is paramount, and validate the effectiveness of the multi-judge approach in providing a balanced, nuanced assessment that prioritizes accuracy and faithfulness alongside structural integrity.
Conclusion. This work demonstrates that reliable, automated code documentation is achievable by integrating diverse LLMs for generation and evaluation. Through SynapseDoc and a multi-judge framework with hallucination penalties, we established a methodology for selecting optimal models in safety-critical contexts. Gemini emerged as the best choice for healthcare applications, while the modular framework enables ongoing model selection as LLMs evolve. This approach sets a benchmark for AI-generated documentation, paving the way for more intelligent software engineering tools. Future work includes scaling to diverse codebases, IDE integration, and incorporating multimodal elements.
Keywords

Baklouti, Y. B., Echcherqaoui, T., & Abdeljaoued-Tej, I. (2025). Smart enterprise document access with a chatbot powered by azure openai. 10th International Congress on Information and Communication Technology (ICICT’2025). Springer. https://doi.org/10.1007/ 978-981-96-6932-5_35
Benali, H., Melliti, M., & Abdeljaoued-Tej, I. (2024). Modèle de langage de vision pour l’extraction de texte en arabe: detection dans des cartes d’identité et reconnaissance faciale. 11ème Conférence Internationale en Automatique & Traitement de Signal (ATS’2024).
Bengio, Y., et al. (2003). A Neural Probabilistic Language Model. Journal of Machine Learning Research, 3, 1137–1155.
Biggs, S., Jennings, M., Swerdloff, S., Chlap, P., Lane, D., Rembish, J., ... & Sobolewski, M. (2022). PyMedPhys: A community effort to develop an open, Python-based standard library for medical physics applications. Journal of Open Source Software, 7(78), 4555. https://doi.org/10.21105/joss.04555
Chakrabarty, S., & Pal, S. (2024). Free and customizable code documentation with LLMs: A fine-tuning approach. Preprint arXiv. https://arxiv.org/abs/2412.00726
Charaa, T., Hamdeni, T., & Abdeljaoued-Tej, I. (2025). Editing fashion images with precision: A controlled inpainting method. The International Journal of Multimedia & Its Applications (IJMA), 17(3).                       
https://doi.org/10.5121/ijma.2025.17302
Charalampidou, S. (2019). Managing technical debt through software metrics, refactoring and traceability. [Thesis fully internal (DIV), University of Groningen]. University of Groningen. https://research.rug.nl/files/84473801/Chapter_4.pdf
Chen, S. F., & Goodman, J. (1999). An Empirical Study of Smoothing Techniques for Language Modeling. Computer Speech & Language.
Chen, L., Guo, Q., Jia, H., Zeng, Z., Wang, X., Xu, Y., Wu, J., ... & Zhang, S. (2025). A survey on evaluating large language models in code generation tasks. Preprint arXiv. https://arxiv.org/abs/2408.16498
Chu, Z., et al. (2024). History, Development, and Principles of Large Language Models: An Introductory Survey. Preprint arXiv:2402.06853.
Chung, J., et al. (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. Preprint arXiv:1412.3555.
Dagenais, B., & Robillard, M. P. (2010). Creating and evolving developer documentation: understanding the decisions of open source contributors. Proceedings of the Eighteenth ACM SIGSOFT International Symposium on Foundations of Software Engineering (p. 127–136). Association for Computing Machinery. https://doi.org/10.1145/1882291.1882312
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4171–4186.
Fattah, M., Ghobadi, N., & Ziari, M. (2025). Emerging Paradigms in Supply Chain Resilience Under Crisis Circumstances: A Roadmap for Future Investigations. https://doi.org/10.22034/ISS.2025.9049.1038
Fu, X.-Y., Laskar, M. T. R., Chen, C., & Tn, S. B. (2023). Are large language models reliable judges? a study on the factuality evaluation capabilities of LLMs. Dans S. Gehrmann et al. (Éds.), Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) (p. 310–316). Association for Computational Linguistics. https://aclanthology.org/2023.gem-1.25/
Hamdi, I., Lam-Weil, J., Martz, E., Abdeljaoued-Tej, I., & Capobianco, A. (2025). An automatic scoring method for responses to the hayling test. Proceedings of the 37th IEEE International Conference on Tools with Artificial Intelligence (ICTAI’2025). https://doi.org/10.1109/ICTAI66417.2025.00181
Jelodar, H., Meymani, M., & Razavi-Far, R. (2025). Large language models (LLMs) for source code analysis: Applications, models and datasets. Preprint arXiv. https://arxiv.org/abs/2503.17502
Khan, M. F. (2024). Evaluation of quantitative and qualitative metrics for assessing hallucination phenomena in LLMs.  Nuvern Applied Science Reviews, 8(10), 38–47. https://nuvern.com/index.php/nasr/article/view/2024-10-16
Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780.
Landolsi, H., & Abdeljaoued-Tej, I. (2025). The future of financial assistance: Leveraging LLMs for personalized, human-like interactions. International Journal of Web & Semantic Technology (IJWesT), 16(1).  https://doi.org/10.5121/ijwest.2025.16102
Malin, B., Kalganova, T., & Boulgouris, N. (2025). A review of faithfulness metrics for hallucination assessment in large language models. IEEE Journal of Selected Topics in Signal Processing. http://doi.org/10.1109/JSTSP.2025.3579203
Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., & Khudanpur, S. (2010). Recurrent neural network based language model. In Interspeech (Vol. 2, No. 3, pp. 1045-1048).
Mintlify. (2024). Beautiful docs for your codebase. https://mintlify.com/
Musah, L. (2025). Enhancing Supply Chain Performance and Agility in the Healthcare Industry Through Big Data Analytics: A Complex Adaptive Systems Theory Approach. Management Science and Information Technology, 2(1), 1-19. https://msit.refconf.com/article_2662.html
Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. OpenAI.
Rosenfeld, R. (2002). Two decades of statistical language modeling: Where do we go from here?. Proceedings of the IEEE, 88(8), 1270-1278.
Sanders, R., & Kelly, D. (2008). Dealing with risk in scientific software development. IEEE Software, 25(4), 21–28. http://doi.org/10.1109/MS.2008.84
Shen, J. H., & Tamkin, A. (2026). How AI impacts skill formation. Preprint arXiv:2601.20245. https://arxiv.org/abs/2601.20245
Sherstinsky, A. (2020). Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network. Physica D: Nonlinear Phenomena, 404, 132306.
Tabib, M., Abdeljaoued-Tej, I., & Raouafi, H. (2026). Enhancing large language models with retrieval augmented generation for a trusted help desk solution. Lecture Notes in Networks and Systems (LNSS), 1606.  https://link.springer.com/chapter/10.1007/978-3-032-03695-7_3
Tan, W. S., Wagner, M., & Treude, C. (2024). Detecting outdated code element references in software repository documentation. Empirical Software Engineering, 29(1), 5.
https://doi.org/10.1007/s10664-023-10397-6
Tyapukhin, A. P. (2025). Classification, Digitalization and Structure of Integrated Logistics Chains. Management Science and Information Technology, 2(3), 189-215. https://doi.org/10.22034/ISS.2025.9029.1031
Van der Spek, P., Klusener, S., & van de Laar, P. (2010). Complementing software documentation. In Views on evolvability of embedded systems (pp. 37-51). Dordrecht: Springer Netherlands. https://doi.org/10.1007/978-90-481-9849-8_3
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 5998–6008
Zhang, H., Haskell, J., & Frost, Y. (2025). Flow state: Humans enabling AI systems to program themselves. Preprint arXiv. https://arxiv.org/abs/2504.03771
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., ... & Wen, J. R. (2023). A survey of large language models. Preprint arXiv:2303.18223, 1(2), 1-124.

Articles in Press, Accepted Manuscript
Available Online from 15 October 2026