End-to-End Machine Learning Pipelines Delivered Through Secure Microservices for Autonomous Operations in Digital Infrastructure Platforms
Keywords:
Machine Learning, Microservices, Digital Infrastructure, Secure Pipelines, Autonomous Operations, Service Mesh, Containerization, Edge ComputingAbstract
As digital infrastructures evolve in scale and complexity, the need for intelligent, autonomous operations has become imperative. This paper explores an architecture combining machine learning (ML) pipelines, microservices, and security frameworks to deliver end-to-end autonomous operations. We propose a modular, scalable system that leverages microservice containers to orchestrate ML models for infrastructure optimization. In this 2018 context, the integration of containerization (via Docker/Kubernetes), basic ML model deployment, and initial service mesh protocols is examined. The study highlights challenges, including data pipeline integrity, latency, and security, and proposes solutions using layered architecture and service-based isolation.
References
1. Fernandez, J., & Keshav, S. (2015). Microservices Architecture in Cloud Systems. ACM Computing Surveys, 47(2), 1-24.
2. Munnelly, D., & Banerjee, S. (2017). Distributed Microservices for Edge-Based Machine Learning Pipelines. Journal of Cloud Computing, 6(3), 45-57.
3. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep Learning. Nature, 521(7553), 436–444.
4. Zaharia, M., et al. (2016). Apache Spark: Unified Engine for Big Data Processing. Communications of the ACM, 59(11), 56–65.
5. Gummadi, V. P. K. (2019). Microservices architecture with APIs: Design, implementation, and MuleSoft integration. Journal of Electrical Systems, 15(4), 130–134. https://doi.org/10.52783/jes.9328
6. Kumar, A., & Singh, P. (2016). Securing Microservices with Service Mesh. IEEE Internet Computing, 20(6), 89–94.
7. Hashimoto, T. (2017). Monitoring for ML Pipelines. IEEE Transactions on Neural Networks and Learning Systems, 28(6), 1450–1462.
8. Li, K., & Zhang, Y. (2016). Machine Learning Pipelines for Real-Time Systems. Proceedings of the VLDB Endowment, 9(13), 1349–1361.
9. Ghosh, R., & Das, S. (2017). Autonomous Cloud Infrastructure. Future Generation Computer Systems, 73, 105–115.
10. Singh, R. (2016). Secure Container Deployment Practices. Journal of Cyber Security Technology, 1(1), 1–14.
11. Patel, S. (2017). Edge AI Deployment via Microservices. IEEE Access, 5, 12341–12352.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Josep Pla Saramago (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
