Machine Learning-Based Intelligent Data Engineering Using Serverless Cloud and API-Driven Architectures
DOI:
https://doi.org/10.15662/IJEETR.2026.0805001Keywords:
Machine Learning, Intelligent Data Engineering, Serverless Cloud Computing, API-Driven Architecture, Data Pipelines, Data Quality, Cloud AnalyticsAbstract
The rapid growth of cloud applications, distributed data sources, and real-time digital services has created a strong demand for intelligent data-engineering architectures that can process, integrate, and analyze information efficiently. Traditional data-engineering platforms often require continuous infrastructure provisioning, manual workflow configuration, and complex resource management. This paper proposes a Machine Learning-based intelligent data-engineering framework that combines serverless cloud computing, API-driven architectures, automated data pipelines, and machine learning techniques. The proposed framework enables dynamic data ingestion, transformation, quality assessment, feature engineering, anomaly detection, workload prediction, and intelligent resource optimization without requiring organizations to maintain continuously running infrastructure. API-driven components provide standardized interfaces for connecting enterprise applications, databases, IoT platforms, SaaS services, and external data sources. Machine learning models analyze pipeline behavior, identify data-quality problems, predict workload changes, and recommend or automate optimization decisions. Serverless functions dynamically execute data-processing tasks according to demand, improving scalability and reducing infrastructure management overhead. The methodology evaluates the proposed architecture using data-processing latency, throughput, resource utilization, cost efficiency, data-quality accuracy, scalability, and fault-recovery performance. Security mechanisms including API authentication, authorization, encryption, monitoring, and access policies are integrated into the architecture. The proposed framework provides an adaptive foundation for modern data engineering by combining intelligent analytics with event-driven and serverless cloud technologies.
References
1. Mohan, A. (2025). Recommender Systems in the Insurance Sector: Personalizing Customer Experiences. Journal of Computer Science and Technology Studies, 7(5), 129-133.
2. Chundi, V. R. K., Agarwal, V., & Arya, P. (2025, November). Green AI for Sustainable Supply Chains: Challenges in Emerging Economies. In 2025 International Conference on Computational Engineering, Sensing Technology and Management (ICCETM) (pp. 1-5). IEEE.
3. Vemireddy, S. (2022). Modernizing enterprise financial platforms through distributed cloud architectures. International Journal of Science, Research and Technology (IJSRAT), 5(2), 7420–7426.
4. Rajula, A. (2024). Drift-aligned personalization for privacy-preserving edge health monitoring. International Journal of Engineering & Extended Technologies Research (IJEETR), 6(5), 8895–8906.
5. Tatavarthi, S., Koilakonda, R. R., Bikkavolu, V., Tarakampet, S. K., & Gudala, M. (2026, April). Enterprise Governance for Generative AI: Prompt Lifecycle and Secure Middleware. In 2026 International Conference on Artificial Intelligence, Systems, and Emerging Technologies (ICAISET) (pp. 1-6). IEEE.
6. Bellundagi, M. (2023). Design of an Intelligent Clinical Decision Support System Using Machine Learning Techniques. International Journal of Research and Applied Innovations, 6(6), 10075-10081.
7. Narra, S. L. (2025). Cybersecurity and Digital Equity: Why Strong Identity Management is a Public Good. Journal Of Engineering And Computer Sciences, 4(7), 308-314.
8. Alvi, Y. M., Kumar, A., & Goel, S. (2026, April). Optimal Clustering with Deep Reinforcement Learning for Supply Chain Management. In 2026 International Conference on Frontiers of Engineering and Emerging Technologies (FET) (pp. 1-5). IEEE.
9. Sabin Begum, R., & Sugumar, R. (2019). Novel entropy-based approach for cost-effective privacy preservation of intermediate datasets in cloud. Cluster Computing, 22(Suppl 4), 9581-9588.
10. Sureshkumar, A., Maragatharajan, M., Jangiti, K., Karuppasamy, M., Jayabalan, K., Raut, P. T., & Sivakumar, N. R. (2026). A lattice-integrated AES framework for ultra-secure biometric protection on resource-constrained edge devices. Scientific Reports, 16(1), 7254.
11. Gopinathan, V. R. (2025). Enhancing Clinical Data Reliability through Predictive Machine Learning-Driven Intelligent DevOps Frameworks. International Journal of Research Publications in Engineering, Technology and Management (IJRPETM), 8(4), 12564-12571.
12. Challa, R. (2024). High-Availability HPC Cluster Design for Mission-Critical Public Infrastructure: Lessons from Energy Grid and Government. International Journal of Engineering & Extended Technologies Research (IJEETR), 6(6), 9305-9309.
13. Nisar, K. (2025). Designing a multi-criteria decision framework for enterprise language model adaptation. International Journal of Science, Research and Technology (IJSRAT), 8(6), 15449–15465.
14. Ferdausi, N. S., Fatema, N. K., Mahmud, N. M. R., Hoque, N. R., & Ali, N. M. (2025). Transforming telehealth with Artificial Intelligence: Predictive and diagnostic advances in remote patient care. World J Adv Eng Technol Sci, 16(1), 355-65.
15. Mathew, A., & Romasco, L. (2024). Forensic Investigation of Artificial Intelligence Systems. Research Updates in Mathematics and Computer Science Vol. 4, 154-164.
16. Gopakumar, S. (2026, April). Tenancy-Aware AI Automation for B2B SaaS Admin Workflows. In 2026 IEEE International Conference on Smart Sustainable Systems for Computer and Engineering Applications (3SCEA) (pp. 77-83). IEEE.
17. Anand, L. (2022). Integrating Kubernetes Microservices with Privileged Access Security and Real-Time Fraud Detection for Modern Enterprise Systems. International Journal of Research Publications in Engineering, Technology and Management (IJRPETM), 5(5), 7453-7461.
18. Patel, M., & Korat, U. (2026, June). Low-Power Sub-GHz Transceiver Design for Long-Range, Low-Bitrate Wireless Networks. In 2026 IEEE 18th International Conference on Computational Intelligence and Communication Networks (CICN) (pp. 98-103). IEEE.
19. Koganti, H. (2024). The computational complexity of hybrid algorithms: When branch-and-bound meets machine learning heuristics. International Journal of Research and Applied Innovations (IJRAI), 7(4), 11184–11190.
20. Bandaru, P. K. (2021). Leveraging hardware-in-the-loop simulation for continuous automotive software testing. International Journal of Engineering & Extended Technologies Research (IJEETR), 3(5), 3730–3735.
21. Tyagi, N. (2024). Deep reinforcement learning for algorithmic trading strategies. International Journal of Research and Applied Innovations, 7(2), 10415-10422.
22. Chowdhury, S. A., Hasan, M., & Hoque, M. J. (2026). Analyze How AI Improves Incident Response Time, Alert Prioritization, and Analyst Productivity in SOC Environments. American Journal of Technology Advancement, 3(6).
23. Yepuri, V. K., Polamarasetty, V. K., Donthi, S., & Gondi, A. K. R. (2023). Containerization of a polyglot microservice application using Docker and Kubernetes. arXiv preprint arXiv:2305.00600
24. Raja, G. V. (2022). AI Enabled Cloud and Data Engineering Frameworks for Secure, Scalable, and Intelligent Cyber Physical Analytics Systems. International Journal of Research and Applied Innovations, 5(4), 7395-7409.
25. Awopejo, T. E., Adigun, P. O., Oyekanmi, T. T., Azeez, N. A. A., Adekanye, M. A., & Obisesan, A. (2025). Machine learning-based prediction of magnetic properties from hysteresis curves: A comparative study of Random Forest, Gradient Boosting, XGBoost, LightGBM and Support Vector Regressions. International Journal of Research Publications in Engineering, Technology and Management, 8(6), 13456–13479.
26. Vimal Raja, G. (2024). Intelligent data transition in automotive manufacturing systems using machine learning. International Journal of Multidisciplinary and Scientific Emerging Research, 12(2), 515-518.
27. Padmanabham, S. (2022). Enterprise identity and access management architecture for large financial institutions. International Journal of Research and Applied Innovations, 5(1), 9486-9490.
28. Mohile, A., Yadav, A. L., Pandey, P. K., & Reddy, R. R. (2026, May). Automated Detection of Human Trafficking Networks Using Multimodal Data Analytics. In 2026 International Conference on Computational Robotics, Testing and Engineering Evaluation (ICCRTEE) (pp. 1-6). IEEE.
29. Sudhan, S. K. H. H., & Kumar, S. S. (2016). Gallant Use of Cloud by a Novel Framework of Encrypted Biometric Authentication and Multi Level Data Protection. Indian Journal of Science and Technology, 9, 44.
30. Balaraman, N. K., Patel, K., & Reddy, N. (October 2025). Smart Water Management: Integrating PLC and SCADA Technologies for Sustainable Urban Infrastructure. In Proceedings of the 22nd International Conference on Informatics in Control, Automation and Robotics (ICINCO) (Vol. 1, pp. 305–312). SciTePress.
31. Mirani, A. (2026). Designing AI-native financial systems: Architecture patterns for intelligent enterprise platforms. IPHO-Journal of Advance Research in Science and Engineering, 4(4), 21–33.
32. Jayaraman, S., Rajendran, S., & P, S. P. (2019). Fuzzy c-means clustering and elliptic curve cryptography using privacy preserving in cloud. International Journal of Business Intelligence and Data Mining, 15(3), 273-287.
33. Mali, R. K. (2026, July). AI-Driven Cloud-Native Banking Platforms: A Scalable Architecture for Real-Time Financial Services. In 2026 International Conference on Intelligent and Sustainable AI Systems (ICOSAAS) (pp. 749-756). IEEE.
34. Awopejo, T. E., Adigun, P. O., Oyekanmi, T. T., Azeez, N. A. A., Adekanye, M. A., & Obisesan, A. (2025). Machine learning-based prediction of magnetic properties from hysteresis curves: A comparative study of Random Forest, Gradient Boosting, XGBoost, LightGBM and Support Vector Regressions. International Journal of Research Publications in Engineering, Technology and Management, 8(6), 13456–13479.
