DRDP : dynamically re-configurable data pipeline in the EDGE network

Abstract

Pipelines are a highly discussed topic in today’s technological world. There are different variations of pipelines; Data Science pipelines, DevOps pipelines, and DevSecOps pipelines, etc. A data science pipeline usually comes with a fixed architecture, which can be problematic in a fast-growing tech industry.Traditional data science pipelines may struggle to handle the volume, velocity, and variety of data at the edge, necessitating more dynamic and adaptable approaches. Many advancements are happening to bring the technology to the edge due to substantial data points generated in the sensor networks at the edge; from factory floors to log streams. So, in this thesis we first discuss the existing literature in the data pipeline domain under three main topics; data pipeline challenges, data pipeline architectures, and data pipeline security. Then we propose a methodology for dynamically re-configurable data pipeline architecture in the edge network. This way we expect to achieve more efficiency, controllability, and scalability of the data across networks.The emerging field of edge architecture presents opportunities for innovative approaches to data pipelines, enabling organizations to harness the full potential of edge data for advanced analytics, machine learning, and real-time decision-making.Further, we propose a prototype with Raspberry Pi-based programs to discuss the effectiveness of this novel method. Using this proposed architecture we have evaluated the results and later discussed how this benefits the current and future data pipeline implementation. We hope this contributes to the emerging edge architecture subject area. Keywords: data science, pipeline, architecture, edge