Data Engineer

Overview:As a Data Engineer specializing in Cloud Image and Edge Device handling, you will play a crucial role inthe development and maintenance of our data infrastructure, particularly focused on managing imagedata from edge devices and processing it efficiently in cloud environments. You will be responsible fordesigning, implementing, and optimizing data pipelines to ensure seamless ingestion, storage, andanalysis of image data from various sources. Additionally, you will collaborate closely with crossfunctionalteams to integrate data-driven solutions into our products and services. Responsibilities:Data Pipeline Development: Design, implement, and maintain scalable data pipelines for ingesting,processing, and analyzing image data from edge devices to cloud-based storage and computingplatforms. Data Architecture: Design and Architect a Data Lake for structured and unstructured data over cloud(e.g., AWS, Azure, Google Cloud) for efficient storage, processing, and retrieval of data, ensuring highavailability, reliability, and performance. Edge Device Integration: Develop strategies and mechanisms for seamless integration of edge deviceswith cloud-based systems, including data synchronization, device management, and security protocols. Image Data Processing: Implement algorithms and techniques for real-time processing, transformation,and analysis of image data, optimizing for speed, accuracy, and resource efficiency. Data Quality and Governance: Implement data quality checks, validation processes, and governanceframeworks to ensure the integrity, consistency, and security of image data throughout its lifecycle. Performance Optimization: Identify performance bottlenecks in data pipelines and cloud infrastructure,and implement optimizations to improve throughput, latency, and cost-effectiveness. Collaboration and Communication: Collaborate with cross-functional teams, including softwareengineers, data scientists, and domain experts, to understand requirements, prioritize tasks, and deliverintegrated solutions. Documentation and Best Practices: Document design decisions, implementation details, and bestpractices for data engineering processes, ensuring knowledge sharing and continuous improvementwithin the team. Qualifications:Bachelor's or Master's degree in Computer Science, Engineering, or related field. Proven experience as a Data Engineer, preferably with specialization in handling image data. Strong proficiency in cloud computing platforms (e.g., AWS, Azure, Google Cloud) and related services(e.g., S3, EC2, Lambda, Kubernetes). Experience with data engineering tools like DataBrick, Snowflake, Glue etc. Proficiency in programming languages commonly used in data engineering (e.g., Python, Scala, Java) andfamiliarity with relevant libraries and frameworks (e.g., Apache Spark, TensorFlow, OpenCV). Solid understanding of data modeling, schema design, and database technologies (e.g., SQL, NoSQL,data warehouses). Familiarity with DevOps practices, CI/CD pipelines, and containerization technologies (e.g., Docker,Kubernetes). Strong problem-solving skills, analytical thinking, and attention to detail. Excellent communication and collaboration skills, with the ability to work effectively in a cross-functionalteam environment.