Data Formats for Data Engineering, Big Data and AI

Understand data formats used in modern data engineering, big data, analytics systems, and artificial intelligence

What you’ll learn
Understand the major data format categories used in modern data platforms and analytics systems.
Learn how tabular data formats such as Comma Separated Values, spreadsheets, and structured files are used for data storage and analysis.
Understand data serialization formats such as JavaScript Object Notation, Extensible Markup Language, Apache Avro, and Protocol Buffers.
Learn how big data storage formats such as Apache Parquet, Apache Optimized Row Columnar, and Apache Arrow are used in large scale analytics systems.
Understand modern data lake table formats such as Delta Lake, Apache Iceberg, and Apache Hudi.
Learn how artificial intelligence and vector embedding formats are used in machine learning and semantic search systems.

Requirements
No programming experience required. This course focuses on understanding data formats used in modern data platforms.
Basic familiarity with data, analytics, or databases is helpful but not required.
A computer with internet access to watch the lessons.
Interest in data engineering, analytics, big data, or artificial intelligence systems.

Description
Modern data platforms use many different data formats to store, exchange, and process information across systems. These formats are the foundation of data engineering pipelines, analytics platforms, distributed systems, and artificial intelligence applications.

This course provides a clear and practical overview of the most important data formats used in modern data platforms. Instead of focusing on programming implementation, this course explains what these formats are, why they exist, where they are used, and how they are typically processed in real-world data systems.

You will learn about structured and tabular data formats such as CSV (Comma Separated Values), TSV (Tab Separated Values), and spreadsheet formats commonly used for storing and sharing structured datasets.

The course also introduces widely used data serialization formats such as JSON (JavaScript Object Notation), XML (Extensible Markup Language), Apache Avro, Protocol Buffers, BSON, and MessagePack that are commonly used in APIs, distributed systems, and streaming platforms.

Next, we explore big data storage formats including Apache Parquet, Apache ORC, Apache CarbonData, Apache Arrow, Feather, and HDF5, which are designed for efficient analytics and large-scale data processing in modern big data environments.

You will also learn about modern data lake table formats such as Delta Lake, Apache Iceberg, Apache Hudi, and Apache Paimon that enable reliable data management in modern lakehouse architectures.

In addition, the course introduces media formats, graph and knowledge graph formats, and vector embedding formats used in artificial intelligence systems and machine learning applications.

Throughout the course, you will understand where each format is used and how common tools such as Python, data processing frameworks, and analytics systems interact with these formats in real-world data platforms.

By the end of this course, you will have a strong conceptual understanding of the major data formats used in modern data engineering, big data analytics, and artificial intelligence ecosystems.

Who this course is for
Data analysts who want to understand how modern data formats are used in analytics systems.
Aspiring data engineers who want to learn the formats used in data pipelines and big data platforms.
Machine learning engineers interested in vector embeddings and artificial intelligence data formats.
Students and beginners exploring careers in data engineering, big data, and artificial intelligence.
Software developers who want to understand how data is stored, exchanged, and processed across modern data systems.