ETL with Power Query: Import, Transform, and Load Data Efficiently

What is ETL?

ETL stands for Extract, Transform, Load — a process used to gather data from various sources, clean and shape it, and load it into a target system like a data warehouse or a Power BI data model.

ETL in Power BI with Power Query

Power BI performs ETL operations using Power Query Editor, which is a built-in tool for data preparation.

1. Extract (E)

Get Data

  • You pull data from various sources like Excel, SQL Server, SharePoint, Web APIs, Azure, etc.

  • In Power BI: Click Home > Get Data to import your data.

2. Transform (T)

Clean and Shape Data

  • This is the core strength of Power Query.

  • You can:

    • Filter rows (e.g., remove nulls)

    • Rename columns

    • Merge or split columns

    • Change data types

    • Unpivot or pivot data

    • Remove duplicates

    • Create custom columns using M code

  • Every action becomes a step in the query, which is repeatable and refreshable.

3. Load (L)

Push Data to Power BI Model

  • After transformation, load the clean data into Power BI’s data model.

  • This data is now ready for creating visuals, measures, and reports.

  • You can also load it to Power BI Dataflows if building centralized models.

 Example:

Suppose you have messy Excel files from five departments. Using Power Query, you can:

  • Extract all files from a folder,

  • Merge them into a single table,

  • Remove duplicates and correct date formats,

  • Then load the cleaned dataset into Power BI for dashboarding.

Key Benefits of ETL in Power Query:

  • No need for external ETL tools for many cases.

  • Visual interface, no-code/low-code.

  • Reusable and refreshable queries.

  • Seamless integration with Power BI.


Comments

Post a Comment

Popular posts from this blog

Decode and Fix: “This Data Type is Unsupported for Evaluation” in Power Apps

Image
 One common frustration when building Canvas Apps over Dataverse is encountering this cryptic error:  “This data type is unsupported for evaluation” This message typically appears when Power Apps cannot process or render a specific Dataverse column type within a control or formula. While it may seem vague, it’s usually tied to specific patterns involving complex column types or runtime data structure mismatches. In this blog, we'll explore common causes, scenarios, and practical fixes from both a developer and architectural standpoint. Root Causes of the Error 1. Polymorphic Lookups Dataverse supports polymorphic relationships (e.g., `Owner`, `Customer`, `Regarding`) that can point to multiple entity types. These lookups can't be directly evaluated in Power Apps unless you cast the reference explicitly. Use: If(     IsType(ThisItem.Customer, Account),     AsType(ThisItem.Customer, Account).Name,     AsType(ThisItem.Customer, Contact).FullName ) 2....
Read more

Effective Strategies for Debugging Plugins in Dynamics CRM

Image
In a recent interview, I was asked about debugging plugins in Dynamics CRM. The interviewer specifically wanted to know my approach to plugin debugging. In response, I mentioned using the Plugin Registration Tool profiler. However, I later realized there are multiple methods to debug plugins in Dynamics CRM/Dataverse, each essential for efficient troubleshooting. Debugging plays a crucial role in development, and having a solid understanding of various debugging approaches can help streamline issue resolution in Dynamics 365. Here, I'll explore different techniques for debugging plugin code. There are below options to use in the plugin in Dynamics CRM/Dataverse ITracingService : In Dynamics CRM plugin development, ITracingService is a service provided within the plugin's execution context that logs information for tracing and debugging. It allows developers to capture real-time. Plugin Profiler : A plugin profiler in Dynamics CRM (or Dataverse) is a debugging tool in the Plugi...
Read more

Dynamics 365 Plugin Life Cycle Simplified for Business Users and Developers

Image
The Dynamics 365 Plugin Life Cycle describes the series of steps that a plugin follows when a record is created, updated, deleted, or otherwise changed in Dataverse. In simple terms, a plugin is like an automated rule that wakes up when something happens in the system. As the event occurs, the plugin passes through different pipeline stages— Pre-Validation, Pre-Operation, Post-Operation, and Asynchronous Processing —each with its own purpose. Pre-Validation allows very early checks before the system applies business rules, Pre-Operation lets you validate or modify data before it is saved, Post-Operation triggers after the record is committed to the database, and Async handles background tasks like integrations or long-running operations. Understanding this life cycle helps ensure business logic runs at the right moment, prevents bad data from entering the system, and enables smooth automation and integrations within Dynamics 365. Plugins in Dynamics 365 are pieces of custom code that ...
Read more

Connecting the Dots: FetchXML and Web API Integration in Dataverse

Image
 In Dataverse, FetchXML and the Web API are two powerful mechanisms for retrieving data and interacting with entities. Here's a brief overview of each and how they can be integrated: 1.    FetchXML :      -   Definition:   FetchXML is a proprietary query language used in Microsoft Dynamics 365 and Dataverse for querying data from entities.    -   Usage:   FetchXML queries can be used to retrieve data based on complex criteria, perform aggregations, and filter records.    -   Features:   FetchXML supports various query expressions, including conditions, sorting, grouping, and aggregation functions.    -   Integration:   FetchXML queries can be executed programmatically using client-side scripting, server-side code, or through tools like the Dynamics 365 Web API. 2.    Web API :      -   Definition:  ...
Read more

Step-by-Step Guide to Building Custom APIs in Dataverse

Image
Custom APIs in Microsoft Dataverse allow developers and architects to create their own reusable, secure, and well-defined service endpoints that extend the standard Dataverse Web API. Instead of relying solely on out-of-the-box messages (like Create, Update, or Retrieve ), Custom APIs let you define your own business operations — complete with input and output parameters — that encapsulate complex logic and can be called from Power Automate, Power Apps, or external systems. These APIs are registered once and behave like native Dataverse operations, making them ideal for implementing domain-specific logic, integrating with external services, or simplifying client-side development. From a technical standpoint, a Custom API can be implemented through a plugin class, where the backend logic is written in C#. Developers can control security through privilege definitions and ensure consistency by centralizing key business rules. Custom APIs are particularly valuable in enterprise environmen...
Read more

Managing Parallel Development and Hotfixes in Dataverse Like a Pro

Image
Managing parallel development and hotfixes in Dataverse means keeping new features and urgent fixe s moving forward at the same time without breaking the system. Because multiple teams often work in different solution layers , it becomes important to separate work clearly using branching, small modular solutions, and structured environments. While feature teams work on future releases in their own branches, hotfixes must be created quickly from the production branch, packaged in focused solutions, tested, and deployed with minimal impact. After a hotfix goes live, it must always be merged back into the development stream so it isn’t overwritten in the next release. With a good CI/CD pipeline, versioning, and clear release governance, organizations can deliver changes safely, avoid solution conflicts, and maintain stability even when several teams are building and fixing at the same time. Working with Dataverse often means several developers, consultants, and teams are customizing the...
Read more

Stop Struggling with Plugins: Learn IOrganizationService the Smart Way

Image
Have you ever deployed a Dataverse plugin, only to watch it fail silently or act unpredictably? Most of the time, the missing piece is understanding how to communicate with Dataverse the smart way—using IOrganizationService . In the world of Dataverse, every plugin waits for its turn when something changes. The plugin looks around and wonders, “How do I talk to the database?” That’s when IOrganizationService steps in like a trusted messenger, ready to carry out requests and bring back results. “ Need to create a record? I can handle that,” it says. “ Want to update or delete data safely ? Leave it to me.” It even speaks the secret language of Dataverse messages like Assign and SetState . But it never works alone— IOrganizationServiceFactory brings it to life, deciding whether it should act as the user or as the powerful system account. Working together with IPluginExecutionContext , it always knows who triggered the action, which record is involved, and when to respond. What is IOrgan...
Read more

How Every Plugin Action Travels Through Request, Service, and the Pipeline

Image
In Dataverse plugin development, a request is simply a message that tells Dataverse what action to perform. This could be a basic action like Create, Update, Delete, Retrieve , or a special action like Assign or SetState . Understanding requests is important for developers and architects because they affect how the plugin runs, how transactions work, and how multiple record updates stay consistent. Think of it this way:  A request says what needs to happen  A service (`IOrganizationService`) is the worker that does the job Together, they are the core of how plugins work in Dataverse. They:  Allow you to design plugins that work in a single transaction  Make sure changes are consistent and traceable  Work smoothly with the plugin pipeline to follow business logic For architects:   Service = the executor   Request = the action   Plugin pipeline = the traffic controller that manages transactions and triggers during plugin execution. Service (`IOrgan...
Read more

Bridging Dataverse and External APIs with Custom API

Image
Modern business applications rarely operate in isolation. Organizations increasingly need their Microsoft Dataverse environments to interact with external systems such as CRMs, ERPs, banking systems, SAP, ServiceNow, payment gateways, logistics platforms, or AI-enabled services. Custom API in Datavers e acts as the perfect bridge—a controlled, secure, and scalable layer that connects Dataverse to external APIs.  What Does “Bridging Dataverse and External APIs” Mean? To “bridge” Dataverse with an external API means: Dataverse triggers a process (from a plugin, Power Automate, Canvas App, Model-Driven App, or even external systems) Custom API receives the request Custom API executes server-side code (Plugin) written in C# The code makes an outbound call to an external REST/SOAP endpoint Returns the processed result back to Dataverse , or updates records This creates a secure, governed, and high-performance integration layer—without exposing Dataverse directly to outside systems. Wh...
Read more

Model-Driven Apps Command Bar: A Guide to PrimaryControl and SelectedControl

Image
The command bar in Power Apps model-driven apps is a toolbar that lets users perform actions. You can customize it by adding buttons that run your own logic, either using Power Fx or JavaScript. This logic can work with a single data record or multiple records, depending on where the button is placed. The CrmParameter in the ribbon of Dynamics CRM (now Dynamics 365) is a special parameter used in the ribbon's button definitions or commands. It provides contextual information about the entity or record within the CRM system when the ribbon button is clicked. Purpose of CrmParameter CrmParameter is primarily used to pass specific CRM-related data to a custom JavaScript function, URL, or command action. It helps in retrieving dynamic data based on the current context, such as entity IDs, selected records, or even the page's user interface settings. 1. PrimaryControl Definition : Refers to the main form context or entity record currently being displayed or interacted with in the ap...
Read more