{"id":98,"date":"2019-02-06T19:40:00","date_gmt":"2019-02-06T18:40:00","guid":{"rendered":"http:\/\/daxter.nl\/?p=98"},"modified":"2020-09-11T20:21:04","modified_gmt":"2020-09-11T19:21:04","slug":"refresh-any-oauth2-api-from-powerbi-online-dataflows-using-azure-functions","status":"publish","type":"post","link":"http:\/\/daxter.nl\/en\/refresh-any-oauth2-api-from-powerbi-online-dataflows-using-azure-functions\/","title":{"rendered":"Refresh (any) OAuth2 API from PowerBI Online \/ Dataflows, using Azure Functions"},"content":{"rendered":"\n<p>Some time ago, I had an intake with a new client; he wanted to analyze his financial data coming from Exact Online, using Power BI. After a quick search, I thought there was no problem. There were many other companies who offered this service and the <a href=\"https:\/\/support.exactonline.com\/community\/s\/knowledge-base#All-All-DNO-Content-restrefdocs\" rel=\"noreferrer noopener\" target=\"_blank\">API documentation<\/a> was comprehensive. I even found a <a href=\"http:\/\/blog.refine-it.nl\/self-service-data-ophalen-vanuit-exact-online-in-power-bi\/\" rel=\"noreferrer noopener\" target=\"_blank\">step-by-step tutorial<\/a>.&nbsp;<br>But, during development, there were many problems to overcome in order to make it work. To share my experience with the rest of the world, I decided to write this blog (my first one!). I\u2019ll show you how to connect an API which uses OAuth2 authentication to PowerBI, refresh the data online and probably the most beautiful part: for free*. Key to this solution is the use of Azure Functions, to act as a gateway between the PBI Service and the API.<\/p>\n\n\n\n<p><em>Note: all assumptions and conclusions about (the lack of) technical features are subject to change. Power BI is developing very fast, and by the time you read this, it could be obsolete.<\/em><\/p>\n\n\n\n<p><strong>The Problem<\/strong><br>The first problem is that PowerBI <a href=\"https:\/\/ideas.powerbi.com\/forums\/265200-power-bi-ideas\/suggestions\/13195278-oauth\" rel=\"noreferrer noopener\" target=\"_blank\">out-of-the-box doesn\u2019t support OAuth2 API sources<\/a>. It can\u2019t handle the authentication flow, using the standard <em>Get Data &gt; From Web<\/em> option. There are <a href=\"http:\/\/angryanalyticsblog.azurewebsites.net\/index.php\/2016\/05\/16\/api-strategies-with-power-bi\/\" rel=\"noreferrer noopener\" target=\"_blank\">some options<\/a> get the data via complex M (PowerQuery) code, but then online refresh would still be an issue. If you want to use OAuth2, you must write (or buy) a Custom Connector. Unfortunately, with a custom connector, you can only refresh the data with an On-Premise Data Gateway, which requires a (windows) pc online at refresh time. I was looking for a solution to connect a SaaS application to Power BI, without the need of any resource on-premise.<\/p>\n\n\n\n<p><strong>The Solution<\/strong><br>To refresh the data directly and get rid of the OAuth2 auth flow, I\u2019ve used <a href=\"https:\/\/docs.microsoft.com\/azure\/azure-functions\/functions-overview\" rel=\"noreferrer noopener\" target=\"_blank\"><em>Azure Functions<\/em><\/a>. In short: with Azure Functions, you can execute code in the cloud (Azure). More beautiful; it can act as an API which you can use to send or receive data.&nbsp;<br>In plain text: the original request will not be sent directly to the API, but to the Azure Function. This piece of code will handle the authentication and sends the original request to the API. The response will be returned as result of the Azure Function. The Azure Function acts as a gateway between the caller (PowerBI) the service.<\/p>\n\n\n\n<figure class=\"wp-block-image alignwide\"><img src=\"https:\/\/cdn-images-1.medium.com\/max\/1200\/1*cIp_iIz3ltcCaz7-JKY_DQ.png\" alt=\"\"\/><\/figure>\n\n\n\n<p><strong>The Code<br><\/strong>The Azure Function contains three parts; the main function call with the logic, a separate access code routine to perform the authentication request and a code block to store the response refresh code into the Table Storage.<\/p>\n\n\n\n<p>To call the function, you must provide the original API call in the <em>name <\/em>query parameter. So the call to&nbsp;<br><code>https:\/\/yourfunctionurl\/function?name=\/financial\/GLAccounts<\/code><br>will be translated to<br><code>https:\/\/start.exactonline.nl\/api\/v1\/division\/financial\/GLAccounts<\/code><\/p>\n\n\n\n<p>These steps will follow:<\/p>\n\n\n\n<ul><li>Get latest refresh token from Table Storage<\/li><li>Request new AccessCode (with previous <em>refresh token<\/em>, <em>client id<\/em> &amp; <em>secret<\/em>)<\/li><li>Perform request to the given API (<em>base_url<\/em>), with the division code AND provided API call (<em>name<\/em>)<\/li><li>Return response<\/li><\/ul>\n\n\n\n<p>To perform the API request, I\u2019m using the <a href=\"https:\/\/www.nuget.org\/packages\/RestSharp\/\" rel=\"noreferrer noopener\" target=\"_blank\">RestSharp<\/a> library.<\/p>\n\n\n\n<p>Function Call:<\/p>\n\n\n\n<script src=\"https:\/\/gist.github.com\/GhandiHarder\/259689220b86cc6045c70228f27045ed.js\"><\/script>\n\n\n\n<p><\/p>\n\n\n\n<p>Access Token part:<\/p>\n\n\n\n<script src=\"https:\/\/gist.github.com\/GhandiHarder\/cef5254f4c9c0e693190761e584bfc37.js\"><\/script>\n\n\n\n<p><\/p>\n\n\n\n<p>Add Token part:<\/p>\n\n\n\n<script src=\"https:\/\/gist.github.com\/GhandiHarder\/7dc546e59e70dfc6c670887a9245186d.js\"><\/script>\n\n\n\n<p><strong>The Result<\/strong><\/p>\n\n\n\n<p>To test the function, I\u2019m calling the function from the browser.&nbsp;<br><em>Please note that in order to make this work, you have to send the client id &amp; secret with the request.<\/em><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/1*1U4PQz2Df9IQhyCUdHRspg.png\" alt=\"\"\/><\/figure>\n\n\n\n<p>Check the function logging from the Azure Portal. The translated request is red marked.<\/p>\n\n\n\n<figure class=\"wp-block-image alignwide\"><img src=\"https:\/\/cdn-images-1.medium.com\/max\/2560\/1*3iNXjP1yWNFT4UhwVS-Bng.png\" alt=\"\"\/><\/figure>\n\n\n\n<p><strong>Power BI<br><\/strong>Now (finally!) we can build our report. No complex M code, just a simple Web Request.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/cdn-images-1.medium.com\/max\/1200\/1*YMNawv6xRI0q00i4LB3UAg.png\" alt=\"\"\/><\/figure>\n\n\n\n<p>And, after publishing the file to the Power BI Service, we hit Refresh to check if it\u2019s working as expected:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/1*kAmMlo3XJpNFqfV0hVBFQw.png\" alt=\"\"\/><\/figure>\n\n\n\n<p>Voila&nbsp;\ud83d\ude42<\/p>\n\n\n\n<p>Now, we can even make use of the new <a href=\"https:\/\/powerbi.microsoft.com\/nl-nl\/blog\/introducing-power-bi-data-prep-wtih-dataflows\/\" rel=\"noreferrer noopener\" target=\"_blank\">DataFlows<\/a> in the Power BI Service:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/cdn-images-1.medium.com\/max\/1200\/1*nabLTmwWE-RtQo8NRuCVCg.png\" alt=\"\"\/><\/figure>\n\n\n\n<p><strong>Considerations<\/strong><\/p>\n\n\n\n<p>Of course, security is an issue. Since you\u2019re no longer using the OAuth2 flow (between Power BI and Function), you must implement your own. Please do not save the sensitive credentials (client id, secret, division) hardcoded in the Function, but sent them in the (secured) header. You can force the Function to use only SSL. To secure the Azure Functions, there are multiple options, please refer to the following resources:<\/p>\n\n\n\n<ul><li><a href=\"https:\/\/www.linkedin.com\/pulse\/whitelisting-azure-functions-mike-douglas\/\" rel=\"noreferrer noopener\" target=\"_blank\">https:\/\/www.linkedin.com\/pulse\/whitelisting-azure-functions-mike-douglas\/<\/a><\/li><li><a href=\"https:\/\/vincentlauzon.com\/2017\/12\/04\/azure-functions-http-authorization-levels\/\" rel=\"noreferrer noopener\" target=\"_blank\">https:\/\/vincentlauzon.com\/2017\/12\/04\/azure-functions-http-authorization-levels\/<\/a><\/li><\/ul>\n\n\n\n<p>Besides the security, the Function is not \u2018production ready\u2019; there are several features that need to be implemented, like error handling and API paging. You could handle the API paging inside the Function, but its runtime is limited to 10 minutes (5 by default, see below) so it might fail when there\u2019s a lot of data. You could handle paging by using M code (PowerQuery), I suggest reading this blog from Chris Webb: <a href=\"https:\/\/blog.crossjoin.co.uk\/2016\/08\/16\/using-the-relativepath-and-query-options-with-web-contents-in-power-query-and-power-bi-m-code\/\" rel=\"noreferrer noopener\" target=\"_blank\">https:\/\/blog.crossjoin.co.uk\/2016\/08\/16\/using-the-relativepath-and-query-options-with-web-contents-in-power-query-and-power-bi-m-code\/<\/a><\/p>\n\n\n\n<p>Another, very important issue; you have to make sure your Function is only running once instance a time. When you request an access code, your previous refresh code gets invalid\u200a\u2014\u200athat\u2019s why you must store it immediately for the next request. If you run two or more Functions at the same time, you end up in doing requests with old refresh tokens. To make sure you can only run one Function, you can set some settings in the host.json file:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">{<br>\"version\": \"2.0\",<br><strong>\"functionTimeout\": \"00:10:00\",<\/strong><br>\"extensions\": {<br>       \"http\": {<br>          \"routePrefix\": \"api\",<br>          \"maxOutstandingRequests\": 50,<br>          <strong>\"maxConcurrentRequests\": 1,<\/strong><br>          \"dynamicThrottlesEnabled\": true<br>          }<br>       }<br>}<\/pre>\n\n\n\n<p>As you can see, this is also the place to increase the default timeout from 5 to 10 minutes (max). If you want to implement paging and need a bigger timeout, you must switch from Consumption Plan to App Service Plan. See pricing below.<\/p>\n\n\n\n<p><strong>Really free?<\/strong><\/p>\n\n\n\n<p>Well, if you run your function in the Consumption Plan, then it\u2019s free up to 1 million (!!) calls. There is a small fee (almost nil) for the used table storage. See the Azure Function Pricing <a href=\"https:\/\/azure.microsoft.com\/pricing\/details\/functions\/\" rel=\"noreferrer noopener\" target=\"_blank\">here<\/a>, and <a href=\"https:\/\/azure.microsoft.com\/pricing\/details\/storage\/tables\/\" rel=\"noreferrer noopener\" target=\"_blank\">this<\/a> for the Table Storage.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\"><p>To wrap it up; by using an Azure Function as a gateway, you can connect and refresh Power BI Online \/ Dataflow with any OAuth2 source you\u2019d like, without the need of a custom connector, or on-premise server for (almost) free.<\/p><\/blockquote>\n\n\n\n<p><em>Of course, the code used in this example only works for Exact Online, but I hope you get the point.<\/em><\/p>\n\n\n\n<p><em>This was my first blog. Thanks for reading! If you have any complaints or suggestions (both technical and\/or about my writing skills), please let me know!<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Some time ago, I had an intake with a new client; he wanted to analyze his financial data coming from Exact Online, using Power BI. After a quick search, I thought there was no problem. There were many other companies who offered this service and the API documentation was comprehensive. I even found a step-by-step&hellip;&nbsp;<a href=\"http:\/\/daxter.nl\/en\/refresh-any-oauth2-api-from-powerbi-online-dataflows-using-azure-functions\/\" class=\"\" rel=\"bookmark\">Read More &raquo;<span class=\"screen-reader-text\">Refresh (any) OAuth2 API from PowerBI Online \/ Dataflows, using Azure Functions<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"","neve_meta_content_width":0,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"http:\/\/daxter.nl\/en\/wp-json\/wp\/v2\/posts\/98"}],"collection":[{"href":"http:\/\/daxter.nl\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/daxter.nl\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/daxter.nl\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/daxter.nl\/en\/wp-json\/wp\/v2\/comments?post=98"}],"version-history":[{"count":2,"href":"http:\/\/daxter.nl\/en\/wp-json\/wp\/v2\/posts\/98\/revisions"}],"predecessor-version":[{"id":100,"href":"http:\/\/daxter.nl\/en\/wp-json\/wp\/v2\/posts\/98\/revisions\/100"}],"wp:attachment":[{"href":"http:\/\/daxter.nl\/en\/wp-json\/wp\/v2\/media?parent=98"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/daxter.nl\/en\/wp-json\/wp\/v2\/categories?post=98"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/daxter.nl\/en\/wp-json\/wp\/v2\/tags?post=98"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}