The Data Engine Room

 This is part two in the series of all of the September announcements for both fabric and power bi. For those that haven't read it please go and have a look at part one on the following link . This parts starts with the announcements around real-time intelligence. I'm not going to touch on the maps feature that was part of the keynotes, please go and see that separate blog for any of those announcements. Fabric continued. Azure monitor diagnostic logs integration via event stream With this new feature, we can now bring metrics and logs from as your Azure sources into fabric event streams. Creating a single location for advanced analytics on monitor diagnostic logs data. Private link support for event stream at Workspace level For a limited number of sources and destinations, we can now use private links at a Workspace level. If you need to use this level of security please do go and check the documentation. External parameter support for dataflow Gen 2 Relatively small update, ...

This month I thought I'd combine the Fabric release notes along with the Power BI release notes to give you a full view of what has been announced in the September releases.  This blog will cover features that have gone GA and for preview features there will be a separate blog to follow.  First up Fabric.  Fabric  General platform announcements The first section in this month's blog covers the general announcements about the platform. Multitasking UI updates Microsoft have put this one further down there release now it's however I'm gonna push it right to the top. The reason for that is though this particular feature is in preview, a lot of you will have already seen the changes that it has brought into the user interface. For example, we will start to see tabs across the top for Soda every time you open a new fabric object you get a new tab that is color coded based upon the Workspace that object is in. That's the most noticeable change from most developers and one...

About a year ago I wrote a blog on workspace topology . Since then, I have spoken to a number of customers and engineers at various meetups on capacities and the challenges they can present.  From those conversations, I've come to realise that a follow on is needed to to help demystify capacity management. For those that haven't read the previous blog, I clearly called out:   In this blog, I'm going to explain why this is the case and how you should go about managing workspace creation. The dangers of creating workspaces without planning Whilst Fabric makes it really easy to standup a new workspace on an existing capacity through the UI, a number of reasons exist today as to why we shouldn't be doing this. Financial management To understand the dangers of workspace creation, we need to first review the ways that Fabric items can be billed: Consumption unit (CU) purchased via SKU with Burst and Smooth functionality enabled (Default). This is the approach we're all us...

As the biggest European Fabric conference of the year kicks off in Vienna today we've had a number of key announcements that have been made this morning. Sadly I wasn't able to make this years as I chose to move jobs, but will push for next - it was the highlight of the year last year and have definately missed it this year. This blog covers all you need to know from this morning, as well as my thoughts. Graph databases As the digital twin functionality hinted at, we now get graph databases as a native object in Fabric. The unique power this brings is that data doesn't need to be duplicated into a new object to create these databases - unlike the likes of Neo4j, TigerGraph, etc. Beyond a couple of specific uses cases (e.g. supply chain analysis, manufacturing lines, tracking data for SAR requests, etc), I've always felt that these are a solution looking for a problem. But now that we have agentic systems, they add a whole load of context that traditional row or columner...

Well, it looks like my predictions as to what Microsoft would do didn't happen. What is normally a quite release period given summer holidays has been pretty full on and we got an August release drop as well. Fabric platform New view in deployment pipelines Sadly this is about as much as we get in the CI/CD space for Fabric - and it doesn't address what is currently the weakest part of the platform in my opinion (I've been saying this for a while now). All this one does is add a new view to make it easier to select components across multiple folders. Great if you are an SME with a small enough team to make deployment pipelines work for you, but it doesn't address the complexities that come with enterprise grade deployments still. API specifications For those using the Fabric APIs, we now have a Github repository that contains all the specifications for the available APIs. Data Engineering Auto-scale billing for spark I know a number of people haven't been happy with...

 Just when I thought the product team might be slowing down for year end and the summer break, they've only gone against my expectations and not only dropped a July feature - but a couple of updates since. Given this one wasn't expected, I'm not going to be able to do my usual breakdown of what is GA and what's not - but I'll try to call it out as we go along. Up first we've got a fairly major update to the Spark compute - one that changes the billing approach Autoscale billing for spark (GA - Generally Available) For those that haven't seen it, this is a pretty big shift in the way that the spark compute is billed. Most of you are probably familiar with the SKU based approach that Fabric follows - that is that you bay for a particular license level and you get a number of compute units (think of it as credits) to spend on the activities you do in Fabric. The idea being that it gets around the difficulty of budgeting for the CFO that consumption based proces...