The Data Engine Room

About a year ago I wrote a blog on workspace topology . Since then, I have spoken to a number of customers and engineers at various meetups on capacities and the challenges they can present.  From those conversations, I've come to realise that a follow on is needed to to help demystify capacity management. For those that haven't read the previous blog, I clearly called out:   In this blog, I'm going to explain why this is the case and how you should go about managing workspace creation. The dangers of creating workspaces without planning Whilst Fabric makes it really easy to standup a new workspace on an existing capacity through the UI, a number of reasons exist today as to why we shouldn't be doing this. Financial management To understand the dangers of workspace creation, we need to first review the ways that Fabric items can be billed: Consumption unit (CU) purchased via SKU with Burst and Smooth functionality enabled (Default). This is the approach we're all us...

As the biggest European Fabric conference of the year kicks off in Vienna today we've had a number of key announcements that have been made this morning. Sadly I wasn't able to make this years as I chose to move jobs, but will push for next - it was the highlight of the year last year and have definately missed it this year. This blog covers all you need to know from this morning, as well as my thoughts. Graph databases As the digital twin functionality hinted at, we now get graph databases as a native object in Fabric. The unique power this brings is that data doesn't need to be duplicated into a new object to create these databases - unlike the likes of Neo4j, TigerGraph, etc. Beyond a couple of specific uses cases (e.g. supply chain analysis, manufacturing lines, tracking data for SAR requests, etc), I've always felt that these are a solution looking for a problem. But now that we have agentic systems, they add a whole load of context that traditional row or columner...

Well, it looks like my predictions as to what Microsoft would do didn't happen. What is normally a quite release period given summer holidays has been pretty full on and we got an August release drop as well. Fabric platform New view in deployment pipelines Sadly this is about as much as we get in the CI/CD space for Fabric - and it doesn't address what is currently the weakest part of the platform in my opinion (I've been saying this for a while now). All this one does is add a new view to make it easier to select components across multiple folders. Great if you are an SME with a small enough team to make deployment pipelines work for you, but it doesn't address the complexities that come with enterprise grade deployments still. API specifications For those using the Fabric APIs, we now have a Github repository that contains all the specifications for the available APIs. Data Engineering Auto-scale billing for spark I know a number of people haven't been happy with...