Monthly Archives: September 2019

A comprehensive guide to Power BI performance tuning

Promotion: Use code PERFBLOG to save 10% off my course. Module 1 is free.

While it’s easy to make reports in Power BI, it can be a pain to troubleshoot and tune them. Properly performance tuning Power BI reports requires identifying the bottleneck and using a handful of external applications. This article covers how to narrow down the performance problem, as well as general best practices. Once you understand the fundamentals it gets a lot easier.

Which part is slow?

As a report developer, it can be frustrating a report developer, knowing that something is slow, but not being able to put your finger on it. In my mind, there are 4 main areas where there might be a slowdown:

  1. Data refresh
  2. Model calculations
  3. Visualization rendering
  4. Everything else

Identifying which one of these is the problem is the first step to improving performance. In most cases, if a report is slow it’s an issue with step 2, your data model.

Tuning the data refresh

The refresh is rarely the problem because the user never sees the refresh process if you are using Import mode. So usually you are going to see a slow refresh when you are authoring the report or if a scheduled refresh fails. Still, it’s important to tune your data refresh to avoid timeouts and minimize how much data your are loading.

Verify that query folding is working

If you are querying a relational database, especially SQL Server, then you want to make sure that query folding is being applied. Query folding is when M code in PowerQuery is pushed down to the source system, often via a SQL query. One simple way to confirm that query folding is working is to right click on a step sand select View Native Query. This will show you the SQL query that will be run against the database. If you have admin privileges on the server, you can also use extended events to monitor the server for queries. If you are doing this, you should limited the monitoring to the Data Mashup engine, which is the application name for Power Query.

Some transformation steps can break query folding, making any steps after them unfoldable. Finding out which steps break folding is a matter of trial and error. But simple transformations, such as filtering rows and removing columns, should be applied as early

Minimize the data you are loading

If you don’t need certain columns, then remove them. If you don’t need certain rows of data, then filter them out. This can improve performance when refreshing the data AND when modelling it later on, which is a win-win.

If your Power BI file is more than 100MB, there is a good chance you are going to see a slowdown due to the data size. Once it gets bigger than that it is important to either work on your DAX code, or look into an alternative querying/hosting method such as DirectQuery or Power BI Premium.

Consider performing joins in DAX, not in M

If you need to establish a relationship purely for filtering reasons, such as using a dimension table, then consider creating the relationship in DAX instead of in PowerQuery. DAX is blazing fast at applying filters, whereas Power Query can be very slow at applying a join, especially if that join is not being folded down to the SQL level. That being said, sometimes the data makes more sense as a flattened table for usability purposes.

Review your applied steps

Because Power Query is such a graphical tool, it can be easy to make changes and then forget about them. I had one customer who would often sort his data and then accidently leave the step in there. This was terrible for performance.

Make use of SQL indexes

If your data is in a relational database, then you want to make sure there are indexes to support your queries. If you are using just a few columns it may be possible to create a covering query that covers all of the columns you need. Work with your DBA to see what your options are. If you are using SQL Server, you can use execution plans to identify if the query is using an index. You will need the SHOW_PLAN permission on the server and you will need to use the View Native Query functionally to get the underlying SQL query

Tuning the model calculations

When someone says that a Power BI report is slow, it is usually an issue with the DAX modelling. Unfortunately, that fact isn’t obvious to the user and it can look like the visuals themselves are slow. Thankfully, there is a tool to identify the difference: the Power BI Performance Analyzer.

Use the Power BI Performance Analyzer

If your report is slow, the very first thing you should do is run the Power BI performance analyzer. This will give you detailed measurements of which visuals are slow as well as how much of that time is spent running DAX and how much is spent rendering the visual. Additionally, this tool gives you the actual DAX code being run behinds the scenes, which you can run manually with DAX Studio. When I am dealing with a tricky performance problem, this is my go to tool.

Confirm that the storage engine is being used

When using DAX studio, you can see how much time is spent with the storage engine versus the formula engine. The storage engine can only handle simple operations, but is multi-threaded and very fast. Whenever possible, it’s better to make use of the storage engine.

Remove data you don’t need

Because of the way the data is stored in Power BI, the more columns you have the worse compression and performance you have. Additionally, unnecessary rows can slow things down as well. Two years of data is almost always going to be faster than 10 years of the same data.

Additionally, avoid columns with a lot of unique values such as primary keys. The more repeated values in a column, the better the compression because of run-length encoding. Unique columns can actually take up more space when encoded for Power BI than the source data did.

Avoid iterator functions

Iterator functions will calculate a result row by agonizing row, which is not ideal for a columnar data store like DAX. There are two ways to identify iterator functions. The aggregation functions generally end in an X: SUMX, MAXX, CONCATENATEX, etc. Additionally, many iterators take in a table as the first parameter and then an expression as the second parameter. Iterators with simple logic are generally fine, and sometimes are secretly converted to more efficient forms.

Use a star schema

Using a star schema, a transaction table in the center surrounded by lookup tables, has a number of benefits. It encourages filtering based on the lookup tables and aggregating based on the transaction table. The two things DAX is best at is filtering and aggregating. A star schema also keeps the relationships simple and easy to understand.

Visualization Rendering

Sometimes the issue isn’t necessarily the data model but the visuals. I’ve seen this when a user tries to put >20 different things on a page or has a table with thousands of rows.

Lean towards aggregation

The DAX engine, Vertipaq, is really good at two things: filtering and aggregations. This means it’s ideal for high level reporting like KPIs and traditional dashboards. This also means it is not good at very detail-heavy and granular reporting. If you have a table with 10,000 rows and complex measures being calculated for each row, it’s going to be really slow. If you need to be able to show detailed information, take advantage of drill-though pages or report tooltips to pre-filter the data that is being shown.

Filter what is being shown

Unless you are using simple aggregations, it’s not advisable to show all of the data at once. One way to deal with this is to apply report or page filters to limit how many rows are being rendered at any one time. Another options is to use drill-through pages and report tool-tips to implicitly filter the data being shown.

Limit how many visualizations are on screen. The part of Power BI that renders the visualization is single-threaded and can be slow sometimes. Whenever possible, try to not have more than 20 elements on screen. Even simple lines and boxes can slow down rendering a little bit.

Everything else

Review your query method. This article assumes that you are using the import method to pull in your data. However, DirectQuery or live connections might be much faster depending on the size of your data.

Sometimes a report can seem slow when you are developing it. Make sure that there aren’t any applications on your PC that are consuming all of your resources and making Power BI Desktop seem slow.

Resources for Freelancers – Free and Paid

In my experience , working for yourself provides a unique set of challenges and required skills. You need to learn how to work on your business, not just in your business. You have to become your marketing, sales and HR departments. Without useful resources, I undoubtedly would have failed as a freelancer. Below is a list of resources I would recommend to anyone getting started as a freelancer.

Blog posts

As I’ve made the leap to working for myself, I’ve written a set of “lessons learned” posts.

Podcasts

Podcasts are my most valuable resource, because they provide a slow drip of insight and expertise. Listening to a podcast provides a regular insight on how other think about this stuff, without a large commitment.

  • Business of Authority. This is my #1 recommendation in podcasts. Rochelle and John are focused and understandable. This isn’t some 2 hour rambling podcast. They provide expertise from very different types of businesses. Every week I enjoy listening to this.
  • Ditching Hourly. John continually beats the drum of “hourly work is bad”. Even if you never switch to a flat-rate model, this podcast will help you think of yourself as a business, not as a technician.
  • Startups for the Rest of Us. While the topic is startups, the actual focus is small businesses. If you are a business, this podcast will have something for you.
  • The Freelancer’s Show. More of a casual chat format, this show regularly has lived experiences from developers working for themselves.
  • Finish your Damn Course. If you are a freelancer, you should consider offering products, not just services. Janelle interviews a wide variety of guest who are making training courses on wildly different subjects.
  • Creative Class. This podcast is generally only active when the Creative Class course is open for enrollment. I love the easy style Kaleigh and Paul have when talking about freelancing.
  • Building a Story Brand. Marketing isn’t always fun or intuitive work, but it is important. Donald Miller explains marketing in wonderful down to earth terms.

Books

Sometimes you need to read a book to get a core idea to stick. Many of these books are also available as audio books, so even if you are busy you can listen to them while exercising or cleaning.

    • Consulting
      • Getting Started in Consulting. If you travel, you probably have a checklist of everything to pack. This book is a big checklist of everything for starting your consulting business.
      • The Secrets of Consulting. Consulting is the art of telling people they are wrong without them having to admit it. This is a fun read on all the weirdness of being a consultant.

  • Running a business
    • The E-Myth. So many of us go freelance because we are great technicians. But we also have to be managers and entrepreneurs, and much as we might hate it. This book covers that through a great parable.Company of One. I absolutely love Paul’s approach to running your own company. This is a great book for thinking about the unique advantage of running a company of one.
    • Building a Story Brand. If you buy one book on marketing, buy this. It is the most intuitive, plain-English explanation of what marketing is that I have ever seen.
    • Built to Sell. While it’s almost impossible to sell a one person business, it’s important to think about it anyway. Much of your work can be automated or outsources, but you have to work hard to identify which parts.
  • Productivity
    • Getting Things Done. The best book I ever read on productivity. A series of simple guidelines for defining and organizing your work.
    • Atomic Habits. One of the hardest parts of working for myself was rebuilding structure and routine. This is a great book on the subject, very practical.
    • The Phoenix Project. While technically a book about IT management, there is insight for everyone. I found this book tremendous in helping me think of work as a flow and not just concrete tasks.
  • Communication

Courses

I’ve only taken one course for freelancing and the was the Creative Class course by Kaleigh and Paul Jarvis. If you are brand new to freelancing, I would consider this but definitely listen to their podcast before you buy. The content is easy to follow and enjoyable, but quite introductory. If you’ve been doing this a year or more, you likely won’t get as much value out of this course.

Software

There are tons of pieces of software out there for running a business, but here are some I have had experience with.

  • Toggl. I use toggle to track my time spent. Having a weekly timesheet makes it easy to track billable hours and review what my focus is.
  • Trello. If you need to collaborate or track todos, Trello is a great free tool.
  • Quickbooks. If you are running a business, then you need to track your books and send invoices.
  • Mailchimp. I send my weekly newsletter using Mailchimp. It’s simple to use and fairly cheap to start out.
  • Buffer. whenever I have a presentation or a course coming up, I like to schedule social media posts. For that, I use buffer.
  • Emergent Task Planner. Not actually software, but I find this note pad to be useful for planning and tracking my day.

Summary

When you work for yourself, you need to manage the entire pipeline Marketing –> Sales –> Delivery. You need to think of yourself as a business, and you need to have a plan for growing your business even if you always stay a “Company of One”. You need to rebuild structure and routine and find a way to focus on your work.

Consulting is a unique job. Freelancing is a unique job. With the right resources you can succeed at both.

Lessons learned from being self-employed, 12 months in

This months marks one year of working for myself, and undoubtedly I feel mixed about it. I’ve made a number of mistakes that I can’t help feeling dumb about. In retrospect, I probably wasted three months learning these lessons the hard way. And yet, it’s a lot like a hot stove. People can tell you it’s hot, but until you touch it you won’t quite grasp it. Here is what I’ve learned from a year of self-employment

What I learned

Don’t force your job into your dream job

When I quit my job, I had ambitions of doing all the things that give me life. I was going to attend more conferences. I was going to study all of the time. I was going to write more. I would work whatever hours I wanted, whenever I wanted. These were all things I imagined I would do once I quit my job.

And while I did many of these things, I was putting the cart before the horse. What I learned is that I needed to get the fundamentals in place first. I had to learn how work from home. I had to learn how to rebuild the boundaries and structure of a normal job. I had to learn how to create a new routine.

And as a result, I probably spent the first two months a bit unfocused. I also made a number of commitments that I am finally coming out the other side of. While I’m proud of all the presenting and speaking I’ve done this year, I wish that I had put first things first.

Respect your human fragility

I underestimated just how human I am and how hard this would be. Often it was dumb stuff, like needing to set office hours. I’m used to just being productive, just getting things done. But in reality, there was so much invisible scaffolding that had been supporting me, that I had interpreted as my own strength.

When you have a day job, you often have set working hours. You have coworkers and expectations. Consequences are often direct and measurable. Most people work in a different environment than they live in, so our brains have context clues.

All of this goes out the window when you work for yourself and work from home. Consequences are both diffuse and existential. No one will yell at you if you put off marketing or sales, and yet your whole life depends on it. There’s a vagueness there that is unsettling. The temptation is to just work harder, to push harder if things aren’t working. But without some structure to push off against, it is exhausting.

Find a way to rebuild that structure

Through trial and error, I found a way to rebuild the structure I had given up. I set office hours of 9AM-6PM, with an hour lunch. I peppered exercise through the day. I worked on breaking things into concrete tasks. I started building a python app to track personal todos and basic things like checking my blood sugar.

I feel like I’m starting to get the hang of it, but I’m not quite there. I find mixing consulting work with creative work to be a challenge. It reminds me of Paul Graham’s essay on Manager’s schedules versus Maker’s schedules.

The contrast of urgency and distant deadlines, of deep work and quick meetings, can be quite jarring. In theory I’d like to have dedicated days for creative work, but it never seems to quite work.

Join a peer group

One of the smartest things I did during this process was a lucky accident. I was friends with a wonderful peer in the Pluralsight space. He lived near me and was a full time author. And when I quit, he invited me to join his mastermind group. We have a Slack chat and we have a Skype call every other week.

Working for yourself is incredibly and brutally lonely, and alienating. You are making all of these decisions and going through new experiences. Having someone to bounce ideas off of is critical. It is so relieving to have a group of peers going through the same thing.

Take a vacation

Schedule a vacation. Make it happen. It took me 8 months but I had a real vacation. By planning it far in advance, I could warn clients and build it into my schedule.

When you work for yourself, there is a creeping sense of opportunity cost. Let’s say your billable rate is $100. That means every movie is $200 you could be earning. That board game is $100 you could have made. And that vacation is $2,000 you didn’t bill someone.

And that is why you need set office hours. That’s why your need strong boundaries. And that’s why you need to schedule a  vacation. Because otherwise there is a temptation to work yourself into the ground.

Was it worth it?

So, was it worth it? Would I do it again? Let’s look at it financially first.

Financial benefits

At my prior job, I made about $65,000. This year I made $85,000 in gross revenue. So, big improvement right? Well…

So, when you work for yourself you have to pay all of your own health insurance, 401k and vacation. You also have to pay both halves of the payroll taxes. Realistically, it turns out to be a bit of a wash, when you factor in all of these costs. Overall, it has been a modest improvement, at the cost of monthly consistency.

If I really wanted to make more money, I could have taken a job with “Senior” in the title and gotten paid $80-$100k AND benefits. At the same time, It probably would involve being on call or quite a bit of stress.

Personal benefits

The personal benefits are huge, and I will probably not take a regular job for the next ten years. When you work for yourself, you decide what you want to focus on and specialize in. You do set your own hours and eventually you can fire your worst clients. There really is something soothing in knowing you have control over your career.

But beyond that, working for myself has allowed me to be a caretaker for my mother. I’m able to visit her 3 days a week and take her in to a Medicare replacement program. I’ve been able to see a marked improvement in her quality of life as a result.

So yes, it was worth it and I would do it again. I just wish I had a time machine so I could do the first year the easy way, haha.