Things to consider when using Flows for data migration

Image for post
Image for post

“Show me a ten-foot wall and I’ll show you an eleven-foot ladder” ― Peter Bevelin

Using Flow for data migration is easy but has limitations you should consider, these limitations can cause problems left the data migration in limbo, whilst waiting for the api limits for each day and Flows waking up and running

When you hit the daily limit for Flows, they stop working until the next day (when limit resets).

Considerations

  • Data migration can have lots of records and could take many days
  • Flows don’t fail, they queue them up
  • The only way to stop Flows triggering is to delete them or put the Dynamics 365 environment into admin mode.

There are examples where the Flow limits meant the Flows have been in a waiting state for 4 days due to hitting the throttling limit on a daily basis.

Flows are not just on Flow runs by actions inside the flow.

How did we get here with Flows

Microsoft initially had workflows which were hosted by Microsoft and used Async service on the Dynamics 365 server. There were no limits on workflows but from Microsoft’s perspective this isn’t great because these run on Microsoft resources and costs them to host and run them.

Microsoft created Flows, with greater functionality and built on Logic apps. Flows are a scalable, enterprise solution and with connections allow you to create Low code solutions and link systems together.

The benefit to Microsoft is the Flow runs are easily counted and Flows are hosted in Azure which is scalable. Microsoft pushed Dynamics 365 users to use Flows and warn that workflows will be deprecated sometime in the future. Flows allow Microsoft to estimate and control how much it costs to run Flows. Dynamics 365 online is a service, so Microsoft don’t want to have unlimited resource/compute for people.

Flows allow people to scale up and pay for what they use in classic Azure costing process. This initially seems unfair because we were used to no limits but paying for what you use is ultimately fair.

You get taxed one way or another with functionality and it’s not something you can resist. It’s important to align your solutions with Microsoft’s road map. If you resist and continue to use workflows because they are free then in the future you are creating a huge upgrading task.

Flow first

I have a Flow first attitude and workflow should not created on projects going forward. Many Dynamics 365 professionals resist this because they have Workflow skills and might not have created a flow yet. By default we don’t like change and stick with what we know.

Make them do it in a flow, it will be slow to start with but they will soon start to love Flows because they are more powerful

How to cancel flows

There isn’t any way to bulk cancel flows, it’s a one by one event. Flows queue up runs even if they are not enabled!! The only way to stop them is to delete them (and there is no way to bulk delete flows!!)

if you cancel a flow it doesn’t stop the flow running when the api threshold limit is refreshed when you hit a new day.

It’s possible you could be left with 1000’s of instances of flows left in a running state, with you waiting for refreshes of API limits. These could be days!

Deleting flows is only a choice if you have a solution with the flows in and more recommended with an automated release.

Logics apps?

Microsoft have created functionality to run Logic apps on your local machine and some talk that we should all be moving to Logic apps. This combined with an improved editor should make it easier for no code professionals to create logic apps.

I’m not sure about this or if it’s the direction of travel but at the moment Flows are easier because of the current environment connection that means the Flow can work in the environment it’s in and is easily deplorable.

What’s the usual data migration choice

The common tool I see used for data migration on large projects is Kingswaysoft. I haven’t used Scribe online but I believe it has similar functionality, so it depends on the expertise of your team.

Microsoft is moving to count the actions within a flow and step in a flow will be an action which you will have a limit

Dynamics 365 and replicating to an SQL database for reporting

Flow limit

The Flow limit is a topic that will increasingly come up in Dynamics 365 projects, as more projects hit the limit.

My feeling is the limit is a bit low, particularly considering the number of environments you can have in a large project.

Conclusion

Flows are awesome and combined with connections they are powering the no/low code revolution in Dynamics 365/Power Platform projects. Flows and Power Platform are replacing the bespoke .Net applications and Excel spreadsheets that can grow in companies.

Microsoft is starting to be tighter with the Flow limits and charging more in the future. This seems unfair because we are coming from not paying much but in reflection I tend to view the Azure/usage pricing as fair.

Currently the way Flows have some odd ways of work and the turning off functionality can be a nuisance because the runs are queued, waiting for you turn the flows back on. The only way round this is deleting the flow.

Written by

Have been working with Dynamics 365 since version 4 and enjoy reading and delivering enterprise projects

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store