SFTP Exports

Last week, we worked on SFTP imports, where customer data was imported in CSV/XLS/XLSX format from the client’s SFTP server. This week, we’re working on the opposite – that when a customer is updated, the changes should be pushed out (exported) to a client’s SFTP server.

This is for fairly large clients that have their own internal customer relationship management software, that want any updates in our workflow management system to feed into theirs.

We were considering doing this in real-time, so that changes were exported the moment they were made, but this can cause network issues if there are a lot of changes. And when you think about it, if all the fields in a customer (name, address, email, phone, etc) are changed one by one, you’re really only interested in the end product – that one export of the whole customer after all changes are done.

So, we’re doing this on an hourly basis instead. If you have an export set up, then every hour, any customers that have been edited within the last hour will be exported from our job management software to your own server.

We also had to consider the end file – will it be a single large file containing all the changed customers, or lots of individual files? A big problem with the large file solution is that if you have an importer on the client server that is reading the file at the same time as we are writing it, then there is a potential loss of data there.

Instead, we will create individual files. That way the chance of read/write collisions is much smaller. One of the problems with paperless office solutions is that things happen so fast there are there “race conditions” that we start worrying about.

New import system coming, replacing Flash

We have imports in a number of places throughout the system, building each one separately. Some are better than others. The Jobs page, for example, has a nice importer which will let you upload a sheet of jobs in any reasonable format and let you “map” your own layout to our internal layout. You can even record that mapping so that you can use it again next time you need to do an import.

I was asked to replicated that method throughout the site, and thought about it for a bit before diving in, so I have a few other ideas about it that will improve it further.

File uploads in Web 2.0 systems have traditionally been done using Flash uploaders such as Uploadify, because up until recently, there was no good way to handle file uploads that didn’t look clunky and Web 1.0. However, Flash is a deprecated technology and will be removed completely by Adobe in 2020.

Some of the other paperless office software solutions are going to be hit really badly by this, as I hear that at least one of the larger workflow management system companies is built entirely on a Flash infrastructure. They’re going to have to rebuild completely within the next two years!

Google Chrome are hinting heavily at that as well, by the way that they disable Flash by default, to the point that when any of our clients say that they can’t upload a file into our system, one of our things to check is “Have you enabled Flash?”. It happens so often that we even made a video showing how to do it!

So, Flash has got to go.

Thankfully, the new HTML5 file uploaders are much better at handling uploads gracefully!

Along with replacing flash as the uploader technology, we’re also thinking about the actual flow of an import.

When a file is imported into the Customers part of the customer relationship management software, for example, we will run the import and then tell you how many were imported. If your spreadsheet contained 2000 customers and we report that 1999 of them imported, what happened to the other one? Why did that not import? Well… because of how imports are currently done, it’s very hard to pass error messages back to the browser, so no – we literally can’t tell you what happened to that one, so you need to now go through your file and see for yourself what didn’t upload. Ick! We are really sorry about that shoddiness (which is implicit in the technology, so hard to get around), and hopefully the new importer makes up for it.

How we’re fixing that in the new importer is that when you upload a file, we then send the contained data back to the browser, which then uploads the data again, but one at a time so it can keep tabs on the progress, and keep records of what errors popped up. After the import has finished, the browser then knows what rows failed to import, and offers you a download of those failed rows. Much neater!

Depending on how busy we get with other stuff, I expect this to be all completed some time next week, and available to clients who are on our “bleeding edge” paperless office solutions server. Those on the “stable” servers will need to wait until the next public release of FieldMotion, which I expect will be in only a few months, as we’re closing in on completing the major parts of it.

Imports, emails, purchase orders

Over the last few weeks, we’ve been working away on various little projects to improve our field service management software.

Yesterday, I finished the first prototype of a new importing system where our clients (you, maybe?) export their customer or job data in whatever common format they like (CSV, XLS, XLSX for example), and use our new import mapper to easily link your headers to ours and do your import.

To demonstrate it, we set up our system to check an SFTP server (like Dropbox, but more fundamental) at a specific location. We put a demo customer file in there in CSV format, and the system was able to detect that and import it automatically.

How did it work?

We put a demo file there first, then ran an “import map wizard”, which found the file and read it. It listed the column headers of the file headers on one side of the wizard, and our own internal customer headers (how we record customer data in the relationship management software) on the right. We then clicked the appropriate fields on the left side and right side to match them, until they were all matched.

Sometimes, there is a mismatch. For example, in our field service software, customer names are recorded as a single field, “customer_name”, but in your database, you may have them separated into multiple fields (title, first name, surname). So, we added the ability to automatically concatenate fields while importing, to join the various fields together and form a single match that we can then import.

Also sometimes, there will be a field in the import which you really want to import but don’t have a matching custom field in the workflow management system already. So, we add that you can automatically add that as a new custom field.

After setting up the import map, we saved that so that the management service software knew how to handle future files.

On the emails front, we’ve been diagnosing issues with sending emails to companies such as Google. Because job management software sometimes involves sending a lot of emails purporting to be from our clients’ domains, we are classified as “bulk senders”.

Google makes it very clear that there are certain rules they want bulk senders to follow.

Today, we made a change to our outgoing emails so they now have a From header saying “no-reply@fieldmotion.com”. This is because of Google’s rule 3: “Use the same address in the ‘From:’ header on every bulk mail you send.”. We can’t use a client’s email address because every other client would then think the email was spam and mark it so, making it very difficult for us to deliver our reports!

Google says very clearly that “unauthenticated emails with attachments may be outrightly rejected, for security reasons.” Since our emails tend to have reports attached, if we do not authenticate them, following their rules, then we risk having the emails completely rejected.

And so, if you receive a report that has “no-reply@fieldmotion.com” written in it, don’t panic.

On the Purchase Orders side, I’ve just had a chat with the lead on that project and she says that what we were (my fault) initially trying to do is way more complete than what most market leaders actually do. Most system handle just basic order tracking, so we’re pushing in that direction instead to get that done. The way we count progress is through issues created/completed, and we’re at about 4 completed, 7 incomplete. Some of those incomplete look small, but you can never tell! Programming is like solving math – sometimes simple-looking things end up being hard (Fermat’s last theorem, for example).

As an example, I had it initially that when you make an order, you can order any stock at all, and the system would break it apart into separate orders for each supplier that might be needed to fulfil that order. For example, if you were using FieldMotion as pest control software, you might order bait boxes and pellets in the same order, even though the two might actually be supplied by different suppliers. It turns out that usually when an order is being made, it’s from a specific supplier – I was just complicating things. So, we’ve managed to simplify it a lot.

In programming, we tend to write out the major points as single issues, and as we dig deeper, we write separate issues to handle any side-projects needed to complete them. The fact that we’re on only 11 issues after a few weeks into this project is pretty good. Means we’ve not had to build structure elsewhere to support what we’re doing here.

And then as usual, we have the smaller day-to-day jobs going on. We work with external developers, for example, to help them use our API efficiently. This sometimes involves us writing new entry points if it will help the developers, or making some existing points more consistent.

FieldMotion: the book of the system

I’ve been tasked with writing a book about this field service management system thing that we built. Most of the books I’ve written have been code-related and general in nature. Writing a specific “how to” of a single system will be a bit more challenging. Especially as the system we have is really so large that condensing it all into one book will probably make the book wither much too long to read, or much too dense to read.

So, I think the best thing to do is to write a general overview of the various parts within the system, and how to use them from a basic point of view. I will intentionally avoid detailing the use of the more complex parts of the workflow management software, and will return to those either in later chapters, or in follow-up books.

Why write a book?

I’m a big fan of written tutorials. I would much rather read instructions on paper than watch a video. Printed instructions and explanations can get a lot more in-depth than videos. Also, it’s easier to highlight lines in a book, or refer back to earlier pages.

Videos tend to have accents as well. Even within the single English language, there is enough disparity in accents that it can be hard for a person in the US to understand someone with a Scottish accent (for example). Written text does not have accents.

It’s also easier to translate a book than a video. With a book, it’s a simple matter of having the text translated by a technical writer. With video, though, the entire thing must be re-done.

I will be publishing the book in this blog as I write it.

When I wrote my other technical books, I stuck to a general prescribed format – about 13-14 chapters per book, each book should be 15-20 pages long, and all concepts should be presented with visual diagrams if possible. I’ll do the same here.

Proposed chapter list:

  1. Introduction to FieldMotion
  2. General Usage
  3. Customers
  4. Jobs
  5. Assets
  6. Stock
  7. Using OnCompletes to setup WorkFlow
  8. Financial Reports
  9. Dynamic Scheduling
  10. Recurring Jobs
  11. Outsourcing Jobs
  12. Linking to Xero
  13. Using FieldMotion with Zapier

After completion, we hope to give out electronic copies of the book for free to people that ask us for a demo of our job scheduling software, and will give a free printed copy to all new customers.

job scheduling software

Fieldmotion is a field service management system, which means that it is used by office workers to manage the jobs that field workers perform in the field.

The most straight-forward use of an FSM would be to directly create jobs and give them out to the workers as they are created, but sometimes you will want to schedule the jobs. This can be done in a number of ways. For example;

  • by setting the meeting time for the job
  • a reactionary job (say, a job that happens two weeks after another different job)
  • a recurring job (a job that happens on a periodic basis)
  • as a dynamically scheduled job (where the time is picked by computer)

Setting a specific job time is obviously the simplest method here. You simply set a date and time for the job, and that gets sent to the field worker’s phone/tablet. In order
to avoid information overload, the device only shows jobs within a set range of today (for example, 3 days in the past to 7 days in the future). This way the worker doesn’t feel overwhelmed, and can concentrate on just what’s in front of them.

Reactionary jobs are those that are created in reaction to the completion of other jobs. For example, a plumber may want to have a 6-month visit scheduled automatically
after a job has been done, in order to check on how the job is holding up. But you don’t want to have to manually create it, as you may forget to do so, and in 6 months, the last thing on your mind will be to go check on something from 6 months before that you barely remember. So, we have a handy “on complete” system in the field service scheduling software which allows you to create any number of follow-up jobs in many different ways.

Recurring jobs are those that you want to schedule every week or month or year. You can tell the field scheduling software to schedule a job every Tuesday, or on the 17th of every month, or specifically on the 3rd of March every year, or every 19 days. There are many ways that people use it.

Our dynamic scheduler can schedule any number of jobs based on the requirements of the job, the distance travel required for the workers, the skills the workers have, the cost of travel, the wages of the workers. Even if the solutions it comes up are then changed to be more in-keeping with what you like, they still save any medium sized company hours every week in scheduling.

If you’d like to talk to us about our service industry scheduling software, please ask for a demo.

Xero, purchase orders and customer imports, oh my!

This week, we finished off the public-authenticated version of Xero. We already had a private-authenticated version running well before that, but Xero demand a public-authenticated method in order that we partner with them, so we did it. Personally, I don’t understand why they insist on that. Public-authenticated methods only allow for about a 30 minute login window, which means that if you do something in the morning and in the evening that both need to push to Xero, you’ll need to authenticate separately for both of them. With a private authentication, you basically set up a key that gives your FieldMotion account permanent (until you delete it) permission to push and pull information to your Xero account.

We are nearly done with the basics of our purchase ordering system. This is so our service management software users can order new stock items and manage their distribution as they come in. I wrote the basic draft myself, got 90% done on the spec, and realised there were some obvious bits missing, and went to do those. When I mentioned those parts to our resident accounting expert, he said that what I intended to do was something that even Sage doesn’t do. So I reigned back a bit and passed the project onto another of my developers to finish off so I could get onto the next big one.

We handle customer imports through API and through manual upload of files. Some larger companies, though, want our field service job management software to also handle imports via spreadsheets pulled from FTP servers. I’m working on that right now, making sure it’s something that can be used by any client that needs that kind of import. We have an upgrade planned for the manual file-based import as well, where we will be “mapping” fields from the uploaded files to our internal format. This gets rid of the need for our customers to follow our rigid file-upload format – they can upload in whatever format they want and we’ll just link to the right fields on-the-fly.

When these are all done, we’ll be starting on the next big project; linking to Quickbooks directly instead of through Zapier, as Quickbook’s Zapier plugin is …lacking. We looked at that a year or two back when we were starting on Xero, and decided to concentrate on Xero as it was the easier of the two to get going. Now that we have Xero going well, we’ll address Quickbooks, reusing as much code as possible to speed this long. I’m not going to put a date on this, but I expect it to happen very very quickly. Our field service management software UK users tend to use Xero, but in the US, Quickbooks is still king, it seems, so we’ll address that.

Another thing we’re doing is to add alerts for when a customer has remained as a specific customer type too long. This is part of our customer relationship management software which will let you set a customer to a specific “type” (for example, “awaiting forms”) and set a deadline for that. If the deadline passes and the “type” hasn’t changed to something else (for example, “in form-building”), then an email is sent to the customer manager to go find out what’s going on. One of the advantages of paperless office software solutions over the traditional systems is that we can automate this stuff so you don’t have to keep everything in your head.

There’s always something new being built here – talk to us and we’ll arrange a demo

1,000,000 jobs completed

A milestone moment.

A few weeks ago, we were doing a little analysis on our historic data and noticed that the number of jobs completed by our field service management software customers was getting close to a million. After a little work, we came up with a quadratic equation which closely modeled our growth. It predicted that we would hit 1,000,000 jobs completed over the weekend.

Well, this morning I checked and we were at 1,000,202!

It’s very difficult to find the exact job that was the one millionth, as they are spread out over all of our databases (we keep our customer databases separate to ensure there is no chance the data can “leak”).

Another nice thing we noticed based on the analysis is that our growth is accelerating. Unlike one of our competitors who has been advertising the exact same number of “completed jobs last month” since early last year, our numbers are growing.

Slow but steady wins the race.

The market for field service business management software is huge. It’s estimated by some that the industry was worth 1.76 billion USD in 2016 and will be worth 3.61 billion in 2021, 4.45 billion in 2022. Others are a little more conservative and suggest 4.1 billion in 2025. No matter who you believe, though, they all say the market is huge.

We have been working our way into this industry for the past 5 years, and are steadily getting more and more of the pie, simply because we work based on what the majority of our clients need, we try to keep it as simple as possible, and we started out right from the beginning with the most common issues in mind.

Our apps work whether you are online or offline, for example, syncing in the background automatically when data is available. There is no need for the user to know how that works – as far as they’re concerned, it just works. Other FSM companies have apps that only work when you are online (and how useful is that when your work is in an area that has no network coverage?). Yet more have a “synchronise” button that you need to press to upload your work and download new work (which the users forget to press).

On the CRM (customer relationship management software) side, we have a core set of features that we try to keep from growing. Yes, we do add some things when they are obviously useful, but we mostly try to not add everything we can think of, because that just ends up complicating the product. Paperless office software solutions, after all, are about getting away from complications – not making more of them.

We anticipate the next million to complete within a year, and the next after that in only a few months. The future is interesting. Talk to us about yours.

FieldMotion as a customer relationship management system

This week, we were concentrating on adding new tools to the CRM (customer relationship management) part of our system.

We recently added that when a customer is changed from one “type” of customer to another, an email could be sent out to that customer. You get to define the email of course – we wouldn’t just send a random email out!

This is useful in cases where all customers need to receive a “welcome pack” email, for example, which includes links to video tutorials, etc.

We got that working well a short while ago. This week, we’re making sure that the history of sent emails is recorded next to each customer, so you can say for definite “this email was sent to you on 2017-xxx at xx AM”.

The fact that FieldMotion can be used as a CRM is something we don’t usually advertise, because it’s mainly designed to simply be the best field service software. However, since we actually use it internally for our own sales, and have experience with other CRMs pre-FieldMotion, we know that it’s up there with the best of the rest.

One of our sales guys was chatting with a potential client recently who had been tasked with procuring a field services tool for the company – the company had already looked into getting a bespoke application built and had balked at the massive number involved (hundreds of thousands). During the course of the chat, our salesperson mentioned that we use FieldMotion internally for pretty much everything we do, even customer relationships, and he suddenly perked up “oh? that’s another off my todo list!”

All field service management software companies should take into account that all companies need to manage their customers. It’s not like they magically appear and stay around – you need to have a method of setting reminders (callbacks) for yourself to keep in touch with them – you need to have a list of notes and files in one handy place where you can read or update them at a moment’s notice. Service job management software needs to record what you did for the customer before, what they said, anything related to them.

If you’d like to learn more about the CRM part of our paperless office software solutions, please ask for a demo.

stages of release in FieldMotion

I wanted to talk about how why FieldMotion’s software process makes us one of the most robust and reliable field service management software companies on the market.

Startup companies race to complete features and ship them as soon as possible. We don’t do that, because we are aware that new features are never right. There is always a period of tweaking after a release to match what we built, to how people are actually using it.

How we work is that the development team will have a list of big projects to do, which they’ll work on for a few months. Each of the projects appear to naturally come to a close near six months into development, so we’ll tidy up all the new projects and announce them all as a new release.

The current release candidate will be our tenth official release since we started about 5 years ago, and will have three major products in it: a stronger Dynamic Scheduler (route optimisation for multiple vehicles, departments, etc), strong Xero integration (both Public and Private API versions), and a new Purchase Orders system.

We’ve already planned out the bones of what’s coming for the next six months after this, and we think it will be our biggest and most important release; not because of new products in it (we’re thinking Quickbooks, and multiple languages, for a start), but because we’re mostly dedicating our development resources into making sure the system is scalable (we want millions of people using it), fast even with all those users, never goes down for any reason, and is so simple to use that you’ll wonder why we bother making tutorial videos!

We’re already at the stage where most parts of the system are redundant, so if a server or two (or even a full datacentre – hello Digital Ocean San Francisco I’m looking at you…) goes offline for a while, we can reroute people to different areas that are still up and running. In most cases, there won’t even be a pause (like when SF went offline, and there wasn’t even moment where anyone noticed but our development team), but sometimes we will have short period where parts of the system are offline.

In our own case, how it usually happens is that we’ll see an issue, we’ll solve it in a quick way that’s not perfect but gets the job done for now, and then we’ll solve it in a more permanent way that takes longer but is more robust.

Yesterday, for example, we had an issue where people logging into one particular server in our network were experiencing a large delay in their requests. We looked at this, couldn’t see the immediate cause, and figured the best thing to do at that moment was to increase memory and CPU (we multiplied the number of CPUs by 4, and the amount of RAM by 16 – hey, go big or go home!) to see if that fixed the immediate issue. It did, forcing people using that server (some customer relationship management software users, no mobile workers) to log in again, which was a price that had to be paid at that time to get the problem solved. The more permanent solution in this case is that over the next six month development period, we will be tearing that server type apart into separate “microarchitecture” pieces that can be managed separately, and making sure that each of those new pieces can be turned on/off without affecting people that are logged in. Obviously, this will take time to plan and build, so the quick solution (throw more memory and CPU at it!) was the right answer in the mean time.

Yesterday’s issue took that particular server offline for three minutes (I was timing it) while we upgraded it. Three minutes is better than how long Google Drive was out last week, or how long the entire country of Japan was offline for on August 25th (thanks again, Google), or how Melbourne’s train network was shut-down for hours on July 13th.

We’re at that stage in a company’s life where we’re pretty happy that we have the product we need, and that there is no need to keep on adding more and more and more shiny stuff to it, which is why we’re happy now to spend the next 6 months or more just making sure it is the absolute strongest, fastest, most reliable, and best all-round cloud field service management software.

purchase ordering system

This week, we’re polishing the purchase ordering part of our field service management software. This is a system which allows you to create purchase orders, mark off the items as they come in, and automatically update the stock levels of the destinations as you mark them off. This helps our clients take yet another step towards having a truly paperless office system.

As an example, let’s say a user has asked for 15 of product A and 10 of product B. You create the purchase order, list the items you’re ordering, and then as the orders are delivered to you, you mark them fulfilled, which will automatically increase the stock-take of product A and B for the user.

If only a few of the items were delivered to you, you just mark in the amount of arrived, which leaves the order in the status “Part Delivered”.

You record the cost and price of the items, so there’s an option in the future of maybe expanding this into something a bit more accounty than it is at the moment.

To round this off, we’re adding in the ability to record what stock is with what customer; something we’ve been asked for a few times. This means that you can set your purchase order up with the customer as the end target, then mark off the products as they are delivered to the customer, meaning that you have a record of all deliveries and part-deliveries.

We hope to have that done by the end of the week. If you’re interested in beta-testing this (along with the rest of the system!), please contact us and ask for a demo.