New features

Wow – it’s been two weeks since my last blog! Shows how busy we are here in FM Central. Well, I suppose it’s time for a catchup on the last two weeks, then.

Up until recently, we recorded the features and bugs of our field service management software in an internal database, marking then done as we went. This was not ideal, of course, as that database is really only useful to developers. So, we’ve started also recording new features specifically in their own page on our internal Wiki.

The most recent developments are:

New API functions to allow creation of multiple jobs simultaneously. One of our UK field service software API users was complaining that it took too long to import jobs into the system (we limit API connections to 1 per second, to avoid accidental network-flooding DDOS attacks), so we wrote a function to let them upload a lot of jobs at the same time. It was perfect timing, really, as I was about to do it anyway as part of our new imports system.

The new imports system is completed. You can now import various things, like customers, assets, etc, with whatever file format you use (XLS, CSV, ODS – common spreadsheet formats), and go through a mapping wizard to link the data into our system. It will then upload the data in batches, with a nice progressbar showing you how much is done, how much is left to do, and a count-down which is probably more accurate than the Windows download time calculations 😉

Once the import is finished, it will then tell you if any failed to import, and will give you a downloadable file containing those broken entries.

A little addition to the field service software app. If you’re working on an asset such as a boiler, and you want to have a table of all the contained assets (de-aerators, pumps, traps, etc), you can do that now. You can set up your service job management software so that if you select the asset, all its sub-assets will be displayed in a table under it. You can even have their custom values pre-fill into the table, ready for you to adjust as necessary.

Also in the app, you can now have a field copy another field, so that if you change the first, then the second will automatically fill with the new value. This is useful where you might have a long form where some values are needed in various places in it.

We previously only allowed PDF reports to be sent at the end of a job, as one of the points of job management software reports is that the customer should not be able to just open them and edit them. PDF reports were limited, though, because of how they work, so a lot of our recent reports have been in Doc format. We can now send those reports, by automatically converting from Doc to PDF just before the email is sent.

Our new Purchase Ordering system is nearing completion. Looking through the list of features, I see we can now automatically break Requisitions apart into separate Purchase Orders, and any POs associated with Contracts will be displayed on the Contracts pages.

A nice new field service scheduling software trick which will help speed up very large companies – previously, in order to create a job, you would need to select the customer, the job form(s), the department (if any), and the user. We’ve done some work speeding that up. You can now associate customers and forms with departments, so if you select a customer, its department is automatically selected, which then filters down the list of forms and users available to select. If there is only one form available, or one user, then they are automatically selected. We also added a “select by default” option to forms so if they’re available after filtering for department, then they will be automatically preselected. This way you can cut down the creation of a job right down to just “select customer. click Create”, in some cases.

More field service management app magic – we added comparison operators (less than, equal to, greater than, etc) to the Calculations module. You can use these to do some nice tricks. I demonstrated how to use just the new ‘<‘ operator, for example, to fill a field with the word “Yes” if today’s date is between two other dates. Contact us for a demo and we’ll show you this trick and much more.

new manual site

Last month, we proposed writing a book to explain how to use FieldMotion, but realised that a much more useful reference would be similar to WikiPedia, where every subject has its own article which we can expand as we need to, with links branching off to related subjects.

In the late 90s, a company I worked with used an internal documentation system called “WikiWikiWeb”, which was written in Perl – a hard-to-read programming language which I’m glad to never have to touch again! The WikiWikiWeb software had been created only a year or two beforehand by a programmer, Ward Cunningham, who based his ideas on Hypercards, which was invented on the Mac. Wikipedia is an example of a WikiWikiWeb implementation.

The idea of a collaborative encyclopaedia or internal documentation store that has links off to other related subjects is useful, because people don’t think in straight lines. I’m sure you’ve found yourself on Wikipedia many times, intending to read about stress concentration in vertical structures and finding yourself clicking on interesting bylines until you have to stop yourself at 3am when you realise you’re studying the usage of Bulgarian wedding songs in Japanese anime soundtracks. Or is that just me?

Up until recently, we were using Google Drive for sharing documents internally, but they’re really not very useful for that purpose. I have friends in Google who refuse to use it themselves because it’s really not fit for purpose. You upload a useful file that you think people should be able to reference easily. But how to share it? You need to individually share it out to people, but that’s not the end of it – how do you organise the files? The address is different for each person so there’s no easy way to make a link that can be embedded in a web-page, and even if you did, you now have two systems to take care of – Google Drive and the system you use to organise the files.

By using a Wiki, you create a central repository of knowledge which any member of staff can update at will, and it acts as its own reference. Each “document” in a wiki is a plain text file that can contain links off to other articles of interest.

In FieldMotion, we have two wikis – we have an internal Wiki which is for writing about internal processes, and an external one which we intend to be read by customers and will contain information about all aspects of how to use FieldMotion. Very soon, if you ask us a question, we will either point to a wiki page, or we will create a wiki page and then point to it 🙂

The wiki (which I’ll link to as soon as we’re ready with it) will serve as an answer to any questions that come in, but it will also be a source of new knowledge.

In our internal wiki, we keep a list of new features, so that our own workers are aware of all the new things that FieldMotion can do. We will do something similar on the new public wiki so that all our customers are aware of the features of the system.

import system rewritten

An issue that comes up with our field management service software clients more and more often lately is that most of our file uploads and importers are written using Flash to enable the file upload. The reason for this is that up until recently, there was no other way to upload files without having to reload the entire page.

Because we try to be the best field service software and keep everything as easy and smooth as possible, forcing a reload of the page every time a file was uploaded was simply not on. The only alternative, though, was a Flash-based uploader. So, that’s what we did.

Now, though, web technology has advanced enough that we can upload files using just JavaScript. No flash. That’s very important because Flash reaches its end of life in 3 years. This is a problem for some service management software solutions I could name that rely heavily on Flash. Luckily, our own investment in the technology was minimal.

The first thing to be rewritten was the importer for Customers. I have yet to plug it into the Customers page properly (it needs to be passed by our testing department first), but it’s working well enough that I can describe what we improved.

Firstly, we had the issue that even though we provide example files illustrating how we need customer imports to be formatted, a lot of our field service management clients had trouble fitting everything into those formats, so we would end up doing it ourselves, or the clients would try anyway and potentially break some of their already imported customers.

To fix this, the import process now includes a mapper. You upload in whatever format your own exports are in (XLS/XLSX/CSV/ODS), and match your headings against our own.

Once that’s done, click Save and the system will ask you what to name your new map (if you hadn’t chosen a pre-existing map), so that next time you upload from a file in the same format, it will already know how to handle it.

After this, the system starts the import properly.

It starts by uploading one at a time, then two at a time, etc., until it figures out how many you can upload at a time so each batch of uploads finishes in 5 seconds. A five second interval is enough for the system to keep you apprised of what’s going on, with uptodate numbers of completed or failed imports, and a reasonably correct timer that will tell you how long you have to get that cup of tea. It can take a while to import customers, as we do a lot of checks to make sure you’re not uploading duplicates or obviously incorrect data (names in the lat/long fields, for example).

When the import is finished, any customers that failed to be imported will be grouped together and you can download a CSV from the result page, to see what was wrong with them and try again.

This system is currently in with our test team so they can do a thorough review of it before update the workflow management system and replace the old flash-based one.

It’s a huge improvement, though! If you’d like to see it in action (along with the rest of the best field service management software on the market), please ask us for a demo.

new Export system coming

The import system we’re building lets you import from any sensible data format, and map your own column names into the column/field names that we use internally here.

While writing the prototype for an FTP exporter, I realised that the same method we use for converting incoming data can be used for outgoing data. For now, I have a very basic proof of concept that runs in one corner of FieldMotion. We’re going to expand on that.

Exporting data is almost the same as importing. As one of the bosses joked, “can you not just write the code backwards?” (yes, he was joking – we’re all pretty savvy here!). In essence, he’s kind of right. The sequence of events is reversed (convert file to data then data to internal map, becomes convert internal map to data then data to file).

So, as soon as we finish the importing system and push that out to all parts of our paperless office solutions, we will do the same for exports.

And as is usually the case with things we have just done or are just about to do, there are already potential clients waiting for this feature!

One nice side-effect of this work is that we will soon be able to hook into external reporting engines, providing data in whatever format they want. This, again, is something our clients are clamouring for. We have simple reporting built in, and have built Access-based report engines where some of our clients needed more bespoke work, but if we could hook into a full-on data analysis package that already exists, this would save us some time. So, we’ll do it.

The proof of concept export that I mentioned is an hourly export to FTP of any customers that have been changed in the customer relationship management software within that hour. You set up details of the server on which the data should be dumped, and then FieldMotion will push the data to that. We could potentially expand that even further to push to Dropbox or Google Drive, etc.

The export format for the hourly thing is CSV, but I will be making that configurable. At first, CSV/XLS/XLSX/ODS, and eventually we may offer XML as well.

SFTP Exports

Last week, we worked on SFTP imports, where customer data was imported in CSV/XLS/XLSX format from the client’s SFTP server. This week, we’re working on the opposite – that when a customer is updated, the changes should be pushed out (exported) to a client’s SFTP server.

This is for fairly large clients that have their own internal customer relationship management software, that want any updates in our workflow management system to feed into theirs.

We were considering doing this in real-time, so that changes were exported the moment they were made, but this can cause network issues if there are a lot of changes. And when you think about it, if all the fields in a customer (name, address, email, phone, etc) are changed one by one, you’re really only interested in the end product – that one export of the whole customer after all changes are done.

So, we’re doing this on an hourly basis instead. If you have an export set up, then every hour, any customers that have been edited within the last hour will be exported from our job management software to your own server.

We also had to consider the end file – will it be a single large file containing all the changed customers, or lots of individual files? A big problem with the large file solution is that if you have an importer on the client server that is reading the file at the same time as we are writing it, then there is a potential loss of data there.

Instead, we will create individual files. That way the chance of read/write collisions is much smaller. One of the problems with paperless office solutions is that things happen so fast there are there “race conditions” that we start worrying about.

New import system coming, replacing Flash

We have imports in a number of places throughout the system, building each one separately. Some are better than others. The Jobs page, for example, has a nice importer which will let you upload a sheet of jobs in any reasonable format and let you “map” your own layout to our internal layout. You can even record that mapping so that you can use it again next time you need to do an import.

I was asked to replicated that method throughout the site, and thought about it for a bit before diving in, so I have a few other ideas about it that will improve it further.

File uploads in Web 2.0 systems have traditionally been done using Flash uploaders such as Uploadify, because up until recently, there was no good way to handle file uploads that didn’t look clunky and Web 1.0. However, Flash is a deprecated technology and will be removed completely by Adobe in 2020.

Some of the other paperless office software solutions are going to be hit really badly by this, as I hear that at least one of the larger workflow management system companies is built entirely on a Flash infrastructure. They’re going to have to rebuild completely within the next two years!

Google Chrome are hinting heavily at that as well, by the way that they disable Flash by default, to the point that when any of our clients say that they can’t upload a file into our system, one of our things to check is “Have you enabled Flash?”. It happens so often that we even made a video showing how to do it!

So, Flash has got to go.

Thankfully, the new HTML5 file uploaders are much better at handling uploads gracefully!

Along with replacing flash as the uploader technology, we’re also thinking about the actual flow of an import.

When a file is imported into the Customers part of the customer relationship management software, for example, we will run the import and then tell you how many were imported. If your spreadsheet contained 2000 customers and we report that 1999 of them imported, what happened to the other one? Why did that not import? Well… because of how imports are currently done, it’s very hard to pass error messages back to the browser, so no – we literally can’t tell you what happened to that one, so you need to now go through your file and see for yourself what didn’t upload. Ick! We are really sorry about that shoddiness (which is implicit in the technology, so hard to get around), and hopefully the new importer makes up for it.

How we’re fixing that in the new importer is that when you upload a file, we then send the contained data back to the browser, which then uploads the data again, but one at a time so it can keep tabs on the progress, and keep records of what errors popped up. After the import has finished, the browser then knows what rows failed to import, and offers you a download of those failed rows. Much neater!

Depending on how busy we get with other stuff, I expect this to be all completed some time next week, and available to clients who are on our “bleeding edge” paperless office solutions server. Those on the “stable” servers will need to wait until the next public release of FieldMotion, which I expect will be in only a few months, as we’re closing in on completing the major parts of it.

Imports, emails, purchase orders

Over the last few weeks, we’ve been working away on various little projects to improve our field service management software.

Yesterday, I finished the first prototype of a new importing system where our clients (you, maybe?) export their customer or job data in whatever common format they like (CSV, XLS, XLSX for example), and use our new import mapper to easily link your headers to ours and do your import.

To demonstrate it, we set up our system to check an SFTP server (like Dropbox, but more fundamental) at a specific location. We put a demo customer file in there in CSV format, and the system was able to detect that and import it automatically.

How did it work?

We put a demo file there first, then ran an “import map wizard”, which found the file and read it. It listed the column headers of the file headers on one side of the wizard, and our own internal customer headers (how we record customer data in the relationship management software) on the right. We then clicked the appropriate fields on the left side and right side to match them, until they were all matched.

Sometimes, there is a mismatch. For example, in our field service software, customer names are recorded as a single field, “customer_name”, but in your database, you may have them separated into multiple fields (title, first name, surname). So, we added the ability to automatically concatenate fields while importing, to join the various fields together and form a single match that we can then import.

Also sometimes, there will be a field in the import which you really want to import but don’t have a matching custom field in the workflow management system already. So, we add that you can automatically add that as a new custom field.

After setting up the import map, we saved that so that the management service software knew how to handle future files.

On the emails front, we’ve been diagnosing issues with sending emails to companies such as Google. Because job management software sometimes involves sending a lot of emails purporting to be from our clients’ domains, we are classified as “bulk senders”.

Google makes it very clear that there are certain rules they want bulk senders to follow.

Today, we made a change to our outgoing emails so they now have a From header saying “”. This is because of Google’s rule 3: “Use the same address in the ‘From:’ header on every bulk mail you send.”. We can’t use a client’s email address because every other client would then think the email was spam and mark it so, making it very difficult for us to deliver our reports!

Google says very clearly that “unauthenticated emails with attachments may be outrightly rejected, for security reasons.” Since our emails tend to have reports attached, if we do not authenticate them, following their rules, then we risk having the emails completely rejected.

And so, if you receive a report that has “” written in it, don’t panic.

On the Purchase Orders side, I’ve just had a chat with the lead on that project and she says that what we were (my fault) initially trying to do is way more complete than what most market leaders actually do. Most system handle just basic order tracking, so we’re pushing in that direction instead to get that done. The way we count progress is through issues created/completed, and we’re at about 4 completed, 7 incomplete. Some of those incomplete look small, but you can never tell! Programming is like solving math – sometimes simple-looking things end up being hard (Fermat’s last theorem, for example).

As an example, I had it initially that when you make an order, you can order any stock at all, and the system would break it apart into separate orders for each supplier that might be needed to fulfil that order. For example, if you were using FieldMotion as pest control software, you might order bait boxes and pellets in the same order, even though the two might actually be supplied by different suppliers. It turns out that usually when an order is being made, it’s from a specific supplier – I was just complicating things. So, we’ve managed to simplify it a lot.

In programming, we tend to write out the major points as single issues, and as we dig deeper, we write separate issues to handle any side-projects needed to complete them. The fact that we’re on only 11 issues after a few weeks into this project is pretty good. Means we’ve not had to build structure elsewhere to support what we’re doing here.

And then as usual, we have the smaller day-to-day jobs going on. We work with external developers, for example, to help them use our API efficiently. This sometimes involves us writing new entry points if it will help the developers, or making some existing points more consistent.

stages of release in FieldMotion

I wanted to talk about how why FieldMotion’s software process makes us one of the most robust and reliable field service management software companies on the market.

Startup companies race to complete features and ship them as soon as possible. We don’t do that, because we are aware that new features are never right. There is always a period of tweaking after a release to match what we built, to how people are actually using it.

How we work is that the development team will have a list of big projects to do, which they’ll work on for a few months. Each of the projects appear to naturally come to a close near six months into development, so we’ll tidy up all the new projects and announce them all as a new release.

The current release candidate will be our tenth official release since we started about 5 years ago, and will have three major products in it: a stronger Dynamic Scheduler (route optimisation for multiple vehicles, departments, etc), strong Xero integration (both Public and Private API versions), and a new Purchase Orders system.

We’ve already planned out the bones of what’s coming for the next six months after this, and we think it will be our biggest and most important release; not because of new products in it (we’re thinking Quickbooks, and multiple languages, for a start), but because we’re mostly dedicating our development resources into making sure the system is scalable (we want millions of people using it), fast even with all those users, never goes down for any reason, and is so simple to use that you’ll wonder why we bother making tutorial videos!

We’re already at the stage where most parts of the system are redundant, so if a server or two (or even a full datacentre – hello Digital Ocean San Francisco I’m looking at you…) goes offline for a while, we can reroute people to different areas that are still up and running. In most cases, there won’t even be a pause (like when SF went offline, and there wasn’t even moment where anyone noticed but our development team), but sometimes we will have short period where parts of the system are offline.

In our own case, how it usually happens is that we’ll see an issue, we’ll solve it in a quick way that’s not perfect but gets the job done for now, and then we’ll solve it in a more permanent way that takes longer but is more robust.

Yesterday, for example, we had an issue where people logging into one particular server in our network were experiencing a large delay in their requests. We looked at this, couldn’t see the immediate cause, and figured the best thing to do at that moment was to increase memory and CPU (we multiplied the number of CPUs by 4, and the amount of RAM by 16 – hey, go big or go home!) to see if that fixed the immediate issue. It did, forcing people using that server (some customer relationship management software users, no mobile workers) to log in again, which was a price that had to be paid at that time to get the problem solved. The more permanent solution in this case is that over the next six month development period, we will be tearing that server type apart into separate “microarchitecture” pieces that can be managed separately, and making sure that each of those new pieces can be turned on/off without affecting people that are logged in. Obviously, this will take time to plan and build, so the quick solution (throw more memory and CPU at it!) was the right answer in the mean time.

Yesterday’s issue took that particular server offline for three minutes (I was timing it) while we upgraded it. Three minutes is better than how long Google Drive was out last week, or how long the entire country of Japan was offline for on August 25th (thanks again, Google), or how Melbourne’s train network was shut-down for hours on July 13th.

We’re at that stage in a company’s life where we’re pretty happy that we have the product we need, and that there is no need to keep on adding more and more and more shiny stuff to it, which is why we’re happy now to spend the next 6 months or more just making sure it is the absolute strongest, fastest, most reliable, and best all-round cloud field service management software.

Iterative development

From our own point of view, and from the points of view of our customers, it is nice to be at a point in our service management software development where we are developing iteratively instead of coming out with new tricks every week.

image: clean, rinse, repeat

While I love it when we develop something new and exciting, it can be nerve-wracking for us for the following weeks, as we try to find out what these new shiny toys have done to our solid, stable, well-tested field service management engines.

New tricks never affect the existing workflow management system customers, because they are on servers that we do not edit except for the occasional bug-fix. The only customers that get to see the new shiny stuff are those that specifically asked for it, or those that asked for other new things such that we were forced to place them on our testing server while we worked on the next release.

FieldMotion is now so flexible that whenever we’re asked to develop anything new, it more than likely ends up that we already are able to do it, so my job is sometimes simply to listen to the request, and then point out how it can done already with the workflow management software.

Sometimes, though, we are asked to do something that’s just slightly beyond what we can do at the moment. It’s never far away; just slightly.

For example, today I was asked if it was possible to do a certain thing with RFID tags. After thinking about it, I suggested a way we could do it that would involve adding maybe only 20-30 lines of code to the field service management app, and yet it opens up our possibilities to yet another broad channel of potential customers.

This kind of thing happens often enough that every six months, we have enough new little tricks that we can release a new version of FieldMotion’s field service engineer software, confident that there is enough newness in there to merit the release, and yet it is similar enough to the previous release that our current clients won’t be shocked at the difference.

Iterative development allows us to “tune” the workflow system to fit better with the customers, knowing for sure that the system already works very well for them, and we’re just adding enhancements, not adding whole new sections that need manuals.

I was explaining earlier today to a new developer that when we create a new widget or page for the field management software, we need to make sure it is as absolutely simple and obvious as possible. He was telling me how he liked the power that the CMS Joomla gives him when he creates a website. Yes, it gives you a lot of power, but it’s at the expense of usability. Every time I have to work with a website that uses Joomla as its engine, I have to learn all over again how it works. That is bad user experience.

When you use any part of FieldMotion’s workflow software, it is straightforward and obvious how it works, whether you’ve been using it every day since you got your account, or this is your first time ever seeing it.

In my old life as a web developer, I would say to clients that “If you need a manual, I’ve built it wrong”. I stick to that slogan and make sure that everything we produce in our system is clean and obvious.

This is also why I love iterative development – instead of developing more and more and more stuff that piles up on the field service software like turrets and walls on a fairy-tale castle, we carefully expand the system just enough to fit the new trick in, and then just as carefully make sure that it is seamless and easy to understand.

How developers are developed in FieldMotion

No two projects are the same. A person coming moving from FaceBook to Google will be lost for the first month. A person moving from Uber to Yahoo would also be lost. The same is true for all tech companies.

When we hire a developer for our workflow management software, we follow a “boot camp” process which gives the developer a grounding in how the workflow system works, and how the code is laid out.

New developers spend a month working in the Implementation department, where they develop digital forms for clients and get to understand how the field management software works from the user’s perspective.

The first month is crucial. New employees are encouraged to spot inefficiencies in how the paperless mobile forms work, and to come up with plans on how they would improve the system if they had the chance. Their plans are added to our internal issues tracker, and worked on over time, either by other developers, or by the new developer.

The next month is spent in handling issues. This can be anything from fixing bugs, to reading logs to figure out what happened at specific times. This month teaches the new employee how the system is spread out, and how we arrange our code. Again, this is invaluable, as you cannot be a good developer if you don’t have a good understanding of the system.

Afterwards, the developer will “gravitate” towards one or more projects to specialise in. Some of our paperless office software solutions overlap (the field service management app and the mobile server, for example), but there’s plenty of room in the system for any developer to “own” a section that they can become expert in.

We encourage our developers to overlap in their knowledge, so that if Alice is out of the office today, Bob can take over if something comes up.

At FieldMotion, we design systems that help you plan your work, but as you can see, we do our best to plan our own internal workings just as rigorously. We intend to be the best workflow management software company in the world, and work constantly towards that goal.

image: we’ve found that pizza helps round them out