10 Oct 2017
Imports, emails, purchase orders
in Technical Blog by Kae Verens
Over the last few weeks, we've been working away on various little projects to improve our field service management software. Yesterday, I finished the first prototype of a new importing system where our clients (you, maybe?) export their customer or job data in whatever common format they like (CSV, XLS, XLSX for example), and use our new import mapper to easily link your headers to ours and do your import. To demonstrate it, we set up our system to check an SFTP server (like Dropbox, but more fundamental) at a specific location. We put a demo customer file in there in CSV format, and the system was able to detect that and import it automatically. How did it work? We put a demo file there first, then ran an "import map wizard", which found the file and read it. It listed the column headers of the file headers on one side of the wizard, and our own internal customer headers (how we record customer data in the relationship management software) on the right. We then clicked the appropriate fields on the left side and right side to match them, until they were all matched. Sometimes, there is a mismatch. For example, in our field service software, customer names are recorded as a single field, "customer_name", but in your database, you may have them separated into multiple fields (title, first name, surname). So, we added the ability to automatically concatenate fields while importing, to join the various fields together and form a single match that we can then import. Also sometimes, there will be a field in the import which you really want to import but don't have a matching custom field in the workflow management system already. So, we add that you can automatically add that as a new custom field. After setting up the import map, we saved that so that the management service software knew how to handle future files. On the emails front, we've been diagnosing issues with sending emails to companies such as Google. Because job management software sometimes involves sending a lot of emails purporting to be from our clients' domains, we are classified as "bulk senders". Google makes it very clear that there are certain rules they want bulk senders to follow. Today, we made a change to our outgoing emails so they now have a From header saying "firstname.lastname@example.org". This is because of Google's rule 3: "Use the same address in the 'From:' header on every bulk mail you send.". We can't use a client's email address because every other client would then think the email was spam and mark it so, making it very difficult for us to deliver our reports! Google says very clearly that "unauthenticated emails with attachments may be outrightly rejected, for security reasons." Since our emails tend to have reports attached, if we do not authenticate them, following their rules, then we risk having the emails completely rejected. And so, if you receive a report that has "email@example.com" written in it, don't panic. On the Purchase Orders side, I've just had a chat with the lead on that project and she says that what we were (my fault) initially trying to do is way more complete than what most market leaders actually do. Most system handle just basic order tracking, so we're pushing in that direction instead to get that done. The way we count progress is through issues created/completed, and we're at about 4 completed, 7 incomplete. Some of those incomplete look small, but you can never tell! Programming is like solving math - sometimes simple-looking things end up being hard (Fermat's last theorem, for example). As an example, I had it initially that when you make an order, you can order any stock at all, and the system would break it apart into separate orders for each supplier that might be needed to fulfil that order. For example, if you were using FieldMotion as pest control software, you might order bait boxes and pellets in the same order, even though the two might actually be supplied by different suppliers. It turns out that usually when an order is being made, it's from a specific supplier - I was just complicating things. So, we've managed to simplify it a lot. In programming, we tend to write out the major points as single issues, and as we dig deeper, we write separate issues to handle any side-projects needed to complete them. The fact that we're on only 11 issues after a few weeks into this project is pretty good. Means we've not had to build structure elsewhere to support what we're doing here. And then as usual, we have the smaller day-to-day jobs going on. We work with external developers, for example, to help them use our API efficiently. This sometimes involves us writing new entry points if it will help the developers, or making some existing points more consistent.