The final entry in my Vault Integration Pattern series will focus on using Vault’s job queue for transferring data. Like I said before, there are more integrations patterns out there, I’m just focusing on the ones that I have used first-hand.
I used this technique for the short-lived integration between Vault and PLM. Let me just get this out of the way upfront: The integration pattern I used had nothing to do with the app being discontinued. The long term vision changed, and the integration was caught in the middle. From a technical standpoint, the integration pattern is sound.
Since PLM is in the cloud and Vault is behind a firewall, there was no easy way for PLM to have a direct view into Vault. Data would have to be copied to PLM. I also wanted a very reliable mechanism for data transfer, so doing it real-time in the Vault clients was not a good fit.
I went with the Vault job queue because it provides a guaranteed mechanism for the data transfer. If it’s on the queue, I can rely on it staying on the queue until it gets properly dealt with. If there is an error, it’s logged in the form of an error job on the queue. If something critical goes wrong, like a power outage, the job is still on the queue in the processing state.
The downside is that the queue is asynchronous. Even if you grab jobs directly off the queue you are still relying on polling. There is no way to instantly know when the job is queued. The other major disadvantage is that it runs without any user interaction. So you can’t prompt the user for information and you can’t rely on problems getting addressed immediately.
If you are replicating data this way, I highly suggest having some kind of re-sync button. It’s great when developing and debugging. It’s also doubles as a bulk-load command, which is usually a requirement for integrations like this. Lastly, it’s a magic fix-all button for a bunch of difficult error cases, such as a user deleting data manually. These re-sync buttons tend to be slow, but that’s OK because things are asynchronous. Also, with the job queue priority feature, you can insure that the re-sync doesn’t interfere with everyday operations.
To summarize...
Advantages:
- Guaranteed delivery - Jobs on the queue are guaranteed to stay there until something process them or the Vault admin removes them.
- Failure recovery - Failed jobs can be re-submitted.
- Background process - Doesn’t interfere or slow down normal operations.
Disadvantages:
- Conflicts - The same data may get updated at the same time in two different systems.
- Asynchronous - You have to wait before you can see results.
- Duplicate data - The same data is now in two different systems.
Subscribe