It's All Just Ones and Zeros

Custom Jobs without Job Processor

If you are building your own custom job, you do not have to plug-in to Job Processor if you don’t want to.  The job queue can be accessed through the Vault web services API, just like with files and folders. 

The recommended approach is to use Job Processor.  That approach is what you see in the SDK documentation and sample apps.  But the Job Processor can’t handle all the cases.  There are some times when you may want to just write your own service.  This article will help you identify those cases and provide some help writing the code.

---

First let’s talk about what Job Processor does well.  It’s a single application than can handle multiple job types.  It has various job handlers loaded out-of-the box, and additional functionality can be plugged in.  For a Vault admin, this is good because everything can be managed from a single app.  If something goes wrong, there is only one place to look, one log file to check, and so on.  From the developer's point of view, Job Processor offers some nice features.  You don’t have to worry about Vault authentication, managing the job queue and logging.  None of these things are very hard, but it’s nice to have the app take care of them.

On the flip side, Job Processor does have some limitations.  First, it’s not a true Windows service.  A user has to log in, launch the app, and leave it running.  Job Processor handles jobs one at a time.  There is no parallel processing.  And you can only run a single process on a machine.  A common problem is that Job Processor gets bogged down with jobs that take a long time, like DWF generate jobs.  This prevents other types from running even if they take less time to run. 

If you are running in a multi-site environment, Job Processor can only process jobs from one site.  Vault requires that a job can only be handled by the site that queued it.  So if you have 5 sites queuing jobs, you would need 5 Job Processors running.

---

By communicating directly with the job queue you can bypass these limitations and build exactly the app you want.  Working with the job queue is easy.  You just need to use the following functions from the JobService:

• ReserveNextJob - This grabs a job and reserves it to you so that nobody else works on it.  You pass in the job types you support and Vault passes back the “next” job based on the priority and the time it was queued.
• UpdateJobSuccess - This tells Vault that you are done with the job, and it can be removed from the queue.
• UpdateJobFailure - This tells Vault that something went wrong.  The job stays on the queue in the error state.  You can pass back information on what went wrong.
• UnReserveJobById - This function is used for cases where a job gets reserved but never updated.  For example, maybe the service crashed or there was a power outage. 

If you want an example, have a look a Q-Tools.  It’s a Windows service that can interact with queues from multiple Vault sites.

Caching

If you’ve looked at any of the source code for my apps, first, let me apologize.  I went for quantity over quality on those apps.  So I didn’t structure the code for people not named Doug Redmond.  Even if you are a professional programmer, I’m guessing you don’t want to spend hours of your time deciphering a bunch of uncommented C#.  It’s probably best if I explain some techniques in the more human-readable blog format.

For this article, I’d like to talk about a caching technique that I’ve found useful in some of my apps.  Caching is pretty much a necessity when working with Vault.  You are constantly running into cases where you have an ID and you need to look up the object.  For things like Categories, Property Definitions and Lifecycle States the objects are unlikely to change.  So the optimial thing to do is store them in memory to reduce the number of API calls.

I like to call my cache Cache.  It contains two main things: The WebServiceManager and collections of cached Vault data.

For the collections of cached data, I usually use lazy initialization.  That way I don’t look up something if I don’t have to.  The first request for the data triggers the API call.  Any further requests for the data use the cache.

The data structure I use depends on how I want to use it.  Just because Vault returns an array doesn’t mean I have to store it that way in the cache.  Frequently I’ll store it as a dictionary using the ID or system name as the key.  The ToDictionary feature in LINQ is a nice little trick.

Example code from DECO:

public class Cache {     private Lazy<Dictionary<string, CustEntDef>> m_custEntDefsBySysName;     public Dictionary<string, CustEntDef> CustEntDefsBySysName     {         get { return m_custEntDefsBySysName.Value; }     }             public Cache(WebServiceManager mgr)     {         m_custEntDefsBySysName =            new Lazy<Dictionary<string, CustEntDef>>(GetCustEntDef_SysName_Map);     }       private Dictionary<string, CustEntDef> GetCustEntDef_SysName_Map()     {         Dictionary<string, CustEntDef> retVal = null;         CustEntDef[] defs = Mgr.CustomEntityService.GetAllCustomEntityDefinitions();           retVal = defs.ToDictionary(n => n.Name);         return retVal;     } }

My Cache object also contains a WebServiceManager.  First, the lazy initializers need to run API commands.  Second, I find it easier to manage.  If the manager and the cache were two separate things, that’s two things I have pass around all the time and keep track of in the code.  By putting the manager in the cache, I only have one object to track.  The objects work well together too.  For example, if you ever need to dispose your WebServiceManager, you will probably want to clear out your cache too.

There are some other cases where you will want to clear your cache.  Mainly, if the data you are caching gets updated.  In my Effective Permissions app, I cache all the all the user and group information.  But it’s reasonable that an admin might run Effective Permissions, update the user and group settings, then run Effective Permissions again.  So I need to clear my cache, but the Vault API doesn’t give me a way to know that users or groups were updated. 

My solution was to clear the cache whenever the Global Settings command is invoked.  Even though I don’t know for sure that anything changed, I clear the cache anyway, just to be safe.  This technique isn’t 100%.  It’s possible that another custom command updated the data, or another admin updated the data from their Vault client.  In the case of Effective Permissions, I didn't feel a 100% solution was needed.

Effective Permissions 2013

I’m currently in an amazingly interesting DB training class.  So I decided to do some Vault app programming so that my heart wouldn't explode from all the excitement.  The result is Effective Permissions 2013.  It’s an update of my Effective Folder Permissions 2.0.  The ‘Folder’ part of the name has been dropped since it now does more than just folder permissions.  Files and custom objects can also be managed.

Check out the video:

The main new addition is a view/edit window for object permissions.  You can now edit an Access Control List and see the effect before you commit the change.  It’s also smart enough to determine the source of the ACL and make the appropriate updates.

All the old features are still there.  So you can still see the Administrators List, and you still have the Permissions Overview, which shows the user-based permissions for a set of folders.

---

Requirements:

• Vault 2013 Workgroup / Collaboration / Professional
• Administrative Rights

Click here to download the application
Click here to download the source code

Click here for 2011 version

As with all the samples on this site, the legal disclaimer applies.

Vault Integration Pattern - Real-time Events

Welcome to part 2 of my 3-part series on Vault integration patterns.  There are many more than 3 patterns, but I’m going to stick to ones that I’ve actually used on an app or product. 

Today’s pattern is the idea that Vault communicates with the other system, in real time, whenever things change.  I used this pattern for Vault ♥ Bluestreak.  In that app, I hooked to events from Vault Explorer and Events.  This allowed the app to “listen” to what the user is doing and send out notifications to Bluestreak.  Bluestreak would then post on the notification on it’s Activity Stream, which is similar to a Facebook wall.

Maybe it will be clearer if you see the demo video (no audio):

The general idea here is that the integration app listens to events and interacts with the other system as soon as the event comes in.  Vault doesn’t have server side events, so the app has to be client-side.  In this case, the communication is one way.  I’m sending alerts to Bluestreak, but I don’t need to get any sort of response.  So the code makes heavy use of Post events.  The communication with Bluestreak is done in another thread for performance reasons.

If you have a case where you need to read data from the other system, this pattern still works.  The only difference is that you would be using the GetRestrictions or Pre events.  Multi-threading probably won’t work here since you need to halt the Vault operation while waiting for the other system to respond.

---

Advantages:

• Operations occur in real time - No need to keep hitting the refresh button, like when working with the Job Processor.
• User Interaction - If you need input from the user, you can do it.  You can also pop up errors immediately if something goes wrong.

Disadvantages:

• Events are not guaranteed - Because the operations are client-side, they occur outside the database transaction.  This means that there is a chance that the event happens, but the Post event never fires.  For example, a file changes lifecycle state, and the power goes out right before the Post event fires.  In this case, the Post even is lost forever.  Along the same lines, a Pre event may fire, but the actual operation my never get a chance to run.  So if you need 100% guarantee on your data, this is not the pattern to use.
• Client side install - If you have 100 Vault users that need to integrate with the other system, you need to install the app on 100 machines.  If an app starts failing for a user, it may not be immediately obvious.

Features across different object types

Let’s start off with a quick quiz.  Look at the image and answer the following questions:
Are the file and the folder in the same category?
Are the file and the folder in the same lifecycle state?

Answers:  No, they are in different categories, and yes, they are in the same lifecycle state.

It can be a bit confusing when you have concepts that span multiple objects types.  Sometimes you can share behavior across different types and sometimes you can’t.  In this article, I’ll attempt to explain what the scope of each feature.

Custom entities make things more tricky.  In case you don’t know already, all custom entities are part of the same entity class (CUSTENT).  When you create your own custom entity definition, it’s actually a sub-type of CUSTENT.  So that adds another level of complexity when defining scope.

---

Let me go down feature-by-feature and explain the scope...

Categories - A category is specific to an entity class.  It cannot be shared across different entity classes.  So a File and a Folder can never have the same category.  You can give set the display names the same on a File and Folder categories, like in my quiz, but they are still different underlying objects.

Categories are applied at the entity class level.  So, CUSTENT categories can be used by any of the custom entity sub-types.  So it is possible for a Foo object and a Bar object to have the same category.

Property Definitions - A property definition can span multiple entity classes.  If you go to the admin settings, you will see that it’s common for Files and Folders to share a property definition.  Categories can provide additional behavior, but it’s not required.  If a Property Definition is associated to the FILE entity class, then any file object can have that property, regardless of category.

If a property definition is associated to the CUSTENT entity class, then it’s available to all sub-types.  There is no way to have a property definition specifically for Foo objects.

Searching - A search is specific to an entity class.  This is pretty obvious at the API level since the functions are named FindXBySearchConditions, where the X is the entity class.

For custom entities, a search will span all CUSTENT objects, regardless of sub-types.  So it’s possible for a single search to return both Foo and Bar objects.  Most of the time, you don’t want this, so you need to add a search condition that narrows the search to a specific sub-type.

Lifecycle Definitions - Files, folders and custom entities all share the same lifecycle engine.  Any lifecycle definition can be used across these 3 entity types and any custom object sub-types.  The only restriction is that the entity must be in a category that is associated with the lifecycle definition.

Revision Schemes - They work similar to lifecycle definitions.  You can share them across different entity classes, but the entity must be in a category that is associated with the lifecycle definition.

Roles and Permissions - The role usually mentions the entity class directly, such as Item Editor (Level 1).  So it’s pretty clear that roles are specific to an entity class. 

For custom entities, a role applies to all custom entity sub types.  So somebody with Custom Object Editor (Level 1), can edit both “foo” and “bar” objects.  If you want the user to be able to edit “foo” but not “bar”, you need to give that user the edit role, then use ACL security to restrict edit on “bar” objects.

Custom Commands - These can be bound to a specific entity type.  In the case of custom entities, they can be bound to specific to a sub-type.  A command can work with to different types/sub-types by binding the command to multiple command sites.  Another option is to specify null for the NavigationTypes property, which has the effect of making the command always available.

Custom Tab Views - These are always bound to a specific entity type or custom entity sub-type.  You can’t share a file tab with an item tab or a “foo” tab with a “bar” tab.

---

Speaking of bar tabs, it’s payday and my shift is over.  I’ll leave you with this this handy chart, summarizing everything.

	Entity class scope	Custom entity sub-type scope
Categories	Specific to a single entity class	Always shared across all sub-types
Property Definitions	May be shared with multiple entity classes	Always shared across all sub-types
Searching	Specific to a single entity class	Always shared across all sub-types
Lifecycle Definitions	May be shared with multiple entity classes (dependent on category)	May be shared with multiple sub-types (dependent on category)
Revision Schemes	May be shared with multiple entity classes (dependent on category)	N/A
Roles and Permissions	Specific to a single entity class	Always shared across all sub-types
Custom Commands	May be shared with multiple entity classes	May be shared with multiple sub-types
Custom Tab Views	Specific to a single entity class	Specific to a single sub-type

Vault Integration Pattern - Provide a View

By far, the most common use of the Vault API is to provide some sort of integrations with something else.  Sure there are some good utilities up on the App Store, but they don’t represent the majority of the cases.  Consulting is where it’s at, and there are a surprising number of systems out there to connect Vault to.

Each integration is different, so there is no way for me to tell you everything you ever need to know.  However, there are certain patterns that emerge, and it’s the patterns that I will be focusing on.

The first pattern is the one I used when designing the SharePoint 2010 integration.  One system provides a view of data in another system.  In this case, SharePoint 2010 provides a view Vault data.

If you do it right, the end user should not know (and not care) that the data is coming from an outside system.  The SP integration does this very well.  Here is a game you can play.  Look at the following video and try to figure out what is the SharePoint data and what is the Vault data.

One cool thing about BCS, the SP feature I used for the integration, is that it doesn’t just provide a view of external data.  BCS allows you to link SharePoint data with external data.  This is a pretty useful feature for integrations like this.  If you can’t link the data between the two systems, then the integration loses a lot of value.  The end user may as well just use two separate applications.

---

I’ll end the article with some quick bullets on the advantages and disadvantages of this pattern.  If I left anything out, please let me know in the comments section.

Advantages:

• No duplication of data - Everything is just cleaner when data lives in one and only one place.  You don’t have to worry about stale data, and you don’t have to worry about conflicts

Disadvantages:

• Performance - Constantly pulling data from another source is time consuming.  No matter how much you optimize things, it will never be as fast as having the data in the same system.  For SharePoint, we found that the distance between the SP server and the Vault server was the biggest factor in how quick the Vault pages came up in SP.
• Network Dependency - If the network ever goes down between the two systems, the integration is effectively broken.  People who rely on the data will no longer be able to access it.

Vault Query 2013

This month's app is an update of Vault Query.  It’s a cute little program that reads Vault data at the API level and displays the results in a similar format to the Visual Studio debugger.  It’s good for things like figuring out what the system name is for the UDP you want to read.

New for 2013, you can export the results to an XML file.  A lot of times, I find it easier to sift through an XML file instead of dealing with a tree view.

(click image for full view)

---

Requirements:

• To use the EXE
  • .NET 4.0
  • Vault 2013 (any product)
• To use the source code
  • Visual Studio 2010
  • Expression Blend 3 or 4

Click here to download the application
Click here to download the source code

Click here for 2011 version.  Sorry, there is no 2012 version.

---

To use:
On the left you have a list of operations.  Each operation represents a single web service call with specific input parameters.  Double click on the operation and it runs.  The output is presented in the right pane.

I set up a bunch of operations out-of-the-box, but you can add your own.  You can either use the "Add Actions" command or edit the Operations.xml file.  In theory, any API call can be made into an operation.  However the UI is picky about what input types it supports.  You may need to edit Operations.xml for many cases.

As with all the sample apps on this site, the legal disclaimer applies.

The post-AU blog article

AU is now over, and I’m almost fully recovered.  My limp is barely noticeable and my right eye is mostly in focus.  It will take a few more weeks for the slot-machine ringing noises to fade away, but that’s to be expected.  This is a good time to go over Vault API related topics from AU.

 

Vault wins at Apphack
Autodesk Developer Network put on an event called Apphack.  The idea is that people could submit their apps, and the public votes on which one should win.  I think there were around 20 entries.  The winning entry was a Vault App!

Rob Oud from CAD & Company submitted Vault Project Portfolio.  It’s a Vault Browser app for Windows 8 tablet.  I was able to see a quick demo between classes, and the app looks very cool.

 

My classes were not recorded, again
Every year a rumor goes around that every class will be recorded, and every year it turns out to be a lie.  I don’t think I’ve ever had an AU lecture recorded.

Anyway, my two classes “Programming Integrations Between Autodesk Vault and Other Systems” and “Programming Custom Objects with the Autodesk Vault API” will be converted into blog posts over the next few months.  It’s probably for the best anyway.  You can’t Google for content within a video very well.

 

Process is critical
I like to attend as many Vault classes as I can.  Mainly I just sit and listen.  One common theme this year was process.  If you don’t get your processes defined properly, your Vault deployment will be a failure.  That got me to thinking what I usually think about:  Can I write any apps for this? 

It’s a bit of a “chicken and the egg” problem.  How can I write a Vault app that helps with Vault deployment?  Any suggestions would be welcome.

Asking for Help - Do’s and Don’ts

Although my official job title is “Software Engineer,” my API knowledge makes me an honorary member of the tech support team.  More often than not, I read a problem and don’t have enough information to answer it properly.  This requires me to respond asking for more formation.  Usually one or two more back-and-forth sessions are needed before the question is finally answered.  This process is a wasteful, especially when different time zones are involved.  If you figure that each back-and-forth takes 24 hours, there is a definite benefit to reducing these round-trips.

So here is a list of things you can do to make the process more efficient.  The goal is to provide all the needed information upfront so that I don’t have to go back and ask you to elaborate in more detail.

---

DO... include the Vault product and version
Is it Vault Basic?  Vault Professional?  Is is the 2013 version?  Each product has a different feature set and the API may change between release years.

DON’T... use the phrase “It doesn’t work”
That statement is way too vague.  You need to be more specific like “it crashes the program” or “it locks up and I have to restart the app” or “it times out” or “it returns error 303”

DO... include the error code
If the server returns an error, don’t just say “the server throws an error”.  Tell me what the error code is.  Don’t leave me in suspense.  I even have some code snippets you can use to extract the error and restriction codes (if any).

DON’T... provide too much source code
Sometimes people will post their entire project and expect me to debug it.  That’s not a good use of anybody’s time.  If you are going to include source code, it should be isolated to a single function at most.

DO... isolate the line of code that fails
If you are sending a block of code and are getting Exceptions, you need to indicate clearly which line is causing the breakage.

DO... take screenshots
It’s amazing how much information is conveyed in a screenshot.  Even if the Vault is in a different language, I can usually figure out what is going on based on the icons and dialogs.  Videos are great too.  If you find yourself taking 4 or more screenshots, it may be easier to just record a video.

DO... use Exception.ToString()
ToString is the best thing to happen to debugging since the print statement.  Don’t be afraid to use it and include the full stack trace in your problem report.  Even if you don’t understand it all, it may make sense to me since I have access to the Vault source code.

The BOM Pipeline

I want to explain a bit how bill of material data works in Vault.  It’s not very intuitive looking at the API.  I’ll also attempt to answer the question, “Why did they do it that way?”

The Primary Workflow

Let me know over the “primary workflow,” which I’m defining as the workflow most responsible for the architecture.  There are plenty of other valid workflows, but I’m not going to focus on them much here.

It’s best if I list out the sequence of events in the primary workflow:

1. The CAD user adds a file to Vault.
2. The CAD plug-in uploads a BOM object along with the file.  I’ll refer to this object as the “file BOM”.
3. The “file BOM” object gets stored in the database and mostly stays dormant.
4. A Vault user selects the file and runs the “Assign Item” command.
5. Internally, the “file BOM” gets read and becomes the template for creating or updating Vault Items.  In other words, the “file BOM” fills out most the information in the Assign Item wizard.
6. When the Assign Item wizard is complete, the Item objects are created and hooked together to form an “Item BOM”.  The Item BOM is what user sees and interacts with in the Item Master.
7. The “file BOM” is still around in the database, but it goes back to being dormant.  It would only be used again if there is an update to the Item BOM or a need to re-create the Item BOM.

Impact

There are some interesting consequences as a result of this workflow.

• The item structure has nothing to do with the file assembly structure.  The file BOM is the only factor.
• The file BOM is uploaded, even if the product is Vault Basic.  The idea is that the product may get upgraded to Vault Professional in the future.  If the upgrade never happens, the BOM data stays dormant forever.
• The file BOM is not static data.  It may get altered by the server.  For example, data gets “filled in” during an Assign Item operation.  This way things are optimized if the Items ever need to be updated.
• Results from GetBOMByFileId are not guaranteed to be the same, even when called on the same file version. See previous bullet.
• GetBOMByFileId is not actually used anywhere in the process.  It’s an API-only feature intended for testing and troubleshooting purposes.  And yes, I’m well aware that many people are using this function beyond it’s intended scope.

File BOM vs. Item BOM

Here is a quick rundown of the two different BOM representations.

	File BOM	Item BOM
Purpose	A template for the Item BOM	Represent an accurate Bill of Materials
Visible to end user	No	Yes
File scope	Attached to 1 and only 1 file.	Can be associated with 0 to many files.
Can be directly edited	No	Yes